Space-time process simulation method and system based on stacked space-time memory unit

Abstract

The application provides a kind of spatiotemporal process simulation method based on stacked spatiotemporal memory unit, characterized in that, including: obtaining spatiotemporal dataset;Initial spatiotemporal process deep learning model carries out preliminary expression to the spatiotemporal dependence of spatiotemporal dataset, obtains first stage deep learning model;Robustness strategy expression is carried out to first stage deep learning model, obtains second stage deep learning model;Parameter optimization is carried out to second stage deep learning model, obtains the spatiotemporal process deep learning model of completing a round of training;Iterative training is carried out, obtains trained spatiotemporal process deep learning model after meeting preset condition;Adjustment is obtained after completion Final spatiotemporal process deep learning model.The application makes the spatiotemporal process deep learning model have strong robustness by stacking time memory unit, has long-term dynamic modeling capability, and spatiotemporal memory flow enhances the modeling capability of network to short-term dynamic.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of space-time simulation, and particularly relates to a space-time process simulation method and system based on stacked space-time memory units. BACKGROUND

[0002] At present, with the rapid development of satellite remote sensing and sensor networks, space-time data is growing explosively. With the development of data-driven deep learning models, great opportunities have been brought to the analysis and mining of space-time processes, making up for the limited ability of traditional parameter models and traditional machine learning models to process data, and overcoming the shortcomings of being suitable for small amounts of data and small sample data. The space-time simulation and prediction method based on deep learning model has obvious advantages in prediction accuracy and processing large-scale data sets. There are important progress in analyzing and mining static objects from large-scale text, image, graph, sequence, remote sensing image and numerical multi-source big data. However, how to efficiently and intelligently analyze and mine dynamic space-time processes is still a difficult problem in scientific data analysis and mining. The existing deep learning model cannot directly and effectively express the complex interaction relationship between the space-related elements in the time sequence in the space-time process.

[0003] The space-time process mainly refers to the change of geographical phenomena in the time process and the distribution characteristics in the spatial range, such as the evolution law of typhoon, flood and natural disease transmission in time and space. This law involves the physical causes of geographical phenomena and the influence of complex elements, and also poses a challenge to the dynamic analysis and mining of space-time processes.

[0004] Existing deep learning spatiotemporal process mainly falls into three categories: deep learning based time series simulation method, deep learning based spatial feature prediction, and deep learning simulation method combining spatiotemporal features. Deep learning based time series simulation method such as recurrent neural network (RNN), long short-term memory (LSTM), gated recurrent unit (GRU) and temporal convolutional network (TCN) can extract the time sequence features in the data, but when it is applied to spatiotemporal process simulation, the spatial correlation of spatiotemporal sequence data is often ignored. When using deep learning method to learn spatial dimension, in order to mine the spatial features between grid spatiotemporal data, it can be modeled as a series of picture sequences, and then convolutional neural network (CNN), graph convolutional neural network (GCN) or residual neural network (ResNet) is used to extract spatial features, but such model often ignores the time correlation of spatiotemporal sequence data. However, in practice, the spatial correlation between spatial regions often changes with time, for example, in the rainy season and the dry season, the runoff generated under the same rainfall intensity is also quite different, and the risk of flood disaster is also different. Therefore, in the simulation of spatiotemporal process, it is particularly important to express the features of time, space dependence and the relationship between them.

[0005] In the aspect of deep learning simulation combined with spatiotemporal characteristics, Shi et al. proposed to apply deep learning method to short-term precipitation forecast simulation in 2015, and proposed ConvLSTM combining CNN and LSTM, which inspired a series of models introducing CNN structure into spatiotemporal sequence simulation and prediction. In order to combine the advantages of convolution and recursive structure, Wang et al. proposed a predictive recurrent neural network (PredRNN) in 2017 and first applied it to precipitation nowcast simulation. Wang believes that spatiotemporal simulation learning should remember the appearance of space and the change of time in a unified memory pool, and adopts a new spatiotemporal LSTM (ST-LSTM) unit which can extract and remember the representation of space and time at the same time. Lin et al. proposed a self-attention memory mechanism (SAM) to capture long-term spatial dependencies based on the self-attention mechanism in 2020, and used depth separable convolution to replace the standard convolution operation in order to further reduce the number of parameters and computational complexity.

[0006] In summary, most models treat spatiotemporal sequence data as a series of time-attribute pictures or directly model them as a graph to learn their spatial features. Although spatial features can be extracted, the physical process of spatiotemporal evolution is ignored. Spatiotemporal processes have the characteristics of high dynamic process and high complex relationship. Taking the flood inundation process as an example, the inundation area is changing at any moment. The inundation process is not only related to the previous inundation situation, but also has a complex relationship with the present hydrology, meteorology and other factors. These relationships include nonlinear relationships, multiple associated factor dependency relationships, time series dependency relationships and spatial dependency relationships, etc. Existing deep learning models cannot directly and effectively express the complex action relationships between spatially related elements in time series in spatiotemporal processes, and lack high-precision spatiotemporal process analysis and mining models.

[0007] The above content is only used to assist in understanding the technical solutions of the present application and does not represent an acknowledgement of the above content as prior art. SUMMARY

[0008] To solve the above technical problems, the present application provides a spatiotemporal process simulation method based on stacked spatiotemporal memory units, comprising:

[0009] S1: obtaining multi-source big data, preprocessing the multi-source big data to obtain a spatiotemporal data set;

[0010] S2: constructing an initial spatiotemporal process deep learning model, the initial spatiotemporal process deep learning model preliminarily expressing the spatiotemporal dependency relationship of the spatiotemporal data set to obtain a first-stage deep learning model;

[0011] S3: Express a robust policy for the first-stage deep learning model to obtain the second-stage deep learning model;

[0012] S4: Optimize the parameters of the second-stage deep learning model to obtain a spatiotemporal process deep learning model that has completed one round of training;

[0013] S5: Repeat steps S2 to S4 for iterative training. Once the preset conditions are met, a well-trained spatiotemporal process deep learning model is obtained.

[0014] S6: Adjust the sensitivity of the trained spatiotemporal process deep learning model through sensitivity analysis. After adjustment, the final spatiotemporal process deep learning model is obtained. The spatiotemporal process simulation results are obtained through the final spatiotemporal process deep learning model.

[0015] Preferably, step S2 specifically includes:

[0016] S21: The initial spatiotemporal process deep learning model is constructed by stacking multiple spatiotemporal memory units. The initial spatiotemporal process deep learning model includes n+1 spatiotemporal memory columns, and each spatiotemporal memory column includes L layers of spatiotemporal memory units.

[0017] S22: Transfer the spatiotemporal data X from the spatiotemporal dataset. t Input the spatiotemporal memory sequence at time step t, and use the hidden state of layer l at time step (t-1). Time memory state and the spatial memory state of layer l-1 at time step t Perform a preliminary representation of the spatiotemporal dependencies to obtain the hidden state of layer l at time step t. Time memory state and spatial memory state Then increment the value of t by one; where t is the time step number and l is the layer number of the spatiotemporal memory unit;

[0018] S23: Repeat step S22 to complete the preliminary expression of the spatiotemporal dependencies of all spatiotemporal memory columns, and obtain the first-stage deep learning model.

[0019] Preferably, step S22 specifically includes:

[0020] S221: Using the spatiotemporal data X at time step t t The hidden state of layer l at time step t-1 and time memory state Calculate and obtain the time memory state of layer l at time step t. The calculation formula is:

[0021]

[0022]

[0023]

[0024]

[0025] wherein, * represents 3D convolution operation, represents Hadamard product, represents sigmoid activation function, represents tanh activation function, f t is a forget gate, i t is an input gate, g t is an input modulation gate, W xi is the weight of spatiotemporal data in the input gate, W hi is the weight of hidden state in the input gate, W xg is the weight of spatiotemporal data in the input modulation gate, W hg is the weight of hidden state in the input modulation gate, W xf is the weight of spatiotemporal data in the forget gate, W hf is the weight of hidden state in the forget gate;

[0026] S222: obtain the hidden state of the l-th layer at the t-th time step X t and the hidden state of the l-th layer at the t-1-th time step and the temporal memory state of the l-th layer at the t-th time step calculate the spatial memory state of the l-th layer at the t-th time step and the hidden state The calculation formula is:

[0027]

[0028]

[0029]

[0030]

[0031]

[0032]

[0033]

[0034] wherein, o t is an output gate, i′ t is an additional input gate, g′ t is an additional input modulation gate, o′ t is an additional output gate, is the hidden state in the process, Z is the fused feature, W xo is the weight of the spatiotemporal data in the output gate, W ho is the weight of the hidden state in the output gate, W m；zi is the weight of the spatial memory state in the additional input gate, W m；hi is the weight of the spatial memory state corresponding hidden state in the additional output gate, W m；zg is the weight of the spatial memory state in the additional input modulation gate, W m；hg is the weight of the spatial memory state corresponding hidden state in the additional input modulation gate, W m；zo is the weight of the spatial memory state in the additional output gate, W m；ho is the weight of the spatial memory state corresponding hidden state in the additional output modulation gate.

[0035] Preferably, step S3 is specifically:

[0036] The first-stage deep learning model is expressed by a robust strategy through a reverse sampling strategy, and a second-stage deep learning model is obtained.

[0037] Preferably, step S4 is specifically:

[0038] The second-stage deep learning model is parameter-optimized and weight-updated by an Adam optimizer, and a spatiotemporal process deep learning model after one round of training is obtained.

[0039] Preferably, step S6 is specifically:

[0040] S61: determining input parameters and sample ranges of the trained spatiotemporal process deep learning model;

[0041] S62: generating input samples of the trained spatiotemporal process deep learning model by a sample function;

[0042] S63: inputting the input samples into an analyze model to obtain analyze output;

[0043] S64: running an analyze function on the analyze output to obtain an optimal sensitivity index, and adjusting sensitivity of the trained spatiotemporal process deep learning model to the optimal sensitivity index.

[0044] A spatiotemporal process simulation system based on a stacked spatiotemporal memory unit, comprising:

[0045] A data acquisition module is configured to acquire multi-source big data, pre-process the multi-source big data, and obtain a spatiotemporal data set.

[0046] The first stage training module is used for constructing an initial spatiotemporal process deep learning model, the initial spatiotemporal process deep learning model preliminarily expresses the spatiotemporal dependence relationship of the spatiotemporal data set, and a first stage deep learning model is obtained;

[0047] The second stage training module is used for expressing a robustness strategy for the first stage deep learning model, and a second stage deep learning model is obtained;

[0048] The third stage training module is used for parameter optimization of the second stage deep learning model, and a spatiotemporal process deep learning model after one round of training is obtained;

[0049] The iterative training module is used for iterative training, and a trained spatiotemporal process deep learning model is obtained after a preset condition is met;

[0050] The sensitivity adjustment module is used for adjusting the sensitivity of the trained spatiotemporal process deep learning model through sensitivity analysis, and a final spatiotemporal process deep learning model is obtained after adjustment, and a spatiotemporal process simulation result is obtained through the final spatiotemporal process deep learning model.

[0051] The present application has the following advantages:

[0052] 1. A spatiotemporal process deep learning model based on a stacked time memory unit is constructed, a gated mechanism and an attention mechanism are used to establish a transmission mode of spatiotemporal features in a spatiotemporal memory stream, long-term time series time sequence dependence is considered by decoupling time and space memory, and a self-attention feature aggregation mechanism is used to calculate a pair of similarity scores, so that the current and memory features are fused together and used to remember features with long-term spatial and temporal dependence.

[0053] 2. The stacked time memory unit enables the spatiotemporal process deep learning model to have strong robustness and long-term dynamic modeling capability, and the spatiotemporal memory stream enhances the network's modeling capability for short-term dynamics.

[0054] 3. The data, methods, features and knowledge used for constructing the spatiotemporal process deep learning model are scalable, and the focus of the present application is to extract features by decoupling time and space dependence, which is suitable for various spatiotemporal processes, and in principle, there is no restriction on the data, methods, features and knowledge used by the model. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 The present application is an embodiment method flowchart;

[0056] Figure 2 It is a multi-source big data preprocessing schematic diagram;

[0057] Figure 3 It is a structure diagram of the spatiotemporal process deep learning model;

[0058] Figure 4 Structure diagram of a spatiotemporal memory unit;

[0059] Figure 5 First comparison diagram of simulation effects of various deep learning methods;

[0060] Figure 6 Second comparison diagram of simulation effects of various deep learning methods;

[0061] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0062] It should be understood that the specific embodiments described herein are merely intended to explain the present application and are not intended to limit the present application.

[0063] With reference to Figure 1 The present application provides a spatiotemporal process simulation method based on a stacked spatiotemporal memory unit, comprising:

[0064] S1: obtaining multi-source big data, preprocessing the multi-source big data to obtain a spatiotemporal data set;

[0065] S2: constructing an initial spatiotemporal process deep learning model, the initial spatiotemporal process deep learning model preliminarily expressing a spatiotemporal dependency relationship of the spatiotemporal data set to obtain a first-stage deep learning model;

[0066] S3: expressing a robustness strategy for the first-stage deep learning model to obtain a second-stage deep learning model;

[0067] S4: optimizing parameters of the second-stage deep learning model to obtain a spatiotemporal process deep learning model after one round of training;

[0068] S5: repeating steps S2 to S4 for iterative training, and obtaining a trained spatiotemporal process deep learning model after satisfying a preset condition;

[0069] S6: adjusting sensitivity of the trained spatiotemporal process deep learning model through sensitivity analysis, obtaining a final spatiotemporal process deep learning model after the adjustment, and obtaining a spatiotemporal process simulation result through the final spatiotemporal process deep learning model.

[0070] Specifically, the multi-source big data selects the classic data set MovingMNIST, which contains 10,000 video sequences, each of which is composed of 20 frames; in each video sequence, two numbers move independently around the frame, and the spatial resolution of the frame is 64*64 pixels; the numbers often intersect and bounce off the edges of the frame; for each time sequence, the first 10 frames are used to simulate the prediction of the last 10 frames, and the training set, test set and validation set are divided for training, testing and validation, respectively;

[0071] The spatio-temporal process has non-stationarity in the time dimension and the space dimension, that is, it shows certain heterogeneity in different ways; in order to ensure the stability of the model input data, the pre-processing process of the multi-source big data is strictly specified, such as Figure 2 As shown, the multi-source big data is sequentially de-dimensioned, normalized, rasterized, and unified in time sequence and data fusion, and the interpolation method is used to interpolate the time and space missing values, respectively, to obtain a spatio-temporal data set with consistent time sequence and spatial range;

[0072] 1) Expression of spatio-temporal dependence, for data containing spatio-temporal information, each time of observation data needs to be expressed as a 2D (only one observation) or 3D vector (containing multiple observations) according to its spatial coordinates and observation, where the index of spatial latitude is M, N, and the index of observation is P;

[0073] 2) Spatio-temporal interpolation and sampling, in order to ensure that the spatio-temporal process analysis meets the required spatio-temporal resolution, the spatio-temporal data collected should meet the requirements of spatio-temporal resolution as much as possible; for spatio-temporal data that do not meet the requirements, interpolation needs to be performed in both time and space dimensions; the spatial interpolation methods include inverse distance weighting, Kriging interpolation, spline function method, etc. When interpolating in time, the observation at the same spatial position is a set of time series data, which can be interpolated by taking the front and rear adjacent data, or by weighted average according to the distance between the front and rear time, or by linear interpolation;

[0074] 3) Log smoothing, spatio-temporal data has non-stationarity in both time dimension and space dimension. In order to enable the spatio-temporal process deep learning model to better capture the stationary and non-stationary characteristics in the spatio-temporal data, the data is log smoothed before input, so as to reduce the non-stationary characteristics as much as possible.

[0075] 4) Normalization, in order to make data of different sources and different dimensions comparable, normalization needs to be performed on the spatio-temporal data in the data pre-processing process, so that the characteristics are unified in a roughly same numerical interval. At the same time, data normalization can accelerate the convergence process of deep learning algorithm.

[0076] 5) Data fusion, after preprocessing the data from different sources, the observation data at each time can be spliced according to the dimension of the observation to obtain the fused spatio-temporal data.

[0077] In this embodiment, the structure of the spatio-temporal process deep learning model is as shown in the figure Figure 3 , which learns the time and space dependence and its feature expression from the input spatio-temporal data set (X0, X1,..., X n ) by stacking spatio-temporal memory units STA-LSTM Unit (Spatiotemporal Attention Long Short-Term Memory Unit, STA-LSTM Unit). The input gate, input modulation gate, forget gate and output gate of the spatio-temporal memory unit are no longer dependent on the hidden state and time memory state of the previous time step in the same layer, but dependent on the hidden state time memory state and spatial memory state provided by the previous layer at the current time step (l∈1,…,L); in this way, the corresponding states of each layer are updated in the entire spatio-temporal process deep learning model, and the architecture of the entire spatio-temporal process deep learning model updates the memory state in the form of spatio-temporal memory flow, and the feature knowledge is efficiently transmitted from the input to the output in the form of "zigzag" direction; the spatio-temporal memory unit respectively expresses the long-term and short-term time and space dependence, and the spatio-temporal memory flow provides a circular expressway, which can reduce the information loss from the bottom layer to the upper layer, and enhances the network's modeling ability for short-term dynamics;

[0078] Step S2 is specifically:

[0079] S21: An initial spatio-temporal process deep learning model is constructed by stacking multiple spatio-temporal memory units, and the initial spatio-temporal process deep learning model includes n+1 spatio-temporal memory columns, and each spatio-temporal memory column includes L layers of spatio-temporal memory units;

[0080] S22: The spatio-temporal data X t in the spatio-temporal data set is input into the spatio-temporal memory column at the t-th time step, and the hidden state time memory state and the spatial memory state of the l-th layer at the t-1-th time step and the hidden state time memory state and spatial memory state of the l-1-th layer at the t-th time step are used to preliminarily express the spatio-temporal dependence relationship to obtain the hidden state of the l-th layer at the t-th time step, and then the value of t is increased by one; wherein t is the number of time steps, and l is the layer number of the spatio-temporal memory unit;

[0081] S23: Repeat step S22, after completing the preliminary expression of the spatio-temporal dependence relationship of all spatio-temporal memory columns, obtain the first stage deep learning model.

[0082] In this embodiment, the structure of the spatio-temporal memory unit is as shown in Figure 4 Step S22 is specifically:

[0083] S221: Calculate the time memory state of the l-th layer at the t-th time step by the spatio-temporal data X t , the hidden state of the l-th layer at the t-1-th time step and the time memory state The calculation formula is:

[0084]

[0085]

[0086]

[0087]

[0088] Wherein, * represents 3D convolution operation, represents Hadamard product, represents sigmoid activation function, represents tanh activation function, f t is the forget gate, i t is the input gate, g t is the input modulation gate, W xi is the weight of spatio-temporal data in the input gate, W hi is the weight of hidden state in the input gate, W xg is the weight of spatio-temporal data in the input modulation gate, W hg is the weight of hidden state in the input modulation gate, W xf is the weight of spatio-temporal data in the forget gate, W hf is the weight of hidden state in the forget gate.

[0089] S222: Calculate the spatial memory state of the l-th layer at the t-th time step by the spatio-temporal data X t and the hidden state of the l-th layer at the t-1-th time step and the time memory state of the l-th layer at the t-th time step The calculation formula is:

[0090]

[0091] ​​​

[0092]

[0093]

[0094]

[0095]

[0096]

[0097] where o t is the output gate, i′ t is the additional input gate, g′ t is the additional input modulation gate, o′ t is the additional output gate, is the hidden state in the process, Z is the fused feature, W xo is the weight of the spatio-temporal data in the output gate, W ho is the weight of the hidden state in the output gate, W m；zi is the weight of the spatial memory state in the additional input gate, W m；hi is the weight of the spatial memory state corresponding to the hidden state in the additional output gate, W m；zg is the weight of the spatial memory state in the additional input modulation gate, W m；hg is the weight of the spatial memory state corresponding to the hidden state in the additional input modulation gate, W m；zo is the weight of the spatial memory state in the additional output gate, W m；ho is the weight of the spatial memory state corresponding to the hidden state in the additional output modulation gate.

[0098] Specifically, in each time step, the fused feature Z is the feature fusion of Z h and Z m , Z m is the query aggregation of the previous layer , which is mapped to the key K mk and the value V hk through 1x1 convolution with weights W m and W m , then the similarity score between the input and the memory is calculated through the matrix multiplication between the query Q h and the key K m as follows:

[0099]

[0100] All weights used to aggregate the features are obtained by applying the SoftMax function along each row, and the function formula is as follows:

[0101]

[0102] Then, the feature Z m The "pixel" in the i-th position in the V m The weighted sum of all positions is calculated, and the formula is as follows, For the memory of the j-th column:

[0103]

[0104] Finally, by analogy, Z = W z [Z h ; Z m ].

[0105] In this embodiment, step S3 is specifically:

[0106] The first-stage deep learning model is expressed by a robust strategy through a reverse sampling strategy to obtain a second-stage deep learning model.

[0107] Specifically, the reverse sampling strategy is a learning strategy applied to the input frame of the encoding time step, which forces the recurrent model to learn long-term dynamics from historical observations and improves the long-term modeling capability.

[0108] The reverse sampling strategy specifically includes reading the corresponding tensor of the spatio-temporal data set, setting the model parameters, including the data space resolution, time series length, hidden layer dimension, learning rate, sampling strategy, and the like, reading the corresponding dimension, and putting it into the spatio-temporal process simulation model STPNet for training and learning.

[0109] In this embodiment, step S4 is specifically:

[0110] The second-stage deep learning model is parameter-optimized and weight-updated by an Adam optimizer to obtain a spatio-temporal process deep learning model after one round of training.

[0111] Specifically, in order to adapt to different scenes and spatio-temporal data sets, the spatio-temporal process deep learning model can automatically optimize the parameters and hyperparameters of the model, and when optimizing the model parameters, an Adaptive Moment Estimation (Adam) optimizer is used to update the weights and optimize the parameters.

[0112] In this embodiment, step S6 is specifically:

[0113] S61: determining the input parameters and sample range of the trained spatio-temporal process deep learning model;

[0114] S62: generating the input sample of the trained spatio-temporal process deep learning model by a sample function;

[0115] S63: inputting the input sample into the analyze model to obtain an analyze output;

[0116] S64: running an analyze function on the analyze output to obtain an optimal sensitivity index, and adjusting the sensitivity of the trained spatiotemporal process deep learning model to the optimal sensitivity index.

[0117] Specifically, in the analysis of the sensitivity of the physical influencing factors in the spatiotemporal process, the One-at-a-time (OAT) method is used. First, for the input physical observation, the time resolution, the spatial resolution, and the physical variable of each are modified in turn, and the model is run respectively. Next, the influence of the table on the output of the model is observed. Through the sensitivity analysis of the input variables, the input variables that have no influence on the input of the spatiotemporal process deep learning model can be eliminated, so as to reduce the difficulty of data collection and reduce the computational complexity of the model.

[0118] The accuracy verification dataset can be used to verify the accuracy of the spatiotemporal process simulation results of the final spatiotemporal process deep learning model, and the evaluation indexes are mean square error (MSE), structural similarity (SSIM), peak signal-to-noise ratio (PSNR), and learning perceptual image block similarity (LPIPS).

[0119] In order to further verify the advantages of the final spatiotemporal process deep learning model of the present application relative to other conventional models, and verify its effectiveness and performance accuracy, a plurality of spatiotemporal feature deep learning models are selected for comparison. Under the same input conditions, the spatiotemporal process simulation effects of different models are compared, and the improvement degree of the proposed spatiotemporal process simulation model compared with other deep learning models is verified.

[0120] The MovingMNIST simulation result evaluation index is shown in Table 1. The final spatiotemporal process deep learning model STPNet (Spatiotemporal Process Network) of the present application is the best in the four index corresponding effects, the mean square error is reduced by 22.14% compared with ConvLSTM, 13.3% compared with PredRNN, 7.8% compared with MIM, and 7.3% compared with PredRNN-v2. The KTH Action dataset simulation result evaluation index is shown in Table 2, and especially the LPIPS is reduced by 42.4% compared with ConvLSTM. The simulation effects of the two datasets are shown in Figures 1 and 2, respectively. Figure 5 、 Figure 6The improvement of multiple indexes proves that the learning efficiency and feature extraction of the final spatio-temporal process deep learning model of the application are better than those of other models, the application has high generalization ability, and the stacking connection between the multiple spatio-temporal memory units guarantees the effective transmission of error back propagation and spatio-temporal memory flow in time and space, ensures the dynamic adjustment of the spatio-temporal memory unit to the spatial correlation continuation and reservation, so that the model further strengthens the spatio-temporal fusion ability on the basis of the recurrent neural network, and has high dynamic modeling and long-term modeling ability.

[0121] Table 1 Comparison results of each model simulation of the MovingMNIST dataset (test set)

[0122]

[0123] In addition, the KTH Action dataset is tested and verified, the KTH dataset is one of the most standard datasets, which contains six actions: walking, jogging, running, punching, waving and clapping. In order to illustrate the subtle difference in performance, each action is performed by 25 different individuals, and each action of each actor is systematically changed. The setting changes include: outdoor (S1), outdoor with scale changes (S2), wearing different clothes (S3) and indoor (S4). These changes test the ability of each algorithm to recognize actions independent of background, appearance of participants and participant scale. For each time series, the first 10 frames are predicted to predict the last 20 frames, so as to verify the simulation of the model to the long-term case.

[0124] Table 2 Comparison results of each model simulation of the KTH Action dataset (test set)

[0125]

[0126] The application provides a spatio-temporal process simulation system based on stacked spatio-temporal memory units, comprising:

[0127] The data acquisition module is used for acquiring multi-source big data, pre-processing the multi-source big data, and obtaining a spatio-temporal dataset;

[0128] The first stage training module is used for constructing an initial spatio-temporal process deep learning model, the initial spatio-temporal process deep learning model preliminarily expresses the spatio-temporal dependence relationship of the spatio-temporal dataset, and a first stage deep learning model is obtained;

[0129] The second stage training module is used for expressing a robustness strategy for the first stage deep learning model, and a second stage deep learning model is obtained;

[0130] The third stage training module is used for parameter optimization of the second stage deep learning model, and a spatio-temporal process deep learning model after one round of training is obtained;

[0131] an iterative training module configured to perform iterative training, and obtain a trained spatiotemporal process deep learning model when a preset condition is met;

[0132] a sensitivity adjustment module configured to adjust sensitivity of the trained spatiotemporal process deep learning model through sensitivity analysis, and obtain a final spatiotemporal process deep learning model after the adjustment is completed, and obtain a spatiotemporal process simulation result through the final spatiotemporal process deep learning model.

[0133] It should be noted that in this document, the terms "comprising", "containing", or any other variant thereof are intended to cover a non-exclusive inclusion, so that a process, method, article, or system that comprises a list of elements does not only include those elements, but can also include other elements not expressly listed, or inherent to such process, method, article, or system. Without more limitations, an element defined by the phrase "comprising a" does not exclude the presence of additional identical elements in the process, method, article, or system that includes the element.

[0134] The above-mentioned embodiment numbers of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments. In the unit claims in which several devices are listed, several of these devices can be embodied by the same hardware item. The use of the words first, second, and third does not represent any order, and these words can be interpreted as identifiers.

[0135] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation, or direct or indirect application in other related technical fields, which is made by using the content of the present application specification and drawings, is also included in the patent protection scope of the present application.

Claims

1. A method for simulating spatiotemporal processes based on stacked spatiotemporal memory units, characterized in that, include: S1: Acquire multi-source big data, preprocess the multi-source big data, and obtain spatiotemporal datasets; S2: Construct an initial spatiotemporal process deep learning model. The initial spatiotemporal process deep learning model provides a preliminary representation of the spatiotemporal dependencies of the spatiotemporal dataset, thus obtaining the first-stage deep learning model. Step S2 is as follows: S21: The initial spatiotemporal process deep learning model is constructed by stacking multiple spatiotemporal memory units. The initial spatiotemporal process deep learning model includes n+1 spatiotemporal memory columns, and each spatiotemporal memory column includes L layers of spatiotemporal memory units. S22: Transfer the spatiotemporal data X from the spatiotemporal dataset. t Input the spatiotemporal memory sequence at time step t, and use the hidden state of layer l at time step (t-1). Time memory state and the spatial memory state of layer l-1 at time step t Perform a preliminary representation of the spatiotemporal dependencies to obtain the hidden state of layer l at time step t. Time memory state and spatial memory state Then increment the value of t by one; where t is the time step number and l is the layer number of the spatiotemporal memory unit; Step S22 is as follows: S221: Using the spatiotemporal data X at time step t t The hidden state of layer l at time step t-1 and time memory state Calculate and obtain the time memory state of layer l at time step t. The calculation formula is: in, This represents a 3D convolution operation. Represents the Hadamard product. f represents the sigmoid activation function, tanh represents the tanh activation function, and f t For the Gate of Oblivion, i t For input gate, g t For input modulation gate, The weights of spatiotemporal data in the input gate, The weights of the hidden state in the input gate, The weights of spatiotemporal data in the input modulation gate, The weights of the hidden state in the input modulation gate, The weights of spatiotemporal data in the forget gate, The weight of the hidden state in the forget gate; S222: Using the spatiotemporal data X at time step t t and the hidden state of layer l at time step t-1 and the time memory state of layer l at time step t. Calculate the spatial memory state of layer l at time step t. and hidden state The calculation formula is: in, For output gate, For additional input gates, For additional input modulation gate, For additional output gates, Z represents the hidden state during the process, and Z represents the fused feature. The weights of spatiotemporal data in the output gate, The weight of the hidden state in the output gate, The weights of the spatial memory states in the additional input gate. The weight of the hidden state corresponding to the spatial memory state in the additional output gate. The weights of the spatial memory state in the additional input modulation gate. The weights of the hidden states corresponding to the spatial memory states in the additional input modulation gate. The weights of the spatial memory states in the additional output gate. The weights of the hidden states corresponding to the spatial memory states in the additional output modulation gate; S23: Repeat step S22 to complete the preliminary expression of the spatiotemporal dependencies of all spatiotemporal memory columns and obtain the first-stage deep learning model; S3: Express a robust policy for the first-stage deep learning model to obtain the second-stage deep learning model; S4: Optimize the parameters of the second-stage deep learning model to obtain a spatiotemporal process deep learning model that has completed one round of training; S5: Repeat steps S2 to S4 for iterative training. Once the preset conditions are met, a well-trained spatiotemporal process deep learning model is obtained. S6: Adjust the sensitivity of the trained spatiotemporal process deep learning model through sensitivity analysis. After adjustment, the final spatiotemporal process deep learning model is obtained. The spatiotemporal process simulation results are obtained through the final spatiotemporal process deep learning model.

2. The spatiotemporal process simulation method based on stacked spatiotemporal memory units according to claim 1, characterized in that, Step S3 is as follows: By employing an antisampling strategy to robustly express the first-stage deep learning model, a second-stage deep learning model can be obtained.

3. The spatiotemporal process simulation method based on stacked spatiotemporal memory units according to claim 1, characterized in that, Step S4 is as follows: The Adam optimizer is used to optimize the parameters and update the weights of the second-stage deep learning model, resulting in a spatiotemporal process deep learning model that has completed one round of training.

4. The spatiotemporal process simulation method based on stacked spatiotemporal memory units according to claim 1, characterized in that, Step S6 is as follows: S61: Determine the input parameters and sample range of the trained spatiotemporal process deep learning model; S62: Generate input samples for the trained spatiotemporal process deep learning model using the sample function; S63: Input the input sample into the analyze model and obtain the analyze output; S64: Run the analyze function on the analyze output to calculate the optimal sensitivity index, and adjust the sensitivity of the trained spatiotemporal process deep learning model to the optimal sensitivity index.

5. A spatiotemporal process simulation system based on stacked spatiotemporal memory units, characterized in that, include: The data acquisition module is used to acquire multi-source big data, preprocess the multi-source big data, and obtain spatiotemporal datasets; The first-stage training module is used to construct an initial spatiotemporal process deep learning model. This initial model provides a preliminary representation of the spatiotemporal dependencies in the spatiotemporal dataset, resulting in the first-stage deep learning model. Specifically: S21: The initial spatiotemporal process deep learning model is constructed by stacking multiple spatiotemporal memory units. The initial spatiotemporal process deep learning model includes n+1 spatiotemporal memory columns, and each spatiotemporal memory column includes L layers of spatiotemporal memory units. S22: Transfer the spatiotemporal data X from the spatiotemporal dataset. t Input the spatiotemporal memory sequence at time step t, and use the hidden state of layer l at time step (t-1). Time memory state and the spatial memory state of layer l-1 at time step t Perform a preliminary representation of the spatiotemporal dependencies to obtain the hidden state of layer l at time step t. Time memory state and spatial memory state Then increment the value of t by one; where t is the time step number and l is the layer number of the spatiotemporal memory unit; Step S22 is as follows: S221: Using the spatiotemporal data X at time step t t The hidden state of layer l at time step t-1 and time memory state Calculate and obtain the time memory state of layer l at time step t. The calculation formula is: in, This represents a 3D convolution operation. Represents the Hadamard product. f represents the sigmoid activation function, tanh represents the tanh activation function, and f t For the Gate of Oblivion, i t For input gate, g t For input modulation gate, The weights of spatiotemporal data in the input gate, The weights of the hidden state in the input gate, The weights of spatiotemporal data in the input modulation gate, The weights of the hidden state in the input modulation gate, The weights of spatiotemporal data in the forget gate, The weight of the hidden state in the forget gate; S222: Using the spatiotemporal data X at time step t t and the hidden state of layer l at time step t-1 and the time memory state of layer l at time step t. Calculate the spatial memory state of layer l at time step t. and hidden state The calculation formula is: in, For output gate, For additional input gates, For additional input modulation gate, For additional output gates, Z represents the hidden state during the process, and Z represents the fused feature. The weights of spatiotemporal data in the output gate, The weight of the hidden state in the output gate, The weights of the spatial memory states in the additional input gate. The weight of the hidden state corresponding to the spatial memory state in the additional output gate. The weights of the spatial memory state in the additional input modulation gate. The weights of the hidden states corresponding to the spatial memory states in the additional input modulation gate. The weights of the spatial memory states in the additional output gate. The weights of the hidden states corresponding to the spatial memory states in the additional output modulation gate; S23: Repeat step S22 to complete the preliminary expression of the spatiotemporal dependencies of all spatiotemporal memory columns and obtain the first-stage deep learning model; The second-stage training module is used to express robust policies for the first-stage deep learning model and obtain the second-stage deep learning model. The third-stage training module is used to optimize the parameters of the second-stage deep learning model to obtain a spatiotemporal process deep learning model that has completed one round of training. The iterative training module is used to perform iterative training and obtain a trained spatiotemporal process deep learning model after meeting preset conditions. The sensitivity adjustment module is used to adjust the sensitivity of the trained spatiotemporal process deep learning model through sensitivity analysis. After adjustment, the final spatiotemporal process deep learning model is obtained, and the spatiotemporal process simulation results are obtained through the final spatiotemporal process deep learning model.

Images