A spatiotemporal process deep learning model real-time stream data incremental learning method
By merging historical and real-time data in a spatiotemporal deep learning model, calculating replay scores, and selecting representative subsets for incremental training, the problem of large model data volume and difficulty in achieving sufficient accuracy is solved, resulting in high training efficiency and improved accuracy.
Patent Information
- Application Number
- CN202310037734.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-01-10
- Publication Date
- 2026-02-17
- Estimated Expiration
- 2043-01-10
AI Technical Summary
Existing technologies face the challenge of simultaneously meeting the requirements of large model data volume and high accuracy when using deep learning models for spatiotemporal process analysis and mining. In particular, traditional methods struggle to efficiently perform incremental training when dealing with a continuous stream of new data.
We employ a real-time streaming data incremental learning method using a spatiotemporal process deep learning model. By merging historical and real-time data, we calculate replay scores and select representative subsets for incremental training. We also apply regularization constraints to train the model, taking into account data distribution patterns and temporal characteristics.
It improves training efficiency and reduces accuracy loss, and is applicable to deep learning models for different spatiotemporal processes. In particular, in the analysis of rainfall and runoff data in the Jinsha River Basin, the training efficiency is improved by 3.95-5.45 times, and the accuracy loss is less than 10%.
Smart Images
Figure CN116011527B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the fields of geographic information analysis and mining and geographic intelligence technology, and specifically relates to a real-time streaming data incremental learning method for a spatiotemporal process deep learning model. Background Technology
[0002] When using deep learning models for spatiotemporal process analysis and mining, the continuous influx of new streaming data often leads to unbearable time overhead when using all historical data for analysis and mining, presenting a dilemma where the model's data volume and accuracy cannot be simultaneously satisfied. Generally, sparsity is a core idea for accelerating the training of various neural networks, referring to reducing some less important input data or intermediate processes in neural network training. From the perspective of reducing model input data, a method can be proposed that, while periodically training with historical data, select a small, representative dataset for incremental training of the deep learning model on real-time streaming data. This method considers the characteristics of spatiotemporal data and draws on the paradigms of replay and regularization methods.
[0003] Replay methods refer to re-inputting a subset of previously learned data into the model for training when new incremental data arrives. Early replay methods were inspired by biology and typically focused on how to use partial historical data for training. Data replayed from the past can be used to constrain parameter updates based on new data, keeping them within the previously feasible range. Replay methods usually set up a buffer to store partial historical data and play random data from the buffer, or replay surviving pseudo-data from a generative model trained on previously learned data. Replay methods can be further divided into original replay methods and generative replay methods, referring to using real historical data and generating pseudo-data, respectively. With the rise of Generative Adversarial Networks (GANs), generative replay methods are often chosen when previous data is difficult to obtain. The fundamental problem of replay is the selection criteria for the replay data. Some successful incremental learning methods store a subset of the previously learned original input in a replay buffer. The sample selection process can be described as an optimization problem of maximizing constrained data diversity; similarity metrics also influence sample selection. Overall selective sampling methods for replay work well in certain specific situations, while uniform random sampling still has strong adaptability for averaging problems. In addition to improving accuracy, selective replay methods can also improve data efficiency, requiring less data for the network to learn new information. After learning other similar tasks, the neural network needs less training time to reach the target accuracy for a new task. These findings on task similarity and network performance in terms of accuracy and time have been investigated and can be further discussed by considering inputs as time series. Regardless of the specific form of replay, combining replay with regularization paradigms can improve model performance.
[0004] Regularization refers to freezing certain parts of the model during training for consecutive tasks. This can be understood as storing knowledge on how to solve different tasks in different parts of the model so that training for subsequent tasks does not interfere with this knowledge. Sometimes, the weights are not completely fixed but regularized, preventing drastic changes when training the model across different tasks. In Elastic Weight Consolidation (EWC), the learning speed slows down when training the model across a series of tasks due to the weights that are important for previous tasks. Specifically, EWC can be extended by reparameterizing the network to approximate the Fisher information matrix of the network parameters by diagonalizing the network parameters. Reparameterization results in a factorial rotation of the parameter space, making the diagonal Fisher information matrix assumption more applicable. Furthermore, other improvements have been proposed for calculating important weights. Memory Aware Synapses (MAS), which has a similar structure to EWC, adds a term to the loss function, obtained from the squared gradient of the l2 norm of the model output.
[0005] Time series similarity is an important indicator for feature analysis of time series data. Its theoretical description was proposed when dealing with time series similarity / dissimilarity searches and has gained widespread acceptance in the field of data mining, leading to the development of various measurement methods. Under different lengths and irregular sampling intervals, different distance measurement methods attempt to specify a similarity formula by simultaneously considering both the time series representation and the original time series. Commonly used methods include Euclidean distance, Hausdorff distance (a modified version of Longest Common Subsequence (LCSS)), and Dynamic Time Warping (DTW). The simplest idea behind these methods is to treat two time series as univariate variables and calculate the distance measurement at all time points. However, time series data, especially Earth science data, often have high dimensionality and complex structures, placing high demands on addressing characteristics, time series length, and representation methods. Based on these three objectives, time series measurement methods can be broadly categorized into time-based methods, shape-based methods, and structure-based methods. Time-based methods consider the similarity at each time step, making the use of basic distance metrics such as Euclidean distance highly suitable. To reduce computation on the original time series, Fourier transform and piecewise aggregation approximation (PAA) are commonly used to measure distance on the transformed time series. Shape-based methods focus more on the similarity of the sequence shape, without considering the temporal features of points. The shape of a time series is a local feature, often reflecting short-term changes. For structure-based methods, Hidden Markov Models (HMMs) and other modeling processes are typically used to measure model parameters and the time series. The structure of a time series has global features, focusing on long-term changes. Although dynamic programming-based methods are time-consuming (the cost of comparing two time series is quadratic of the time series length), they achieve a balance between global and local features and are the most efficient. Preliminary conclusions from the above methods and related literature suggest that the similarity of time series depends on utilizing similar targets. To date, most research has been based on relatively small datasets, proposing various measurement methods based on temporal and global or local structural features. Among these methods, the most commonly used methods, such as Euclidean distance and DTW, still exhibit good performance. Summary of the Invention
[0006] In view of this, the present invention proposes a real-time streaming data incremental learning method for spatiotemporal process deep learning models, comprising the following steps:
[0007] S1. Prepare runoff and rainfall data for a historical period to obtain a data stream; construct a spatiotemporal process deep learning model, input the data stream into the spatiotemporal process deep learning model for periodic training, and obtain model parameters. The data that has been processed and trained in the spatiotemporal process deep learning model is called historical data, and the incremental data that needs to be trained at this moment is called real-time data.
[0008] S2. After merging historical data and real-time data, periodically slice them to obtain multiple subsets. Calculate the difference in distribution parameters and the difference in temporal similarity measure between the subsets and all data to obtain the replay score and sort them.
[0009] S3. Select a subset of data with high replay scores as replay data for incremental training. During incremental training, calculate the importance of parameters, apply regularization constraints to the training loss of the spatiotemporal process deep learning model, and adjust the hyperparameters of the model.
[0010] S4. Obtain the training results using the real-time streaming data incremental learning method of the spatiotemporal process deep learning model.
[0011] Furthermore, step S1 specifically includes:
[0012] S11. Select a suitable spatiotemporal deep learning model and construct a spatiotemporal deep learning model based on different attention-RNN models. The model network structure is composed of an encoder and a decoder, and includes an encoder layer composed of RNNs, an encoder attention layer, a decoder attention layer, a decoder layer composed of RNNs, and two fully connected layers. The RNN units are in the form of three types: original RNN, LSTM, and GRU, which respectively form three spatiotemporal deep learning models: attention-RNN, attention-LSTM, and attention-GRU.
[0013] S12. Obtain the trained model and parameters. Data is continuously input into the spatiotemporal process deep learning model in the form of a data stream for training, with different moments serving as elements of the time series. , i =1,2,…, the difference between two time points equals the incremental training interval required for real-time streaming data, for time point… Data that has been processed and trained in a deep learning model is called historical data, denoted as . The incremental data that needs to be used for training is called real-time data, denoted as . Let the spatiotemporal process deep learning model be denoted as ,in The model represents the first j The parameters at time... The value, The combination of historical data and real-time data at any given moment becomes Historical data at any given moment The parameters of the deep model trained at time step 1 become the input parameters of the model at that time step 2; attention-RNN, attention-LSTM, and attention-GRU are used respectively for time step 1. Different attention-RNN models were obtained by training on historical data. As a moment Model and initialize relevant parameters;
[0014] S13. Each time there is new incremental data, repeat steps S11 and S12 and update the model and parameters.
[0015] Furthermore, step S2 specifically includes:
[0016] S21. After merging incremental and historical data, analyze the degree of deviation of the overall data skewness distribution and the time series periodicity. The deviation index is selected from the third and fourth moments of a certain sub-time series dataset with respect to the other sub-datasets. The calculation formula is as follows:
[0017]
[0018]
[0019] in, Describes the third central moments of dataset X. Denotes the fourth central moment of dataset X. n Indicates the number of data points. x i Represents the first... i One data point, This represents the mean of dataset X. S This represents the standard deviation of dataset X;
[0020] S22. Divide the overall data into multiple subsets according to time-series periodic slices, and calculate the difference between the higher-order moment index of the subsets and the whole dataset.
[0021] S23. Calculate the temporal similarity metric between subsets. The standardized Euclidean distance is used as the similarity metric, and the calculation formula is as follows:
[0022]
[0023] in, express a , b Standardized Euclidean distance between datasets; They represent a , b Dataset No. j The first dimension i One component;p , n They represent a , b Dataset dimensions and the total number of its components;
[0024] S24. Determine appropriate weights, taking into account the degree of distribution deviation and similarity metrics, to generate a replay score for each subset of data. The calculation formula is as follows:
[0025]
[0026] in, Indicates the replay score. Indicates the weighting coefficient. This represents the difference in skewness between the subset and the overall dataset. This represents the kurtosis difference between the subset and the overall dataset.
[0027] S25. Select the subset of the target magnitude with the highest score as the replay data.
[0028] Furthermore, step S3 specifically includes:
[0029] S31. Input the replay dataset into the model obtained in S12 and initialize the model with the parameters obtained in S12;
[0030] S32. Calculate the importance of parameters and use them as regularization constraints in the loss function of the spatiotemporal process deep learning model. Adjust the model's hyperparameters through incremental training. The loss function is:
[0031]
[0032] in, express The loss function of the model during incremental training at each time step. express The loss function during model training at each time step. This represents the i-th parameter of the model. This represents the regularization constraint imposed on the loss function;
[0033] The beneficial effects of the technical solution provided by this invention are:
[0034] 1. This invention proposes a real-time streaming data incremental learning method for spatiotemporal process deep learning models: In the case of periodically training deep learning models, a small number of representative data are selected from a large amount of data, combined with the data distribution pattern and temporal characteristics, and the importance of parameters is calculated as a regularization constraint of the loss function for incremental training.
[0035] 2. This invention is applicable to deep learning models for different spatiotemporal processes.
[0036] 3. This invention significantly improves training efficiency when training data for spatiotemporal deep learning models while minimizing accuracy loss. Attached Figure Description
[0037] Figure 1 This is a flowchart illustrating the steps of a real-time streaming data incremental learning method for a spatiotemporal process deep learning model according to an embodiment of the present invention.
[0038] Figure 2 This is an overall structural diagram of an embodiment of the present invention;
[0039] Figure 3 This is a flowchart illustrating the execution of an embodiment of the present invention;
[0040] Figure 4 The standard flowchart for selecting playback data in the embodiments of the present invention is shown below. Detailed Implementation
[0041] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described below with reference to the accompanying drawings.
[0042] refer to Figure 1 , Figure 2 and Figure 3 , Figure 1 This is a flowchart illustrating the steps of a real-time streaming data incremental learning method for a spatiotemporal process deep learning model according to an embodiment of the present invention. Figure 2 This is an overall structural diagram of an embodiment of the present invention; Figure 3 This is a flowchart illustrating the execution of an embodiment of the present invention.
[0043] This invention provides a real-time streaming data incremental learning method for spatiotemporal process deep learning models, comprising the following steps:
[0044] S1. Periodically train the spatiotemporal process deep learning model to obtain model parameters.
[0045] This embodiment uses rainfall and runoff data from five important hydrological stations in the Jinsha River basin. The Jinsha River is another name for the upper reaches of the Yangtze River, and its basin extends from 90°E to 105°E and from 24°N to 36°N. Figure 1 As shown. The Jinsha River basin is characterized by steep mountains and deep valleys, making it prone to floods and geological disasters. Therefore, rapid analysis and prediction of runoff data is of great significance. This invention uses runoff and precipitation data from five hydrological stations over a 19-year period from 1998 to 2016. Runoff data are daily observation averages in cubic meters per second, and precipitation data are CMORPH precipitation products with a temporal resolution of 30 minutes and a spatial resolution of 0.25° × 0.25°. The point-source daily rainfall data series, in millimeters, was extracted from the geographic coordinates of six hydrological stations within the daily rainfall grid data.
[0046] S1.1 Selecting a suitable spatiotemporal deep learning model: This embodiment analyzes and mines the aforementioned spatiotemporal data based on different attention-RNN models. The model network structure is based on an encoder and decoder, sequentially including an encoder layer composed of RNNs, an encoder attention layer, a decoder attention layer, a decoder layer composed of RNNs, and two fully connected layers. The RNN units come in three forms: original RNN, LSTM, and GRU, forming attention-RNN, attention-LSTM, and attention-GRU spatiotemporal deep learning models, respectively.
[0047] S1.2. Obtain the trained model and parameters. As time progresses, a continuous stream of data arrives and is fed into the deep learning model for training. During this process, different moments serve as elements of the time series. , i =1, 2, ... The difference between two time points equals the interval required for incremental training on the real-time streaming data. For time point... Historical data is recorded as Real-time data is recorded as Let the spatiotemporal process deep learning model be denoted as ,in This indicates that the j-th parameter of the model is at time j. The value, The combination of historical data and real-time data at any given moment becomes Historical data at any given moment The parameters of the deep model after training at each time step become the input parameters of the model at that time.
[0048] Under conditions of regular training, attention-RNN, attention-LSTM, and attention-GRU were used for time steps respectively. Different attention-RNN models were obtained by training on historical data. As a moment Model and initialize relevant parameters.
[0049] S1.3. Each time there is new incremental data, repeat steps S1.1 and S1.2 and update the model and parameters.
[0050] S2. Next, by analyzing the data distribution patterns and temporal periodicity of different hydrological stations, an appropriate segmentation size is selected. Considering extreme cases, i.e., reducing the number of iterations, sub-time series with distribution patterns similar to the overall data and high temporal similarity are used as the selection criteria for replay data. Historical and real-time data are merged and periodically sliced to obtain multiple subsets. The differences in distribution parameters and temporal similarity metrics between the subsets and the entire dataset are calculated to obtain replay scores, which are then sorted. (Reference) Figure 4 , Figure 4 The standard flowchart for selecting playback data in the embodiments of the present invention is shown below.
[0051] S2.1 After merging incremental and historical data, analyze the degree of deviation of the overall data skewness distribution and the time series periodicity. The deviation index is selected from the third and fourth moments of a certain sub-time series dataset with respect to the other sub-datasets. The calculation formula is as follows:
[0052]
[0053]
[0054] in, Describes the third central moments of dataset X. Denotes the fourth central moment of dataset X. n Indicates the number of data points. x i Represents the first... i One data point, This represents the mean of dataset X. S This represents the standard deviation of dataset X;
[0055] S2.2 Divide the overall data into multiple subsets according to time-series periodic slices, and calculate the difference between the higher-order moment index of the subsets and the whole dataset;
[0056] S2.3 Calculate the temporal similarity metric between subsets. The standardized Euclidean distance is used as the similarity metric, and the calculation formula is as follows:
[0057]
[0058] in, express a , b Standardized Euclidean distance between datasets; They represent a , b Dataset No. j The first dimension i One component; p , n They represent a ,b Dataset dimensions and the total number of its components;
[0059] S2.4 Under the same conditions, the order of magnitude of input data for a deep learning model is positively correlated with training time. The order of magnitude of replay data is determined based on the time required for periodic training and the target expected training time. Appropriate weights are determined, taking into account the degree of distribution deviation and similarity metrics, to form a replay score. The subset of data with the highest target order of magnitude is selected as replay data for incremental training. Appropriate weights are determined, taking into account the degree of distribution deviation and similarity metrics, to generate a replay score for each subset of data. The calculation formula is as follows:
[0060]
[0061] in, Indicates the replay score. Indicates the weighting coefficient. This represents the difference in skewness between the subset and the overall dataset. This represents the kurtosis difference between the subset and the overall dataset.
[0062] S2.5 Select the subset of the target order with the highest score as the replay data.
[0063] S3. Select a subset of data with high replay scores as replay data for incremental training. During incremental training, calculate the importance of parameters, apply regularization constraints to the training loss of the spatiotemporal process deep learning model, and adjust the hyperparameters of the model.
[0064] S3.1 Input the replay dataset into the model obtained in S1.2 and initialize the model with the parameters obtained in S1.2;
[0065] S3.2. Apply regularization constraints to the loss function of the spatiotemporal process deep learning model. Specifically, apply regularization terms to the loss functions of attention-RNN, attention-LSTM, and attention-GRU trained on the incremental dataset. Calculate the parameter importance and use it as the regularization constraint term for the loss function of the spatiotemporal process deep learning model. Adjust the model's hyperparameters and perform incremental training, calculating the importance of the model parameters. The loss function is:
[0066]
[0067] in, express The loss function of the model during incremental training at each time step. express The loss function during model training at each time step. The model number is represented by the first... i One parameter, This represents the regularization constraint imposed on the loss function;
[0068] S4. Obtain the training results using the real-time streaming data incremental learning method of the spatiotemporal process deep learning model.
[0069] This invention proposes a real-time streaming data incremental learning method for spatiotemporal process deep learning models. This method involves selecting a small, representative dataset based on data distribution patterns and temporal data similarity metrics, applying regularization constraints to the loss function, and then incrementally training the model using historical data for periodic training. Compared to non-incremental learning methods that use all data, this method improves training efficiency by 3.95-5.45 times, with an accuracy loss of less than 10%, and achieves a percentage R-squared and NSE loss of less than 0.0846.
[0070] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A real-time streaming data incremental learning method for a spatiotemporal process deep learning model, characterized in that, Includes the following steps: S1. Prepare runoff and rainfall data for a historical period to obtain a data stream; A spatiotemporal process deep learning model is constructed. The data stream is input into the spatiotemporal process deep learning model for periodic training to obtain model parameters. The data that has been processed and trained in the spatiotemporal process deep learning model is called historical data, and the incremental data that needs to be trained at this moment is called real-time data. S2. After merging historical data and real-time data, periodically slice them to obtain multiple subsets. Calculate the difference in distribution parameters and the difference in temporal similarity measure between the subsets and all data to obtain the replay score and sort them. S3. Select a subset of data with high replay scores as replay data for incremental training. During incremental training, calculate the importance of parameters, apply regularization constraints to the training loss of the spatiotemporal process deep learning model, and adjust the hyperparameters of the model. S4. Obtain the training results using the real-time streaming data incremental learning method of the spatiotemporal process deep learning model; Step S2 is as follows: S21. After merging incremental and historical data, analyze the degree of deviation of the overall data skewness distribution and the time series periodicity. The deviation index is selected from the third and fourth moments of a certain sub-time series dataset with respect to the other sub-datasets. The calculation formula is as follows: in, Describes the third central moments of dataset X. Let n represent the fourth central moment of dataset X, and let n represent the number of data points. i This represents the i-th data point in dataset X. Let X represent the mean of dataset X, and S represent the standard deviation of dataset X. S22. Divide the overall data into multiple subsets according to time-series periodic slices, and calculate the difference between the higher-order moment index of the subsets and the whole dataset. S23. Calculate the temporal similarity metric between subsets. The standardized Euclidean distance is used as the similarity metric, and the calculation formula is as follows: in, Represents the standardized Euclidean distance between datasets a and b; Let represent the i-th component of the j-th dimension of datasets a and b, respectively; p and n represent the dimensions and the total number of components of datasets a and b, respectively. S24. Determine appropriate weights, taking into account the degree of distribution deviation and similarity metrics, to generate a replay score for each subset of data. The calculation formula is as follows: in, Indicates the replay score. Indicates the weighting coefficient. This represents the difference in skewness between the subset and the overall dataset. This represents the kurtosis difference between the subset and the overall dataset. S25. Select the subset of the target magnitude with the highest score as the replay data.
2. The real-time streaming data incremental learning method for a spatiotemporal process deep learning model according to claim 1, characterized in that, Step S1 is as follows: S11. Select a suitable spatiotemporal deep learning model and construct a spatiotemporal deep learning model based on different attention-RNN models. The model network structure is composed of an encoder and a decoder, and includes an encoder layer composed of RNNs, an encoder attention layer, a decoder attention layer, a decoder layer composed of RNNs, and two fully connected layers. The RNN units are in the form of three types: original RNN, LSTM, and GRU, which respectively form three spatiotemporal deep learning models: attention-RNN, attention-LSTM, and attention-GRU. S12. Obtain the trained model and parameters. Data is continuously input into the spatiotemporal process deep learning model in the form of a data stream for training, with different moments serving as elements of the time series. , i =1,2,…, the difference between two time points equals the incremental training interval required for real-time streaming data, for time point… Historical data is recorded as Real-time data is recorded as Let the spatiotemporal process deep learning model be denoted as ,in The model represents the first j The parameters at time... The value, The combination of historical data and real-time data at any given moment becomes Historical data at any given moment The parameters of the deep model trained at time step 1 become the input parameters of the model at that time step 2; attention-RNN, attention-LSTM, and attention-GRU are used respectively for time step 1. Different attention-RNN models were obtained by training on historical data. As a moment Model and initialize relevant parameters; S13. Each time there is new incremental data, repeat steps S11 and S12 and update the model and parameters.
3. The real-time streaming data incremental learning method for a spatiotemporal process deep learning model according to claim 2, characterized in that, Step S3 is as follows: S31. Input the replay dataset into the model obtained in S12 and initialize the model with the parameters obtained in S12; S32. Calculate the importance of parameters and use them as regularization constraints in the loss function of the spatiotemporal process deep learning model. Adjust the model's hyperparameters through incremental training. The loss function is: = in, express The loss function of the model during incremental training at each time step. express The loss function during model training at each time step. The model number is represented by the first... i One parameter, This represents the regularization constraint imposed on the loss function.