Multivariable time series prediction method based on NMF multi-scale lightweight space-time convolutional neural network

Abstract

The invention discloses a multivariable time series prediction method based on an NMF multi-scale lightweight space-time convolutional neural network, and belongs to the technical field of machine learning and deep learning application. The method specifically comprises the following steps: (1) collecting and processing multivariable time sequence data, and unifying the scale of the data; (2) decomposing the time sequence data by using NMF, extracting basis matrixes and coefficient matrixes of different scales, and constructing a multi-scale feature pyramid; (3) applying a space-time convolution layer on the basis of the output of the feature pyramid, introducing a Bayesian optimization algorithm, and adopting a feature fusion layer fused with a multi-scale attention mechanism; (4) dividing a training data set and a test data set, and carrying out model training; and (5) predicting future time sequence data by using the trained model and outputting an expected value. According to the method, a more powerful and more efficient multivariable time sequence prediction model is effectively constructed, and the method is suitable for analysis and prediction of multivariable time sequence data.

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning and deep learning applications, and specifically to a multivariate time series prediction method based on an NMF multi-scale lightweight spatio-temporal convolutional neural network. Background Art

[0002] With the rapid development of data science and artificial intelligence technologies, time series prediction has become one of the core tasks in multiple fields. Time series data is widely used in multiple fields such as financial markets, weather forecasting, energy demand analysis, traffic flow monitoring, etc. One of the key challenges in these fields is how to accurately predict future multivariate time series data, which is often affected by multiple factors and has strong temporal dependencies and spatial correlations.

[0003] Traditional time series prediction methods, such as ARIMA (Autoregressive Integrated Moving Average Model) and exponential smoothing methods, although perform well in dealing with simple prediction tasks of single variables, are significantly limited in the face of multiple related variables, complex spatio-temporal features, and large-scale data sets. With the development of deep learning technologies, more and more neural network-based prediction methods have been proposed to overcome the deficiencies of traditional methods.

[0004] The successful application of convolutional neural networks (CNNs) in image processing has prompted researchers to extend them to the processing of time series data, especially spatio-temporal convolutional neural networks. Spatio-temporal convolutional neural networks can effectively capture the spatial and temporal dependencies in data and provide more powerful prediction capabilities than traditional models. However, standard convolutional neural networks face problems such as high computational complexity and overfitting when dealing with high-dimensional, multi-scale time series data, especially when the data set is large and the feature dimension is large.

[0005] In recent years, multi-scale spatio-temporal convolutional neural networks have emerged. By extracting spatio-temporal features at different scales, they can capture different levels of dependencies in data and further improve the prediction accuracy of the model. However, existing multi-scale networks still face challenges in terms of structural optimization, computational efficiency, and model generalization ability, especially in how to effectively weight and fuse features at different scales.

[0006] In addition, with the increase in the amount of data, how to improve the computational efficiency and reduce the computational costs of training and prediction while ensuring the model performance has become an urgent problem to be solved. For this reason, the Bayesian optimization algorithm and the adaptive weighting mechanism have gradually become effective means to improve the model performance and efficiency. Bayesian optimization can improve the training efficiency of the model by intelligently searching for the optimal hyperparameter configuration, while the adaptive weighting mechanism can dynamically adjust the weights of each feature according to the importance of the features to optimize the prediction accuracy of the model.

[0007] Therefore, the multivariate time series prediction method based on the NMF multi-scale lightweight spatio-temporal convolutional neural network, combined with the Bayesian optimization and the adaptive weighting mechanism, can not only effectively capture complex spatio-temporal dependence relationships, but also improve the computational efficiency and generalization ability by optimizing the model structure and parameters, which is an effective technical path to solve the computational complexity, overfitting and feature weighting problems faced in spatio-temporal data prediction. Summary of the Invention

[0008] The object of the present invention is to provide a multivariate time series prediction method based on the NMF multi-scale lightweight spatio-temporal convolutional neural network. This method aims to solve the prediction problem of complex spatio-temporal data by constructing an efficient, flexible and lightweight spatio-temporal convolutional neural network, especially when dealing with multivariate time series data, maintaining a low computational complexity and a high prediction accuracy. This method combines technologies such as multi-scale feature extraction, Bayesian optimization, adaptive weighting mechanism and multi-scale attention mechanism, significantly improving the model performance, computational efficiency and generalization ability.

[0009] To achieve the above object, the present invention provides the following technical solution: A multivariate time series prediction method based on the NMF multi-scale lightweight spatio-temporal convolutional neural network, specifically including the following steps:

[0010] S1: Collect multivariate time series data, and perform missing value and outlier processing on the time series data, and at the same time perform normalization processing on the time series data to unify the scale of the data;

[0011] Suppose there is a multivariate time series of N*T, where N represents N variables, T represents the length of the time series, and the multiple variable indicators of each variable n at time t can be represented as a vector

[0012] x n (t)=[x n,1 ,x n,2 ,x n,3 ,…,x n,T

[0013] The N variable indicators at each time t can be represented as a vector ​

[0014] x t (n)=[x 1,t ,x 2,t ,x 3,t ,...,x N,t

[0015] The data collected within the time interval [1, T] can be represented as a matrix

[0016] For the missing values, a linear interpolation method is used to interpolate based on the values at adjacent time points. The specific operations are as follows:

[0017]

[0018] where, X filled represents the filled value, X n ( t -1) and X n (t + 1) are the observed values at the time steps before and after the missing point respectively;

[0019] For the outliers, the z-score method is used to identify outliers; when the z-score of a data point is greater than a preset threshold, that data point is considered an outlier. Specifically, the z-score calculation formula is as follows:

[0020]

[0021] where, x n,t is the value of variable n at time point t, μ n and σ n are the mean and standard deviation of this variable over all time points respectively;

[0022] S2: Use NMF to decompose the time series data, extract the basis matrix and coefficient matrix at different scales, and construct a multi-scale feature pyramid;

[0023] The objective function of the NMF is:

[0024]

[0025] where, ||·|| F represents the Frobenius norm of the matrix, reflecting the error between the original matrix and the reconstructed matrix; NMF solves this optimization problem through an alternating minimization algorithm, such as the multiplicative update rule, to obtain the basis matrix W and the coefficient matrix H; during the optimization process, the update rule for each time is as follows:

[0026] Update the basis matrix W:

[0027] ​

[0028] Update the coefficient matrix H:

[0029]

[0030] The update process is repeated until the error converges to a preset threshold; by selecting different r values, spatio-temporal patterns of data can be captured at different scales, thereby constructing a multi-scale feature pyramid; when using NMF for decomposition, the choice of the implicit feature dimension r directly affects the scale of the decomposition result, that is, the granularity of the data features captured by each decomposition result. Specifically:

[0031] When r is small, NMF captures relatively macroscopic and rough patterns in the data, that is, larger scales;

[0032] When r is large, NMF captures relatively fine-grained and local patterns in the data, that is, smaller scales;

[0033] S3: Apply a spatio-temporal convolutional layer based on the output of the feature pyramid to simultaneously process and analyze complex relationships in the time and space dimensions; introduce a dynamic dual mechanism to enhance the balance between features of different scales; introduce a Bayesian optimization algorithm to automatically adjust the network structure and hyperparameter configuration; adopt a feature fusion layer integrating a multi-scale attention mechanism to adaptively weight features of different scales according to the importance of each scale feature;

[0034] S4: Based on the data and features prepared in step S1 and step S2, divide the training data set and the test data set, perform model training, and use a loss function and an optimizer to improve the model performance;

[0035] The model training uses the mean squared error MSE or the root mean squared error RMSE as the loss function to measure the difference between the model prediction value and the actual value;

[0036] The calculation formulas for the mean squared error MSE and the root mean squared error RMSE are respectively:

[0037]

[0038] where, y i is the actual value, is the model prediction value, and N is the number of samples;

[0039] The optimizer uses the Adam optimizer to adjust the model parameters, and utilizes its characteristic of adaptive learning rate to accelerate the training process and improve the model convergence speed; its update rule is simplified to:

[0040]

[0041] Among them, θ represents the model parameters, L is the loss function, α is the learning rate, is the gradient of L with respect to θ, and Adam() represents the update step of the Adam optimization algorithm;

[0042] Through multiple iterations of training, the model parameters are continuously updated by repeatedly executing the following steps until convergence:

[0043]

[0044] S5: Use the trained model to predict future time series data and output the multivariate expected values within several future time windows.

[0045] Preferably, the collection and preliminary preprocessing of the multivariate time series data in step S1 specifically include the following sub-steps:

[0046] S101: Data collection:

[0047] Collect the multivariate time series data; the source of the time series data is sensors, databases, and online data platforms; the time series data contains time series of multiple variables;

[0048] S102: Perform data cleaning:

[0049] Perform preprocessing operations on the collected time series data, including handling missing values and outliers, to ensure the consistency and usability of the collected data;

[0050] S103: Perform standardization processing of the data:

[0051] To unify the scale of the data and ensure that the time series data meets the analysis requirements:

[0052] For each variable n ∈ {1, 2,..., N}, calculate the value after standardization processing as follows:

[0053]

[0054] where μ n and σ n are the mean and standard deviation of variable n at all time points, respectively.

[0055] Preferably, in step S2, the NMF is used, and the NMF is non-negative matrix factorization, which specifically includes the following sub-steps:

[0056] S201: Matrixize the time series data;

[0057] Represent the collected multivariate time series data in matrix form: Assume that the original multivariate time series data is an N×T matrix, where N is the number of variables and T is the number of time steps. Each element X in the matrix (n,t) represents the observed value of variable n at time step t. To adapt to the processing of the NMF, the original data matrix X is represented as:

[0058] X = W·H

[0059] where W is an N×r basis matrix, H is an r×T coefficient matrix, r is the implicit feature dimension, and r is usually smaller than T and N. The NMF decomposes the original data matrix into two non-negative matrices W and H by constraining all matrix elements to be non-negative, such that X≈W·H;

[0060] S202: Perform the NMF:

[0061] Use the NMF algorithm to decompose the time series data matrix X, with the goal of minimizing the error between the original matrix X and the reconstructed matrix W·H;

[0062] S203: Construct a multi-scale feature pyramid:

[0063] Select k different implicit feature dimensions r 1 , r 2 , r 3 ,..., r k , corresponding to obtaining k of the basis matrices W r1 , W r2 , W r3 ,..., W rk and the coefficient matrices H r1 , H r2 , H r3 ,…, H rk ; Concatenate the basis matrices W ri by columns at different scales to obtain a new matrix W muti-scale :

[0064] W multi-scale = [W r1 , W r2 ,…, W rk

[0065] The obtained W muti-scale is an N×(r 1 + r 2 +…+ r k ) matrix, containing features at different scales.

[0066] Preferably, the specific steps in step S3 include the following sub-steps: ​

[0067] S301: Based on the output of the feature pyramid, perform spatio-temporal convolution operations on the basis matrix W at each scale ri to capture complex dependencies in the temporal and spatial dimensions of the data, and use a combination of a temporal convolutional network (TCN) and a graph neural network (GNN) to process time series data;

[0068] The temporal convolutional network (TCN) is used to model long-term dependencies in the temporal dimension. For the feature matrix W at each scale r , through causal convolution operations, that is, only using the data at the current and previous time steps, calculate the convolutional features in time:

[0069]

[0070] where filter k is the convolutional kernel, K is the convolutional window size, and W r (t - k + 1) is the feature of the basis matrix W r at time step t - k + 1;

[0071] The graph neural network (GNN) is used to model dependencies in the spatial dimension; for the feature matrix H output by each TCN TCN , take it as the node feature of the graph, and perform spatial information transmission through graph convolution operations. The calculation rule of graph convolution is:

[0072]

[0073] where N(n) represents the set of neighbor nodes of node n, W GNN is the weight matrix of graph convolution, b is the bias term, σ is the activation function; through the combination of the temporal convolutional network (TCN) and the graph neural network (GNN), complex features in both the temporal and spatial dimensions can be captured simultaneously, obtaining richer spatio-temporal information;

[0074] S302: Introduce a dynamic duality mechanism to enhance the balance between features at different scales. The dynamic duality mechanism can optimize the relationship between scales through duality variables, ensuring that features at different scales can effectively complement each other during training, rather than simply being superimposed;

[0075] The goal of the dynamic duality mechanism is to minimize the conflict between scales, enabling features at each scale to work in harmony with features at other scales; specifically, introduce duality variables to penalize the differences between each scale:

[0076]

[0077] where H GNN (r) and HTCN (r) are the output features of graph convolution and temporal convolution respectively, and λ r is the dual coefficient, which is used to adjust the balance between the outputs of graph convolution and temporal convolution; through the dynamic dual mechanism, the model can adaptively adjust the influence between features of different scales, so that in the spatio-temporal convolution module, features of multiple scales can complement each other rather than repeat;

[0078] S303: Bayesian optimization is used to automatically search for the optimal network structure and hyperparameter configuration; Bayesian optimization uses a surrogate model to predict the performance of different hyperparameter combinations, so as to automatically adjust the network structure to improve the effect of the model;

[0079] Use Gaussian process regression as a surrogate model to estimate the distribution of the objective function f(θ):

[0080] f(θ) ∼ GP(μ(θ), k(θ, θ′))

[0081] where μ(θ) is the mean function and k(θ, θ′) is the covariance function. In order to select the optimal hyperparameter combination θ * , maximize the acquisition function a(θ):

[0082]

[0083] The acquisition function measures the "information gain" of the hyperparameter combination. By evaluating the newly selected hyperparameter combination, the surrogate model is updated and the optimization continues until convergence to the optimal solution;

[0084] S304: A multi-scale attention weighted fusion layer is introduced to achieve the fusion of features of different scales by dynamically adjusting the weights of features at each scale;

[0085] For each scale r, first calculate its corresponding attention weight a r , which reflects the contribution of the features at this scale to the overall prediction task; the calculation formula of the attention weight is as follows:

[0086]

[0087] where score r is the score calculated based on the input features and the learned parameters, indicating the importance of the features at this scale; the calculation of the attention score score r combines the idea of adaptive weighting:

[0088] score r = β r ·H r

[0089] where β ris an adjustable parameter learned through optimization during training, representing the dynamic weighting coefficient of this scale, H r represents the scale feature; in this way, the model can automatically adjust the feature contributions of each scale, highlight important information, and suppress redundant features;

[0090] All scale features are weighted and fused according to their corresponding attention weights a r to obtain a comprehensive feature representation Hfinal :

[0091]

[0092] Preferably, in step S4, it specifically includes the following sub-steps:

[0093] S401: Before model training, first divide the prepared multivariate time series data to generate a training set and a test set; the division of the data follows the time order, and the time series data is sliced according to the time stamp order, with 80% as the training data set and 20% as the test data set;

[0094] The division of the data set is carried out in the following way:

[0095] X train ,X test = split(X, train_size = 0.8)

[0096] where X represents the original data set, X train is the training data set, X test is the test data set, and the training set and the test set are independent in time to ensure that the data in the test set is not used for training in advance and avoid data leakage;

[0097] S402: The model training uses the mean squared error MSE or the root mean squared error RMSE as the loss function to measure the difference between the model prediction value and the actual value;

[0098] After each iteration, evaluate the performance of the model on the validation set, and use predefined evaluation metrics to measure the prediction ability and stability of the model until the performance of the model on the validation set no longer improves significantly to achieve the best generalization ability.

[0099] Preferably, in step S5, the specific implementation process includes:

[0100] S501: Ensure that the input time prediction sequence data has been properly preprocessed, and the preprocessed data should be in the same format as the data used during training to ensure that the model can correctly process the data and make predictions;

[0101] Input the preprocessed data into the trained multi-scale spatio-temporal convolutional neural network model. The multi-scale spatio-temporal convolutional neural network model will use the previously learned spatio-temporal features and weight parameters to perform forward propagation and generate multi-variable prediction values for multiple future time steps. The multi-scale spatio-temporal convolutional neural network model infers future data through the learned spatio-temporal patterns.

[0102] S502: The trained multi-scale spatio-temporal convolutional neural network model generates multi-variable expected values within several future time windows based on the input future data. The expected values are the prediction results of the multi-scale spatio-temporal convolutional neural network model for future time steps and can be represented by the output of the multi-scale spatio-temporal convolutional neural network model. where is the multi-variable value predicted at time point t. Specifically, the predicted values output by the model include the expected values for multiple time steps. And these predicted values are optimized based on historical data and spatio-temporal patterns.

[0103]

[0104] where f is the model function learned during the training process and T is the number of time steps predicted.

[0105] S503: Based on the multi-variable expected values output in step S502, the user makes actual decisions and applications based on the prediction results. The multi-variable prediction values can provide valuable information for practical applications such as decision support and trend prediction.

[0106] Compared with the prior art, the beneficial effects of the present invention are:

[0107] The present invention solves the problems of the ability to handle complex spatio-temporal dependence relationships, high computational complexity, overfitting problems, feature weighting and fusion problems, and the scalability and flexibility of the model. The present invention provides an efficient, flexible, and well-generalized multi-variable time series prediction method, which can be widely applied to multiple fields such as finance, meteorology, energy, and transportation, and provide accurate prediction support for decision-making in related fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0108] Figure 1 is a schematic diagram of the multi-variable time series prediction method based on the lightweight multi-scale spatio-temporal convolutional neural network of the present invention.

[0109] Figure 2 is the construction flowchart of Application Example 1.

[0110] Figure 3 is the prediction process of Application Example 2.

[0111] Figure 4 The network structure for Application Example 1

[0112] Figure 5 The multi-scale adaptive weighted attention feature fusion for Application Example 1 Detailed implementation manners

[0113] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Apparently, the described embodiments are only a part of the embodiments of the present invention, rather than all of them. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0114] As Figure 1 The schematic diagram of a multi-variable time series prediction method based on the NMF multi-scale lightweight spatio-temporal convolutional neural network proposed in the present invention shown below specifically includes the following steps:

[0115] S1: Collect multi-variable time series data, process the missing values and outliers in the time series data, and at the same time perform standardization processing on the time series data to unify the data scale;

[0116] S2: Use NMF to decompose the time series data, extract the basis matrices and coefficient matrices of different scales, and construct a multi-scale feature pyramid;

[0117] S3: Apply a spatio-temporal convolutional layer based on the output of the feature pyramid to simultaneously process and analyze the complex relationships in the time and space dimensions; introduce a dynamic dual mechanism to enhance the balance between features of different scales; introduce a Bayesian optimization algorithm to automatically adjust the network structure and hyperparameter configuration; adopt a feature fusion layer integrating a multi-scale attention mechanism to adaptively weight features of different scales according to the importance of each scale feature;

[0118] S4: Based on the data and features prepared in steps S1 and S2, divide the training data set and the test data set, train the model, and adopt a loss function and an optimizer to improve the model performance;

[0119] S5: Use the trained model to predict future time series data and output the multi-variable expected values within a number of future time windows.

[0120] The collection and preliminary preprocessing of the multi-variable time series data in step S1 specifically include the following sub-steps:

[0121] S101: Data collection:

[0122] Collect the multivariate time series data; the source of the time series data is sensors, databases, and online data platforms; the time series data contains time series of multiple variables;

[0123] Suppose there is an N*T multivariate time series, where N represents N variables and T represents the length of the time series. The multiple variable indicators of each variable n at time t can be represented as a vector

[0124] x n (t)=[x n,1 ,x n,2 ,x n,3 ,...,x n,T

[0125] The N variable indicators at each time t can be represented as a vector

[0126] x t (n)=[x 1,t ,x 2,t ,x 3,t ,...,x N,t

[0127] The data collected within the time interval [1, T] can be represented as a matrix

[0128] S102: Perform data cleaning:

[0129] Perform preprocessing operations on the collected time series data, including handling missing values and outliers, to ensure the consistency and usability of the collected data;

[0130] For the missing values, the method of linear interpolation is used to interpolate according to the values at adjacent time points. The specific operation is as follows:

[0131]

[0132] Among them, X filled represents the filled value, and X n (t - 1) and X n (t + 1) are the observed values at the time steps before and after the missing point respectively;

[0133] For the outliers, the z-score method is used to identify outliers; when the z-score of a data point is greater than a preset threshold, that data point is considered an outlier. Specifically, the z-score calculation formula is as follows:

[0134] ​​

[0135] where x n,t is the value of variable n at time point t, and μ n and σ n are the mean and standard deviation of this variable over all time points, respectively;

[0136] S103: Perform standardization processing on the data:

[0137] To unify the scale of the data and ensure that the time series data meets the analysis requirements:

[0138] For each variable n ∈ {1, 2,..., N}, calculate the value after standardization processing as follows:

[0139]

[0140] where μ n and σ n are the average value and standard deviation of variable n over all time points, respectively.

[0141] In step S2, the NMF is used, and the NMF is non - negative matrix factorization, which specifically includes the following sub - steps:

[0142] S201: Matrixize the time series data;

[0143] Represent the collected multi - variable time series data in matrix form: Assume that the original multi - variable time series data is an N×T matrix, where N is the number of variables and T is the number of time steps. Each element X in the matrix (n,t) represents the observed value of variable n at time step t. To adapt to the processing of the NMF, the original data matrix X is represented as:

[0144] X = W·H

[0145] where W is an N×r basis matrix, H is an r×T coefficient matrix, r is the latent feature dimension, and r is usually less than T and N. The NMF decomposes the original data matrix into two non - negative matrices W and H by constraining all matrix elements to be non - negative, such that X≈W·H;

[0146] S202: Perform the NMF:

[0147] Use the NMF algorithm to decompose the time series data matrix X. The goal is to minimize the error between the original matrix X and the reconstructed matrix W·H. The objective function of the NMF is:

[0148]

[0149] where ||·||F Denotes the Frobenius norm of the matrix, reflecting the error between the original matrix and the reconstructed matrix; NMF solves this optimization problem through an alternating minimization algorithm, such as the multiplicative update rule, to obtain the basis matrix W and the coefficient matrix H; during the optimization process, the update rule for each time is as follows:

[0150] Update the basis matrix W:

[0151]

[0152] Update the coefficient matrix H:

[0153]

[0154] The update process is repeated until the error converges to a preset threshold; by selecting different r values, the spatio-temporal patterns of the data can be captured at different scales, thereby constructing a multi-scale feature pyramid; when using NMF for decomposition, the selection of the implicit feature dimension r directly affects the scale of the decomposition result, that is, the granularity of the data features captured by each decomposition result. Specifically:

[0155] When r is small, NMF captures the more macroscopic and rough patterns in the data, that is, the larger scale;

[0156] When r is large, NMF captures the more fine-grained and local patterns in the data, that is, the smaller scale;

[0157] S203: Construct a multi-scale feature pyramid:

[0158] Select k different implicit feature dimensions r 1 , r 2 , r 3 ,..., r k , corresponding to obtaining k of the basis matrices W r1 , W r2 , W r3 ,..., W rk and the coefficient matrix H r1 , H r2 , H r3 ,..., H rk ; Stack the basis matrices W at different scales ri by columns to obtain a new matrix W muti-scale :

[0159] W multi-scale = [W r1 , W r2 ,..., W rk

[0160] The obtained W muti-scale ​is an N×(r 1 +r 2 +…+r k ) matrix, which contains features at different scales.

[0161] The specific steps in step S3 include the following sub-steps:

[0162] S301: Based on the output of the feature pyramid, perform spatio-temporal convolution operations on the basis matrix W at each scale ri to capture the complex dependencies in the temporal and spatial dimensions of the data, and use a combination of a temporal convolutional network (TCN) and a graph neural network (GNN) to process time series data;

[0163] The temporal convolutional network TCN is used to model long-term dependencies in the temporal dimension. For the feature matrix W at each scale r , through causal convolution operations, that is, only using the data at the current and previous time steps, calculate the convolutional features in time:

[0164]

[0165] where filter k is the convolution kernel, K is the convolution window size, and W r (t - k + 1) is the feature of the basis matrix W r at time step t - k + 1;

[0166] The graph neural network GNN is used to model dependencies in the spatial dimension; for the feature matrix H output by each TCN TCN , use it as the node feature of the graph, and perform spatial information transmission through graph convolution operations. The calculation rule of graph convolution is:

[0167]

[0168] where N(n) represents the set of neighbor nodes of node n, W GNN is the weight matrix of graph convolution, b is the bias term, σ is the activation function; through the combination of the temporal convolutional network TCN and the graph neural network GNN, complex features in both the temporal and spatial dimensions can be captured simultaneously, obtaining richer spatio-temporal information;

[0169] S302: Introduce a dynamic duality mechanism to enhance the balance between features at different scales. The dynamic duality mechanism can optimize the relationship between scales through duality variables, ensuring that features at different scales can effectively complement each other during training, rather than simply being superimposed;

[0170] The goal of the dynamic dual mechanism is to minimize the conflict between scales, enabling the features of each scale to work in harmony with those of other scales. Specifically, dual variables are introduced to penalize the differences between each scale:

[0171]

[0172] where H GNN (r) and H TCN (r) are the output features of graph convolution and temporal convolution respectively, and λr is the dual coefficient used to adjust the balance between the outputs of graph convolution and temporal convolution. Through the dynamic dual mechanism, the model can adaptively adjust the influence between features of different scales, enabling features of multiple scales to complement rather than duplicate each other in the spatio-temporal convolution module;

[0173] S303: Bayesian optimization is used to automatically search for the optimal network structure and hyperparameter configuration. Bayesian optimization utilizes a surrogate model to predict the performance of different hyperparameter combinations, thereby automatically adjusting the network structure to improve the model's effectiveness;

[0174] Gaussian process regression is used as the surrogate model to estimate the distribution of the objective function f(θ):

[0175] f(θ) ∼ GP(μ(θ), k(θ, θ′))

[0176] where μ(θ) is the mean function and k(θ, θ′) is the covariance function. To select the optimal hyperparameter combination θ * , the acquisition function a(θ) is maximized:

[0177]

[0178] The acquisition function measures the "information gain" of the hyperparameter combination. By evaluating newly selected hyperparameter combinations, the surrogate model is updated, and optimization continues until convergence to the optimal solution;

[0179] S304: A multi-scale attention weighted fusion layer is introduced to achieve the fusion of features of different scales by dynamically adjusting the weights of features at each scale;

[0180] For each scale r, first calculate its corresponding attention weight a r , which reflects the contribution of the features at this scale to the overall prediction task. The calculation formula for the attention weight is as follows:

[0181]

[0182] where score r is the score calculated based on the input features and the learned parameters, representing the importance of the features at this scale; The attention score scorer The calculation combines the idea of adaptive weighting:

[0183] score r = β r ·H r

[0184] where β r is an adjustable parameter learned through optimization during training, representing the dynamic weighting coefficient of this scale, and H r represents the scale feature; in this way, the model can automatically adjust the feature contribution of each scale, highlight important information, and suppress redundant features.

[0185] All scale features are weighted and fused according to their corresponding attention weights a r to obtain a comprehensive feature representation Hfinal :

[0186]

[0187] In step S4, it specifically includes the following sub-steps:

[0188] S401: Before model training, first divide the prepared multivariate time series data to generate a training set and a test set; the data division follows the time order, and the time series data is sliced in the order of timestamps, with 80% as the training data set and 20% as the test data set; the data set is divided in the following way:

[0189] X train , X test = split(X, train_size = 0.8)

[0190] where X represents the original data set, X train is the training data set, and X test is the test data set. The training set and the test set are independent in time to ensure that the data in the test set is not used for training in advance and to avoid data leakage;

[0191] S402: The model training uses the mean squared error MSE or the root mean squared error RMSE as the loss function to measure the difference between the model prediction value and the actual value;

[0192] The calculation formulas for the mean squared error MSE and the root mean squared error RMSE are respectively:

[0193]

[0194] where y i is the actual value, is the model prediction value, and N is the number of samples;

[0195] The optimizer uses the Adam optimizer to adjust the model parameters, leveraging its characteristic of adaptive learning rate to accelerate the training process and improve the model convergence speed; its update rule is simplified as:

[0196]

[0197] where, θ represents the model parameters, L is the loss function, α is the learning rate, is the gradient of L with respect to θ, and Adam() represents the update step of the Adam optimization algorithm;

[0198] Through multiple iterative trainings, the model parameters are continuously updated, and the following steps are repeatedly executed until convergence:

[0199]

[0200] After each iteration, the performance of the model on the validation set is evaluated, and predefined evaluation metrics are used to measure the prediction ability and stability of the model until the performance of the model on the validation set no longer improves significantly, so as to achieve the best generalization ability.

[0201] In the step S5, the specific implementation process includes:

[0202] S501: Ensure that the input time prediction sequence data has been properly preprocessed, and the preprocessed data should be in the same format as the data used in the training process to ensure that the model can correctly process the data and make predictions;

[0203] Input the preprocessed data into the trained multi-scale spatio-temporal convolutional neural network model. The multi-scale spatio-temporal convolutional neural network model will utilize the previously learned spatio-temporal features and weight parameters to perform forward propagation and generate multi-variable prediction values for multiple future time steps; the multi-scale spatio-temporal convolutional neural network model infers future data through the learned spatio-temporal patterns;

[0204] S502: The trained multi-scale spatio-temporal convolutional neural network model generates multi-variable expected values within several future time windows according to the input future data; the expected values are the prediction results of the multi-scale spatio-temporal convolutional neural network model for future time steps, and can be represented by the output by the multi-scale spatio-temporal convolutional neural network model, where is the multi-variable value predicted at time point t; specifically, the predicted values output by the model include the expected values for multiple time steps and these predicted values are optimized based on historical data and spatio-temporal patterns;

[0205]

[0206] Among them, f is the model function learned during the training process, and T is the number of time steps predicted;

[0207] S503: Based on the multi-variable expected values output in step S502, the user makes actual decisions and applications based on the prediction results; the multi-variable prediction values can provide valuable information for actual applications such as decision support and trend prediction.

[0208] Embodiment Application 1

[0209] Figure 2 This is Application 1 of the embodiments of the present invention: a flowchart of a multi-variable time series prediction system and method provided. This Embodiment Application 1 is applicable to the situation of training a lightweight and highly accurate multi-variable time series data prediction model. Specifically, taking power data analysis as an example, the training method of this power multi-variable time series data prediction model can be executed by a power equipment business data prediction model training device, and this power equipment business data prediction model training device can be implemented in software. Further, the electronic device includes, but is not limited to: desktop computers, laptop computers, and servers, etc.

[0210] As Figure 2 shown, the method specifically includes the following steps:

[0211] S110: Collect the actual data of at least two time series sensors as multi-variable data samples.

[0212] In this Embodiment Application 1, the multi-variable time series data set can be understood as power consumption data from multiple customers (each customer is a variable), covering records of multiple time periods. For example, the actual power consumption data of users is collected once every hour. The data samples include the power consumption records of multiple customers changing over time, which can reflect the state of the power system at different time steps.

[0213] The actual data can be understood as the key data generated and used during the operation of the device or system, which is suitable for analysis as time series data. The actual data includes not only historical data but also the data at the current moment. Preferably, during the training stage of the data prediction model, the actual data should be mainly historical data to ensure that the model can learn the temporal characteristics in the data.

[0214] Among them, the actual data of at least two time series sensors can be collected by an electronic device as multi-variable data samples for subsequent operations.

[0215] It should be noted that, in order to ensure the consistency and availability of the collected actual business data, data cleaning operations can be performed on the actual business data, that is to say, preprocessing operations are performed on the actual business data, and the preprocessing operations can include at least one of missing value processing, outlier processing, and standardization or normalization processing.

[0216] S120. Select multiple different r values, and use the NMF matrix decomposition method to obtain feature matrices of different scales, so as to construct a multi-scale pyramid as the input feature matrix.

[0217] In Application Example 1 of this embodiment, first select multiple scale parameters r 1 , r 2 ,..., r n , and each scale corresponds to a different feature dimension, representing the power consumption characteristics at different time steps and granularities. Through the NMF method, the input original power consumption data is decomposed to obtain the basis matrix and coefficient matrix of different scales. Specifically, for each scale r, use NMF to decompose the original data matrix X:

[0218] X≈W r H r

[0219] where W r is the basis matrix, representing the feature space at different scales, and H r is the coefficient matrix, representing the performance of the time series at this scale.

[0220] By decomposing the features of multiple scales, a set of feature matrices of different scales are obtained These matrices form a multi-scale feature pyramid, which serves as the input features for the subsequent multi-variable time series prediction model. The construction of the multi-scale pyramid can effectively capture different levels of information in the data and improve the model's understanding ability of spatio-temporal data.

[0221] S130. Train a multi-variable prediction model according to the data samples and the feature matrix, and the multi-variable sequence prediction model is used for predicting future sequence data.

[0222] In Application Example 1 of this embodiment, the process of training the multi-variable time series prediction model is divided into the following steps:

[0223] S131. Input the multi-scale feature matrix after data preprocessing and feature pyramid construction into the multi-variable prediction model. The multi-scale feature matrix contains spatio-temporal feature information of multiple scales and can effectively reflect the spatio-temporal characteristics of power consumption.

[0224] S132. Train the multi-scale features through a neural network model. The neural network includes multiple convolutional layers, fully connected layers, and an output layer. The training process adjusts the model parameters through the backpropagation algorithm to minimize the loss function (such as mean squared error, MSE). During this process, the model extracts the temporal dependencies in the features through spatio-temporal convolution, and at the same time learns the mutual influence between different scale features to ensure that the model can effectively learn the multi-level features of spatio-temporal data (the specific network architecture is as shown in Figure 4 ).

[0225] S133. Introduce a dynamic pairing mechanism (Dynamic Pairing) to optimize the relationship between multi-scale features. Specifically, the dynamic pairing mechanism ensures the synergy between different scale features by dynamically pairing and adjusting the scale features. This mechanism can adaptively adjust the influence of each scale feature, enabling the multi-scale features to be better fused together and enhancing the model's ability to model complex spatio-temporal dependencies. The application of the dynamic pairing mechanism in the model helps to improve the cooperation degree and prediction accuracy of each scale feature.

[0226] S134. Use the Bayesian optimization algorithm to optimize the network structure and hyperparameters during the training process. Bayesian optimization searches for the optimal hyperparameter combination by constructing a surrogate model (such as Gaussian process regression), thereby improving the training efficiency and prediction accuracy. The optimization result is the final model structure and parameter configuration.

[0227] S135. Introduce an adaptive weighted multi-scale attention mechanism to fuse multi-scale features. Specifically, through the multi-scale attention mechanism, the model calculates attention weights for each scale feature and automatically adjusts the influence of each scale feature. Higher attention weights correspond to more important features, while lower attention weights reduce the influence of that feature. After weighting, the features of all scales will be fused to obtain a comprehensive feature representation as the input for subsequent prediction ( Figure 5 shows the fusion process of the adaptive weighted multi-scale attention mechanism).

[0228] S136. Use the trained multi-variable prediction model to predict the power consumption data for several future time steps. The trained model can predict future power consumption based on historical data and generate multi-variable prediction values for future time points ( Figure 3 shows the prediction process).

[0229] Example Application 2

[0230] Figure 3A multi-user power consumption prediction method provided for Application 2 of the embodiments of the present invention, the method comprising: collecting historical data and real-time data of the power consumption of at least two users; inputting the power consumption data into a power consumption prediction model to obtain power consumption prediction values of the at least two users in multiple future time periods. In this embodiment, an intelligent prediction model is used to achieve efficient and accurate multi-user power consumption prediction, optimize power load distribution, and improve the efficiency and security of the operation of the power system.

[0231] As Figure 3 shown, the method specifically comprises the following steps:

[0232] S210. Collect historical data of the actual power consumption of at least two users to be predicted.

[0233] In Application 2 of this embodiment, the users to be predicted can be understood as multi-variables to be predicted. The power consumption data indicates the power usage of users within each hour, including but not limited to: indicators such as user load, power consumption, power fluctuation, and voltage change. Optionally, one of the indicators is selected as the prediction indicator. The collected data undergoes preprocessing operations (such as missing value filling, outlier cleaning, and normalization processing) to ensure data consistency and availability.

[0234] S220. Input the power consumption data into a power consumption prediction model to obtain prediction values of the at least two users to be predicted in multiple future time periods.

[0235] Among them, the power consumption prediction model is trained based on the power multi-variable time series data prediction model training method described in Application 1 of the above embodiments. Input the power consumption data into the power consumption prediction model to obtain prediction values of the at least two users to be predicted in at least one future time period.

[0236] Exemplarily, the input of the power multi-variable time series data prediction model can be the actual power consumption measurement data of at least two users. Based on the relationship between the actual data of at least two users, the power multi-variable time series data prediction model can predict the power data in at least one future time period within multiple future time windows.

[0237] A multi-user power consumption prediction method provided for Application 2 of the embodiments of the present invention, the method comprising: collecting historical data of the actual power consumption of at least two users to be predicted; inputting the power consumption data into a power consumption prediction model to obtain prediction values of the at least two users to be predicted in multiple future time periods, wherein the power consumption prediction model is trained based on the power multi-variable time series data prediction model training method described in Application 1 of the above embodiments.

[0238] Finally, it should be noted that the above are only preferred embodiments of the present invention and are not used to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or perform equivalent replacements for some of the technical features. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A multivariate time series prediction method based on NMF multi-scale lightweight spatiotemporal convolutional neural network, characterized by: The specific steps include: S1: Collect multivariate time series data, process missing values ​​and outliers on the time series data, and standardize the time series data to unify the scale of the data; Suppose there is an N*T multivariate time series, where N represents N variables and T represents the length of the time series. The multiple variable indicators of each variable n at time t can be expressed as a vector x n (t)=[x n,1 ,x n,2 ,x n,3 ,...,x n,T ] The N variables at each time t can be expressed as a vector The data collected in the time interval [1, T] can be represented as a matrix For the missing values, the linear interpolation method is used to interpolate according to the values ​​of adjacent time points. The specific operations are as follows: Among them, X filled Indicates the value after filling, X n (t-1) and X n (t+1) are the observed values ​​of the time steps before and after the missing point respectively; For the outliers, the z-score method is used to identify outliers; when the z-score of a data point is greater than a preset threshold, the data point is considered to be an outlier. Specifically, the z-score calculation formula is as follows: Among them, x n,t is the value of variable n at time point t, μ n and σ n are the mean and standard deviation of the variable at all time points, respectively; S2: Decompose the time series data using NMF, extract basis matrices and coefficient matrices of different scales, and construct a multi-scale feature pyramid; The objective function of the NMF is: Among them, ||·|| F represents the Frobenius norm of the matrix, reflecting the error between the original matrix and the reconstructed matrix; NMF solves the optimization problem by an alternating minimization algorithm, such as a multiplication update rule, to obtain the basis matrix W and the coefficient matrix H; during the optimization process, the rule for each update is as follows: Update the basis matrix W: Update coefficient matrix H: The updating process is repeated until the error converges to a preset threshold. By selecting different r values, the spatiotemporal patterns of data can be captured at different scales, thereby constructing a multi-scale feature pyramid. When using NMF for decomposition, the choice of the implicit feature dimension r directly affects the scale of the decomposition result, that is, the granularity of the data features captured by each decomposition result. Specifically: When r is small, NMF captures the more macroscopic and coarse patterns in the data, i.e., larger scales; When r is large, NMF captures finer-grained and local patterns in the data, i.e., smaller scales; S3: Based on the output of the feature pyramid, a spatiotemporal convolution layer is applied to simultaneously process and analyze complex relationships in time and space dimensions; a dynamic duality mechanism is introduced to enhance the balance between features of different scales; a Bayesian optimization algorithm is introduced to automatically adjust the network structure and hyperparameter configuration; a feature fusion layer that integrates a multi-scale attention mechanism is used to adaptively weight features of different scales according to the importance of each scale feature; S4: Based on the data and features prepared in step S1 and step S2, divide the training data set and the test data set, perform model training, and use a loss function and an optimizer to improve model performance; The model training uses mean square error (MSE) or root mean square error (RMSE) as a loss function to measure the difference between the model prediction value and the actual value; The calculation formulas of the mean square error MSE and the root mean square error RMSE are respectively: Among them, y i is the actual value, is the model prediction value, N is the number of samples; The optimizer uses the Adam optimizer to adjust the model parameters and uses its adaptive learning rate feature to speed up the training process and improve the model convergence speed; its update rule is simplified to: i new =θ old -a·Adam(▽ θ L) Among them, θ represents the model parameters, L is the loss function, α is the learning rate, is the gradient of L with respect to θ, and Adam() represents the update step of the Adam optimization algorithm; Through multiple iterations of training, the model parameters are continuously updated, and the following steps are repeatedly performed until convergence: S5: Use the trained model to predict future time series data and output multivariate expected values ​​in several future time windows.

2. According to claim 1, a multivariate time series prediction method based on NMF multi-scale lightweight spatiotemporal convolutional neural network is characterized in that: The collection and preliminary preprocessing of multivariate time series data in step S1 specifically includes the following sub-steps: S101: Data Collection: Collecting multivariable time series data; the time series data source is a sensor, a database, and an online data platform; the time series data includes time series of multiple variables; S102: Execute data cleaning: Preprocessing the collected time series data, including processing missing values ​​and outliers, to ensure the consistency and availability of the collected data; S103: Execute data standardization processing: By unifying the data scale, we can ensure that the time series data meets the analysis requirements: For each variable n∈{1, 2, ..., N}, calculate the normalized value as follows: Among them, μ n and σ n are the mean and standard deviation of variable n at all time points.

3. The multivariate time series prediction method based on NMF multi-scale lightweight spatiotemporal convolutional neural network according to claim 1 is characterized in that: The NMF is used in step S2, and the NMF is a non-negative matrix factorization, which specifically includes the following sub-steps: S201: Matrix processing is performed on the time series data; The collected multivariate time series data are represented as a matrix: Assume that the original multivariate time series data is an N×T matrix, where N is the number of variables, T is the number of time steps, and each element X in the matrix is (n,t) Represents the observed value of variable n at time step t. In order to adapt to the NMF processing, the original data matrix X is represented as: X=W·H Where W is an N×r basis matrix, H is an r×T coefficient matrix, r is an implicit feature dimension, and r is usually smaller than T and N. The NMF decomposes the original data matrix into two non-negative matrices W and H by constraining all matrix elements to be non-negative values, so that X≈W·H; S202: Perform the NMF: Decomposing the time series data matrix X using the NMF algorithm, with the goal of minimizing the error between the original matrix X and the reconstructed matrix W·H; S203: Construct a multi-scale feature pyramid: We select k different implicit feature dimensions r1, r2, r3, ..., r k , corresponding to the k basis matrices W r1 ,W r2 ,W r3 ,...,W rk and the coefficient matrix H r1 ,H r2 ,H r3 ,...,H rk ; The basis matrix W at different scales ri Concatenate columns to get a new matrix W muti-scale : IN multi-scale =[W r1 ,IN r2 ,...,IN rk ] The obtained W muti-scale is an N×(r1+r2+…+r k ) contains features of different scales.

4. The multivariate time series prediction method based on NMF multi-scale lightweight spatiotemporal convolutional neural network according to claim 1, characterized in that: The step S3 specifically includes the following sub-steps: S301: Based on the output of the feature pyramid, the basis matrix W of each scale is ri Perform spatiotemporal convolution operations to capture the complex dependencies in the time and space dimensions of the data, and use a combination of the time series convolutional network (TCN) and the graph neural network (GNN) to process time series data; The temporal convolutional network TCN is used to model long-term dependencies in the time dimension. For each scale feature matrix W r , through the causal convolution operation, that is, only using the current and previous time step data, the temporal convolution features are calculated: Among them, filter k is the convolution kernel, K is the convolution window size, W r (t-k+1) is the basis matrix W r Features at time step t-k+1; The graph neural network GNN is used to model the dependencies in the spatial dimension; for each feature matrix H output by TCN TCN , taking it as the node feature of the graph, and transferring spatial information through graph convolution operation. The calculation rule of graph convolution is: Among them, N(n) represents the set of neighbor nodes of node n, W GNN is the weight matrix of graph convolution, b is the bias term, σ is an activation function; through the combination of the temporal convolutional network TCN and the graph neural network GNN, the complex features in the time and space dimensions can be captured at the same time to obtain richer spatiotemporal information; S302: Introduce a dynamic duality mechanism to enhance the balance between features of different scales. The dynamic duality mechanism can optimize the relationship between scales through dual variables, ensuring that features of different scales can effectively complement each other during training rather than simply superimposed. The goal of the dynamic dual mechanism is to minimize the conflict between scales so that the features of each scale work in coordination with the features of other scales; specifically, a dual variable is introduced to penalize the difference between each scale: Among them, H GNN (r) and H TCN (r) are the output features of graph convolution and temporal convolution, respectively, r It is the duality coefficient, which is used to adjust the balance between the outputs of graph convolution and temporal convolution. Through the dynamic duality mechanism, the model can adaptively adjust the influence between features of different scales, so that in the spatiotemporal convolution module, features of multiple scales can complement each other instead of being repeated. S303: Bayesian optimization is used to automatically search for the optimal network structure and hyperparameter configuration. Bayesian optimization uses a proxy model to predict the performance of different hyperparameter combinations, thereby automatically adjusting the network structure to improve the model's performance. Use Gaussian process regression as a proxy model to estimate the distribution of the target function f(θ): f(θ)~GP(μ(θ),k(θ,θ′)) Among them, μ(θ) is the mean function, k(θ,θ′) is the covariance function, in order to select the optimal hyperparameter combination θ * , maximize the acquisition function a(θ): The acquisition function measures the "information gain" of the hyperparameter combination, updates the proxy model by evaluating the newly selected hyperparameter combination, and continues to optimize until it converges to the optimal solution; S304: A multi-scale attention weighted fusion layer is introduced to achieve the fusion of features of different scales by dynamically adjusting the weight of each scale feature; For each scale r, first calculate its corresponding attention weight a r , the weight reflects the contribution of the scale feature in the overall prediction task; the calculation formula of the attention weight is as follows: Among them, score r It is a score calculated based on the input features and the learned parameters, indicating the importance of the scale feature; attention score r The calculation of combines the idea of ​​adaptive weighting: score r =β r ·H r Among them, β r It is an adjustable parameter learned through optimization during the training process, representing the dynamic weighting coefficient of the scale, H r Represents scale features; in this way, the model can automatically adjust the feature contribution of each scale, highlight important information, and suppress redundant features; All scale features are calculated according to their corresponding attention weights a r Perform weighted fusion to obtain a comprehensive feature representation Hfinal :

5. The multivariate time series prediction method based on NMF multi-scale lightweight spatiotemporal convolutional neural network according to claim 1, characterized in that: The step S4 specifically includes the following sub-steps: S401: Before model training, the prepared multivariate time series data is first divided to generate a training set and a test set; the data division follows the time order, and the time series data is divided in the order of timestamps, 80% as a training data set and 20% as a test data set; The data set is divided in the following way: X train ,X test =split(X,train_size=0.8) Among them, X represents the original data set, X train is the training data set, X test For testing data sets, the training set and the test set are kept independent in time to ensure that the data of the test set is not used for training in advance to avoid data leakage; S402: The model training uses mean square error (MSE) or root mean square error (RMSE) as a loss function to measure the difference between the model prediction value and the actual value; After each iteration, the performance of the model on the validation set is evaluated, and the predictive ability and stability of the model are measured using pre-defined evaluation indicators until the performance of the model on the validation set is no longer significantly improved to achieve the best generalization ability.

6. The multivariate time series prediction method based on NMF multi-scale lightweight spatiotemporal convolutional neural network according to claim 1, characterized in that: In step S5, the specific implementation process includes: S501: Ensure that the input time prediction series data has been properly preprocessed, and the preprocessed data should be consistent with the data format used in the training process to ensure that the model can correctly process the data and make predictions; The preprocessed data is input into a trained multi-scale spatiotemporal convolutional neural network model, which uses previously learned spatiotemporal features and weight parameters to perform forward propagation and generate multivariate prediction values ​​for multiple future time steps; the multi-scale spatiotemporal convolutional neural network model infers future data through the learned spatiotemporal patterns; S502: The trained multi-scale spatiotemporal convolutional neural network model generates multivariate expected values ​​in several future time windows according to the input future data; the expected values ​​are the prediction results of the multi-scale spatiotemporal convolutional neural network model for future time steps, which can be output by the multi-scale spatiotemporal convolutional neural network model. Indicates that is the multivariate value predicted at time point t; specifically, the predicted value output by the model includes the expected values ​​at multiple time steps And these forecasts are optimized based on historical data and spatiotemporal patterns; Where f is the model function learned during training, and T is the number of predicted time steps; S503: According to the multivariate expected values ​​outputted in step S502, the user makes actual decisions and applications based on the prediction results; the multivariate predicted values ​​can provide valuable information for practical applications such as decision support and trend prediction.

Images