Multivariable time sequence prediction method and system based on implicit neural network

Abstract

The invention discloses a multivariable time sequence prediction method and system based on an implicit neural network. The method comprises the following steps: 1) collecting data and preprocessing the data; 2) performing window division on the standardized or normalized multivariable time sequence and determining the length of a to-be-predicted window; 3) performing variable correlation coding on the input window to obtain a variable-level feature vector; 4) the implicit neural network based on time attention predicts target parameters by using the variable features in the step 3), and implicit neural representation of the target sequence is modeled through the parameters; 5) taking the output of the implicit nerve representation and the original input window as the input of the multi-head attention predictor, and obtaining a prediction result through cross-sequence cross attention calculation performed in the implicit space and multi-layer perceptron conversion output dimension; and 6) training and optimizing model parameters, calculating a mean square error of a prediction result and a real result, taking the mean square error as a loss function, carrying out back propagation to optimize trainable parameters of the variable correlation coding module, the implicit neural network module multi-head attention predictor and the multi-layer perceptron, and then repeating the steps 3) to 6) to obtain the multi-head attention predictor. Until the preset number of iterations is reached or the error of the model on the verification set meets the requirement of early stop; and 7) performing prediction by using a model of training convergence, and performing reverse normalization on a prediction result to obtain a final prediction result. The method has good generalization, and meanwhile, the interpretability of the attention mechanism is remarkably improved by generating the hidden space characteristics of the trend component and the season component.

Description

Technical fields:

[0001] This invention relates to a method and system for multivariate time series prediction based on implicit neural networks, belonging to the field of computer artificial intelligence. Background technology:

[0002] Multivariate time series forecasting is a critical task with profound implications in fields such as financial market analysis, environmental monitoring, and industrial process control. In this field, a significant amount of research has focused on modeling the complex correlations between multiple variables using attention mechanisms. Attention mechanisms, which selectively focus on different parts of the input data through attention matrices, have shown potential in capturing the interdependencies between variables. For example, in financial multivariate time series, these mechanisms help identify how stock prices, exchange rates, and economic indicators of different companies interact.

[0003] Since attention mechanisms were originally designed for natural language processing tasks, those obtained by computing long-distance lexical units are beneficial for autoregressive tasks. However, these attention-based methods have a significant limitation: applying attention mechanisms, whether on the variable or time dimension, inevitably leads to the loss of some information. Temporal correlations are crucial because they encode sequential patterns and dependencies that change over time. For example, in environmental monitoring, understanding the continuous changes in temperature, humidity, and air quality over days, weeks, or months is essential for accurate prediction. Attention-based models that downplay these temporal relationships may fail to fully capture the dynamic characteristics of time series, resulting in poor predictive performance. Applying attention mechanisms only on the variable or time dimension cannot fully uncover the intrinsic features of multivariate time series, and flattening the variable and time dimensions before applying attention introduces significant time and space complexity to the algorithm, making it an unconventional solution. Summary of the Invention:

[0004] To address the problems and challenges mentioned above, this invention proposes a multivariate time series prediction method and system based on implicit neural networks.

[0005] This invention proposes a method that retains the advantage of calculating attention along the variable dimension by integrating a variable correlation encoder. First, the invention learns the implicit representation of the historical window using the variable correlation encoder. Then, it predicts the parameters of the implicit neural standard continuous functions by using a standard Transformer that calculates attention along the time dimension. In this variable-based implicit neural representation, these continuous functions map time indices to predicted values ​​at corresponding moments in the sequence. The continuous functions of the implicit neural representation contain both trend and seasonal components, and the sum of the outputs of the two functions serves as the implicit representation of the target sequence. Finally, a Transformer-based predictor takes the implicit representations of the target sequence and the historical window as input and performs multi-head attention calculation between them.

[0006] The multivariate time series prediction method based on implicit neural networks proposed in this invention includes the following innovative modules:

[0007] M1: Variable correlation encoder, which can capture the endogenous correlation between multivariate variables;

[0008] M2: A variable-based implicit neural representation module that models the latent features of time series through continuous functions;

[0009] M3: A Transformer-based predictor that combines multi-head attention mechanism to achieve cross-sequence feature fusion and final prediction.

[0010] The method of the present invention includes the following steps:

[0011] S1: Collect data and preprocess the data.

[0012] S1-1: Determine the types and number of variables included in the collected dataset. This invention is used to solve the prediction task of multivariate time series data. Taking common weather data prediction as an example, important weather indicators include: air pressure, temperature, humidity, wind speed, and wind direction.

[0013] S1-2: Data cleaning and outlier handling. The raw data obtained may contain missing or erroneous information, so it is necessary to clean the raw data, handle missing values, outliers, and erroneous data, and ensure data quality.

[0014] S1-3: Data Standardization or Normalization. In this step, the data needs to be standardized or normalized to ensure that different types of data have similar scales and distributions. This helps prevent certain features from having an excessive impact on the results during model training and also helps to speed up model convergence. Commonly used methods include Z-score standardization and Min-Max normalization. Choosing the appropriate method requires considering the data distribution and the model's requirements.

[0015] S2: Window the normalized multivariate time series and determine the length of the prediction window. In multivariate time series forecasting tasks, different input window lengths and prediction window lengths will affect the algorithm's prediction performance. Window partitioning can use a sliding window (suitable for short-term dependencies) or an expanded window (suitable for long-term trends). The input window (length L) and prediction window (length F) are generated with a fixed step size, and the partitioning is performed in chronological order to ensure no future information leakage. The prediction window length F needs to be based on business requirements (e.g., short-term, medium- to long-term forecasts), data periodicity (matching integer multiples of the period), and model capabilities.

[0016] S3: The preprocessed and segmented input window is fed into a variable correlation encoder, which performs embedding encoding on each variable to generate a variable-level feature vector. Multi-layer attention is then applied to this feature vector to update the feature matrix, explicitly modeling the coupling relationships of multiple variables through a variable-dimensional attention mechanism. Subsequently, a multi-layer perceptron maps the feature output to a specific size dimension as input for the next step.

[0017] S4: Input the variable features output by the variable correlation encoder into the function parameters of a Transformer to predict implicit neural representations. The output parameters serve as weights and biases in the function, and the latent space feature vector predicted by the implicit neural representations can be obtained by directly inputting time stamps. By utilizing implicit neural representations to model time series as a combination of continuous functions, the ability to express complex patterns is improved.

[0018] S5: The output of the implicit neural representation and the original input window are used together as the input of the multi-head attention predictor. After cross-sequence cross-attention calculation in the latent space and multilayer perceptron transformation of the output dimension, the prediction result of this method is obtained.

[0019] S6: During training, the mean squared error between the predicted and actual results is calculated as the loss function and backpropagated to optimize the trainable parameters of each module. Then, steps S3 to S6 are repeated until the preset number of iterations is reached or the error of the model on the validation set reaches the requirement of early stopping. During prediction, the prediction results of S5 are inversely normalized to obtain the final prediction results.

[0020] Given a historical multivariate time series X = {x1, x2, ..., x...} T}∈R T×D The task of multivariate time series forecasting is to predict the future Y = {y1, y2, ..., y3}. T+U}∈R U×D Where D is the number of variables, and T and U are the input window length and the prediction window length, respectively. This framework constructs a model that maps X to Y, formalized as follows:

[0021] F θ :R T×D →R U×D (1)

[0022] Use F θ This refers to the prediction model constructed by this framework, where θ represents the trainable parameters in the entire model, which are continuously updated during training through steps S3 to S6:

[0023]

[0024] Here θ * These are the updated model parameters required during training; E is the preset number of training iterations; the loss function... It is the mean squared error function.

[0025] Preferably, the data used in S1 is the actual collected dataset and the existing dataset.

[0026] In S2, a widely used research and application scenario is to use an input window of length 96 to predict target windows of lengths 96, 192, 336, and 720.

[0027] Preferably, the variable embedding encoding used in S3 can be formalized as follows:

[0028]

[0029] Embedding′ α′ The time dimension was reduced (α) E (Referring to the trainable parameters of this encoding module), which eliminates the time dimension from the original input window of length 96 after encoding, resulting in the variable feature dimension. For variable d, the generated feature vector h is of length E. d ∈R E Then, the features of all variables are concatenated to obtain the feature-based matrix H = {h1, h2, ..., h...} of the input window. D}∈R E×D The attention module is then applied to this matrix to calculate the correlation between the variables in the multivariate sequence:

[0030]

[0031] Where L refers to the number of layers in the attention module, α A This represents the trainable parameters in the attention module. At the end of the variable correlation encoder, a multilayer perceptron is used to map the output of the multilayer attention module onto a specific dimension to align with the inputs of subsequent algorithmic flows.

[0032]

[0033] Ultimately, z x ∈R K×D This is the output of the variable correlation encoder, aligned with the inputs of the modules predicting implicit neural representations in dimension K and S4. α = {α M ,α A ,α E} represents all trainable parameters in the variable correlation encoder.

[0034] In step S4, the Transformer is used to predict the function parameters of the implicit neural representation:

[0035]

[0036] in These are randomly initialized weight parameters, which, along with the output of the relevance encoder, are used as input to predict two parts of the implicit neural representation function through a 3-layer attention module: former As a latent vector of trend features in time series, the latter This is used to characterize the seasonal variations in a time series. For the input time stamp, the latent vectors for the two parts of the target sequence's features are constructed using the parameters described above:

[0037]

[0038] The trend component should capture long-term patterns with relatively low frequency; therefore, a multinomial continuous function is used as the trend prediction function. Let represent the polynomial coefficients corresponding to the i-th term, and p represent the polynomial degree (usually taken as 2 or 4 to simulate low-frequency information features in the trend). Seasonal components are used to capture relatively high-frequency latent patterns; therefore, a Fourier-based periodic function is used to fit the seasonal pattern, where... Here are the Fourier coefficients, and q represents the frequency range of the Fourier series (for common time series data, this frequency range should be greater than 100).

[0039] The sum of the two functions characterizes the latent features of the predicted target window, and is finally input into the predictor in step S5 for the final prediction. This is achieved by continuously optimizing the trainable parameters β={β...} during training. T ,β I This allows the implicit neural representation results to fit the target features:

[0040]

[0041] Preferably, for the predictor used in S5, the framework can integrate most existing Transformer-based multivariate time series forecasting methods, specifically:

[0042] The iTransformer model is a time series prediction model that independently embeds the entire time series of each variable into a token. It captures the correlation between multiple variables through a self-attention mechanism and learns the global representation of the sequence using layer normalization and feedforward network modules. It performs well on real-world datasets.

[0043] The Autoformer model enhances the ability to model time series data through sequence decomposition and autocorrelation mechanisms.

[0044] Reformer is an improvement on the Transformer architecture. It learns long-term dependencies of multivariate time series by using the attention mechanism and introduces LSH technology to group and index key-value pairs, thereby reducing computational complexity and improving the efficiency of long sequence processing.

[0045] The Informer model improves the efficiency of long-sequence, high-dimensional time-series prediction through optimization strategies such as sparse attention, self-attention filtering, and one-time multi-step prediction.

[0046] All of these provide effective solutions for multivariate time series forecasting, each with its own advantages in different application scenarios. Currently popular Transformer-based benchmark models can be divided into two categories: encoder-only modules and encoder-decoder modules. Of the four time series forecasting models mentioned above, iTransformer and Reformer belong to encoder-only modules, while Autoformer and Informer belong to encoder-decoder modules. To accommodate both types of modules, two adaptation modes are designed within the predictor framework:

[0047] 1. When using the encoder-only module as the predictor, the output of the S4 implicit neural representation and the original input window are aligned and then concatenated. Then, the multi-head attention mechanism in the encoder is applied to the concatenated input.

[0048] 2. When using the predictor of the encoder-decoder module, the results of the implicit neural representation and the original input window are used to calculate cross-attention.

[0049] Specifically, for the first mode, assuming the number of heads is h, and each head corresponds to a dimension d. k / h (where d) k For multi-head attention, the target dimension is calculated after feature concatenation. For each head, the subspace matrix Q is... i K i V iExecute the attention algorithm independently to obtain

[0050] head i =Attention(Q) i ,K i V i (10)

[0051] The Attention function is the attention function, and the subspace Q is... i ,K i V i These are obtained from the original query and the key-value matrix after linear transformation by the projection matrix, respectively: Wherein, projection matrix The dimension is d model The total dimension representing the input features (implicit neural representations and the original input window after alignment).

[0052] The calculation results of all attention heads {head1,head2,…,head h The data is then concatenated and output through the projection matrix W. O After performing a linear transformation, the final result of the multi-head attention calculation is obtained:

[0053] MultiHead(Q,K,V)=Concat(head1,head2,…,head h W O (11)

[0054] Wherein, the output projection matrix W O The dimension is This enables the projection and stitching of multi-head attention, achieving the extraction and integration of attention information from multiple dimensions and subspaces of the input features.

[0055] For the second mode, the difference in cross-attention calculation lies in the subspace Q. i ,K i V i Before the linear transformation, the key values ​​K and V will come from the implicit neural representation result and the original input window, respectively. The decoder retrieves relevant information from the encoder's K and V through Q, realizing "conditional query". When using iTransformer and Reformer as predictors in the method framework, the first mode will be used to concatenate the result of step S4 and the original input window and perform multi-head attention; while when using predictors with encoder-decoder structures such as Autoformer and Informer, the second mode based on the cross-attention mechanism is used to calculate the correlation between the input window and the target window in the latent space in the decoder.

[0056] In summary, the equation for obtaining the final prediction result in step S5 is as follows:

[0057]

[0058]

[0059] Where AdaptedAttention represents the attention mechanism of the two modes discussed above, Projection maps the result of the attention mechanism to the temporal dimension of the final target window, and Embedding... ″ The representative will z x , The module for dimension alignment. γ = {γ E ,γ P ,γ A} represents the trainable parameters in the predictor.

[0060] Preferably, for step S6, the mean square error between the predicted result and the true value is calculated during training. The mean squared error loss function is used as the loss function, and then backpropagation updates the trainable parameters in the three modules θ = {α, β, γ}. The mean squared error loss function can be formalized as:

[0061]

[0062] The method F is constructed using the trained model parameters. θ The target window for predicting future series from a multivariate time series input from a historical window. During prediction... After inverse normalization, it can be used as a prediction result for the future window.

[0063] A second aspect of the present invention relates to a multivariate time series prediction system based on an implicit neural network, comprising:

[0064] The data processing module is used to preprocess the collected multivariate time series forecast data;

[0065] The variable correlation encoding module is used to embed and encode multivariate time series data along the variable dimension to obtain time-independent variable features. The purpose of variable correlation encoding is to extract and learn variable features from multivariate series data, thereby improving the model's ability to discover the relationships between variables.

[0066] The implicit neural representation module is used to generate and simulate latent space features of multivariate time series over time; it promotes the alignment of generated features with the target to be predicted by generating latent space features of both trend and seasonal components separately; the generation module is usually implemented by using Transformer to predict the parameters of continuous functions;

[0067] The target window prediction module uses the output of the implicit neural representation model and the original input window as input to the multi-head attention predictor. After cross-sequence cross-attention calculation in the latent space, the output dimension is transformed by a multilayer perceptron to obtain the prediction result of the method.

[0068] The training module uses the pre-defined training set data to train the first three modules;

[0069] The prediction module, during the usage phase, applies all the trained modules to the data to be predicted, and performs predictions on multivariate time series.

[0070] The results output module is used to output the accuracy and other evaluation metrics of multivariate time series prediction on the test set to evaluate the overall performance and effectiveness of the system.

[0071] This invention comprises three innovative modules: a variable correlation encoder, a variable-based implicit neural representation module, and a Transformer-based predictor. First, multivariate time series data is preprocessed and windowed. Then, the variable correlation encoder captures the endogenous correlations between multiple variables. An implicit neural network is used to model the latent features of the time series through continuous functions. Finally, a Transformer-based predictor combined with a multi-head attention mechanism achieves cross-sequence feature fusion and prediction. During training, the model parameters are optimized using a mean squared error loss function, and the results are inversely normalized during prediction. This invention significantly improves the prediction accuracy of multivariate time series through joint modeling, especially in long-sequence prediction. The framework supports both encoder-decoder and encoder-only attention architectures, is compatible with various Transformer-based time series prediction methods, and exhibits good generalization. Furthermore, by generating latent space features for both trend and seasonal components, the interpretability of the attention mechanism is significantly improved.

[0072] The present invention has the following advantages:

[0073] (1) By jointly modeling the variable correlation encoder and the implicit neural representation, the model can significantly improve the accuracy of multivariate time series, especially in long series prediction.

[0074] (2) The framework designed in this invention supports both encoder-decoder and encoder-only architectures, and has good generalization ability. It is compatible with various Transformer-based time series prediction methods such as iTransformer and Autoformer, and can improve the performance of each method;

[0075] (3) By generating latent spatial features of two components, trend and seasonality, the low-frequency trend and high-frequency periodic components in the time series are explicitly separated, which significantly improves the interpretability of the attention mechanism. Attached image description:

[0076] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0077] Figure 1 This is the overall flowchart of the method of the present invention.

[0078] Figure 2 This is a structural diagram of the system of the present invention. Detailed implementation method:

[0079] Exemplary embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art based on the present invention are within the scope of protection of the present invention.

[0080] Example 1

[0081] Reference Figure 1 This embodiment relates to a multivariate time series prediction method based on implicit neural networks, including the following steps:

[0082] S1: Collect and preprocess data.

[0083] This example uses five multivariate time series datasets from weather, traffic, residential electricity consumption, power system transformer temperature monitoring, and exchange rates. These datasets contain time series data from different fields and at different scales, and they are all publicly available.

[0084] The weather dataset contains local climate data for nearly 1,600 regions in the United States, spanning four years from 2010 to 2013, with data points collected hourly. Each data point includes 11 variable features such as "humidity" and "temperature," totaling over 50,000 continuous observations that can be used to divide sample windows.

[0085] The traffic dataset contains hourly data from the California Department of Transportation, showing road occupancy as measured by various sensors on highways in the San Francisco Bay Area. The dataset includes 862 variables and over 17,544 observation points.

[0086] The residential electricity consumption dataset collected electricity consumption (kWh) data from 321 customers, containing 321 variables. When using this dataset, it was converted into two years of hourly electricity consumption data, totaling 26,304 time points.

[0087] Transformer oil temperature in power supply systems is a key indicator in long-term power deployment. This dataset collects two years of data from two different counties in China. To explore data granularity in long-sequence time prediction problems, the data collectors created different subset datasets: ETTh1 and ETTh2 for 1-hour timeframes, and ETTm1 and ETTm2 for 15-minute timeframes. Each data point contains the target value "oil temperature" and six power load features, for a total of seven variables.

[0088] The exchange rate dataset contains daily exchange rate data for eight countries—Australia, the United Kingdom, Canada, Switzerland, China, Japan, New Zealand, and Singapore—from 1990 to 2016, totaling eight variables and 7,588 time points.

[0089] S1-1: Determine the number of variables, used to define and initialize the input and output module dimensions of the prediction framework. Based on the statistics of each dataset, the number of input and prediction variables are as follows:

[0090] Dataset weather transportation Electricity consumption Transformer temperature exchange rate Number of variables 11 862 321 7 8

[0091] S1-2: Data Cleaning and Outlier Handling. Sensor-based natural statistical data inevitably contains missing and erroneous information, thus requiring filtering and imputation techniques for cleaning. First, all data points outside the normal distribution range are filtered out, and then the mean is used to fill in the gaps left by the filtering.

[0092] S1-3: Divide the dataset into training, validation, and test sets, and standardize all data based on the training set. Use z-score standardization along the time dimension for each variable's sequence to transform the training set into data that conforms to a standard normal distribution in time series. This allows variables with different numerical values ​​to be uniformly input into this framework for modeling.

[0093] S2: Use an input window of length 96 to predict target windows of lengths 96, 192, 336, and 720. Divide the training set, validation set, and test set according to the lengths of the input and output windows to obtain samples with one-to-one correspondence between input and output.

[0094] S3-S5: The variable correlation encoder encodes the input window, generates variable feature vectors through embedding, and updates the feature matrix through the multi-layer attention mechanism of equation (4) to explicitly model the endogenous relationship of variables; the output is input to the function parameters of the Transformer to predict the implicit neural representation, the latent space feature vector is obtained through time stamping, and the ability to express complex patterns is improved by combining continuous functions; finally, together with the original input window, it is used as the input of the multi-head attention predictor, and the prediction result is output after the latent space cross-sequence cross-attention calculation and multi-layer perceptron dimensional transformation.

[0095] S6: Calculate the mean squared error between the predicted and actual results as the loss function and backpropagate to optimize the trainable parameters of each module. Then repeat steps S3 to S6 until the preset number of iterations is reached or the model's error on the validation set meets the early stopping requirement.

[0096] During forecasting, the forecast results of S5 are inversely normalized to obtain the final multivariate time series forecast results.

[0097] Example 2

[0098] This embodiment relates to a power load forecasting method based on a multivariate time series forecasting framework using an implicit neural network, according to the present invention, including:

[0099] S1: Application Scenarios and Data Background

[0100] Taking multi-node power load forecasting of a city's power grid as an example, the goal is to predict load changes over the next 24 hours using historical power data (including six variables: voltage, current, oil temperature, active power, reactive power, and ambient temperature). The data uses the ETT dataset format, with a time granularity of 1 hour and an input window size of 96 hours (4 days). It includes a 12-month training set, a 4-month validation set, and a 4-month test set. Preprocessing steps include Z-score normalization and sliding window partitioning.

[0101] S2: Specific Implementation Steps

[0102] S2-1: Variable Correlation Encoder Processing

[0103] Input window: Input 96 hours of 6-dimensional power data (dimension 96×6) into the encoder.

[0104] Embedding encoding: Each variable (such as voltage and current) is embedded and encoded with a dimension of 128 to generate a variable-level feature vector (dimension of 6×128).

[0105] Multi-level attention update: Apply the multi-head attention mechanism (8 heads, 16 dimensions per head) of equation (4), update the feature matrix to 6×128 through attention calculation of variable dimensions (such as the coupling relationship between voltage and active power), and explicitly model the endogenous dependence between variables (such as the effect of oil temperature rise on load fluctuation).

[0106] Dimension mapping: The feature matrix is ​​compressed to 6×64 through a 2-layer MLP (hidden layer dimension 128→64) and used as input to the implicit neural representation module Transformer.

[0107] S2-2: Prediction of Implicit Neural Representation Parameters

[0108] Transformer structure: A Transformer using a 6-layer encoder, with the input being the 64-dimensional features output from S2-1, predicts the function parameters (weights W∈R) of the implicit neural representation through a self-attention mechanism. 64×32 Bias b∈R 32 ).

[0109] Time stamp input: The time stamp contains hours (0-23) and date offset (0-365), encoded as a 32-dimensional vector, which is combined with parameters W and b to form an implicit function: f(t) = σ(W·t+b), where t is the time stamp and σ is the ReLU activation function, and the output is a 24-dimensional latent space feature vector (corresponding to the prediction dimension of the next 24 hours).

[0110] S2-3: Multi-head attention predictor output

[0111] Input combination: The 24-dimensional output of the implicit neural representation is concatenated with the 96×6-dimensional data of the original input window to form a [96×6, 24×6]-dimensional input.

[0112] Cross-sequence attention: In the latent space, using 24-dimensional predicted features as queries and original input window features as key-value pairs, cross-attention with 8 heads (64 dimensions) is used to capture the association between historical sequences and predicted sequences (such as the impact of historical load peaks on future trends).

[0113] Dimensional transformation: The output dimension is mapped to 24×6 (multivariate predicted values ​​for the next 24 hours) through a 2-layer MLP (hidden layer dimension 64→24).

[0114] S2-4: Model Training and Prediction

[0115] Training process:

[0116] The formula for calculating the mean squared error loss between the predicted value and the actual load is as follows:

[0117] The trainable parameters of the encoder, Transformer, and multilayer perceptron are updated using backpropagation with the Adam optimizer (learning rate 1e-4).

[0118] Stop early when training has been iterated to 100 rounds or when the mean squared error loss on the validation set decreases by less than 0.01 for 5 consecutive rounds.

[0119] Prediction process: The output of S5 is denormalized, and the true load value is reconstructed using the mean (μ = 500kW) and standard deviation (σ = 100kW) of the training set.

[0120] Example 3

[0121] Reference Figure 2 This embodiment relates to the present invention and also to a multivariate time series prediction system based on implicit neural representation, comprising:

[0122] The data processing module is used to preprocess the collected multivariate time series forecast data.

[0123] The variable correlation encoding module is used to embed and encode multivariate time series data along the variable dimension to obtain time-independent variable features. The purpose of variable correlation encoding is to extract and learn variable features from multivariate time series data, thereby improving the model's ability to discover the relationships between variables.

[0124] The implicit neural representation module is used to generate and model latent space features of multivariate time series over time. It facilitates the alignment of generated features with the target to be predicted by generating latent space features for both trend and seasonal components separately. The generation module is typically implemented using Transformer to predict the parameters of a continuous function.

[0125] The target window prediction module uses the output of implicit neural representation modeling and the original input window as input to the multi-head attention predictor. After cross-sequence cross-attention calculation in the latent space, the output dimension is transformed by a multilayer perceptron to obtain the prediction result of the method.

[0126] The training module uses the pre-defined training set data to train the first three modules.

[0127] The prediction module, during the usage phase, applies all the trained modules to the data to be predicted, and performs predictions on multivariate time series.

[0128] The results output module is used to output the accuracy and other evaluation metrics of multivariate time series prediction on the test set to evaluate the overall performance and effectiveness of the system.

[0129] Example 4

[0130] The present invention also relates to a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the method of embodiment 1.

[0131] The embodiments described in this specification are merely examples of implementations of the inventive concept. The scope of protection of this invention should not be considered as limited to the specific forms stated in the embodiments. The scope of protection of this invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A multivariate time series prediction method based on implicit neural networks, characterized in that, Includes the following steps: S1: Collect data and preprocess the data; S2: Divide the standardized or normalized multivariate time series into windows and determine the length of the window to be predicted; S3: Encode the input window using variable correlation to obtain variable-level feature vectors; S4: The implicit neural network based on time attention uses the variable features from step S3 to predict target parameters and models the implicit neural representation of the target sequence through parameters. S5: The output of the implicit neural representation and the original input window are used together as the input of the multi-head attention predictor. After cross-sequence cross-attention calculation in the latent space and multilayer perceptron transformation of the output dimension, the prediction result is obtained. S6: Train and optimize the model parameters, calculate the mean squared error between the predicted results and the actual results as the loss function, and backpropagate to optimize the trainable parameters of the variable correlation encoding module, the implicit neural network module multi-head attention predictor, and the multilayer perceptron. Then repeat steps S3 to S6 until the preset number of iterations is reached or the error of the model on the validation set reaches the requirement of early stopping. S7: Use the model that has converged after training in S6 to make predictions, and then inversely normalize the prediction results of S5 to obtain the final prediction results.

2. The multivariate time series prediction method based on implicit neural networks according to claim 1, characterized in that, Step S1 specifically includes: S1-1: Determine the types and number of variables included in the collected dataset; S1-2: Data cleaning and outlier handling, handling missing values, outliers and erroneous data to ensure data quality; S1-3: Data standardization or normalization makes different types of data have similar scales and distributions.

3. The multivariate time series prediction method based on implicit neural networks according to claim 1, characterized in that, The window division described in step S2 includes a sliding window suitable for short-term dependencies and an extended window suitable for long-term trends. The input window and prediction window are generated with a fixed step size and divided in chronological order to ensure no leakage of future information.

4. The multivariate time series prediction method based on implicit neural networks according to claim 1, characterized in that, Step S3 specifically includes: S3-1: The preprocessed and segmented input window is fed into the variable correlation encoder, which embeds each variable to generate variable-level feature vectors. h d =Embedding ′ (X) (3) X is a given multivariate time series, and the embedding encoding reduces the time dimension of the input data. S3-2: By applying multi-layer attention to update the feature matrix on this feature vector, the coupling relationship of multivariate variables is explicitly modeled through the attention mechanism of the variable dimension: H l+1 *Warning l (H l ),l=0,1,…,L-1 (4) H 0 The matrix represents the concatenation of all variable features, and L refers to the number of attention layers in the attention module. S3-3: Subsequently, a multilayer perceptron is used to map the feature output onto a specific dimension as input for the next step: With x =MLP(H L ) (5) Finally, z was obtained x This outputs the results of the variable correlation encoder.

5. The multivariate time series prediction method based on implicit neural networks according to claim 1, characterized in that, Step S4 specifically includes: S4-1: Input the variable features output by the variable correlation encoder into the function parameters of the target representation predicted by the implicit neural network based on time series attention; S4-2: The output parameters serve as weights and biases in the function. By directly inputting time stamps, the latent space feature vector predicted by the implicit neural representation can be obtained.

6. The multivariate time series prediction method based on implicit neural networks according to claim 5, characterized in that: Step S4 introduces an implicit neural network to model the latent space features of the target to be predicted, and uses it as an intermediate process for multivariate time series prediction. Specifically, step S4-1 inputs the variable features output by the variable correlation encoder into the Transformer model, and uses the Transformer model to predict the function parameters of the implicit neural representation. W β The weight parameters represent randomly initialized values, and the output z of the correlation encoder. x Together they serve as input; the resulting function parameters include weights and biases used to construct the continuous mapping function; As a latent vector of trend features in time series Used to characterize seasonal variations in time series; Step S4-2: Input the time stamp into the continuous mapping function, and calculate the latent space feature vector using the function parameters. Let represent the polynomial coefficients corresponding to the i-th term, and p represent the degree of the polynomial. Here, q represents the frequency range of the Fourier series; the time series is modeled as a combination of continuous functions of trend features and seasonal features through the latent space feature vector, so as to improve the ability to express complex patterns in the time series. The Transformer model captures the dependencies of variable features through an attention mechanism and aligns their temporal features in the implicit neural space. The continuous mapping function maps time stamps to the latent space based on the function parameters, realizing the transformation of time series representation from discrete points to continuous functions. The sum of the two functions characterizes the latent features of the target window to be predicted, and is finally input into the predictor in step S5 for the final prediction.

7. The multivariate time series prediction method based on implicit neural networks according to claim 1, characterized in that: Step S5 uses the output of the implicit neural representation and the original input window together as the input to the multi-head attention predictor. Different input methods are adopted for attention algorithms with different structures, thereby integrating with existing Transformer-based prediction methods. A simplified attention algorithm is used to illustrate: head = Attention(Q,K,V) (10) Q, K, and V represent the query vector, key (identifier vector), and value (information vector) in the attention mechanism; the resulting output head represents the attention head result. For prediction methods that only include an encoder module, the original input window is aligned and concatenated with the implicit neural representation; the result is then used as the input to the multi-head attention algorithm. Embedding further encodes the concatenated input and applies it to the multi-head attention algorithm: MultiHead(Q,K,V)=Concat(head1,head2,…,head h )W (11) The output projection matrix W is used to project and concatenate the multi-head attention results, and the original Q, K, V are derived from the feature matrix H of all variables after embedding and encoding. ′ ={h ′ 1,h ′ 2,…,h ′ D } is obtained through projection transformation; while {head1,head2,…,head h The subspace Q derived from multiple independent projection matrices i ,K i V i The results were obtained by calculation using formula (10); For encoder-decoder prediction methods (such as Autoformer and Informer), two parts of the input are used to calculate cross-attention: AdaptedAttention represents a cross-attention mechanism. Projection maps the results of the attention mechanism to the time dimension of the final target to obtain the prediction result. For the input of the attention function, the architecture containing only the encoder module uses the concatenated features as a unified query Q and key-value pairs K and V; while in the encoder-decoder structure method, the encoder's K and V come from the implicit neural representation results and the original input window, respectively, and the decoder retrieves relevant information from the encoder's output K and V through Q to realize "conditional query".

8. The multivariate time series prediction method based on implicit neural networks according to claim 7, characterized in that: Step S5 involves multi-head attention calculation in the latent space and multilayer perceptron transformation of the output dimension to obtain the prediction result; By integrating existing Transformer-based prediction methods as attention head predictors for multi-head attention computation, the prediction accuracy of multivariate time series can be maximized.

9. The multivariate time series prediction method based on implicit neural networks according to claim 1, characterized in that, Step S6 training adopts an end-to-end approach. The model to be trained, which maps historical multivariate sequence windows to future models, can be formalized as follows: F θ :R T×D →R U×D (1) F in formula (1) θ In formula (2), F refers to the constructed multivariate time series prediction model, where the parameter θ includes the trainable parameters of the modules to be trained in steps S3 to S5, and T, U, and D represent the input historical window length, the future window length to be predicted, and the number of variables, respectively; formula (2) summarizes the parameter update equation for the iterative training process in step S6, where θ * These are the updated model parameters required during training; E is the preset number of training iterations. loss function It is the mean squared error function, in the form of: X∈R T×D ,Y∈R U×D Representative model F θ The input window and the real label window during training; In step S3, the embedding encoding, multi-layer attention mechanism, and multi-layer perceptron of the variable correlation encoder can be summarized into the module VariteEncoder. α In step S4, the implicit neural network can be summarized as module INR. β In step S5, the predictor can be summarized as the Forecaster module. γ In step S6, iteratively train model F. θ At that time, the parameters θ = {α, β, θ} of each module will be updated simultaneously.

10. A multivariate time series prediction system based on implicit neural networks, characterized in that, include: The data processing module is used to preprocess the collected multivariate time series forecast data; The variable correlation encoding module is used to embed and encode multivariate time series data along the variable dimension to obtain time-independent variable features. The purpose of variable correlation encoding is to extract and learn variable features from multivariate series data, thereby improving the model's ability to discover the relationships between variables. Implicit neural representation module for generating and simulating latent spatial features of multivariate time series over time; By generating latent space features of both trend and seasonal components separately, the alignment between the generated features and the target to be predicted is facilitated; the generation module is typically implemented by using Transformer to predict the parameters of a continuous function; The target window prediction module uses the output of the implicit neural representation model and the original input window as input to the multi-head attention predictor. After cross-sequence cross-attention calculation in the latent space, the output dimension is transformed by a multilayer perceptron to obtain the prediction result of the method. The training module uses the pre-defined training set data to train the first three modules; The prediction module, during the usage phase, applies all the trained modules to the data to be predicted, and performs predictions on multivariate time series. The results output module is used to output the accuracy and other evaluation metrics of multivariate time series prediction on the test set to evaluate the overall performance and effectiveness of the system.

Images