Time sequence prediction method based on visual pre-training model

Abstract

The invention discloses a time sequence prediction method based on a visual pre-training model, and the method comprises the steps: carrying out the interpolation and normalization processing of input data, and dividing the data into a univariable sequence and a multivariable sequence; converting the univariate sequence into an image through segmentation, rendering and alignment operations, and reconstructing the converted image by using the pre-training MAE to obtain a predicted value corresponding to the univariate sequence; converting the multivariable sequence into an image through clustering, rendering and alignment operations, reconstructing the converted image by using the pre-training MAE, and restoring the reconstructed image to obtain a predicted value corresponding to the multivariable sequence; and respectively endowing different learnable weight parameters for the predicted value corresponding to the single-variable sequence and the predicted value corresponding to the multivariable sequence, and dynamically integrating time features and space features. According to the method, the cooperative relationship among the multivariate sequences is considered, the influence of interpolation errors on the model can be reduced, and the prediction performance is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of multivariate time series prediction, and in particular to a time series prediction method based on a visual pre-training model. Background Art

[0002] Currently, in the field of multivariate time series prediction, one approach is to segment and concatenate single-variable sequences into images, reconstructing them using a pre-trained visual model called MAE (Masked AutoEncoder) to obtain predicted values. However, this approach cannot simultaneously integrate multivariate sequences into a single image, resulting in an inability to model synergistic relationships between variables. Another approach uses memory networks and adversarial training to jointly simulate local and global temporal dynamics and interpolate missing values. However, this approach suffers from error accumulation, which reduces the model's predictive performance.

[0003] Existing solutions for missing data mostly involve building end-to-end models that integrate interpolation and prediction based on non-pretrained models. This involves first interpolating missing values ​​and then using the interpolated sequences as training data for the model. However, interpolation errors can interfere with the model fitting process, leading to cumulative errors. Summary of the Invention

[0004] The purpose of the present invention is to provide a time series prediction method based on a visual pre-training model, which simultaneously considers the synergistic relationship between multivariate sequences and can also reduce the impact of interpolation errors on the model, thereby improving the prediction performance.

[0005] The purpose of the present invention is achieved through the following technical solutions: A time series prediction method based on a visual pre-training model, the method comprising: Step 1: First, interpolate and normalize the multivariate time series data with a sequence length of L and a total number of variables of N. Step 2: Based on the Temporal View, the univariate sequence is converted into an image through segmentation, rendering, and alignment operations. The converted image is reconstructed using the visual pre-trained model MAE to obtain the predicted value corresponding to the univariate sequence. The Temporal View is a module that can perform feature transformation and reconstruction on univariate time series. Step 3: Based on the Spatial View, the multivariate sequence is converted into an image through clustering, rendering, and alignment operations. The converted image is reconstructed using the visual pre-trained model MAE, and the reconstructed image is restored to obtain the predicted value corresponding to the multivariate sequence. The Spatial View is a module that can perform feature transformation and reconstruction on multivariate time series. Step 4: Assign different learnable weight parameters to the predicted values ​​corresponding to the univariate sequence and the multivariate sequence, dynamically integrate the temporal and spatial features, and achieve adaptive optimization of the model and improvement of the prediction performance.

[0006] It can be seen from the technical solution provided by the present invention that the above method simultaneously considers the synergistic relationship between multivariate sequences, and can also reduce the impact of interpolation errors on the model, thereby improving the prediction performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0008] Figure 1 A schematic flow chart of a time series prediction method based on a visual pre-training model provided by an embodiment of the present invention; Figure 2 Schematic diagram of natural images and sequence images according to an embodiment of the present invention. DETAILED DESCRIPTION

[0009] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments, and do not constitute a limitation of the present invention. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0010] like Figure 1 FIG2 is a flow chart of a time series prediction method based on a visual pre-training model provided by an embodiment of the present invention, wherein the method includes: Step 1: First, interpolate and normalize the multivariate time series data with a sequence length of L and a total number of variables of N. In this step, the K-nearest neighbor regression (KNN) algorithm is used to interpolate missing values ​​in the input data. KNN is an example-based supervised learning algorithm that can be used for regression tasks. Specifically, given a current sample point, the n nearest neighbors to the sample point are found in the training set, and the weighted average of these n neighbors is used to interpolate the current sample point. Considering that the neighbors closer to the current sample point have a deeper impact on the current sample point, a distance-based weighted method is used to predict the value of the current sample point, that is, higher weights are given to samples closer to the current sample point, and lower weights are given to samples farther away. The specific calculation formula is: in, is the current sample x to be predicted and the i-th nearest neighbor The Euclidean distance of Represents the predicted value of the current sample x; Indicates the weight size; In order to make the model converge faster during training, the input data needs to be normalized. The normalization formula is: in, Represents the value of variable i at the jth moment; Represents the normalized value; represents the mean; represents variance; is a minimum value, preventing is 0; is a hyperparameter less than 1, which can achieve better performance after model normalization; L is the length of the input sequence.

[0011] Step 2: Based on the Temporal View, the univariate sequence is converted into an image through segmentation, rendering, and alignment operations. The converted image is reconstructed using the visual pre-trained model MAE to obtain the predicted value corresponding to the univariate sequence. The Temporal View is a module that can perform feature transformation and reconstruction on univariate time series. In this step, the segmentation operation is as follows: first, according to the set period The univariate sequence is segmented and then the segmented subsequences are spliced ​​into a two-dimensional matrix. The purpose of this is mainly to meet the input requirements of MAE and capture the time characteristics within each cycle and the correlation characteristics between different cycles. The univariate sequence after segmentation and splicing is expressed as ,in Indicates the length of the input sequence; The rendering operation is: Since the input of the visual pre-training model MAE is three-channel image data, in order to match the channel dimension, Duplicate three times in the first dimension and render as grayscale ; Alignment operation: Since it is The subsequences are spliced ​​in the (periodic) direction, so the part that needs to be predicted corresponds to the grayscale image On the right side, so we need to Add a mask area to the right side of , and use the reconstruction ability of MAE to get the predicted value. However, since the size of the grayscale image is generally difficult to be exactly the same as the image size required by MAE, it is necessary to use bilinear interpolation to convert the grayscale image The size is aligned to the standard input size of the pre-trained MAE. Specifically, assuming that the visual pre-trained model MAE divides the input image into patches, each patch is of size , the number of visible patches is expressed as , the number of mask patches is expressed as ; Use bilinear interpolation to convert the grayscale image The dimension from Interpolation ;in, is the length of the input sequence and the predicted length The specific calculation formula is: in It is a hyperparameter used to adjust the size of the visible patches and enhance the reconstruction ability of the model; Reconstruction operation: Use the visual pre-trained model MAE to reconstruct the aligned grayscale image into a complete grayscale image without mask patches; take the average of the three channels of the reconstructed grayscale image to obtain a two-dimensional matrix; restore the obtained two-dimensional matrix to the segmented state through bilinear interpolation; finally, extract the sequence reconstructed by the mask patches for subsequent feature fusion and prediction.

[0012] Step 3: Based on the Spatial View, the multivariate sequence is converted into an image through clustering, rendering, and alignment operations. The converted image is reconstructed using the visual pre-trained model MAE, and the reconstructed image is restored to obtain the predicted value corresponding to the multivariate sequence. The Spatial View is a module that can perform feature transformation and reconstruction on multivariate time series. In this step, the clustering operation is: based on the Spatial View, the sequence values ​​of all multivariate sequences within a period of time are spliced ​​into a two-dimensional matrix. The advantage of this is that it can effectively model the dependencies across variables, thereby realizing the representation learning of complex covariation structures (referring to the dynamically changing dependencies between variables in high-dimensional data). However, the sequence images generated by the random arrangement of multivariate sequences lack the inherent regularity of natural images (the spatial correlation between different pixels), and cannot effectively simulate the spatial distribution characteristics existing in real visual scenes, such as Figure 2 The figure shows a schematic diagram of a natural image and a sequence image according to an embodiment of the present invention. Figure 2 Comparative analysis reveals that in natural images, the closer the image segments are, the greater the similarity, while the farther the image segments are, the less similar they are. Therefore, image segments can be considered as distinct variables, with variables with high similarity clustering together to form a cluster. Therefore, to enable the pre-trained MAE to better recognize this feature, K-means clustering (KMeans) is used to cluster multivariate sequences, transforming the originally irregularly arranged sequence images into a sequence that more closely resembles natural images. The rendering operation is as follows: Unlike the Temporal View, the Spatial View does not require the splitting and splicing of the single variable sequence, but the two-dimensional matrix composed of the multivariate sequence Duplicate three times in the first dimension and render as grayscale ; The alignment and reconstruction operations are: Since the input sequences of different variables are spliced ​​in the variable direction, the part to be predicted corresponds to the grayscale image The lower side, so it is necessary to A mask area is added to the lower side of ; the parameters used for alignment and reconstruction in Spatial View are consistent with those in Temporal View, and the pre-trained MAE required for reconstruction is shared with Temporal View; The restoration operation is as follows: Since the original input variables are clustered and rearranged in the clustering operation, the original order of the variables is disrupted. In order to keep the reconstructed eigenvalues ​​obtained from the Spatial View and the reconstructed eigenvalues ​​obtained from the Temporal View aligned with the variables, the reconstructed eigenvalues ​​obtained from the Spatial View are restored, that is, the original order of the input variables is restored.

[0013] Step 4: Assign different learnable weight parameters to the predicted values ​​corresponding to the univariate sequence and the multivariate sequence, dynamically integrate the temporal and spatial features, and achieve adaptive optimization of the model and improvement of the prediction performance.

[0014] In this step, although the output of Temporal View and the output of Spatial View Both serve as the input feature sources for the final prediction, but there are differences in their information contribution and they may contain a certain degree of redundant information. In order to improve the model's ability to discriminate and utilize key information, this step introduces a weight-based feature fusion mechanism, which assigns different learnable weight parameters to the two types of features, thereby achieving adaptive adjustment of feature importance and enhancing the model's selectivity and dynamic planning capabilities in the prediction stage.

[0015] Because the output of Temporal View and the output of Spatial View The historical values ​​are included in the data, where B is the batch size; P is the period; D is the number of periods after reconstruction; N is the total number of variables; S is the sequence length of each variable after reconstruction; and the prediction uses only the length after the historical value. The reconstructed sequence, so the final predicted value for: in and Represent the length of the lookback window (historical sequence) and the prediction length respectively; and Represents the corresponding weight parameters; MLP represents a multi-layer perceptron, which is used to fuse the output from the time dimension and spatial dimensions The output of , its calculation process is expressed as: in, represents the input vector, i.e., the feature after weighted fusion, and d represents the representation latitude of the feature space of the input data; and are the weight matrices of the first and second layers respectively, d1 and d2 represent the representation latitudes of the feature space of the first and second layers respectively; and are the bias terms for the first and second layers respectively; Represents the nonlinear activation function ReLU; Dropout() is a regularization function used to prevent overfitting of neural networks, which enhances the generalization ability of the model by randomly dropping neurons.

[0016] A large number of experimental results show that, under the same interpolation conditions, compared with other baselines in the existing technology, the time series prediction method described in the embodiment of the present invention (i.e., ST-ViTS) can achieve the best prediction performance on six public datasets and different data missing ratios.

[0017] It should be noted that the contents not described in detail in the embodiments of the present invention belong to the prior art known to those skilled in the art.

[0018] Based on the above method embodiment, an embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the method.

[0019] An embodiment of the present invention further provides a computer storage medium, wherein the computer storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing the method.

[0020] In summary, the method described in the embodiment of the present invention converts univariate sequences and multivariate sequences into images, while considering the temporal patterns within the variables and the collaborative dependencies between variables, and utilizes the noise resistance of the visual pre-training model to reduce the impact of interpolation errors on prediction performance, thereby improving prediction performance.

[0021] In addition, those skilled in the art will understand that all or part of the steps in the above-mentioned embodiment method can be implemented by instructing the relevant hardware through a program, and the corresponding program can be stored in a computer-readable storage medium. The above-mentioned storage medium can be a read-only memory, a disk or an optical disk, etc.

[0022] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or any form of implication that the information constitutes prior art already known to those skilled in the art.

Claims

1. A time series prediction method based on a visual pre-training model, characterized in that: The method comprises: Step 1: First, interpolate and normalize the multivariate time series data with a sequence length of L and a total number of variables of N. Step 2: Based on the Temporal View, the univariate sequence is converted into an image through segmentation, rendering, and alignment operations. The converted image is reconstructed using the visual pre-trained model MAE to obtain the predicted value corresponding to the univariate sequence. The Temporal View is a module that can perform feature transformation and reconstruction on univariate time series. Step 3: Based on the Spatial View, the multivariate sequence is converted into an image through clustering, rendering, and alignment operations. The converted image is reconstructed using the visual pre-trained model MAE, and the reconstructed image is restored to obtain the predicted value corresponding to the multivariate sequence. The Spatial View is a module that can perform feature transformation and reconstruction on multivariate time series. Step 4: Assign different learnable weight parameters to the predicted values ​​corresponding to the univariate sequence and the multivariate sequence, dynamically integrate the temporal and spatial features, and achieve adaptive optimization of the model and improvement of the prediction performance.

2. The time series prediction method based on the visual pre-training model according to claim 1 is characterized in that: In step 1, the K-nearest neighbor regression algorithm is used to interpolate the missing values ​​of the input data. Specifically, given a current sample point, the n nearest neighbors to the sample point are found in the training set, and the weighted average of the n neighbors is used to interpolate the current sample point. The value of the current sample point is predicted using a distance-based weighted method, that is, samples with closer distances are given higher weights, and samples with farther distances are given lower weights. The specific calculation formula is: in, is the current sample x to be predicted and the i-th nearest neighbor The Euclidean distance of Represents the predicted value of the current sample x; Indicates the weight size; In order to make the model converge faster during training, the input data needs to be normalized. The normalization formula is: in, Represents the value of variable i at the jth moment; Represents the normalized value; represents the mean; represents variance; is a minimum value, preventing is 0; is a hyperparameter less than 1, which can achieve better performance after model normalization; L is the length of the input sequence.

3. The time series prediction method based on the visual pre-training model according to claim 1 is characterized in that: In step 2, the segmentation operation is: first, according to the set period The univariate sequence is segmented and then the segmented subsequences are spliced ​​into a two-dimensional matrix, capturing the time characteristics within each period and the correlation characteristics between different periods. The univariate sequence after segmentation and splicing is expressed as ,in Indicates the length of the input sequence; The rendering operation is: Duplicate three times in the first dimension and render as grayscale ; Alignment operation: Use bilinear interpolation to The size is aligned to the standard input size of the pre-trained MAE. Specifically, assuming that the visual pre-trained model MAE divides the input image into patches, each patch is of size , the number of visible patches is expressed as , the number of mask patches is expressed as ; Use bilinear interpolation to convert grayscale image The dimension from Interpolation ;in, is the length of the input sequence and the predicted length The specific calculation formula is: in It is a hyperparameter used to adjust the size of the visible patches and enhance the reconstruction ability of the model; Reconstruction operation: Use the visual pre-trained model MAE to reconstruct the aligned grayscale image into a complete grayscale image without mask patches; take the average of the three channels of the reconstructed grayscale image to obtain a two-dimensional matrix; restore the obtained two-dimensional matrix to the segmented state through bilinear interpolation; finally, extract the sequence reconstructed by the mask patches for subsequent feature fusion and prediction.

4. The time series prediction method based on the visual pre-training model according to claim 1 is characterized in that: In step 3, the clustering operation is as follows: based on the Spatial View, the sequence values ​​of all multivariate sequences within a period of time are spliced ​​into a two-dimensional matrix; the multivariate sequences are clustered using K-means clustering to convert the original irregularly arranged sequence images into sequence images close to natural images; The rendering operation is: a two-dimensional matrix composed of multivariate sequences Duplicate three times in the first dimension and render as grayscale ; The alignment and reconstruction operations are: Since the input sequences of different variables are spliced ​​in the variable direction, the part to be predicted corresponds to the grayscale image The lower side, so in the grayscale image A mask area is added to the lower side of ; the parameters used for alignment and reconstruction in Spatial View are consistent with those in Temporal View, and the pre-trained MAE required for reconstruction is shared with Temporal View; The restoration operation is: in order to keep the reconstructed eigenvalues ​​obtained from the Spatial View and the reconstructed eigenvalues ​​obtained from the Temporal View aligned with each other, the reconstructed eigenvalues ​​obtained from the Spatial View are restored to their original order.

5. The time series prediction method based on the visual pre-training model according to claim 1 is characterized in that: In step 4, the process of step 4 is specifically as follows: Because the output of Temporal View and the output of Spatial View The historical values ​​are included in the data, where B is the batch size; P is the period; D is the number of periods after reconstruction; N is the total number of variables; S is the sequence length of each variable after reconstruction; and the prediction uses only the length after the historical value. The reconstructed sequence, so the final predicted value for: in and Represent the length of the lookback window and the prediction length respectively; and Represents the corresponding weight parameters; MLP represents a multi-layer perceptron, which is used to fuse the output from the time dimension and spatial dimensions The output of , its calculation process is expressed as: in, represents the input vector, i.e., the feature after weighted fusion, and d represents the representation latitude of the feature space of the input data; and are the weight matrices of the first and second layers respectively, d1 and d2 represent the representation latitudes of the feature space of the first and second layers respectively; and are the bias terms for the first and second layers respectively; Represents the nonlinear activation function ReLU; Dropout() is a regularization function used to prevent overfitting of neural networks.

6. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 5.

7. A computer storage medium, characterized in that The computer storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the method according to any one of claims 1 to 5.

Images