A time series large model data processing method and system for realizing multivariate prediction

Abstract

The application provides a time sequence large model data processing method and system for realizing multivariate prediction, and belongs to the technical field of artificial intelligence, and comprises the following steps: acquiring time sequence data, and performing normalization processing to obtain multivariate data; uniformly sampling the multivariate data to mix the multivariate data into single variable data; performing patch division on the single variable data to divide the single variable data into a plurality of non-overlapping continuous patches; converting each continuous patch into a fixed-size vector, adding position encoding, length encoding and resolution encoding information, and obtaining vector data; and inputting the vector data into a transformer network to perform model training and prediction; the model learns the correlation information between different variables through an attention mechanism, realizes time sequence prediction of multivariate data, and improves the ability of the model in processing complex time sequence data.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence, and particularly relates to a time series large model data processing method and system for realizing multivariate prediction. BACKGROUND

[0002] Large language models use large-scale self-supervised learning, pre-training through a large amount of unlabeled data, using the Transformer structure and the core attention mechanism, so that the model can effectively capture context information, can process multi-modal data, and performs well in the generation task. Inspired by large language models, more and more researchers have begun to explore large pre-training time series models, which can improve the generalization ability and performance of the model by pre-training on large-scale data sets. The current representative model architectures include Encoder-Only, Decoder-Only, and Encoder-Decoder architectures.

[0003] However, in the field of time series data processing, especially in scenarios involving multivariate input, existing Decoder-Only architecture large models mostly rely on single variable input, which limits the ability of the model to process complex time series data. In order to overcome this limitation, researchers have begun to explore methods for processing multivariate data, but most of the methods focus on feature fusion or parallel processing. SUMMARY

[0004] The present application is based on the existing Decoder-Only architecture time series large model, and proposes a method for efficiently mixing multivariate data into single variable data. The mixed data is used to train or fine-tune the time series large model, so that the model can effectively capture and learn the relevant information between different variables through the attention mechanism, and realize the prediction of multivariate time series.

[0005] In order to achieve the above purpose, the technical solution adopted by the present application is as follows:

[0006] In a first aspect, the present application provides a time series large model data processing method for realizing multivariate prediction, comprising:

[0007] Obtaining time series data and performing normalization processing to obtain multivariate data;

[0008] Uniformly sampling the multivariate data to mix it into single variable data;

[0009] Dividing the single variable data into a plurality of non-overlapping continuous patches through patch division;

[0010] The continuous patch is converted into a fixed-size vector, and position encoding, length encoding and resolution encoding information are added to obtain vector data;

[0011] The vector data is input into a transformer network for model training and prediction.

[0012] Further, the normalization processing includes scaling the time series data to a range with a mean of 0 and a variance of 1, so that the time series data distribution is close to a normal distribution.

[0013] Further, the uniform sampling and mixing into single variable data step includes sequentially taking one time point data of each variable of the multi-variable data, and mixing the multi-variable data into one single variable.

[0014] Further, the patch division step includes dividing the single variable data into a plurality of non-overlapping continuous patches, determining the number of time points of each patch, and traversing the entire time series to take out a fixed number of continuous time point data as a patch each time.

[0015] Further, the position encoding uses a combination of sine and cosine functions to provide the model with information about the position of each time point data in the sequence.

[0016] Further, the length encoding includes masking part or all of the input patches by random masking to enable the model to adapt to different context lengths.

[0017] Further, the resolution encoding includes setting 0, 1 and 2 to represent high frequency, medium frequency and low frequency data respectively, and randomly initializing a resolution sequence of patch length, and outputting a resolution encoding vector of dim dimensions through a multi-layer perception.

[0018] In a second aspect, the present application provides a time series large model data processing system for realizing multi-variable prediction, comprising:

[0019] The acquisition module is configured to acquire time series data and perform normalization processing to obtain multi-variable data.

[0020] The data mixing module is configured to uniformly sample and mix the multi-variable data into single variable data.

[0021] The division module is configured to divide the single variable data into a plurality of non-overlapping continuous patches.

[0022] The vectorization module is configured to convert each continuous patch into a fixed-size vector, and add position encoding, length encoding and resolution encoding information to obtain vector data;

[0023] The training prediction module is configured to input the vector data into a transformer network for model training prediction, and the model learns the correlation information between different variables through an attention mechanism to realize multivariate time series prediction.

[0024] In a third aspect, the present application provides a computer-readable storage medium comprising a stored program, wherein the program, when executed, controls the device in which the computer-readable storage medium is located to perform the time series large model data processing method for realizing multivariate prediction according to the first aspect.

[0025] In a fourth aspect, the present application provides an electronic device comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to perform the time series large model data processing method for realizing multivariate prediction according to the first aspect.

[0026] Compared with the prior art, the present application has the advantages and positive effects that:

[0027] The present application provides a time series large model data processing method for realizing multivariate prediction, comprising: obtaining time series data and performing normalization processing to obtain multivariate data; uniformly sampling the multivariate data to mix into single variable data; patch dividing and vectorizing the single variable data; inputting the vector data into a transformer network for model training prediction to realize multivariate time series prediction. BRIEF DESCRIPTION OF DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0029] Figure 1 It is a flowchart of a method for efficiently mixing multivariate into single variable in time series prediction. Detailed Implementation

[0030] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described below in conjunction with the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0031] Numerous specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways than those described herein, and therefore the invention is not limited to the specific embodiments disclosed in the following specification.

[0032] Example 1, such as Figure 1 As shown, this application proposes a method for efficiently mixing multivariate data into univariate data in time series forecasting. First, the time series data is normalized. The processed multivariate data is then uniformly sampled and mixed into univariate data. The data is divided into patches according to fixed time points. Each patch is then vectorized, with location encoding, length encoding, and resolution encoding added. These vectors are fed into a subsequent transformer network for model learning and prediction. The model effectively captures and learns relevant information between different variables through an attention mechanism, achieving multivariate time series forecasting.

[0033] The specific implementation steps are as follows:

[0034] ① Time series data, such as weather temperature data, can be recorded hourly. Many variables are related to weather temperature, such as humidity and air pressure data, which are also collected hourly. The data is then normalized, scaling it to a range with a mean of 0 and a variance of 1, making the data distribution approximate a normal distribution. The specific normalization method involves calculating the mean and variance, and then using the formula...

[0035] Where z: standardized value; x: original value; μ: feature mean; σ: standard deviation of feature.

[0036] ② Perform multivariate data mixing. Take the data values ​​corresponding to the same time point for each variable in turn, and mix the multiple variables into a single variable.

[0037] ③Patch division. The one-dimensional time series of the previous step is divided into multiple non-overlapping continuous segments (patches), the number of time points of each patch is determined, and the entire time series is traversed, each time taking out a fixed number of continuous time point data as a patch. For time series whose length is not a multiple of the fixed number of time points, add zero padding to the nearest multiple of the fixed number of time points, or ignore the insufficient part when input.

[0038] ④Embedding layer realizes vectorization. Time series vectorization converts each patch into a fixed-size vector for input into the Transformer layer. Each patch is output as a vector by a multi-layer perceptron (MLP) whose dimension matches the input dimension (dim) specified by the model.

[0039] Position encoding. Since the Transformer model itself does not have the ability to process sequence order, position encoding needs to be added to the vector of each patch. Position encoding can be a combination of sine and cosine functions to provide the model with information about the position of each data point in the sequence.

[0040] The position encoding formula is as follows:

[0041] (1)

[0042] (2)

[0043] where: pos is the position index, i is the dimension index of the position encoding vector, is the dimension of the model (i.e., the dimension of the embedding vector).

[0044] Length encoding. By randomly masking part or all of the input patches, the model can adapt to different context lengths. Specifically, in each training batch, a starting point r is randomly selected, r takes a value less than the number of patches, representing the part from this point to the end of the patch that is masked. This masking strategy ensures that the model can see all possible context lengths during training. A binary mask vector is created with the same length as the input time series, where 1 indicates that the corresponding time point data should be considered by the model, and 0 indicates that the corresponding time point data should be ignored.

[0045] Resolution encoding. Set 0, 1, and 2 to represent high, medium, and low frequency data, respectively. High frequency represents data with frequencies of minutes, hours, and days, medium frequency represents data with frequencies of weeks and months, and low frequency represents data with frequencies of quarters and years. Randomly initialize the resolution sequence of the patch length, and output a dim-dimensional resolution encoding vector through a multi-layer perceptron (MLP).

[0046] 5. The model is trained and predicted in the transformer network, the model learns the correlation information between different variables through the attention mechanism, and the time series prediction of the multivariate is realized.

[0047] Embodiment 2 provides a time series large model data processing system for realizing multivariate prediction, comprising:

[0048] The acquisition module is configured to acquire time series data and perform normalization processing to obtain multivariate data.

[0049] The data mixing module is configured to uniformly sample and mix the multivariate data into single variable data.

[0050] The division module is configured to divide the single variable data into a plurality of non-overlapping continuous patches.

[0051] The vectorization module is configured to convert each continuous patch into a fixed-size vector, and add position encoding, length encoding and resolution encoding information to obtain vector data.

[0052] The training and prediction module is configured to input the vector data into the transformer network to train and predict the model, the model learns the correlation information between different variables through the attention mechanism, and the time series prediction of the multivariate is realized.

[0053] Embodiment 3 provides a computer readable storage medium, the computer readable storage medium comprises a stored program, wherein when the program runs, the computer readable storage medium controls the device where the computer readable storage medium is located to execute the time series large model data processing method for realizing multivariate prediction in embodiment 1.

[0054] Embodiment 4 provides an electronic device, comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute the time series large model data processing method for realizing multivariate prediction in embodiment 1.

[0055] The electronic device can include a processor, a memory and a communication unit. These components communicate through one or more buses, and those skilled in the art can understand that the structure of the electronic device does not limit the embodiments of the present application, which can be bus-shaped structure, star-shaped structure, or combination of some components, or different component arrangement.

[0056] The communication unit is configured to establish a communication channel, so that the electronic device can communicate with other devices, receive user data sent by other devices, or send user data to other devices.

[0057] The processor is the control center of the electronic device, and is connected with various parts of the electronic device through various interfaces and lines. The processor executes or runs software programs and / or modules stored in the memory and calls data stored in the memory to perform various functions of the electronic device and / or process data. The processor can be composed of an integrated circuit (IC), for example, a single packaged IC or a plurality of packaged ICs with the same or different functions connected together. For example, the processor can only include a central processing unit (CPU). In the embodiments of the present application, the CPU can be a single operation core or can include a plurality of operation cores.

[0058] The memory is configured to store execution instructions of the processor. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0059] When the execution instructions in the memory are executed by the processor, the electronic device can perform part or all of the steps of the embodiment 1.

[0060] The above is only the preferred embodiment of the present application, and does not limit the form of the present application. Any skilled person in the art can modify or change the above disclosed technical content to equivalent embodiments applied to other fields, but any simple modification, equivalent change and modification made according to the technical essence of the present application to the above embodiments without departing from the technical solution of the present application shall fall within the protection scope of the present application.

Claims

1. A method for processing large time-series model data for multivariate prediction, characterized in that, include: Time series data is acquired and normalized to obtain multivariate data. The time series data includes weather temperature data, which is recorded hourly. Variables related to weather temperature include humidity data and air pressure data, which are also collected at a frequency of hourly recording. Multivariate data is uniformly sampled and mixed into univariate data; The univariate data is divided into multiple non-overlapping continuous segments using the patch function; Each consecutive segment patch is converted into a fixed-size vector, and position encoding, length encoding, and resolution encoding information are added to obtain vector data; Vector data is input into the transformer network for model training and prediction. The model captures and learns the relevant information between different variables through the attention mechanism, thereby achieving multivariate time series prediction. The normalization process includes scaling the time series data to a range with a mean of 0 and a variance of 1, making the time series data distribution approximate a normal distribution. Specifically, the normalization method involves calculating the mean and variance, and then using the formula... Where z: the standardized value; x: the original value; μ: Mean of the feature; σ: Standard deviation of the feature; The position encoding uses a combination of sine and cosine functions to provide the model with information about the position of each data point in the sequence; the position encoding formula is as follows: （1） （2） Where: pos is the position index, and i is the dimension index of the position encoding vector. It is the dimension of the model; The length encoding includes: randomly masking part or all of the input patches to enable the model to adapt to different context lengths; specifically, in each training batch, a starting point r is randomly selected, where the value of r is less than the number of patches, representing that the part from that point to the end of the patch is masked. This masking strategy ensures that the model can see all possible context lengths during training; a binary mask vector with the same length as the input time series is created, where 1 indicates that the corresponding time point data should be considered by the model, and 0 indicates that the corresponding time point data should be ignored.

2. The method for processing large time-series model data for multivariate prediction according to claim 1, characterized in that, The step of uniformly sampling and mixing into univariate data includes: sequentially taking one time point data point for each variable of the multivariate data, and mixing the multivariates into a single variable.

3. The method for processing large time-series model data for multivariate prediction according to claim 1, characterized in that, The patch division step includes: dividing the univariate data into multiple non-overlapping continuous segments patch, determining the number of time points in each patch, traversing the entire time series, and taking out a fixed number of consecutive time point data as a patch each time.

4. The method for processing large time-series model data for multivariate prediction according to claim 1, characterized in that, The resolution encoding includes: setting 0, 1, and 2 to represent high-frequency, mid-frequency, and low-frequency data, respectively; randomly initializing a resolution sequence of patch length; and outputting a resolution encoding vector of dimension dim through a multilayer perceptron.

5. A time-series large-scale model data processing system for multivariate prediction, characterized in that, include: The acquisition module is configured to acquire time series data and perform normalization processing to obtain multivariate data. The time series data includes weather temperature data, which is recorded hourly. Variables related to weather temperature include humidity data and air pressure data, which are also collected hourly. The data mixing module is configured to uniformly sample and mix multivariate data into univariate data. The partitioning module is configured to: divide univariate data into multiple non-overlapping continuous segments; The vectorization module is configured to convert each consecutive patch into a fixed-size vector and add position encoding, length encoding, and resolution encoding information to obtain vector data. The training and prediction module is configured to: input vector data into the transformer network for model training and prediction; the model captures and learns relevant information between different variables through an attention mechanism to achieve multivariate time series prediction. The normalization process includes scaling the time series data to a range with a mean of 0 and a variance of 1, making the time series data distribution approximate a normal distribution. Specifically, the normalization method involves calculating the mean and variance, and then using the formula... Where z: the standardized value; x: the original value; μ: Mean of the feature; σ: Standard deviation of the feature; The position encoding uses a combination of sine and cosine functions to provide the model with information about the position of each data point in the sequence; the position encoding formula is as follows: （1） （2） Where: pos is the position index, and i is the dimension index of the position encoding vector. It is the dimension of the model; The length encoding includes: randomly masking part or all of the input patches to enable the model to adapt to different context lengths; specifically, in each training batch, a starting point r is randomly selected, where the value of r is less than the number of patches, representing that the part from that point to the end of the patch is masked. This masking strategy ensures that the model can see all possible context lengths during training; a binary mask vector with the same length as the input time series is created, where 1 indicates that the corresponding time point data should be considered by the model, and 0 indicates that the corresponding time point data should be ignored.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the time-series large model data processing method for multivariate prediction as described in any one of claims 1-4.

7. An electronic device, characterized in that, It includes a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute the time series large model data processing method for multivariate prediction as described in any one of claims 1-4.

Images