Time sequence prediction model construction method based on large language model weight fine tuning

Abstract

The invention discloses a time sequence prediction model construction method based on large language model weight fine tuning, and the method comprises the steps: carrying out the pre-training model weight of an open source large language model GPT2, and carrying out the network parameter freezing and two-stage pre-training model parameter adjustment. Enabling the large language model to adapt to a source load prediction task in a power system scene in a weight adjustment process; a model input layer driven by a natural language instruction is constructed, efficient combination of time sequence data and natural language input is achieved, and the effect of a time sequence prediction model in a time sequence prediction task is further improved. The time sequence prediction model constructed on the basis of the method has remarkable advantages in generalization performance, and the prediction precision under special prediction scenes such as small samples and zero samples is remarkably improved compared with a traditional prediction method.

Description

Technical Field

[0001] The present invention relates to a method for constructing a time series prediction model based on fine-tuning of large language model weights, and belongs to the field of power system time series prediction. Background Art

[0002] The output of renewable energy in power systems is significantly affected by weather factors and exhibits strong randomness and volatility. Source-load forecasting can vary significantly in time and space depending on the scenario. Furthermore, computational models used for source-load forecasting face challenges such as massive input data volumes, high-dimensional features, and strong coupling. Traditional forecasting methods require the development of dedicated models for specific scenarios, which is time-consuming and labor-intensive, with poor generalization capabilities, high operational and maintenance costs, and complex technology. Therefore, there is an urgent need for a forecasting model with strong generalization capabilities, high prediction accuracy, and applicability to a variety of forecasting scenarios.

[0003] The rapid development of large language model technology has driven technological advancements in fields such as computer vision and natural language processing. Research has demonstrated that the weights of large language model systems, with their massive parameter size, demonstrate powerful learning, reasoning, and pattern recognition capabilities across multiple domains, outperforming traditional forecasting schemes in extreme scenarios such as zero-shot and small-shot predictions. Therefore, time series pre-training models based on large language model weights hold great potential for building more efficient general-purpose models for power system time series forecasting.

[0004] The patent application with publication number CN117634740A discloses a method for power load forecasting based on a large language model. The method is oriented to natural language task training and generation. During the training and reasoning process, its input and output content are in the form of text and characters. If it is necessary to apply the large language model to the time series prediction task, the key technical difficulty lies in how to make the weights of the backbone network of the large language model adapt to the specific task of time series prediction. In time series prediction, the input and output are continuous data points, not text and characters. The coding table established by the original model (i.e., the mapping relationship from text characters to vector space) is no longer applicable. The weights within the large language model need to be adjusted in a targeted manner to adapt to the source load prediction task within the power system.

[0005] Although patent application CN117634740A also attempts to address the problem of adapting the weights of the large language model backbone network to the specific task of time series prediction, its approach involves converting time series into a structured text input model and then introducing a loss function to update the weight parameters of the entire model during gradient descent to adapt it to the task of time series prediction. This approach suffers from three shortcomings: 1. Time series are discrete data points, significantly different from text, making lossless conversion impossible. This inevitably results in information loss and reduced prediction performance. 2. The model structure lacks adaptive design and improvement for time series prediction, retaining the text input-text output structure. 3. During gradient updates, model parameters are updated through supervised fine-tuning, but instead of deeply exploring the specific differences between time series and text input scenarios to optimize the weights of the large language model, all model parameters are updated. Parameter updates to the FFN and multi-head self-attention mechanisms during time series prediction fine-tuning may disrupt the knowledge graph learned by the large language model during pre-training, leading to performance degradation. Summary of the Invention

[0006] The purpose of the present invention is to address the deficiencies of the existing technology and provide a method for constructing a time series prediction pre-training model based on large language model weights.

[0007] The object of the present invention is achieved through the following technical solutions:

[0008] A method for constructing a time series prediction model based on fine-tuning the weights of a large language model includes the following steps:

[0009] Step 1: Data Collection. The weights of the large language model need to be adjusted through training to gradually adapt them to time series prediction tasks. Therefore, rich time series data is required to support the weight adjustment of the large language model. This step requires collecting publicly available datasets from the internet and compiling basic information about these open source datasets, including the number of time series data points, the number of variables, the time interval, the time span, and the domain to which the data belongs.

[0010] Step 2: Data cleaning and dataset construction. Based on step 1, the collected data is processed at a deeper level. This step requires preprocessing the time series of each dataset, including marking and removing missing values and outliers to improve data quality. When constructing the dataset, the principle of uniform sampling is followed to ensure that each batch of data input to the model comes from different fields (electricity, transportation, finance, weather, etc.) and the data ratio is close to balanced. The purpose is to use rich time series data to train large language models.

[0011] Step 3: One-stage training, adjusting the parameters of the large language base model based on the dataset constructed in step 2. For example, GPT2 is an open-source large language base model based on the Transformer architecture, which is trained and generated using rich natural language data. To adapt it to tasks in time series prediction scenarios, the weights of the GPT2 base model need to be adjusted using the gradient descent algorithm during training. During the training process, some model parameters are frozen, and some network layers of the large language model (specifically, the input embedding layer, normalization layer, and position encoder) are trained and adjusted to enhance the model's adaptability to time series data.

[0012] Step 4: Two-stage training, construction of the model input layer driven by natural language instructions and its parameter adjustment. In order to give full play to the potential of the large language model in time series prediction tasks. This step constructs a model input layer driven by natural language instructions based on the parameter weights of the large language model adjusted in step 3, and efficiently combines time series data with natural language input to further improve the model reasoning ability. The natural language instruction-driven model input layer is trained using power system source and load data. Based on the above steps, the large language model can adapt to a variety of tasks in time series prediction scenarios. In order to further improve the performance of the model in power system source and load prediction scenarios, this step can use wind turbines, photovoltaic power stations and load data from buses, feeders, and substations to optimize and adjust the parameters of the model input layer driven by natural language instructions until the model parameters converge and reach the optimal prediction state, and the time series prediction model can be obtained.

[0013] In the above technical solution, further, in step 3, the parameters of the input embedding layer, the normalization layer, and the position encoder are updated during the training process. The specific method is:

[0014] 1) Let the total parameter set of the large language model be θ, of which the parameter set that needs to be updated is θ u , the set that is frozen and does not participate in parameter update is θ f , and the parameter sets satisfy θ=θ u ∪θ f and The constraint relationship of θ u Including position encoding parameters PE, embedding layer parameters W E And each normalization layer parameter LayerNorm; θ f Including multi-head attention mechanism model parameters and feedforward neural network model parameters;

[0015] 2) During the training process, the time series is used as the model input, and the prediction result Y output by the model is used pred Calculate the loss of the current prediction with the true value Y N BRepresents the number of all samples in a batch, and N represents the number of input variables of the time series;

[0016] 3) Based on the loss calculation results, the back propagation algorithm is applied to calculate the loss function with respect to the parameter set to be updated θ u Gradient

[0017] Set the learning rate α, the decay rate of the first-order moment estimate β1, the decay rate of the second-order moment estimate β2, and introduce a constant ε to avoid the denominator being zero;

[0018] 4) Based on the loss function L about the model parameter θ u Gradient Update the first-order moment estimate m t and the second moment estimate v t :

[0019]

[0020] 5) Due to m t With v t The initial value of m is 0, so t With v t Perform bias correction to obtain the corrected first-order moment estimate And the second-order moment estimation results

[0021]

[0022] and are the decay rates of the next-order moment estimate and the second-order moment estimate at the t-th iteration, respectively;

[0023] 6) Use the corrected first-order moment and second-order moment to update the model parameters θ u :

[0024]

[0025] Furthermore, in step four, the natural language instruction driven model input layer receives natural language input and time series data input through the text input embedding layer and the time series input embedding layer respectively, and the text input embedding layer and the time series input embedding layer output natural language vectors and time series vectors respectively, and the natural language vectors and time series vectors are spliced and the splicing result is used as the output of the natural language instruction driven model input layer.

[0026] Furthermore, the natural language input includes a basic description of the data set and a description of the prediction task;

[0027] The basic description of the dataset specifically includes the field to which the dataset belongs, the time span of the dataset, the resolution of the data, the number of variables, the target variable, and the relationship between other variables in the dataset and the target variable;

[0028] The prediction task description specifically includes the input time series length, the predicted time series length, the mean information of the input time series data, the standard deviation information of the input time series data and the trend information of the input data.

[0029] The beneficial effects of the present invention are:

[0030] The present invention uses the open source GPT2 large language basic model parameters, and can generate a time series prediction model with strong generalization ability and adaptability to multiple downstream prediction scenarios through iterative training with limited resources; it provides a complete solution from data set construction to model parameter training iteration to ensure the prediction performance of the generated time series prediction model; it constructs a model input layer driven by natural language instructions, realizes the efficient combination of time series data and natural language input, and further improves the effect of the time series prediction model in time series prediction tasks.

[0031] In addressing the problem of "how to adapt the weights of the backbone network of a large language model to the specific task of time series prediction," this paper breaks down the adaptation process into two stages. The first stage focuses on targeted weight adjustments within the large language model, retaining the key knowledge graph within the large language model and freezing the weight parameters of key structures such as the feedforward neural network and the multi-head attention mechanism. Only the position encoding and normalization layers are involved in parameter updates, adapting the parameter dimensions of the time series prediction input during large-scale open-source time series data training. The second stage focuses on reconstructing the input and output layers of the large language model. Unlike existing methods that convert time series into text for processing, the reconstructed input layer proposed in this paper directly takes time series as input and designs a structured instruction generation scheme that can automatically generate corresponding natural language instructions based on the time series. Subsequently, through two-stage training, the time series input and text input are aligned in the vector space, further improving the prediction effect of the time series prediction model. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a flowchart of a method for building a time series prediction model based on fine-tuning the weights of a large language model.

[0033] Figure 2 It is the parameter adjustment status of the large language model in one stage of training.

[0034] Figure 3 It is the model input layer structure design and model parameter adjustment in the second stage training. DETAILED DESCRIPTION

[0035] The present invention will be described in detail below with reference to the accompanying drawings.

[0036] like Figure 1 The present invention provides a method for constructing a time series prediction model based on fine-tuning the weights of a large language model, comprising the following steps:

[0037] Step 1: Collect open source time series data.

[0038] The initial weights of large language models are designed for various language tasks. This requires collecting time series data from diverse fields such as transportation, finance, electricity, and weather. Fine-tuning the weights of these large language models in two stages allows the model to fully leverage its cross-domain learning and reasoning capabilities, improving its generalization performance. When collecting time series data, basic information about the dataset should be collected, including the number of time series data points, the number of variables, the time series interval, the time span, and the domain to which the data belongs.

[0039] Step 2: Data cleaning and dataset construction.

[0040] Because open-source time series data contains a large amount of low-quality data, including missing, distorted, and erroneous values, this step requires processing the time series of each dataset based on the datasets collected in Step 1. This includes labeling outliers and preprocessing the data to improve the overall quality of the dataset. During dataset construction, the first-stage model parameter update should adhere to the principle of uniform sampling. Each batch of data input to the model should include time series from different fields and be evenly distributed in terms of proportion.

[0041] Step 3: In the first stage of training, some weight parameters of the large language model are updated and adjusted during the training process.

[0042] The GPT2 model is based on the Transformer architecture, and its open source model structure is as follows Figure 2 As shown in , the model consists of multiple encoders and decoders connected in series. The basic network units of the encoder and decoder include multi-head attention, layer normalization, and feedforward neural network layers. Figure 2 As shown, the specific calculation formula corresponding to the basic network unit is as follows:

[0043] Assume that the input time series data is X∈R N×T , where N represents the number of input variables of the time series and T represents the time step. First, the input time series is reversibly normalized so that the mean of the data is 0 and the standard deviation is 1. The processed data is X norm . norm Divide into several pieces of length L p Each fragment is represented by Each fragment passes through the input embedding layer to form an embedding vector Ep ∈R d , d is the parameter dimension of the GPT2 backbone model, W E is the linear layer network parameter of the input embedding layer. The embedded vector is superimposed with the encoding information PE output by the position encoder to obtain the encoded vector E pos :

[0044] E pos =E p +PE

[0045] Vector E containing position information pos Input into the GPT2 model, the input E is calculated by the multi-head attention mechanism pos In this process, the query matrix Q of the i-th attention head is first calculated. i , key-value matrix K i and weight matrix V i :

[0046]

[0047] in, and are the mapping weight matrices corresponding to the query matrix, key matrix, and weight matrix of the i-th attention head respectively.

[0048] Based on Q i , K i , V i Three matrices are used for attention calculation:

[0049]

[0050] head i Represents the operation result of the i-th attention head.

[0051] In the multi-head attention calculation, the final attention output result O att The calculation results of h attention heads are concatenated and calculated:

[0052] O att =concat(head1,head2,...,head h )·W o

[0053] W o It is the linear change matrix after concatenating the outputs of multiple attention heads.

[0054] Output of multi-head attention O att After layer normalization and residual connection to prevent the gradient from disappearing during training, the operation process is as follows:

[0055] Onorm1 =LayerNorm( att +E pos )

[0056] Where LayerNorm represents the normalization layer in the GPT2 model. The normalized result is O norm1 Input feedforward neural network, the operation process is:

[0057] O ffn =ReLU(O norm1 W ffn1 +b ffn1 )·W ffn2 +b ffn2

[0058] As can be seen from the formula, the feedforward neural network consists of two layers of fully connected networks, where W ffn1 , W ffn2 are the weight matrices of the inner and outer fully connected networks, b ffn1 , b ffn2 are the bias terms of the inner and outer fully connected networks respectively. The output of the feedforward neural network O ffn After the normalization layer again:

[0059] O norm2 =LayerNorm( ffn +O norm1 )

[0060] The above is a data operation process in a basic network unit. The GPT2 model uses multiple identical basic network units to serially process input data to achieve deep feature extraction at different scales. After processing by N layers of basic network units, the final output result can be expressed as O final ∈R P×d Finally, the time series prediction result Y is obtained through the output linear layer pred ∈R 1×H :

[0061] Y pred =O final W out

[0062] Where W out are the output linear layer parameters of the model.

[0063] The self-attention layer and feedforward neural network layer in the large language model contain most of the knowledge from the pre-trained language model and are the basis for the model's strong learning, reasoning, and pattern recognition capabilities. Therefore, during this stage of model training, the weights of the decoder layer, the feedforward neural network in the encoder, and the multi-head attention layer need to be frozen to prevent the weight parameters from changing during the model training process and thus affecting the model's reasoning ability. In this stage of training, the time series data from different fields constructed in step 2 is input to update the weight parameters of the position encoder, layer normalization network, and input embedding layer of the GPT2 large language model. The specific update steps are as follows:

[0064] 1) Let the total parameter set of the GPT2 model be θ, of which the parameter set that needs to be updated is θ u , the set that is frozen and does not participate in parameter update is θ f , and the parameter sets satisfy θ=θ u ∪θ f and The constraint relationship of . Among them, θ u Including position encoding parameters PE, embedding layer parameters W E And each normalization layer parameter LayerNorm; θ f Including multi-head attention mechanism model parameters and feedforward neural network model parameters.

[0065] 2) During the training process, the time series is used as the model input, and the prediction result Y output by the model is used pred Calculate the loss of the current prediction with the true value Y N B Indicates the total number of samples in a batch.

[0066] 3) Based on the loss calculation results, the back propagation algorithm is applied to calculate the loss function with respect to the parameter set to be updated θ u Gradient

[0067] Since θ f The parameters in the set are frozen and no gradients need to be calculated Set the learning rate α, the decay rate of the first-order moment estimate β1, the decay rate of the second-order moment estimate β2, and introduce a constant ε to avoid the denominator being zero.

[0068] 4) Based on the loss function L about the model parameter θ u Gradient Update the first-order moment estimate m t and the second moment estimate v t :

[0069]

[0070] 5) Due to m t With v t The initial value of m is 0, so t With v t Perform bias correction to obtain the corrected first-order moment estimate And the second-order moment estimation results

[0071]

[0072] and are the decay rates of the next-order moment estimate and the second-order moment estimate at the t-th iteration, respectively;

[0073] 6) Use the corrected first-order moment and second-order moment to update the model parameters θ u :

[0074]

[0075] Step 4: Two-stage training, construction of the model input layer driven by natural language instructions and its parameter adjustment. Step 3 adaptively adjusts the weights of the GPT2 large language model through parameter freezing and fine-tuning methods to adapt it to the scale characteristics of time series data. In order to further give full play to the logical reasoning ability of the large language model and introduce additional information of prediction scenarios and task descriptions, this step abandons the original input layer of the GPT2 model and redesigns the model input layer driven by natural language instructions, such as Figure 3 As shown. The vector input to the large language model through the model input layer driven by natural language instructions is composed of two parts. One part is the time series data. After the time series data is fragmented and normalized, the time series input embedding layer (the time series input embedding layer needs to be rebuilt here instead of using the input embedding layer in step 3 GPT2) extracts the time series feature information to form the time series vector E p ∈R d , W E The first part is the network parameters of the time series input embedding layer; the other part is the natural language input. Natural language is a supplementary description of the time series data characteristics and task scenarios, used to provide additional information to improve the prediction effect. After the input passes through the text input embedding layer, it forms a natural language vector. The corresponding formula is expressed as follows:

[0076] Assume that the input text sequence is S and the character table is V. First, the input text S needs to be tokenized according to the character table V to form a tokenization result T = [t1, t2, ..., t n ], where t i represents the i-th token, and n is the number of tokens. Subsequently, each token is mapped to a high-dimensional space through the text input embedding layer to form a natural language vector ES ∈R d , W S Linear mapping network parameters for the text embedding layer.

[0077] The natural language input to the model should correspond to the input time series data and be generated in a fixed format, including "basic dataset description + prediction task description." The basic dataset description specifically includes the dataset's domain, time span, data resolution, number of variables, target variable, and the relationship between other variables in the dataset and the target variable. The prediction task description specifically includes the input time series length, predicted time series length, input time series data mean, input time series data standard deviation, and input data trend information.

[0078] To illustrate the method of generating natural language input, take the open source dataset ETTh1 as an example. For a batch of time series input data from this dataset, the natural language input generated according to the above generation format is as follows:

[0079] -Basic description of the dataset: "The ETTh1 dataset is an open-source dataset in the power sector. It contains two years of data points with an hourly interval. It contains seven groups of variables. The target variable is transformer oil temperature, and the remaining six groups are covariates related to the target variable."

[0080] The prediction task is described as: "Input time series data of 192 points in the past and predict time series data of 32 points in the future."

[0081] -Input time series data mean information: "The mean of the input data is 12.316"

[0082] -Enter the standard deviation information of the time series data: "The standard deviation is 3.263"

[0083] - Trend information of input data: "Increasing trend"

[0084] The result of sequentially concatenating the above information is the natural language input corresponding to the dataset and the current input time series data. The construction of the text input embedding layer introduces additional task instructions in natural language form, provides additional explanations for the prediction task and application scenario, and converts the time series vector E p With the natural language vector E S Splicing along the model backbone dimension d as the input E of the GPT2 model in , further enhancing the model’s reasoning ability for time series concepts.

[0085] E in =concat(E p ,E s )

[0086] During the second stage of training, power system source and load data are used to update the model input layer parameters driven by natural language instructions.

[0087] After the first phase of training, the weights of the large language model are fine-tuned appropriately. After the prediction loss function converges, it can be considered that the current large language model has adapted to various downstream tasks of time series prediction. The second phase of training is based on the first phase, with the goal of optimizing and adjusting the input layer parameters of the model driven by the constructed natural language instructions. Figure 3 As shown in Figure 1, during the training process, the weight parameters of the large language model need to be frozen, and only the parameters of the text input embedding layer and the time series input embedding layer in the model input layer driven by natural language instructions are optimized and adjusted so that it can normalize the time series input and effectively extract the semantic feature information of the time series. The parameter update set of the model is θ s , including the text input embedding layer W E With the temporal input embedding layer W s The training loss function and gradient descent process are consistent with the training of the GPT2 model in step 3.

[0088] After completing the above steps in sequence, the time series prediction model can be obtained.

Claims

1. A method for constructing a time series prediction model based on fine-tuning the weights of a large language model, characterized in that: Here are the steps: Step 1: Obtain an open source dataset and collect statistics on its basic information, including the number of time series data points, the number of variables, the time series interval, the time span, and the field to which the data belongs; Step 2: Mark and remove missing values and outliers in the time series data of each open source dataset, and use the processed time series data to construct a dataset; Step 3: Use the dataset constructed in step 2 to perform a one-stage fine-tuning of the large language model. During the training process, the parameters of the input embedding layer, normalization layer, and position encoder are updated, and other structural parameters are frozen. Step 4: Build a natural language instruction-driven model input layer and use it as the input layer of the large language model after the first stage of fine-tuning in step 3. The natural language instruction-driven model input layer includes a text input embedding layer and a time series input embedding layer; The natural language instruction driven model input layer is trained in two stages using power system source and load data, and the parameters of the natural language instruction driven model input layer are updated until the model parameters converge and reach the optimal prediction state, thereby obtaining a time series prediction model.

2. The method for constructing a time series prediction model based on large language model weight fine-tuning according to claim 1, characterized in that: In step 3, the parameters of the input embedding layer, normalization layer, and position encoder are updated during training. The specific method is as follows: 1) Let the total parameter set of the large language model be θ, of which the parameter set that needs to be updated is θ u , the set that is frozen and does not participate in parameter update is θ f , and the parameter sets satisfy θ=θ u ∪θ f and The constraint relationship of θ u Including position encoding parameters PE, embedding layer parameters W E And each normalization layer parameter LayerNorm; θ f Including multi-head attention mechanism model parameters and feedforward neural network model parameters; 2) During the training process, the time series is used as the model input, and the prediction result Y output by the model is used pred Calculate the loss of the current prediction with the true value Y N B Represents the number of all samples in a batch, and N represents the number of input variables of the time series; 3) Based on the loss calculation results, the back propagation algorithm is applied to calculate the loss function with respect to the parameter set to be updated θ u Gradient Set the learning rate α, the decay rate of the first-order moment estimate β1, the decay rate of the second-order moment estimate β2, and introduce a constant ε to avoid the denominator being zero; 4) Based on the loss function L about the model parameter θ u Gradient Update the first-order moment estimate m t and the second moment estimate v t : 5) Due to m t With v t The initial value of m is 0, so t With v t Perform bias correction to obtain the corrected first-order moment estimate And the second-order moment estimation results and are the decay rates of the next-order moment estimate and the second-order moment estimate at the t-th iteration, respectively; 6) Use the corrected first-order moment and second-order moment to update the model parameters θ u :

3. The method for constructing a time series prediction model based on large language model weight fine-tuning according to claim 1, characterized in that: In step four, the model input layer driven by the natural language instruction receives the natural language input and the time series data input through the text input embedding layer and the time series input embedding layer respectively. The text input embedding layer and the time series input embedding layer output the natural language vector and the time series vector respectively. The natural language vector and the time series vector are spliced and the splicing result is used as the output of the model input layer driven by the natural language instruction.

4. The method for constructing a time series prediction model based on large language model weight fine-tuning according to claim 3, characterized in that: The natural language input includes a basic description of the data set and a description of the prediction task; The basic description of the dataset specifically includes the field to which the dataset belongs, the time span of the dataset, the resolution of the data, the number of variables, the target variable, and the relationship between other variables in the dataset and the target variable; The prediction task description specifically includes the input time series length, the predicted time series length, the mean information of the input time series data, the standard deviation information of the input time series data and the trend information of the input data.

Images