Load forecasting method and system based on variational autoencoder and TIME-LLM model

Abstract

This paper proposes a load forecasting method and system based on a variational autoencoder and a TIME-LLM model. The method includes the following steps: historical data collection and preprocessing; autoencoder training to obtain a trained encoder component; reprogramming training to obtain a reprogramming result vector for input into the TIME-LLM model; customizing the prompt word for the load forecasting problem as a prompt vector for input into the TIME-LLM model; and integrating the student model predictions with the TIME-LLM model through knowledge distillation to form the final load forecast result. The method details the variational autoencoder structure, optimizes the feature extraction process, enhances the model's understanding of the load forecasting task, and, through knowledge distillation and student model integration, addresses the resource constraints of actual deployments.

Description

Technical Field

[0001] The present invention belongs to the field of new energy power technology, and in particular relates to a load forecasting method and system based on a variational autoencoder and a TIME-LLM model. Background Art

[0002] With the continuous development of power systems, power load forecasting is crucial for their stable operation and optimized dispatch. Load forecasting involves analyzing and studying historical data based on known power system, economic, social, and meteorological conditions, exploring the inherent connections and patterns of development and change, and making preliminary load estimates and projections. Load forecasting is fundamental to power system planning, scheduling, electricity consumption, and dispatching, and its importance has long been recognized.

[0003] Existing methods typically rely on traditional time series models or deep learning models, which often require large amounts of data for training and have limited ability to process complex patterns of future load changes. The development of large language models (LLMs), and their success in natural language processing, has brought new ideas for processing time series data. However, LLMs cannot directly process time series data. Researchers explored the application of model reprogramming on large language models (LLMs) and proposed the TIME-LLM framework, which achieves high-precision time series forecasting without modifying the language model. Building on this, we attempt to apply TIME-LLM to a more specific load forecasting domain, leveraging domain data for appropriate optimization. Summary of the Invention

[0004] The present invention proposes a load forecasting method and system based on a variational autoencoder and a TIME-LLM model. The structure of the variational autoencoder is designed in detail to optimize the feature extraction process; the model's understanding of the load forecasting task is enhanced; and the resource constraints of actual deployment are met through knowledge distillation and student model integration.

[0005] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0006] A load forecasting method based on a variational autoencoder and a TIME-LLM model includes:

[0007] S1. Data collection and preprocessing: Collect historical data including meteorological data, load data, and time data, and organize them into patches after RevIN processing;

[0008] S2. Autoencoder training: training the variational autoencoder through the Patch block to obtain the encoder part of the trained variational autoencoder;

[0009] S3. Reprogramming training: The latent variable Patch output by the encoder is aligned with the text prototype describing the trend characteristics of the time series through a cross-attention network to obtain a reprogramming result vector input to the TIME-LLM model;

[0010] S4. Customize prompts for load forecasting problems: The text description and background knowledge of specific load forecasting problems are pre-trained to construct prompt vectors that are input into the TIME-LLM model.

[0011] S5. Forecasting based on the TIME-LLM model: After concatenating the reprogramming result vector and the prompt vector, the vector is input into the TIME-LLM model and trained to obtain a predicted value that conforms to the result format of the load forecasting problem.

[0012] S6, knowledge distillation and student model integration: The model trained in steps S3 to S5 is used as the teacher model, and a small model that meets the on-site resource conditions is initialized as the student model. The student model is trained through knowledge distillation. The training is stopped when the student model prediction value and the teacher model prediction value are less than a certain critical value, and the student model prediction value is integrated as the final load forecast result.

[0013] Furthermore, the historical data in step S1 includes measured meteorological data, predicted meteorological data, time data, holiday data, industry production predicted load data, and industry production actual load data at different meteorological points at n time nodes in the historical period of the area to be predicted.

[0014] Furthermore, the training method in step S2 includes: the encoder part of the variational autoencoder inputs the Patch block, outputs the latent variable Patch, uses the decoder part of the variational autoencoder to decode the latent variable Patch, and outputs the reconstructed Patch; evaluates the mean square error of the input and output and the KL divergence of the difference between the latent variable distribution and the standard normal distribution, and takes the hyperparameter that minimizes the evaluation error.

[0015] Furthermore, the training method in step S4 includes organizing the specific problem of load forecasting, input and output content and form, industry background knowledge about load forecasting, and statistical data of each dimension of the above input data into a prompt word through text description, and inputting it into a small pre-trained LLM module in the TIME-LLM model to obtain the compressed word embeddings Token Embeddings as the Prompt vector passed into the main LLM model of TIME-LLM.

[0016] Furthermore, in step S6, two or more student models with different architectures are designed, and the prediction values of the weighted average integrated student models are used as the prediction results.

[0017] On the other hand, the present invention also proposes a load forecasting system based on a variational autoencoder and a TIME-LLM model, comprising:

[0018] Data collection and pre-processing unit: collects historical data including meteorological data, load data, and time data, and organizes them into patch blocks after being processed by RevIN;

[0019] Autoencoder training unit: training the variational autoencoder through the patch block to obtain the encoder part of the trained variational autoencoder;

[0020] Reprogramming training unit: aligns the latent variable Patch output by the encoder with the text prototype describing the trend characteristics of the time series through a cross-attention network to obtain a reprogramming result vector input to the TIME-LLM model;

[0021] Load forecasting problem prompt customization unit: This unit constructs the text description and background knowledge of the specific load forecasting problem into the prompt vector input to the TIME-LLM model through pre-training;

[0022] The prediction unit based on the TIME-LLM model: After concatenating the reprogramming result vector and the prompt vector, it is input into the TIME-LLM model and trained to obtain the predicted value that conforms to the result format of the load forecasting problem;

[0023] Knowledge distillation and student model integration unit: The models trained in the reprogramming training unit, the load forecasting prompt customization unit, and the TIME-LLM model-based prediction unit are used as teacher models. A small model that meets the on-site resource conditions is initialized as the student model. The student model is trained through knowledge distillation. Training is stopped when the student model prediction value and the teacher model prediction value are less than a certain critical value, and the student model prediction value is integrated as the final load forecast result.

[0024] Furthermore, the historical data in the data collection and preprocessing unit includes measured meteorological data, predicted meteorological data, time data, holiday data, industry production predicted load data, and industry production actual load data at different meteorological points at n time nodes in the historical period of the predicted area.

[0025] Furthermore, the autoencoder training unit includes: the encoder part of the variational autoencoder inputs the Patch block, outputs the latent variable Patch, uses the decoder part of the variational autoencoder to decode the latent variable Patch, and outputs the reconstructed Patch; evaluates the mean square error of the input and output and the KL divergence of the difference between the latent variable distribution and the standard normal distribution, and takes the hyperparameters that minimize the evaluation error.

[0026] Furthermore, the customized prompt word unit for load forecasting problems includes organizing the specific load forecasting problem, input and output content and form, industry background knowledge about load forecasting, and statistical data of each dimension of the above input data into a prompt word through text description, and inputting it into a small pre-trained LLM module in the TIME-LLM model to obtain the compressed token embeddings as the prompt vector passed into the main LLM model of TIME-LLM.

[0027] Furthermore, in the knowledge distillation and student model integration unit, more than two student models with different architectures are designed, and the prediction values of the weighted average integrated student models are used as the prediction results.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] This paper optimizes the feature extraction process by carefully designing the variational autoencoder structure. Customizing Prompt and a small LLM enhances the model's understanding of the load forecasting task. Through knowledge distillation and student model integration, it addresses the resource constraints of actual deployments. Experimental results demonstrate that this paper significantly improves performance and has practical application value in load forecasting tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a schematic diagram of a method flow in an embodiment of the present invention;

[0031] Figure 2 4 is a load forecast comparison result diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0033] like Figure 1 As shown, the method proposed in this embodiment specifically includes:

[0034] 1. Data Collection and Preprocessing: Historical data includes measured meteorological data, forecasted meteorological data, time data, holiday data, industry production load forecast data, and actual industry production load data at n time points during the historical period of the forecast area. Data cleaning is performed, including deletion and filling of missing values, screening and removal of outliers, and data alignment and splicing. Feature engineering based on prior knowledge of load is used to construct features, building derived features from the perspective of physical relationships and window statistics. Ultimately, this data is organized into a common input format that meets the requirements of deep time series models, namely, patches divided after RevIN reversible normalization.

[0035] The operation of dividing the Patch blocks after the reversible normalization processing by RevIN mainly includes normalizing the historical data mentioned above through RevIN. The purpose of this step is to adjust the distribution of the historical data to make it more suitable for model training; the normalized data is then divided into multiple Patch blocks. Each Patch block contains a part of the historical data. The RevIN is a reversible instance normalization method that can not only normalize the data, but also denormalize the normalized data to restore it to the original distribution. It is suitable for time series data. The historical data in this embodiment also belong to data with time series attributes. The normalization parameters of the data can be dynamically adjusted through RevIN to cope with changes in the distribution of each data.

[0036] 2. Autoencoder training:

[0037] A variational autoencoder (VAE) is a probabilistic autoencoder consisting of an encoder and a decoder. It uses the encoder to learn the intrinsic characteristics of data, compressing the data to generate latent variables, and the decoder to reconstruct the latent variables into fitted data similar to the input data.

[0038] Now we design a VAE whose encoder part receives the input Patch generated by S1 and outputs the latent variable Patch, which is a low-dimensional feature vector obtained by compressing and extracting the feature information in the historical data.

[0039] The structure of the VAE encoder includes:

[0040] Input layer: The input dimension is the number of features d of Patch.

[0041] Hidden layer: 3 fully connected layers, the number of neurons is 256, 128, and 64 respectively, and the activation function is ReLU.

[0042] Output layer: Generates the mean μ and variance σ of the latent variable, and the latent variable dimension is set to z_dim (such as 32).

[0043] Next, we optimize the hyperparameters for VAE training. Specifically, we use the decoder to decode the latent variable Patch to obtain the output of the VAE, which is the reconstructed Patch of the original input Patch. We evaluate the mean square error between the input and output and the KL divergence of the difference between the latent variable distribution and the standard normal distribution, and define the loss function:

[0044] L = reconstruction error (MSE) + KL divergence (the difference between the latent variable and the standard normal distribution);

[0045] Take the hyperparameters that minimize the evaluation error to train VAE.

[0046] The purpose of training VAE is to use the encoder part of the trained VAE as the embedding layer in the re-encoding module of TIME-LLM, replacing the original linear Patch Embedder layer.

[0047] Performance comparison test data:

[0048] In the load forecasting task, after using VAE to replace the linear Patch Embedder, the mean absolute error (MAE) of the model was reduced by about 2%, demonstrating the advantage of VAE in feature extraction.

[0049] 3. Reprogramming module training: The reprogramming module includes an autoencoder feature compression module, a text prototype, and an attention network module. The autoencoder module is the encoder part of the variational autoencoder obtained in step 2 above; the text prototype is a linear combination of words that describe the trend characteristics of the time series (including rising, falling, slow, fast, etc.), such as Figure 1 As shown, the word vector represents the large language model's understanding of the time series curve. For example, it can be learned from a pre-trained word vector model (such as GloVe). It is then linearly combined through a linear layer to produce the text prototype. The attention network module is a cross-attention network. This cross-attention network is constructed using a classic multi-head attention network, with 8 heads and 64 dimensions per head. The latent variable patch features passed into the autoencoder module serve as the query, and the text prototype embeddings serve as the key and value. Leveraging multi-head parallel computing, this network can capture the various dependencies between time series data and background knowledge. The cross-attention network aligns the latent variable patch and the text prototype, and finally linearly combines them to produce the reprogramming result vector R1. The results demonstrate that the recoding achieves the transformation from the time series patch to language text understandable by the LLM model.

[0050] 4. Customizing Prompts for Load Forecasting Problems: A prompt is constructed based on the text description and background knowledge of the specific load forecasting problem. Specifically, the specific load forecasting problem, input and output content and format, industry background knowledge about load forecasting, and statistical data on various dimensions of the input data are organized into a textual description. This is then fed into a small pre-trained LLM module in TIME-LLM (such as DistilGPT-2, which has approximately 82MB parameters). The resulting compressed token embeddings serve as the prompt vector R2 that is fed into the main LLM model of TIME-LLM to fully activate the LLM's capabilities for the specified time series task.

[0051] In this embodiment, the Prompt paradigm is assumed to include three key components:

[0052] [Domain]: Describes the industry background of load forecasting.

[0053] [Instruction]: Clarify the prediction tasks and requirements of the model.

[0054] [Statistics]: Provides statistical information of input data, such as mean, variance, trend, etc.

[0055] For load forecasting problems, please refer to the following example for specific prompts:

[0056] Power Load Forecasting (PLF) aims to predict future electricity usage to optimize grid management. Each data point consists of historical load consumption, meteorological data, and seasonal characteristics... The following is information about the input time series:

[0057] [Start Data]

[0058] [Industry Background]: It has been observed that electricity consumption on weekdays is generally higher than on weekends. Due to residential electricity consumption, the peak occurs in the evening.

[0059] [Description]: Based on the past load consumption <t>Step and other contextual features provided to predict the next step <h>step.

[0060] [Statistics]: The minimum value of the input time series is "min val", the maximum value is "max val", and the average value is "mean val". The data shows an upward or downward trend.<lag val> The most significant lagged correlation was observed at .

[0061] [End Data]

[0062] The above explanation is only for the purpose of explaining the Chinese meaning of the Prompt reference example and does not limit it otherwise.

[0063] 5. TIME-LMM-based prediction: The prompt word vector R2 generated in step 4 is used as a prefix and concatenated with the time series patch reprogramming result vector R1 generated in step 3. This is then input into the LLM model body of TIME-LLM. Through the forward propagation of the model, the output patch embeddings are obtained. Finally, the output patches are flattened and concatenated, and then converted into predicted values that conform to the required load forecasting problem result format through a linear layer.

[0064] 6. Knowledge Distillation and Student Model Integration: The TIME-LLM model trained in steps 3-5 serves as the teacher model, and a small model that meets on-site resource requirements is initialized as the student model. The student model is trained through knowledge distillation. Training is terminated when the student model's predicted value and the teacher model's predicted value fall below a certain threshold. The student model results are then integrated to form the final load forecast. This approach takes into account the differences in how different architectures capture feature information. To improve prediction performance and model robustness, two or more small models with different architectures (such as Transformer and LSTM) are designed, and the prediction results are integrated using weighted averaging.

[0065] Specifically, the TIME-LLM teacher model is first used to predict the training data, obtaining the predicted value for each time series sample as a soft label (the actual value is the hard label). These soft labels not only reflect the model's prediction results but also include TIME-LLM's understanding of the complex time series patterns in the data. This information will guide the student model.

[0066] Next, initialize a Transformer and an LSTM deep time series model as student models. The former is clearly more compatible with the LLM of the same underlying architecture, while the latter effectively captures long-term dependencies in time series. Since the student model does not need to process text, it only needs to take the patch processed in step 1 as input and output a result that conforms to the time series format.

[0067] The specific structure scale is as follows.

[0068] Transformer student model:

[0069] Number of layers: 4-layer Transformer Encoder.

[0070] Number of neurons: The hidden layer dimension is 128.

[0071] Activation function: ReLU.

[0072] LSTM student model:

[0073] Number of layers: 2 LSTM layers.

[0074] Number of neurons: 256 units per layer.

[0075] Then, we design a loss function for knowledge distillation, which consists of two parts: hard label loss (the deviation between the student model's prediction and the hard label, calculated by MSE) and soft label loss (the deviation between the student model's prediction and the teacher model's soft label, calculated by KL divergence). By jointly optimizing these two losses, the student model can not only learn the task objectives corresponding to the true labels, but also learn deeper information about the data distribution from the teacher model. The loss function is specifically defined as follows:

[0076] L_total=α*L_hard+(1-α)*L_soft;

[0077] Where L_hard represents the MSE between the student model output and the true label;

[0078] L_soft represents the KL divergence between the student model output and the teacher model output;

[0079] α represents the weight coefficient, which is initialized to 0.5.

[0080] The student model is then hyperparameter optimized by evaluating the distillation loss function, so that its prediction results are as close as possible to the output of the teacher model while keeping its computational complexity suitable for field deployment.

[0081] Finally, the trained student model is called and its output is weighted averaged for load forecasting. This process is computationally efficient, and because the student model is much less complex than TIME-LLM, it can meet the resource constraints of the business site.

[0082] Experimental results and comparison:

[0083] like Figure 2 As shown in the figure, the prediction comparison results of a load forecasting project are as follows. The prediction accuracy of the new load forecasting method based on variational autoencoder and TIME-LLM model proposed in this invention is 98.32%, and the accuracy of the original load forecasting algorithm is 95.06%, with an accuracy improvement of 3.26%.

[0084] in conclusion:

[0085] This paper optimizes the feature extraction process by carefully designing the variational autoencoder structure. Customizing Prompt and a small LLM enhances the model's understanding of the load forecasting task. Through knowledge distillation and student model integration, it addresses the resource constraints of actual deployments. Experimental results demonstrate that this method significantly improves performance and has practical application value in load forecasting tasks.

[0086] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.< / h> < / t>

Claims

1. A load forecasting method based on variational autoencoder and TIME-LLM model, characterized in that: include: S1. Data collection and preprocessing: Collect historical data including meteorological data, load data, and time data, and organize them into patches after RevIN processing; S2. Autoencoder training: training the variational autoencoder through the Patch block to obtain the encoder part of the trained variational autoencoder; S3. Reprogramming training: The latent variable Patch output by the encoder is aligned with the text prototype describing the trend characteristics of the time series through a cross-attention network to obtain a reprogramming result vector input to the TIME-LLM model; S4. Customize prompts for load forecasting problems: The text description and background knowledge of specific load forecasting problems are pre-trained to construct prompt vectors that are input into the TIME-LLM model. S5. Forecasting based on the TIME-LLM model: After concatenating the reprogramming result vector and the prompt vector, the vector is input into the TIME-LLM model and trained to obtain a predicted value that conforms to the result format of the load forecasting problem. S6, knowledge distillation and student model integration: The model trained in steps S3 to S5 is used as the teacher model, and a small model that meets the on-site resource conditions is initialized as the student model. The student model is trained through knowledge distillation. The training is stopped when the student model prediction value and the teacher model prediction value are less than a certain critical value, and the student model prediction value is integrated as the final load forecast result.

2. The load forecasting method based on variational autoencoder and TIME-LLM model according to claim 1 is characterized in that: The historical data in step S1 includes measured meteorological data, predicted meteorological data, time data, holiday data, industry production predicted load data, and industry production actual load data at different meteorological points at n time nodes in the historical period of the area to be predicted.

3. The load forecasting method based on variational autoencoder and TIME-LLM model according to claim 1 is characterized in that: The training method in step S2 includes: the encoder part of the variational autoencoder inputs the Patch block, outputs the latent variable Patch, uses the decoder part of the variational autoencoder to decode the latent variable Patch, and outputs the reconstructed Patch; evaluates the mean square error of the input and output and the KL divergence of the difference between the latent variable distribution and the standard normal distribution, and takes the hyperparameter that minimizes the evaluation error.

4. The load forecasting method based on variational autoencoder and TIME-LLM model according to claim 1 is characterized in that: In step S4, the training method includes organizing the specific load forecasting problem, input and output content and form, industry background knowledge about load forecasting, and statistical data of various dimensions of the input data into a prompt word through text description, and inputting it into a small pre-trained LLM module in the TIME-LLM model to obtain the compressed word embedding TokenEmbeddings as the Prompt vector passed into the main LLM model of TIME-LLM.

5. The load forecasting method based on variational autoencoder and TIME-LLM model according to claim 1 is characterized in that: In step S6, two or more student models with different architectures are designed, and the prediction values of the weighted average integrated student models are used as the prediction results.

6. A load forecasting system based on variational autoencoder and TIME-LLM model, characterized in that: include: Data collection and pre-processing unit: collects historical data including meteorological data, load data, and time data, and organizes them into patch blocks after being processed by RevIN; Autoencoder training unit: training the variational autoencoder through the patch block to obtain the encoder part of the trained variational autoencoder; Reprogramming training unit: aligns the latent variable Patch output by the encoder with the text prototype describing the trend characteristics of the time series through a cross-attention network to obtain a reprogramming result vector input to the TIME-LLM model; Load forecasting problem prompt customization unit: This unit constructs the text description and background knowledge of the specific load forecasting problem into the prompt vector input to the TIME-LLM model through pre-training; The prediction unit based on the TIME-LLM model: After concatenating the reprogramming result vector and the prompt vector, it is input into the TIME-LLM model and trained to obtain the predicted value that conforms to the result format of the load forecasting problem; Knowledge distillation and student model integration unit: The models trained in the reprogramming training unit, the load forecasting prompt customization unit, and the TIME-LLM model-based prediction unit are used as teacher models. A small model that meets the on-site resource conditions is initialized as the student model. The student model is trained through knowledge distillation. Training is stopped when the student model prediction value and the teacher model prediction value are less than a certain critical value, and the student model prediction value is integrated as the final load forecast result.

7. The load forecasting system based on variational autoencoder and TIME-LLM model according to claim 6, characterized in that: The historical data in the data collection and preprocessing unit includes measured meteorological data, predicted meteorological data, time data, holiday data, industry production predicted load data, and industry production actual load data at different meteorological points at n time nodes in the historical period of the predicted area.

8. The load forecasting system based on variational autoencoder and TIME-LLM model according to claim 6, characterized in that: The autoencoder training unit includes: the encoder part of the variational autoencoder inputs the patch block, outputs the latent variable patch, uses the decoder part of the variational autoencoder to decode the latent variable patch, and outputs the reconstructed patch; evaluates the mean square error of the input and output and the KL divergence of the difference between the latent variable distribution and the standard normal distribution, and takes the hyperparameters that minimize the evaluation error.

9. The load forecasting system based on variational autoencoder and TIME-LLM model according to claim 6, characterized in that: The customized prompt unit for load forecasting problems combines the specific load forecasting problem, input and output content and formats, industry background knowledge about load forecasting, and statistical data on various dimensions of the input data into a prompt through text description. This prompt is then input into a small pre-trained LLM module in the TIME-LLM model. The resulting compressed token embeddings are used as the prompt vector for the main LLM model of TIME-LLM.

10. The load forecasting system based on variational autoencoder and TIME-LLM model according to claim 6, characterized in that: In the knowledge distillation and student model integration unit, two or more student models with different architectures are designed, and the prediction values of the weighted average integrated student models are used as the prediction results.

Images