Load prediction method and system based on variational auto-encoder and TIME-LLM model

Abstract

The invention provides a load prediction method and system based on a variational auto-encoder and a TIME-LLM model. The method comprises the following steps: collecting and preprocessing historical data; carrying out self-encoder training to obtain a trained encoder part; performing reprogramming training to obtain a reprogramming result vector input into the TIME-LLM model; a load prediction problem cue word Prompt is customized and constructed into a Prompt vector input into the TIME-LLM model; and based on TIME-LLM model prediction, through knowledge distillation and student model integration, student model prediction values are integrated as a final load prediction result. According to the invention, the structure of the variational auto-encoder is designed in detail, and the feature extraction process is optimized; the understanding of the model on the load prediction task is enhanced; and through knowledge distillation and student model integration, resource limitation of actual deployment is satisfied.

Description

Technical Field

[0001] The present invention belongs to the technical field of new energy power, and in particular relates to a load forecasting method and system based on a variational autoencoder and a TIME-LLM model. Background Art

[0002] With the continuous development of the power system, power load forecasting is of great significance to the stable operation and optimal dispatching of the power system. Load forecasting refers to the exploration of the internal connection and development and change laws of things based on the known power system, economy, society, meteorology and other conditions through the analysis and research of historical data, and the pre-estimation and speculation of the load. Load forecasting is the basic work of power system planning, planning, electricity consumption, dispatching and other departments, and its importance has long been recognized by people.

[0003] Existing methods usually rely on traditional time series models or deep learning models, which often require a large amount of data for training and have limited ability to process complex patterns of future load changes. With the development of large language models (LLMs), their success in the field of natural language processing has brought new ideas for the processing of time series data. However, LLMs cannot process time series data directly. Researchers explored the application of model reprogramming on large language models (LLMs) and proposed the TIME-LLM framework, which can achieve high-precision time series prediction without modifying the language model. On this basis, we try to apply TIME-LLM to a more subdivided load forecasting field and use domain data for appropriate optimization. Summary of the invention

[0004] The present invention proposes a load forecasting method and system based on variational autoencoder and TIME-LLM model, designs the structure of variational autoencoder in detail, optimizes the feature extraction process, enhances the model's understanding of load forecasting tasks, and meets the resource constraints of actual deployment through knowledge distillation and student model integration.

[0005] To achieve the above object, the technical solution of the present invention is achieved as follows: A load forecasting method based on variational autoencoder and TIME-LLM model, comprising: S1. Data collection and preprocessing: Collect historical data including meteorological data, load data, and time data, and organize them into Patch blocks divided after RevIN processing; S2. Autoencoder training: training the variational autoencoder through the Patch block to obtain the encoder part of the trained variational autoencoder; S3, reprogramming training: the latent variable Patch output by the encoder part and the text prototype describing the trend characteristics of the time series are aligned and trained through the cross attention network to obtain the reprogramming result vector input into the TIME-LLM model; S4. Customization of prompt words for load forecasting problems: The text description and background knowledge of specific load forecasting problems are constructed as prompt vectors for input into the TIME-LLM model through pre-training; S5. Prediction based on TIME-LLM model: After concatenating the reprogramming result vector and the prompt vector, input them into the TIME-LLM model and train to obtain the predicted value that conforms to the result format of the load forecasting problem; S6, knowledge distillation and student model integration: The model trained in steps S3 to S5 is used as the teacher model, and a small model that meets the on-site resource conditions is initialized as the student model. The student model is trained through knowledge distillation. The training is stopped when the student model prediction value and the teacher model prediction value are less than a certain critical value, and the student model prediction value is integrated as the final load forecasting result.

[0006] Furthermore, the historical data in step S1 includes measured meteorological data, predicted meteorological data, time data, holiday data, industry production predicted load data, and industry production actual load data at different meteorological points at n time nodes in the historical period of the area to be predicted.

[0007] Furthermore, the training method in step S2 includes: the encoder part of the variational autoencoder inputs the Patch block, outputs the latent variable Patch, uses the decoder part of the variational autoencoder to decode the latent variable Patch, and outputs the reconstructed Patch; evaluates the mean square error of the input and output and the KL divergence of the difference between the latent variable distribution and the standard normal distribution, and takes the hyperparameters that minimize the evaluation error.

[0008] Furthermore, the training method in step S4 includes organizing the specific problems of load forecasting, input and output content and forms, industry background knowledge about load forecasting, and statistical data of various dimensions of the above input data into a prompt word Prompt through text description, and inputting it into a small pre-trained LLM module in the TIME-LLM model to obtain compressed word embeddings Token Embeddings as the Prompt vector passed into the main LLM model of TIME-LLM.

[0009] Furthermore, in step S6, more than two student models with different architectures are designed, and the prediction values ​​of the weighted average integrated student models are used as the prediction results.

[0010] On the other hand, the present invention also proposes a load forecasting system based on a variational autoencoder and a TIME-LLM model, comprising: Data collection and preprocessing unit: collects historical data including meteorological data, load data, and time data, and organizes them into Patch blocks divided after RevIN processing; Autoencoder training unit: training the variational autoencoder through the Patch block to obtain the encoder part of the trained variational autoencoder; Reprogramming training unit: aligning the latent variable Patch output by the encoder part with the text prototype describing the trend characteristics of the time series through a cross attention network to obtain a reprogramming result vector input into the TIME-LLM model; Prompt customization unit for load forecasting problems: The text description and background knowledge of specific load forecasting problems are constructed as prompt vectors for input into the TIME-LLM model through pre-training; Prediction unit based on TIME-LLM model: After concatenating the reprogramming result vector and the prompt vector, the vector is input into the TIME-LLM model and trained to obtain the prediction value that conforms to the result format of the load forecasting problem; Knowledge distillation and student model integration unit: The trained models in the reprogramming training unit, the load forecasting problem prompt word customization unit, and the prediction unit based on the TIME-LLM model are used as the teacher model, and a small model that meets the on-site resource conditions is initialized as the student model. The student model is trained through knowledge distillation. The training is stopped when the student model prediction value and the teacher model prediction value are less than a certain critical value, and the student model prediction value is integrated as the final load forecasting result.

[0011] Furthermore, the historical data in the data collection and preprocessing unit includes measured meteorological data, predicted meteorological data, time data, holiday data, industry production predicted load data, and industry production actual load data at different meteorological points at n time nodes in the historical period of the predicted area.

[0012] Furthermore, the autoencoder training unit includes: the encoder part of the variational autoencoder inputs the Patch block, outputs the latent variable Patch, uses the decoder part of the variational autoencoder to decode the latent variable Patch, and outputs the reconstructed Patch; evaluates the mean square error of the input and output and the KL divergence of the difference between the latent variable distribution and the standard normal distribution, and takes the hyperparameters that minimize the evaluation error.

[0013] Furthermore, the load forecasting problem prompt word customization unit includes organizing the specific problems of load forecasting, input and output content and form, industry background knowledge about load forecasting, and statistical data of each dimension of the above input data into a prompt word through text description, which is input into a small pre-trained LLM module in the TIME-LLM model to obtain the compressed token embeddings as the Prompt vector passed into the main LLM model of TIME-LLM.

[0014] Furthermore, in the knowledge distillation and student model integration unit, more than two student models with different architectures are designed, and the prediction value of the weighted average integrated student model is used as the prediction result.

[0015] Compared with the prior art, the present invention has the following beneficial effects: The present invention optimizes the feature extraction process by designing the structure of the variational autoencoder in detail; enhances the model's understanding of the load forecasting task by customizing Prompt and small LLM; and meets the resource constraints of actual deployment through knowledge distillation and student model integration. Experimental results show that the present invention has significant performance improvement and practical application value in load forecasting tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a schematic diagram of a method flow of an embodiment of the present invention; Figure 2 4 is a load forecast comparison result diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0018] like Figure 1 As shown, the method proposed in this embodiment specifically includes: 1. Data collection and preprocessing: The historical data includes measured meteorological data, forecasted meteorological data, time data, holiday data, industry production forecast load data, and industry production actual load data at n time nodes in the historical period of the forecast area at different meteorological points. Data cleaning is performed, including deletion and filling of missing values, screening and deletion of outliers, data alignment and splicing, etc. Feature construction is performed based on feature engineering based on prior knowledge of load, and derived features are constructed from the perspective of physical relationships and window statistics. Finally, it is organized into a common input form that meets the requirements of deep time series models, that is, the Patch blocks divided after RevIN reversible normalization processing.

[0019] The operation of dividing the Patch blocks after reversible normalization processing by RevIN mainly includes normalizing the historical data mentioned above through RevIN. The purpose of this step is to adjust the distribution of historical data to make it more suitable for model training; the normalized data is then divided into multiple Patch blocks. Each Patch block contains a part of the historical data. The RevIN is a reversible instance normalization method, which can not only normalize the data, but also denormalize the normalized data to restore it to the original distribution. It is suitable for time series data. The historical data in this embodiment also belong to data with time series attributes. The normalization parameters of the data can be dynamically adjusted through RevIN to cope with changes in the distribution of each data. 2. Autoencoder training: Variational Autoencoder (VAE) is a probabilistic autoencoder consisting of an encoder and a decoder. It can learn the intrinsic characteristics of data through the encoder, compress data to generate latent variables, and reconstruct latent variables into fitting data similar to the input data through the decoder.

[0020] Now we design a VAE, whose encoder part receives the input Patch generated by S1 and outputs the latent variable Patch, which is a low-dimensional feature vector obtained by compressing and extracting the feature information in the historical data.

[0021] The structure of the VAE encoder includes: Input layer: The input dimension is the number of features d of Patch.

[0022] Hidden layer: 3 fully connected layers, the number of neurons is 256, 128, and 64 respectively, and the activation function is ReLU.

[0023] Output layer: Generates the mean μ and variance σ of the latent variable, and the dimension of the latent variable is set to z_dim (such as 32).

[0024] Next, we optimize the hyperparameters for VAE training. Specifically, we use the decoder to decode the latent variable Patch to obtain the output of VAE, that is, the reconstructed Patch of the original input Patch. We evaluate the mean square error of the input and output and the KL divergence of the difference between the latent variable distribution and the standard normal distribution, and define the loss function: L = reconstruction error (MSE) + KL divergence (the difference between the latent variable and the standard normal distribution); Take the hyperparameters that minimize the evaluation error to train the VAE.

[0025] The purpose of training VAE is to use the encoder part of the trained VAE as the embedding layer in the re-encoding module of TIME-LLM, replacing the original linear Patch Embedder layer.

[0026] Performance comparison test data: In the load forecasting task, after using VAE to replace the linear Patch Embedder, the mean absolute error (MAE) of the model was reduced by about 2%, demonstrating the advantage of VAE in feature extraction.

[0027] 3. Reprogramming module training: The reprogramming module includes an autoencoder feature compression module, a text prototype, and an attention network module. The autoencoder module is the encoder part of the variational autoencoder obtained in step 2 above; the text prototype is a linear combination of words (including rising, falling, slow, fast, etc.) that describe the trend characteristics of the time series, such as Figure 1 As shown in the figure, the vocabulary vector is the vector of the large language model's understanding of the time series curve. For example, it can be learned through a pre-trained word vector model (such as GloVe), and then linearly combined through a linear layer to obtain a text prototype. The attention network module is a cross attention network. The cross attention network is composed of a classic Multi-Head Attention cross attention network. The number of heads is set to 8, the dimension of each head is 64, the hidden variable Patch feature passed in by the autoencoder module is Query, and the embedding of the text prototype is used as Key and Value. Using the characteristics of multi-head parallel computing, multiple dependencies between time series data and background knowledge can be captured. The cross attention network aligns the hidden variable Patch and the text prototype for training, and finally linearly combines them to obtain the reprogramming result vector R1; from the results, the recoding realizes the conversion from the time series Patch to the language text that the LLM model can understand.

[0028] 4. Customization of prompts for load forecasting problems: Construct prompts based on the text description and background knowledge of specific load forecasting problems. Specifically, the specific load forecasting problem to be conducted, the input and output content and form, the industry background knowledge about load forecasting, and the statistical data of each dimension of the above input data are organized into a prompt through text description, and input into a small pre-trained LLM module in TIME-LLM (such as DistilGPT-2, with about 82M parameters). The compressed Token Embeddings are used as the prompt vector R2 of the main LLM model of TIME-LLM to fully activate the ability of LLM in the specified time series task. In this embodiment, the Prompt paradigm includes three key components: [Domain]: Describes the industry context of load forecasting.

[0029] [Instruction]: Clarify the prediction tasks and requirements of the model.

[0030] [Statistics]: Provides statistical information of input data, such as mean, variance, trend, etc.

[0031] For load forecasting problems, please refer to the following examples for specific prompts: Power Load Forecasting (PLF) aims to predict future electricity usage to optimize grid management. Each data point consists of historical load consumption, meteorological data, and seasonal characteristics...Here is the information about the input time series: [Start Data] [Industry Background]: It has been observed that electricity consumption on weekdays is generally higher than on weekends, and the peak value occurs in the evening due to residential electricity consumption.

[0032] [Description]: Based on the past load consumption <t>step and other contextual features provided to predict the next step <h>step.

[0033] [Statistics]: The minimum value of the input time series is "min val", the maximum value is "max val", and the average value is "mean val". The data shows an upward or downward trend.<lag val> The most significant lagged correlation was observed at .

[0034] [END DATA] The above explanation is only for explaining the Chinese meaning of the Prompt reference example and there is no other limitation.

[0035] 5. Prediction based on TIME-LMM: The prompt word vector R2 generated in step 4 is used as a prefix and concatenated with the time series patch reprogramming result vector R1 generated in step 3, and then input into the LLM model body of TIME-LLM. Through the forward propagation of the model, the output Output Patch Embeddings are obtained. Finally, the output patches are flattened and concatenated, and then converted into predicted values ​​that conform to the required load forecasting problem result format through a linear layer.

[0036] 6. Knowledge distillation and student model integration: Use the TIME-LLM model trained in steps 3 to 5 as the teacher model, and initialize a small model that meets the on-site resource conditions as the student model. Train the student model through knowledge distillation, stop training when the student model prediction value and the teacher model prediction value are less than a certain critical value, and integrate the student model results as the final load forecast result. Considering that different architecture models capture feature information differently, and in order to improve the prediction effect and model robustness, design more than two small models with different architectures (such as Transformer and LSTM, etc.), and use weighted average to integrate the prediction results.

[0037] Specifically, the TIME-LLM teacher model is first used to predict the training data, and the predicted value of each time series sample is obtained as a soft label (the actual value is a hard label). These soft labels not only contain the results of the model prediction, but also include TIME-LLM's understanding of the complex time series patterns in the data. This information will guide the student model.

[0038] Then, a Transformer construction and a LSTM architecture deep time series model are initialized as student models. The former is obviously more compatible with the LLM of the same infrastructure, and the latter can effectively capture the long-term dependencies in the time series. Since the student model does not need to have the ability to process text, it only needs to take the patch processed in step 1 as input and output the result that meets the time series format.

[0039] The specific structure scale is as follows.

[0040] Transformer student model: Number of layers: 4-layer Transformer Encoder.

[0041] Number of neurons: The hidden layer dimension is 128.

[0042] Activation function: ReLU.

[0043] LSTM Student Model: Number of layers: 2 LSTM layers.

[0044] Number of neurons: 256 units per layer.

[0045] Then, the loss function of knowledge distillation is designed, which consists of two parts: hard label loss (the deviation between the student model's prediction value and the hard label, calculated by MSE) and soft label loss: (the deviation between the student model's prediction value and the teacher model's soft label, calculated by KL divergence). By jointly optimizing these two parts of loss, the student model can not only learn the task objectives corresponding to the true label, but also learn deeper information about the data distribution from the teacher model. The loss function is specifically defined as follows: L_total=α*L_hard+(1-α)*L_soft; Among them, L_hard represents the MSE between the student model output and the true label; L_soft represents the KL divergence between the output of the student model and the output of the teacher model; α represents the weight coefficient, which is initialized to 0.5.

[0046] The student model is then hyperparameter optimized by evaluating the distillation loss function to make its predictions as close as possible to the output of the teacher model while keeping its computational complexity suitable for field deployment.

[0047] Finally, the trained student model is called and the weighted average of the output results of the student model is used for load forecasting. This process has high computational efficiency and can meet the resource constraints of the business site because the complexity of the student model is much smaller than that of TIME-LLM.

[0048] Experimental results and comparison: like Figure 2 As shown in the figure, the prediction comparison results of a load forecasting project are shown. The prediction accuracy of the new load forecasting method based on variational autoencoder and TIME-LLM model proposed in the present invention is 98.32%, and the accuracy of the original load forecasting algorithm is 95.06%, with an accuracy improvement of 3.26%.

[0049] in conclusion: The present invention optimizes the feature extraction process by designing the structure of the variational autoencoder in detail; enhances the model's understanding of the load forecasting task by customizing Prompt and small LLM; and meets the resource constraints of actual deployment through knowledge distillation and student model integration. Experimental results show that this method has significant performance improvement and practical application value in load forecasting tasks.

[0050] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.< / h> < / t>

Claims

1. A load forecasting method based on variational autoencoder and TIME-LLM model, characterized in that: include: S1. Data collection and preprocessing: Collect historical data including meteorological data, load data, and time data, and organize them into Patch blocks divided after RevIN processing; S2. Autoencoder training: training the variational autoencoder through the Patch block to obtain the encoder part of the trained variational autoencoder; S3, reprogramming training: the latent variable Patch output by the encoder part and the text prototype describing the trend characteristics of the time series are aligned and trained through the cross attention network to obtain the reprogramming result vector input into the TIME-LLM model; S4. Customization of prompt words for load forecasting problems: The text description and background knowledge of specific load forecasting problems are constructed as prompt vectors for input into the TIME-LLM model through pre-training; S5. Prediction based on TIME-LLM model: After concatenating the reprogramming result vector and the prompt vector, input them into the TIME-LLM model and train to obtain the predicted value that conforms to the result format of the load forecasting problem; S6, knowledge distillation and student model integration: The model trained in steps S3 to S5 is used as the teacher model, and a small model that meets the on-site resource conditions is initialized as the student model. The student model is trained through knowledge distillation. The training is stopped when the student model prediction value and the teacher model prediction value are less than a certain critical value, and the student model prediction value is integrated as the final load forecasting result.

2. The load forecasting method based on variational autoencoder and TIME-LLM model according to claim 1 is characterized in that: The historical data in step S1 includes measured meteorological data, predicted meteorological data, time data, holiday data, industry production predicted load data, and industry production actual load data at different meteorological points at n time nodes in the historical period of the area to be predicted.

3. The load forecasting method based on variational autoencoder and TIME-LLM model according to claim 1 is characterized in that: The training method in step S2 includes: the encoder part of the variational autoencoder inputs the Patch block, outputs the latent variable Patch, uses the decoder part of the variational autoencoder to decode the latent variable Patch, and outputs the reconstructed Patch; evaluates the mean square error of the input and output and the KL divergence of the difference between the latent variable distribution and the standard normal distribution, and takes the hyperparameters that minimize the evaluation error.

4. The load forecasting method based on variational autoencoder and TIME-LLM model according to claim 1 is characterized in that: The training method in step S4 includes organizing the specific problem of load forecasting, input and output content and form, industry background knowledge about load forecasting, and statistical data of each dimension of the above input data into a prompt word Prompt through text description, and inputting it into a small pre-trained LLM module in the TIME-LLM model to obtain the compressed word embedding TokenEmbeddings as the Prompt vector of the main LLM model of TIME-LLM.

5. The load forecasting method based on variational autoencoder and TIME-LLM model according to claim 1 is characterized in that: In step S6, more than two student models with different architectures are designed, and the prediction values ​​of the weighted average integrated student models are used as the prediction results.

6. A load forecasting system based on variational autoencoder and TIME-LLM model, characterized in that: include: Data collection and preprocessing unit: collects historical data including meteorological data, load data, and time data, and organizes them into Patch blocks divided after RevIN processing; Autoencoder training unit: training the variational autoencoder through the Patch block to obtain the encoder part of the trained variational autoencoder; Reprogramming training unit: aligning the latent variable Patch output by the encoder part with the text prototype describing the trend characteristics of the time series through a cross attention network to obtain a reprogramming result vector input into the TIME-LLM model; Prompt customization unit for load forecasting problems: The text description and background knowledge of specific load forecasting problems are constructed as prompt vectors for input into the TIME-LLM model through pre-training; Prediction unit based on TIME-LLM model: After concatenating the reprogramming result vector and the prompt vector, the vector is input into the TIME-LLM model and trained to obtain the prediction value that conforms to the result format of the load forecasting problem; Knowledge distillation and student model integration unit: The trained models in the reprogramming training unit, the load forecasting problem prompt word customization unit, and the prediction unit based on the TIME-LLM model are used as the teacher model, and a small model that meets the on-site resource conditions is initialized as the student model. The student model is trained through knowledge distillation. The training is stopped when the student model prediction value and the teacher model prediction value are less than a certain critical value, and the student model prediction value is integrated as the final load forecasting result.

7. The load forecasting system based on variational autoencoder and TIME-LLM model according to claim 6, characterized in that: The historical data in the data collection and preprocessing unit includes measured meteorological data, predicted meteorological data, time data, holiday data, industry production predicted load data, and industry production actual load data at different meteorological points at n time nodes in the historical period of the predicted area.

8. The load forecasting system based on variational autoencoder and TIME-LLM model according to claim 6, characterized in that: The autoencoder training unit includes: the encoder part of the variational autoencoder inputs the Patch block, outputs the latent variable Patch, uses the decoder part of the variational autoencoder to decode the latent variable Patch, and outputs the reconstructed Patch; evaluates the mean square error of the input and output and the KL divergence of the difference between the latent variable distribution and the standard normal distribution, and takes the hyperparameter that minimizes the evaluation error.

9. The load forecasting system based on variational autoencoder and TIME-LLM model according to claim 6, characterized in that: The customized prompt word unit for load forecasting problems includes organizing the specific problems of load forecasting, input and output content and forms, industry background knowledge about load forecasting, and statistical data of various dimensions of the above input data into a prompt word through text description, and inputting it into a small pre-trained LLM module in the TIME-LLM model to obtain the compressed token embeddings as the Prompt vector passed into the main LLM model of TIME-LLM.

10. The load forecasting system based on variational autoencoder and TIME-LLM model according to claim 6, characterized in that: In the knowledge distillation and student model integration unit, two or more student models with different architectures are designed, and the prediction values ​​of the weighted average integrated student models are used as the prediction results.

Images