Load prediction method based on pre-training language model

Abstract

The invention provides a load prediction method based on a pre-training language model, which comprises the following steps: S1) constructing a data modeling method suitable for the pre-training language model, taking language description and statistical data as prediction data, and better mining the prediction performance of the model; s2) providing a data enhancement method suitable for a language model based on the data modeling method designed in S1, and reducing the illusion rate in a prediction result; s3) constructing a pre-training language model-based load prediction framework suitable for multiple time scales and multiple scenes; and S4) based on the prediction data formed in S1 and S2 and the load prediction framework constructed in S3, verification of the proposed method is carried out in load prediction tasks under different time scales and different scenes, and finally the prediction precision of the system is given. The strong flexibility and universality of PLMs are utilized, and the method is provided for multi-time-scale and multi-scene load prediction tasks.

Description

Technical Field

[0001] The present invention belongs to the field of power system load forecasting, and specifically relates to a load forecasting method based on a pre-trained language model. Background Art

[0002] Accurate load forecasting is essential to maintain the power balance between generators and consumers, especially with the increasing integration of renewable energy sources, which introduce significant intermittent fluctuations. With the development of data-driven methods, machine learning and deep learning-based models have become the main methods for load forecasting tasks, and various networks have been continuously improved to improve prediction accuracy, such as extreme gradient boosting (XGBoost), random forest (RF), and long short-term memory network (LSTM).

[0003] In recent years, pre-trained language models have made significant progress and demonstrated excellent performance in various fields. This paper proposes a load forecasting method based on a pre-trained language model, which not only has accurate forecasting capabilities but also has general and flexible applicability. In addition, the present invention proposes a data modeling method that can effectively convert load sequence data into natural language for model training. At the same time, the present invention introduces a data augmentation strategy to eliminate the influence of the language model's hallucination on the forecast results.

[0004] The effectiveness of the proposed method has been verified on two real-world datasets. Compared with existing methods, the proposed method shows state-of-the-art performance in all validation metrics.

[0005] The present application is compared with the prior art as follows:

[0006] Technical comparison with patent CN202311631190.2 "Power load forecasting method based on large language model"

[0007] Patent CN202311631190.2 converts structured power load data containing multiple influencing factors into corresponding text descriptions through prompt word templates, and then uses a large language model to perform load forecasting on the corresponding text descriptions. The data directly input by the present invention is structured text descriptions, and the pre-trained language model is directly used for load forecasting tasks.

[0008] Patent CN202311631190.2 uses an efficient fine-tuning method to train parameters of a large language model. In this article, large models with different parameter quantities are compared. The large models are trained using full parameter training and university fine-tuning methods respectively, and the load forecasting results of different methods and different models are compared.

[0009] Technical comparison with patent CN202310711160.6 "A power grid load forecasting method based on large language generation model"

[0010] Patent CN202310711160.6 calls a large language model to start the load forecasting task by establishing a dialogue function. Afterwards, based on the name of the predicted data, the number of data features, and the amount of data training information, the large language model is used to convert the model descriptive text into corresponding executable code, and then the generated code is executed to establish a load forecasting model and perform the load forecasting task. The load prediction code generated by the large language model may be an arbitrary model. The load forecasting method used in the present invention is to directly use the pre-trained language model for load forecasting and directly generate a load forecasting sequence, which ensures the superiority of the prediction model and does not require the generation of code for other models to complete the load forecasting task.

[0011] Technical comparison with patent CN202410001924.7 "Automated building energy load forecasting method based on large language model"

[0012] Patent CN202410001924.7 uses a prompt word function and a Bayesian optimization algorithm to automatically generate and optimize prompt words for a large language model, thereby achieving autonomous generation of high-quality building energy load prediction model training code, and then executing the generated code to perform load prediction tasks. The load prediction method used in the present invention is to directly use a pre-trained language model for load prediction, directly generate a load prediction sequence, and does not require code generation.

[0013] The load forecasting method in patent CN202410001924.7 is only for the energy load of automated buildings. The load forecasting method adopted by the present invention is applicable to multi-time scale and multi-scenario load forecasting. Summary of the invention

[0014] In order to solve the above technical problems, the present invention proposes a load forecasting method based on a pre-trained language model, which utilizes the powerful flexibility and versatility of PLMs itself to provide multi-time scale and multi-scenario load forecasting tasks.

[0015] To achieve the above object, the technical solution adopted by the present invention is:

[0016] A load forecasting method based on a pre-trained language model, the specific steps are as follows:

[0017] S1) A data modeling method suitable for pre-trained language models was constructed, using language descriptions and statistical data as prediction data to better explore the prediction performance of the model;

[0018] S2) Based on the data modeling method designed in S1, a data enhancement method suitable for language model is proposed to reduce the hallucination rate in the prediction results;

[0019] S3) A load forecasting framework based on pre-trained language models suitable for multiple time scales and multiple scenarios is constructed;

[0020] S4) Based on the forecast data formed by S1 and S2 and the load forecasting framework constructed by S3, the proposed method was verified in load forecasting tasks under different time scales and scenarios, and finally the prediction accuracy of the system was given.

[0021] Further, the step S1 includes: the specific contents are as follows:

[0022] S11. Data modeling methods;

[0023] Historical load data is used as input for forecasting and the input data is modeled as a continuous series Where L and d represent the length and dimension of the sequence respectively. Since the data needs to be input in text format, a dataset modeling method is proposed to convert the numerical sequence into a natural language expression X. text , as follows:

[0024]

[0025] Among them, x i is the i-th data in the input sequence, represents the set of real numbers, Represents conversion from real numbers to text;

[0026] In addition, in order to further utilize the advantages of text expression, statistical information X is introduced stat To enhance the feature dimension of the input data, denoted as X ts The statistical information includes the maximum, minimum and average values, using the N values ​​before the prediction time. obs The maximum and minimum values ​​within the step are used to model the global features, and the average value of the input sequence is used to represent the local features;

[0027]

[0028] Among them, X ts Represents PLM input with statistical information, X stat It is a verbal description of statistics containing the maximum, minimum, and average values ​​of X. obs Indicates the number of N before the prediction time obs Historical load data within the time step;

[0029] S12. Construction of multi-time scale datasets:

[0030] In order to evaluate the versatility and accuracy of the proposed method in load forecasting tasks, the following two real datasets with different time scales are selected for study;

[0031] (1) Electric load forecasting dataset ELFD:

[0032] (2) Industrial Customer Load Dataset ICLD.

[0033] Further, the step S2 includes: the specific contents are as follows;

[0034] The hallucination phenomenon in load forecasting tasks is attributed to two aspects:

[0035] 1) When converting numerical data to text description, the length of the loaded data stored in string format showed inconsistency;

[0036] 2) The pre-trained parameters of PLM are derived from natural language training, so they lack the ability to effectively identify pure numerical values;

[0037] Taking advantage of PLM's sensitivity to language description, a data augmentation method is proposed to separate numerical data from text information. ets X-based text Build as follows:

[0038]

[0039] in is a textual expression of a numerical sequence with time information, t i is x i The corresponding i-th time step in the textual expression;

[0040] Given that the output data of the prediction model also exists in text form, the corresponding true value in text format is required during the training process. The corresponding true value Y is generated according to the same process as for each input format X. gt .

[0041] Further, the step S3 includes: the specific contents are as follows;

[0042] S31, basic architecture of pre-trained language model;

[0043] There are three types of pre-trained language models, and different architectures are suitable for different tasks, including:

[0044] 1) Encoder-only architecture. The model is BERT. The model uses a masked language model to learn a bidirectional context encoder. The training goal is to randomly mask the text and then predict the masked words. It is suitable for tasks that do not require sequence generation and only require one-way task scenarios that encode and process the input.

[0045] 2) Decoder-only architecture, also known as generative architecture, the model is GPT and LLaMA, used for sequence generation tasks, the model directly generates the corresponding sequence from the input encoding, at the same time, the decoder-only architecture also has a feature of unsupervised pre-training, good at creative writing;

[0046] 3) Encoder-Decoder architecture, also known as sequence-to-sequence architecture, includes Transformer structure. The models are T5 and BART. The model uses encoder to encode the input sequence, extract its features and semantic information, and pass the encoding result to the decoder. Then, the decoder generates the corresponding output sequence according to the encoding result. The advantage is that it can better handle the relationship between input sequence and output sequence, thereby improving the accuracy of tasks such as machine translation and dialogue generation.

[0047] According to the characteristics of the load forecasting task, PLMs based on Decoder-only and Encoder-Decoder architectures are considered, and PLMs trained in different languages ​​are selected to verify whether natural language expressions affect the forecasting results;

[0048] S32, model training method;

[0049] S321, full parameter training method;

[0050] For language models other than BLOOM, the full parameter training method is used to train the model in the load forecasting dataset. Although this method sacrifices the original problem-solving ability of some of the model's own pre-trained parameters, it performs well in the load forecasting task.

[0051] S322, PEFT, a method for efficient parameter fine-tuning;

[0052] The parameter efficient fine-tuning PEFT method is used in combination with the low-rank adaptation LoRA technique of large language models to fine-tune the model parameters. In this method, low-rank decomposition is used to simulate parameter changes based on the parameter distribution of the original model, thereby indirectly training a large model with the least number of parameters. The selected parameter matrix W in the original model is d×k Perform the following processing:

[0053] W d×k =U d×r ·V r×k r< <d,k (11)

[0054] Among them, r is the low-rank coefficient, U and V are low-rank matrices;

[0055] S33, prediction result evaluation method;

[0056] Since the format of the prediction results is natural language, the numerical sequence in the natural language is extracted and compared with the true value to calculate the prediction accuracy to analyze the performance of the model. The hallucination phenomenon in the prediction results is evaluated by the hallucination rate. In addition, the mean absolute error MAE and root mean square error RMSE are also used to measure the accuracy of the prediction results, which are defined as follows:

[0057]

[0058] Where N is the number of samples, Y i represents the i-th prediction result, Y i,gt represents the corresponding true value, H∈(0,1) represents the hallucination rate, n h Indicates the number of hallucinated samples.

[0059] Beneficial effects:

[0060] (1) A dataset construction method was established to convert sequence format data into natural language to facilitate PLM training and integrate the language description of statistical information to expand the input feature dimension.

[0061] (2) Aiming at the hallucination problem of PLMs in load forecasting tasks, the present invention proposes a data enhancement method accordingly. By properly separating the numerical sequence and the language description, the hallucination rate is significantly reduced to 0%.

[0062] (3) The comprehensive prediction performance of the proposed prediction model is verified on two real data sets. On the load forecasting data sets of different time scales and different scenarios, the mean absolute error of the model is reduced to 40.6 and 4.0, respectively, demonstrating superior prediction accuracy over existing methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is the data distribution diagram of the ICLD data set and the ELFD data set of the present invention;

[0064] Figure 2 It is the overall prediction framework diagram of the present invention;

[0065] Figure 3 is a data flow diagram of the present invention;

[0066] Figure 4 It is a display diagram of the ICLD data set prediction results of the present invention;

[0067] Figure 5 It is a graph showing the prediction results of the ELFD data set of the present invention. DETAILED DESCRIPTION

[0068] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments:

[0069] Embodiment 1:

[0070] A load forecasting method based on a pre-trained language model. The overall forecasting framework is shown in the figure Figure 2 As shown, the following steps are included:

[0071] S1) A data modeling method suitable for pre-trained language models was constructed, using language descriptions and statistical data as prediction data to better explore the prediction performance of the model;

[0072] S2) Based on the data modeling method designed in S1, a data enhancement method suitable for language model is proposed to reduce the hallucination rate in the prediction results;

[0073] S3) A load forecasting framework based on pre-trained language models suitable for multiple time scales and multiple scenarios is constructed;

[0074] S4) Based on the forecast data formed by S1 and S2 and the load forecasting framework constructed by S3, the proposed method was verified in load forecasting tasks under different time scales and scenarios, and finally the prediction accuracy of the system was given.

[0075] The specific contents of step S1 include: constructing a data modeling method suitable for the prediction framework in S1, using language description and statistical data as prediction input data, and better mining the prediction performance of the model. The specific contents are as follows:

[0076] S11. Data Modeling Methods

[0077] In common load forecasting tasks, historical load data are usually used as forecast input. These input data are usually modeled as a continuous series Where L and d represent the length and dimension of the sequence respectively. Since data needs to be input in text format, the present invention proposes a data set modeling method to convert the numerical sequence into a natural language expression X text , as follows:

[0078]

[0079] Among them, x i is the i-th data in the input sequence, represents the set of real numbers, Represents conversion from real number to text.

[0080] In addition, in order to further utilize the advantages of text expression, the present invention introduces statistical information X stat To enhance the feature dimension of the input data, denoted as X ts The statistical information includes the maximum value, minimum value and average value. Specifically, the present invention uses the N values ​​before the prediction time. obsThe maximum and minimum values ​​within the step are used to model the global features, and the average value of the input sequence is used to represent the local features.

[0081]

[0082] Among them, X ts Represents PLM input with statistical information, X stat It is a verbal description of statistics containing the maximum, minimum, and average values ​​of X. obs Indicates the number of N before the prediction time obs Historical load data within a time step.

[0083] S12. Construction of multi-time scale dataset

[0084] In order to evaluate the versatility and accuracy of the proposed method in load forecasting tasks, the present invention selects the following two real data sets with different time scales for research.

[0085] (1) Electricity Load Forecasting Dataset (ELFD): This dataset is an open source dataset on Kaggle, covering more than 40,000 hours of load data in Panama from 2015 to 2020.

[0086] (2) Industrial Customer Load Dataset (ICLD) This real-world dataset contains approximately 9,000 daily load data from 10 industrial customers from 2018 to 2021. The dataset is derived from a real power system in eastern China.

[0087] The data distribution of the two data sets is as follows Figure 1 shown.

[0088] The specific contents of step S2 include: based on the data modeling method designed in S1, a data enhancement method suitable for language model is proposed to reduce the hallucination rate in the prediction results, and the specific contents are as follows;

[0089] The hallucination phenomenon (e.g., missing data or generating extra data) in the load forecasting task can be attributed to two main aspects:

[0090] 1) When converting numerical data to text description, the length of loaded data stored in string format showed inconsistency.

[0091] 2) The pre-trained parameters of PLM are derived from natural language training, and therefore lack the ability to effectively recognize pure numerical values.

[0092] Taking advantage of PLM’s sensitivity to language description, this section proposes a data augmentation method to separate numerical data from text information. ets X-based textBuild as follows:

[0093]

[0094] in is a textual expression of a numerical sequence with time information, t i is x i The corresponding i-th time step in the textual representation.

[0095] Given that the output data of the prediction model also exists in text form, the corresponding true values ​​in text format are required during the training process. The corresponding true value Y is generated according to the same process as for each input format X. gt The overall prediction framework and data flow of the system are as follows: Figure 3 shown.

[0096] The specific contents of step S3 include: constructing a load forecasting framework based on a pre-trained language model applicable to multiple time scales and multiple scenarios, as shown in the abstract figure, and its main contents include:

[0097] S31. Basic architecture of pre-trained language model

[0098] There are three types of pre-trained language models. Different architectures are suitable for different tasks, mainly including:

[0099] 1) Encoder-only architecture, the representative model is BERT, the model is to learn a bidirectional context encoder by using a masked language model, the training goal is to randomly mask the text and then predict the masked words. It is mainly suitable for tasks that do not require sequence generation, but only require one-way task scenarios that encode and process the input, such as text classification, sentiment analysis, etc.

[0100] 2) Decoder-only architecture, also known as generative architecture, is represented by GPT and LLaMA. It is usually used for sequence generation tasks, such as text generation, machine translation, etc. The model directly generates the corresponding sequence from the input encoding. At the same time, another important feature of the Decoder-Only architecture is that it can be pre-trained without supervision and is good at creative writing. It focuses more on expanding new content from existing information (beginning). Its disadvantage is that a large amount of training data is required to improve the quality and diversity of generated text.

[0101] 3) Encoder-Decoder architecture, also known as sequence-to-sequence architecture, includes Transformer structure, and representative models are T5 and BART. The model uses the encoder to encode the input sequence, extract its features and semantic information, and pass the encoding result to the decoder. Then, the decoder generates the corresponding output sequence based on the encoding result. The advantage of this architecture is that it can better handle the relationship between the input sequence and the output sequence, thereby improving the accuracy of tasks such as machine translation and dialogue generation. Accept a piece of text and generate another piece of text from left to right. It is often used in conditional generation tasks such as machine translation, summary generation, factual dialogue, etc. The advantage of this architecture is that it can better handle the relationship between the input sequence and the output sequence, thereby improving the accuracy of tasks such as machine translation and dialogue generation. The disadvantage is that the model complexity is high, and the training time and computing resources consumption are large.

[0102] According to the characteristics of the load forecasting task, PLMs based on Decoder-only and Encoder-Decoder architectures are mainly considered, as shown in Table 1. In addition, PLMs trained in different languages ​​are also selected to verify whether natural language expressions affect the forecasting results.

[0103] Table 1 Pre-trained language models used in the framework

[0104]

[0105] S32. Model training method

[0106] S321, full parameter training method

[0107] For language models other than BLOOM, this section uses the full parameter training method to train the model in the load forecasting dataset. Although this method sacrifices the original problem-solving ability of some of the model's own pre-trained parameters, it performs well in the load forecasting task.

[0108] S322, Parameter Efficient Fine-tuning Method (PEFT)

[0109] For large language models such as BLOOM, a comprehensive understanding of knowledge is already encoded in their pre-trained parameters when they are pre-trained for general tasks. However, training these models entirely on specialized datasets destroys the distribution pattern of their pre-trained parameters and reduces their feasibility in text understanding. Therefore, the present invention adopts a parameter efficient fine-tuning (PEFT) method in combination with a low-rank adaptation (LoRA) technique for large language models to fine-tune the model parameters. In this method, the present invention uses low-rank decomposition to simulate parameter changes based on the parameter distribution of the original model, thereby indirectly training a large model with a minimal number of parameters. The present invention performs a multi-step optimization on the selected parameter matrix W in the original model. d×kPerform the following processing:

[0110] W d×k =U d×r ·V r×k r< <d,k (4)

[0111] Among them, r is the low-rank coefficient, U and V are low-rank matrices.

[0112] S33. Prediction result evaluation method

[0113] Since the format of the prediction results is natural language, this chapter extracts the numerical sequence in the natural language and compares it with the true value to calculate the prediction accuracy to analyze the performance of the model. In order to measure the model, this chapter proposes the hallucination rate to evaluate the hallucination phenomenon in the prediction results. In addition, the mean absolute error (MAE) and root mean square error (RMSE) are also used to measure the accuracy of the prediction results, which are defined as follows:

[0114]

[0115]

[0116] Where N is the number of samples, Y i represents the i-th prediction result, Y i,gt represents the corresponding true value, H∈(0,1) represents the hallucination rate, n h Indicates the number of hallucinated samples.

[0117] Embodiment 2:

[0118] The same parts of this embodiment as those of the first embodiment are not described in detail, except that:

[0119] The present invention calculates the hallucination rate, mean absolute error (MAE) and root mean square error (RMSE) of prediction data with lengths of 860 and 2670 on two different time scales and different validation sets, ICLD and ELFD. The hallucination of the language model may cause missing or redundant data in the prediction results. To ensure the accuracy of the indicator calculation, this chapter solves the above problem through the following settings:

[0120] 1) Missing data are filled with zeros;

[0121] 2) Redundant data is removed to keep the length of all output sequences consistent.

[0122] The present invention adopts the Mengzi-T5 model in the load forecasting framework based on the pre-trained language model and compares it with traditional methods (including XGBoost, random forest RF and LSTM). The hyperparameter settings of the model are shown in Table 2. The PLM method shows the most accurate forecasting performance compared with the three traditional forecasting methods.

[0123] Table 2 Model hyperparameter settings

[0124]

[0125]

[0126] The PLM, PLM-ts, and PLM-ets methods differ only in the input data format, corresponding to X text , X ts and X ets The prediction results show that integrating long-term statistical information into the data can improve the prediction accuracy. In order to intuitively demonstrate the prediction effect, the prediction curves based on the method proposed by the present invention and the comparative method are shown in Figure 4 and Figure 5 The visualization is carried out in Figure 4 is the prediction curve in the ICLD data set, Figure 5 is the prediction curve in the ELFD data set.

[0127] It is worth noting that the skewed parts in Table 3 highlight the hallucination problem in the prediction, especially in the ICLD dataset, which leads to a significantly higher RMSE than normal. The present invention confirms that this problem stems from missing values ​​in the prediction data series. ets When used as input data, the hallucination rate drops to 0%, and the MAE and RMSE are also significantly improved.

[0128] These results show that without preprocessing the raw data, the prediction ability of PLM cannot be fully utilized. The method proposed in Section S2 can effectively eliminate the hallucination and improve the prediction accuracy.

[0129] Table 3 Comparison of PLM prediction results with other methods

[0130]

[0131]

[0132] In addition, the present invention also verifies the versatility of the proposed method on PLMs of different structures, as shown in Table 4. The load forecasting framework based on the pre-trained language model consistently achieves lower MAE and RMSE on different PLMs, among which the Mengzi-T5 model shows the best prediction performance on both datasets. The results also show that the models based on Chinese pre-trained parameters and corresponding datasets perform better in the prediction task. This chapter achieves better prediction results on two Chinese pre-trained models, Mengzi-T5 and BART-CN.

[0133] The BLOOM and BLOOM-CN models based on the data modeling method proposed in this invention combined with the PEFT parameter efficient fine-tuning method performed poorly in the load forecasting task. The output of the model is inaccurate and accompanied by a large number of hallucinations. For example, the output result is "the load forecast value is 1834, 133, 12699,,---, 192,,". Although the text part of its output can reflect some forecast information, the numerical output is significantly distorted. Therefore, the results of these two models are excluded from the statistical data.

[0134] Table 4 Comparison of prediction results of different PLMs

[0135]

[0136] The above description is only a preferred embodiment of the present invention and does not constitute any other form of limitation to the present invention. Any modification or equivalent change made based on the technical essence of the present invention still falls within the scope of protection required by the present invention.

Claims

1. A load forecasting method based on a pre-trained language model, characterized in that: The specific steps are as follows: S1) A data modeling method suitable for pre-trained language models was constructed, using language descriptions and statistical data as prediction data to better explore the prediction performance of the model; S2) Based on the data modeling method designed in S1, a data enhancement method suitable for language model is proposed to reduce the hallucination rate in the prediction results; S3) A load forecasting framework based on pre-trained language models suitable for multiple time scales and multiple scenarios is constructed; S4) Based on the forecast data formed by S1 and S2 and the load forecasting framework constructed by S3, the proposed method was verified in load forecasting tasks under different time scales and scenarios, and finally the prediction accuracy of the system was given.

2. The load forecasting method based on a pre-trained language model according to claim 1, characterized in that: The step S1 includes: the specific contents are as follows: S11. Data Modeling Methods Historical load data is used as input for forecasting and the input data is modeled as a continuous series Where L and d represent the length and dimension of the sequence respectively. Since the data needs to be input in text format, a dataset modeling method is proposed to convert the numerical sequence into a natural language expression X. text , as follows: Among them, x i is the i-th data in the input sequence, represents the set of real numbers, Represents conversion from real numbers to text; In addition, in order to further utilize the advantages of text expression, statistical information X is introduced stat To enhance the feature dimension of the input data, denoted as X ts The statistics include the maximum, minimum and average values, using the N values ​​before the prediction time. obs The maximum and minimum values ​​within the step are used to model the global features, and the average value of the input sequence is used to represent the local features; Among them, X ts Represents PLM input with statistical information, X stat It is a verbal description of statistics containing the maximum, minimum, and average values ​​of X. obs Indicates the number of N before the prediction time obs Historical load data within the time step; S12. Construction of multi-time scale datasets: In order to evaluate the versatility and accuracy of the proposed method in load forecasting tasks, the following two real datasets with different time scales are selected for study; (1) Electric load forecasting dataset ELFD: (2) Industrial Customer Load Dataset ICLD.

3. The load forecasting method based on a pre-trained language model according to claim 1, characterized in that: The step S2 includes: the specific contents are as follows; The hallucination phenomenon in load forecasting tasks is attributed to two aspects: 1) When converting numerical data to text description, the length of the loaded data stored in string format showed inconsistency; 2) The pre-trained parameters of PLM are derived from natural language training, so they lack the ability to effectively identify pure numerical values; Taking advantage of PLM's sensitivity to language description, a data augmentation method is proposed to separate numerical data from text information. ets X-based text Build as follows: in is a textual expression of a numerical sequence with time information, t i is x i The corresponding i-th time step in the textual expression; Given that the output data of the prediction model also exists in text form, the corresponding true value in text format is required during the training process. The corresponding true value Y is generated according to the same process as for each input format X. gt .

4. The load forecasting method based on a pre-trained language model according to claim 1, characterized in that: The step S3 includes: the specific contents are as follows; S31, basic architecture of pre-trained language model; There are three types of pre-trained language models, and different architectures are suitable for different tasks, including: 1) Encoder-only architecture. The model is BERT. The model uses a masked language model to learn a bidirectional context encoder. The training goal is to randomly mask the text and then predict the masked words. It is suitable for tasks that do not require sequence generation and only require one-way task scenarios that encode and process the input. 2) Decoder-only architecture, also known as generative architecture, the model is GPT and LLaMA, used for sequence generation tasks, the model directly generates the corresponding sequence from the input encoding, at the same time, the decoder-only architecture also has a feature of unsupervised pre-training, good at creative writing; 3) Encoder-Decoder architecture, also known as sequence-to-sequence architecture, includes Transformer structure. The models are T5 and BART. The model uses encoder to encode the input sequence, extract its features and semantic information, and pass the encoding result to the decoder. Then, the decoder generates the corresponding output sequence according to the encoding result. The advantage is that it can better handle the relationship between input sequence and output sequence, thereby improving the accuracy of tasks such as machine translation and dialogue generation. According to the characteristics of the load forecasting task, PLMs based on Decoder-only and Encoder-Decoder architectures are considered, and PLMs trained in different languages ​​are selected to verify whether natural language expressions affect the forecasting results; S32, model training method; S321, full parameter training method; For language models other than BLOOM, the full parameter training method is used to train the model in the load forecasting dataset. Although this method sacrifices the original problem-solving ability of some of the model's own pre-trained parameters, it performs well in the load forecasting task. S322, PEFT, a method for efficient parameter fine-tuning; The parameter efficient fine-tuning PEFT method is used in combination with the low-rank adaptation LoRA technique of large language models to fine-tune the model parameters. In this method, low-rank decomposition is used to simulate parameter changes based on the parameter distribution of the original model, thereby indirectly training a large model with the least number of parameters. The selected parameter matrix W in the original model is d×k Perform the following processing: W d×k =U d×r ·V r×k r<<d,k (4) Among them, r is the low-rank coefficient, U and V are low-rank matrices; S33, prediction result evaluation method; Since the format of the prediction results is natural language, the numerical sequence in the natural language is extracted and compared with the true value to calculate the prediction accuracy to analyze the performance of the model. The hallucination phenomenon in the prediction results is evaluated by the hallucination rate. In addition, the mean absolute error MAE and root mean square error RMSE are also used to measure the accuracy of the prediction results, which are defined as follows: Where N is the number of samples, Y i represents the i-th prediction result, Y i,gt represents the corresponding true value, H∈(0,1) represents the hallucination rate, n h Indicates the number of hallucinated samples.

Images