Transform model-based power load prediction method and system, and medium

Abstract

The invention discloses a power load prediction method and system based on a Transform model, and a medium, mainly relates to the technical field of power load prediction, and is used for solving the problems that the precision of a traditional model cannot be realized, the precision of the model is reduced, and a large number of resources need to be consumed when the model is trained again due to the fact that a lightweight Transform model is directly used in an existing scheme. Comprising the following steps: training a second Transform model through a trained first Transform model; when power load prediction is carried out, an input sequence and time information of a power load are obtained, the input sequence and the time information are input into a decoder, and coded representation of the power load is obtained; the coding representation is input into a second Transform model, and output prediction data is obtained; meanwhile, kernel smoothness in the model is updated through coding representation; and updating token data in the model through the output prediction data.

Description

Technical Field

[0001] The present application relates to the technical field of power load forecasting, and in particular to a power load forecasting method, system and medium based on a Transformer model. Background Art

[0002] At present, the prediction research of residential power load has become a research hotspot. According to the statistics, the total electricity consumption of the whole society in 2022 is 8637.2 billion kWh, of which the electricity consumption of urban and rural residents is 1336.6 billion kWh, a year-on-year increase of 13.8%. Therefore, effectively predicting the power load is the key to achieve the reasonable dispatch of power resources, avoid the waste of power resources and save the cost of power supply.

[0003] At present, the main scheme for predicting power load using the Transformer model is to collect and process power load data, including data normalization, feature extraction, etc., to establish a power load data set. The Transformer model is used to capture the long-term dependency characteristics of power load data by taking advantage of its feature extraction capabilities and parallel computing advantages. The Transformer model is trained through an encoder-decoder structure to enable it to have prediction capabilities. The real-time target data of the area to be predicted is input into the trained Transformer model to obtain short-term or long-term load prediction values.

[0004] However, the traditional Transformer model involves many calculation rules and consumes a lot of energy during operation; directly using a lightweight Transformer model cannot achieve the accuracy of the traditional Transformer model. In addition, as time goes by and the environment changes, the prediction accuracy of the Transformer model will gradually decrease. Retraining the Transformer model requires a large amount of training data and time to achieve a higher prediction accuracy. Therefore, there is an urgent need for a power load forecasting method, system and medium based on the Transformer model to solve the above-mentioned problems that directly using a lightweight Transformer model cannot achieve the accuracy of the traditional Transformer model, the accuracy of the Transformer model will gradually decrease, and retraining the Transformer model requires a lot of resources. Summary of the invention

[0005] In response to the above-mentioned deficiencies in the prior art, the present application provides a power load forecasting method, system and medium based on the Transformer model to solve the problem that the existing solution directly uses a lightweight Transformer model, cannot achieve the accuracy of the traditional Transformer model, the accuracy of the Transformer model will gradually decrease, and re-training the Transformer model requires a lot of resources.

[0006] In a first aspect, the present application provides a method for power load forecasting based on a Transformer model, the method comprising: The second Transformer model is trained by the trained first Transformer model to obtain the trained second Transformer model; wherein the model parameters, number of layers, number of hidden units and number of attention heads of the second Transformer model are all smaller than those of the first Transformer model; when performing power load forecasting, the input sequence and time information of the power load are obtained, and the input sequence and time information are input into a decoder to obtain a coded representation of the power load; the coded representation is input into the second Transformer model to obtain output prediction data; at the same time, the kernel smoothness in the second Transformer model is updated through the coded representation; and the token data in the second Transformer model is updated through the output prediction data.

[0007] The power load forecasting method provided in the embodiment of the present application trains the second Transformer model by training the first Transformer model. This method may utilize the idea of ​​transfer learning, so that the second Transformer model can achieve higher prediction accuracy with less training resources and time. Although the scale of the second Transformer model is small, the accuracy of the traditional Transformer model can be approached or reached in this way. Directly training a large Transformer model usually requires a lot of computing resources and time. This method reduces the demand for computing resources by first training a larger first Transformer model and then using it to train a smaller second Transformer model. In addition, the power load data changes dynamically, and the model needs to be able to adapt to such changes. The present application can maintain the prediction performance of the model without retraining the entire model by updating the kernel smoothness and token data in the second Transformer model in real time, thereby maintaining or improving the accuracy and reliability of the prediction.

[0008] In one implementation of the present application, training a second Transformer model by using a trained first Transformer model to obtain a trained second Transformer model specifically includes: The training data is input into the first Transformer model to obtain a soft label for each training data; the training data and the soft label are input into the second Transformer model as sample data to obtain a trained second Transformer model.

[0009] In one implementation of the present application, obtaining the input sequence and time information of the power load specifically includes: Get the data set of input power load { , ,...., } and time information; among them, , j∈[1,m], m represents the number of input data, represents the measured value of the j-th input data in the i-th charge-related term, i∈[1,n], and n represents the total number of charge-related terms; the input data is standardized by standard deviation through Standard technology to obtain an input sequence set; wherein the input sequence set is composed of several input sequences.

[0010] In one implementation of the present application, the input sequence and time information are input into a decoder to obtain a coded representation of the power load, specifically including: By presetting the decoder formula: , calculate and obtain the coded representation V of the power load; where j represents the jth input sequence or the jth time information, the input sequence and the time information have a one-to-one relationship, and m represents the number of total input sequences. represents the input sequence corresponding to the jth time information, represents the jth time information, f( ) represents the autoregressive function, Represents the preset adjustment constant corresponding to the j-th input sequence.

[0011] In one implementation of the present application, updating the kernel smoothness in the second Transformer model by encoding representation specifically includes: By formula: , update kernel smoothness ;in, represents the preset expectation function, Indicates the key matrix corresponding to the k value in the second Transformer model The value of the row; Indicates the key matrix corresponding to the Q value in the second Transformer model The value of the row, V represents the encoding representation.

[0012] In one implementation of the present application, updating the token data in the second Transformer model by using the output prediction data specifically includes: By formula: , update the token data in the second Transformer model ; in, Indicates the preset number of iterations, Represents a preset constant, Indicated in The preset constant of times, represents random error, Indicates the previous token data. There is a preset initial token data.

[0013] In a second aspect, the present application provides a power load forecasting system based on a Transformer model, the system comprising: A training module is used to train a second Transformer model through the trained first Transformer model to obtain a trained second Transformer model; wherein the model parameters, number of layers, number of hidden units and number of attention heads of the second Transformer model are all smaller than those of the first Transformer model; a self-training module is used to obtain an input sequence and time information of the power load when performing power load forecasting, input the input sequence and time information into a decoder to obtain a coded representation of the power load; input the coded representation into the second Transformer model to obtain output prediction data; at the same time, update the kernel smoothness in the second Transformer model through the coded representation; and update the token data in the second Transformer model through the output prediction data.

[0014] In one implementation of the present application, the training module includes a training unit, which is used to input the training data into the first Transformer model to obtain a soft label for each training data; The training data and the soft labels are input as sample data into the second Transformer model to obtain a trained second Transformer model.

[0015] In one implementation of the present application, the self-training module includes an updating unit for updating the self-training module by formula: , update kernel smoothness ;in, represents the preset expectation function, Indicates the key matrix corresponding to the k value in the second Transformer model The value of the row; Indicates the key matrix corresponding to the Q value in the second Transformer model The value of the row, V represents the encoding representation.

[0016] In a third aspect, the present application provides a non-volatile computer storage medium having computer instructions stored thereon, which when executed implement a method for power load forecasting based on a Transformer model such as any of the above.

[0017] Those skilled in the art can understand that the present application has at least the following beneficial effects: The present application discloses a method, system and medium for predicting power load based on a Transformer model. The second Transformer model is trained by a trained first Transformer model. This method may utilize the idea of ​​transfer learning, so that the second Transformer model can achieve higher prediction accuracy with less training resources and time. Although the scale of the second Transformer model is small, the accuracy of the traditional Transformer model can be approached or reached in this way. Directly training a large Transformer model usually requires a lot of computing resources and time. This method reduces the demand for computing resources by first training a larger first Transformer model and then using it to train a smaller second Transformer model. In addition, the power load data changes dynamically, and the model needs to be able to adapt to such changes. The present application can maintain the prediction performance of the model without retraining the entire model by updating the kernel smoothness and token data in the second Transformer model in real time, thereby maintaining or improving the accuracy and reliability of the prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Some embodiments of the present disclosure are described below with reference to the accompanying drawings, in which: Figure 1 It is a flow chart of a power load forecasting method based on a Transformer model provided in an embodiment of the present application.

[0019] Figure 2 It is a schematic diagram of the internal structure of a power load forecasting system based on a Transformer model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] It should be understood by those skilled in the art that the embodiments described below are only preferred embodiments of the present disclosure, and do not mean that the present disclosure can only be implemented through the preferred embodiments. The preferred embodiments are only used to explain the technical principles of the present disclosure, and are not used to limit the protection scope of the present disclosure. Based on the preferred embodiments provided by the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work should still fall within the protection scope of the present disclosure.

[0021] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0022] The technical solution proposed in the embodiments of the present application is described in detail below with reference to the accompanying drawings.

[0023] The embodiment provides a method for predicting power load based on the Transformer model, such as Figure 1 As shown, the method provided in the embodiment of the present application mainly includes the following steps: Step 110: Train the second Transformer model using the trained first Transformer model to obtain a trained second Transformer model.

[0024] It should be noted that the model parameters, number of layers, number of hidden units, and number of attention heads of the second Transformer model are all smaller than those of the first Transformer model. The first Transformer model is a traditional large Transformer model, and the second Transformer model is a lightweight Transformer model.

[0025] This step can be specifically as follows: The training data is input into the first Transformer model to obtain a soft label for each training data; the training data and the soft label are input into the second Transformer model as sample data to obtain a trained second Transformer model.

[0026] It should be noted that in order to ensure that all data have a positive effect on the final prediction, the data is first standardized; then, the pre-training parameters are adjusted and the data is input into the first Transformer model. The model uses the generalization of the Transformer structure to extract the main rules (soft labels) in the power load time series data; next, after the second Transformer model is trained, the low-level parameters of the large model are frozen to ensure that all regularity information is preserved intact, freeing up the high-level fully connected layer parameters so that the model can be fine-tuned slightly based on the current data.

[0027] Step 120: When performing power load forecasting, obtain the input sequence and time information of the power load, input the input sequence and time information into a decoder, and obtain a coded representation of the power load.

[0028] It should be noted that parameters associated with the unit load: wind speed, air temperature, humidity, etc. are selected as the power load information.

[0029] The input sequence and time information of the power load here can be specifically: Get the data set of input power load { , ,...., } and time information; among them, , j∈[1,m], m represents the number of input data, represents the measured value of the j-th input data in the i-th charge-related term, i∈[1,n], and n represents the total number of charge-related terms; the input data is standardized by standard deviation through Standard technology to obtain an input sequence set; wherein the input sequence set is composed of several input sequences.

[0030] It is understood by those skilled in the art that standard normalization is also called standard deviation normalization. Its main idea is to process the standard data that obeys the normal distribution through the mean and variance, and finally obtain data that conforms to the standard normal distribution. Such data is easier to obtain gradients during the training process, making the training effect more obvious.

[0031] It should be noted that the charge-related items may be wind speed, temperature, and humidity data.

[0032] In the step, the input sequence and time information are input into the decoder to obtain the coded representation of the power load, which can be specifically: By presetting the decoder formula: , calculate and obtain the coded representation V of the power load; where j represents the jth input sequence or the jth time information, the input sequence and the time information have a one-to-one relationship, and m represents the number of total input sequences. represents the input sequence corresponding to the jth time information, represents the jth time information, f( ) represents the autoregressive function, Represents the preset adjustment constant corresponding to the j-th input sequence.

[0033] Step 130: input the encoded representation into the second Transformer model to obtain output prediction data; at the same time, update the kernel smoothness in the second Transformer model through the encoded representation; and update the token data in the second Transformer model through the output prediction data.

[0034] In the step, the kernel smoothness in the second Transformer model is updated by encoding, which can be specifically: By formula: , update kernel smoothness ;in, represents the preset expectation function, Indicates the key matrix corresponding to the k value in the second Transformer model The value of the row; Indicates the key matrix corresponding to the Q value in the second Transformer model The value of the row, V represents the encoding representation.

[0035] The token data in the second Transformer model is updated by the output prediction data, which can be specifically: By formula: , update the token data in the second Transformer model ;in, Indicates the preset number of iterations, Represents a preset constant, Indicated in The preset constant of times, represents random error, Indicates the previous token data. There is a preset initial token data.

[0036] It should be noted that the autoregressive next time point prediction target is used as a pre-training task here, so that the model can learn the inherent laws of time series data, thereby achieving higher accuracy in the actual power load forecasting task.

[0037] In addition, this application Figure 2 The present invention provides a power load forecasting system based on a Transformer model. Figure 2 As shown, the system provided in the embodiment of the present application mainly includes: The training module 210 is used to train the second Transformer model through the trained first Transformer model to obtain a trained second Transformer model; wherein the model parameters, number of layers, number of hidden units and number of attention heads of the second Transformer model are all smaller than those of the first Transformer model.

[0038] The training module 210 includes a training unit, Used to input the training data into the first Transformer model to obtain the soft label of each training data; The training data and the soft labels are input as sample data into the second Transformer model to obtain a trained second Transformer model.

[0039] The self-training module 220 is used to obtain the input sequence and time information of the power load when performing power load forecasting, input the input sequence and time information into the decoder to obtain the encoded representation of the power load; input the encoded representation into the second Transformer model to obtain output prediction data; at the same time, update the kernel smoothness in the second Transformer model through the encoded representation; and update the token data in the second Transformer model through the output prediction data.

[0040] The self-training module 220 includes an updating unit, Used by the formula: , update kernel smoothness ; in, represents the preset expectation function, Indicates the key matrix corresponding to the k value in the second Transformer model The value of the row; Indicates the key matrix corresponding to the Q value in the second Transformer model The value of the row, V represents the encoding representation.

[0041] In addition, an embodiment of the present application further provides a non-volatile computer storage medium on which executable instructions are stored. When the executable instructions are executed, a power load forecasting method based on a Transformer model as described above is implemented.

[0042] So far, the technical solutions of the present disclosure have been described in combination with the above multiple embodiments, but it is easy for those skilled in the art to understand that the protection scope of the present disclosure is not limited to these specific embodiments. Without departing from the technical principles of the present disclosure, those skilled in the art can split and combine the technical solutions in the above-mentioned various embodiments, and can also make equivalent changes or replacements to the relevant technical features. Any changes, equivalent replacements, improvements, etc. made within the technical concept and / or technical principle of the present disclosure will fall within the protection scope of the present disclosure.

Claims

1. A power load forecasting method based on Transformer model, characterized in that: The method comprises: By training the second Transformer model with the trained first Transformer model, a trained second Transformer model is obtained; wherein the model parameters, the number of layers, the number of hidden units, and the number of attention heads of the second Transformer model are all smaller than those of the first Transformer model; When performing power load forecasting, an input sequence and time information of the power load are obtained, and the input sequence and time information are input into a decoder to obtain a coded representation of the power load; The encoded representation is input into the second Transformer model to obtain output prediction data; at the same time, the kernel smoothness in the second Transformer model is updated by the encoded representation; and the token data in the second Transformer model is updated by the output prediction data.

2. The power load forecasting method based on the Transformer model according to claim 1 is characterized in that: The second Transformer model is trained by using the trained first Transformer model to obtain the trained second Transformer model, which specifically includes: Input the training data into the first Transformer model to obtain the soft label of each training data; The training data and the soft labels are input as sample data into the second Transformer model to obtain a trained second Transformer model.

3. The power load forecasting method based on the Transformer model according to claim 1 is characterized in that: Obtain the input sequence and time information of the power load, including: Get the data set of input power load { , ,...., } and time information; among them, , j∈[1,m], m represents the number of input data, represents the measured value of the j-th input data in the i-th charge-related term, i∈[1,n], and n represents the total number of charge-related terms; The input data is standardized by standard deviation through Standard technology to obtain an input sequence set; wherein the input sequence set is composed of several input sequences.

4. The power load forecasting method based on the Transformer model according to claim 1 is characterized in that: The input sequence and time information are input into the decoder to obtain the coded representation of the power load, which includes: By presetting the decoder formula: , calculate and obtain the coded representation V of the power load; Among them, j represents the jth input sequence or the jth time information, the input sequence and the time information have a one-to-one relationship, m represents the total number of input sequences, represents the input sequence corresponding to the jth time information, represents the jth time information, f( ) represents the autoregressive function, Represents the preset adjustment constant corresponding to the j-th input sequence.

5. The power load forecasting method based on the Transformer model according to claim 1 is characterized in that: Update the kernel smoothness in the second Transformer model through the encoded representation, specifically including: By formula: , update kernel smoothness ; in, represents the preset expectation function, Indicates the key matrix corresponding to the k value in the second Transformer model The value of the row; Indicates the key matrix corresponding to the Q value in the second Transformer model The value of the row, V represents the encoding representation.

6. The power load forecasting method based on the Transformer model according to claim 1 is characterized in that: Update the token data in the second Transformer model with the output prediction data, including: By formula: , update the token data in the second Transformer model ; in, Indicates the preset number of iterations, Represents a preset constant, Indicated in The preset constant of times, represents random error, Indicates the previous token data. There is a preset initial token data.

7. A power load forecasting system based on Transformer model, characterized in that: The system comprises: A training module, used for training a second Transformer model by using the trained first Transformer model to obtain a trained second Transformer model; wherein the model parameters, the number of layers, the number of hidden units, and the number of attention heads of the second Transformer model are all smaller than those of the first Transformer model; The self-training module is used to obtain the input sequence and time information of the power load when predicting the power load, input the input sequence and time information into the decoder to obtain the encoded representation of the power load; input the encoded representation into the second Transformer model to obtain the output prediction data; at the same time, update the kernel smoothness in the second Transformer model through the encoded representation; and update the token data in the second Transformer model through the output prediction data.

8. The power load forecasting system based on the Transformer model according to claim 7 is characterized in that: The training module includes training units, Used to input the training data into the first Transformer model to obtain the soft label of each training data; The training data and the soft labels are input as sample data into the second Transformer model to obtain a trained second Transformer model.

9. The power load forecasting system based on the Transformer model according to claim 7, characterized in that: The self-training module includes an update unit, Used by the formula: , update kernel smoothness ; in, represents the preset expectation function, Indicates the key matrix corresponding to the k value in the second Transformer model The value of the row; Indicates the key matrix corresponding to the Q value in the second Transformer model The value of the row, V represents the encoding representation.

10. A non-volatile computer storage medium, characterized in that: Computer instructions are stored thereon, and when the computer instructions are executed, the method for predicting electric load based on the Transformer model as described in any one of claims 1 to 6 is implemented.

Images