A pre-training model construction method for power system source and load prediction

Abstract

The present invention discloses a method for constructing a pre-training model for power system source-load prediction, which belongs to the field of power system time series prediction. The pre-training model aims at the problem of uncertainty on both the source and load sides brought about by the access of new energy and the development of new loads, and constructs a heterogeneous combination architecture of a spatiotemporal pattern hybrid expert sub-model and a forecast information hybrid expert sub-model, which can adapt to a variety of downstream prediction scenarios and meet the prediction needs of multiple time scales and multi-variable inputs. The present invention extracts universal spatiotemporal features from massive data through a pre-training model, effectively improving the accuracy and generalization performance of source-load prediction, adapting to the needs of multiple scenarios such as planning and scheduling, and providing reliable technical support for the awareness of the operational risk situation of distribution networks containing a high proportion of distributed resources.

Description

Technical Field

[0001] The present invention relates to a method for constructing a pre-training model for power system source-load prediction. The method is suitable for constructing a pre-training model for multi-temporal and spatial scale prediction tasks of power system source-load time series data, and belongs to the field of power system time series prediction. Background Art

[0002] Power system source-load forecasting is a core foundational technology supporting the safe, economical, and low-carbon operation of modern power systems. It provides a strong guarantee for the accuracy of dispatch decisions and the stability of operational control. With the influx of renewable energy sources connected to the grid, the uncertainty facing the system's power supply side has significantly increased. With the increasing ownership of electric vehicles and the development of new loads such as household energy storage, the uncertainty on the system's load side is also increasing.

[0003] With the massive amount of data and high-dimensional features, the difficulty of achieving accurate source and load forecasting has increased, and higher requirements have been placed on the parameter scale and performance of the forecasting model. In addition, the requirements for source and load forecasting vary in different application scenarios such as planning, scheduling, optimization, and control, and the forecasting tasks in different time and space vary greatly. Traditional time series forecasting methods usually choose to build dedicated small-parameter scale forecasting models for different forecasting tasks and scenarios, which is time-consuming and labor-intensive and has poor performance in terms of forecast versatility and generalization. In the context of a large number of distributed resources being connected to the power system, it is urgent to build a pre-trained model with strong learning ability, high prediction accuracy, strong versatility, and applicable to multi-time and space scale forecasting tasks to improve the predictive perception of the distribution network operation risk situation. Summary of the Invention

[0004] To address the issues outlined in the background art, the present invention proposes a method for constructing a pre-trained model for power system source-load prediction, and designs a pre-trained model framework suitable for power system source-load time series data. This pre-trained model structurally supports the input and prediction output of single and multivariate variables, and can perform prediction tasks at multiple time scales, including ultra-short-term, short-term, and medium- to long-term. During the pre-training process, a prediction model composed of heterogeneous hybrid expert sub-models based on a self-attention mechanism is able to learn general knowledge from massive time series data, resulting in stronger generalization performance and broader downstream application scenarios.

[0005] In the field of power systems, source-load data specifically includes photovoltaic power, wind power, and load data at different voltage levels. The classification of historical data and forecast data, as well as the data input of the pre-trained model in different prediction scenarios, are summarized in the following table:

[0006]

[0007] A method for constructing a pre-trained model for power system source and load prediction is as follows:

[0008] From a framework perspective, the pre-trained model consists of two heterogeneous hybrid expert sub-models: a spatiotemporal pattern hybrid expert sub-model and a forecast information hybrid expert sub-model. These are used to process historical data and forecast data, respectively, in the forecasting task, adapting to complex downstream data scenarios. The sub-model design should also fully consider different forecasting scenarios for single-variable and multi-variable inputs, and adapt to different forecasting tasks. The construction of the pre-trained model specifically includes the following steps:

[0009] Step 1: Construct the spatiotemporal pattern hybrid expert sub-model and the forecast information hybrid expert sub-model respectively.

[0010] The spatiotemporal pattern hybrid expert sub-model uses a Transformer decoder as its backbone and employs a causal masked attention mechanism to learn correlations between input time series data segments. The model's unique characteristics are reflected in its output structure, which employs a multi-degree-of-freedom hybrid expert Student-T distribution output head, each with its own independent parameter mapping network. Multiple degrees of freedom include high, medium, and low degrees of freedom. The high-degree-of-freedom Student-T distribution output head is used to fit the distributional characteristics of stationary load data; the medium-degree-of-freedom Student-T distribution output head is used to fit scenarios with periodic fluctuations and light tails; and the low-degree-of-freedom Student-T distribution output head is used to fit heavy-tailed features with frequent extreme values. The Student-T distribution is chosen as it is more stable in scenarios with outliers or offsets in time series data and is therefore more suitable for pre-training with massive data input. The introduction of distribution output heads with different degrees of freedom improves the model's adaptability during pre-training.

[0011] The forecast information hybrid expert sub-model uses a single feedforward neural network as its basic expert unit and is composed of multiple (e.g., 54) expert models. These models can be further divided into three independent expert model groups: photovoltaic, wind power, and load, each processing forecast information for its respective domain. The structural design of the load, photovoltaic, and wind power expert model groups incorporates numerous basic forecasting units. During pre-training, they learn complex weather-power mappings across different scenarios, enabling strong generalization performance across tasks and scenarios. The forecast information hybrid expert sub-model utilizes a two-stage routing network design, enabling model guidance from task type to fine-grained prediction features. This overcomes the limitations of traditional single-stage forecasting models and better meets the forecasting needs of multi-level, multi-factor, and complex operating conditions in power systems. During the forecast and training process, the incoming forecast information is distributed to the expert model groups in different domains through the main routing network in the first stage. In the second stage, the sub-routing networks within the current expert model group further select the optimal expert model to perform the forecast. The final forecast result is output in a parameterized Gaussian form.

[0012] Step 2: Create a pre-training dataset. The pre-training dataset used for pre-training model training consists of two parts: one is an open-source time series dataset available on the internet (specific sources include energy, Internet of Things, health, network, transportation, and environment); the other is real photovoltaic, wind power, and load data of different voltage levels in the power system. Most open-source datasets do not contain forecast information, and the information that can be provided in the forecasting task is mainly historical data; the real operating data in the power system includes real data on photovoltaic, wind power, and load, as well as supporting meteorological forecast information (including wind speed, irradiance, temperature, and humidity data). In the pre-training dataset construction phase, historical data is extracted from open-source data and power system data to construct a pre-training dataset for the spatiotemporal pattern hybrid expert sub-model, and forecast data is extracted to construct a pre-training dataset for the forecast information hybrid expert sub-model.

[0013] Step 3: Pre-train the two sub-models based on the pre-training dataset constructed in Step 2, using the negative log-likelihood loss function. During pre-training, verify the prediction performance of the pre-trained models until they reach the optimal value on the validation dataset. After training, save the parameters of the two sub-models to obtain the pre-trained models. Using datasets from different domains during sub-model pre-training allows the sub-models to learn different spatiotemporal features.

[0014] Furthermore, the final pre-trained model structure and fine-tuning plan for deployment can be determined based on the hardware resources, forecast requirements, and available data of the downstream deployment application scenario. The data that the downstream application scenario can provide for model prediction is the key to determining the pre-trained model structure and fine-tuning plan.

[0015] The specific solutions are summarized in the table below. In the table, STP represents the spatiotemporal pattern hybrid expert sub-model; MIM represents the forecast information hybrid expert sub-model. When the application scenario only provides historical data for model prediction (for example, small-capacity distributed photovoltaic operators who cannot afford the high cost of weather forecast services), the spatiotemporal pattern hybrid expert sub-model is selected. The pre-trained weights of this sub-model are loaded, and the model is fine-tuned using the target scenario's historical data. When the application scenario has no or limited historical data (for example, a newly commissioned wind farm with limited historical wind power data but a purchased weather forecast service), the forecast information hybrid expert sub-model is selected. The parameters of this sub-model are loaded, and the model is fine-tuned using the target scenario's forecast data. When both historical and forecast data are available and sufficient, a combination of the spatiotemporal pattern hybrid expert sub-model and the forecast information hybrid expert sub-model is selected to achieve the highest prediction accuracy. The integrated fine-tuning technique of the pre-trained model is used to align the scales of the two sub-models and complement their strengths.

[0016]

[0017] In terms of hardware resources and prediction timeliness, the network depth of the model can be adjusted in combination with the computing power and memory limitations of the server deployed in the downstream application scenarios. The solution of partially loading pre-trained model parameters can reduce model complexity and computing power requirements and improve prediction speed.

[0018] Based on the selected deployment plan and model fine-tuning, the model is tested using real data from the application scenario to ensure that it meets acceptance requirements in terms of speed and prediction accuracy. The model is deployed offline to the server and the time series prediction service is launched online.

[0019] The present invention also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the pre-training model construction method for power system source and load prediction.

[0020] The present invention also provides a computer-readable storage medium having computer instructions stored thereon, wherein the computer instructions are used to enable a computer to execute the method for constructing a pre-training model for power system source-load prediction.

[0021] The beneficial effects of the present invention are:

[0022] The present invention proposes a method for constructing a pre-training model for power system source and load prediction. The pre-training model proposed in the present invention is composed of heterogeneous hybrid expert sub-models. While maintaining the simplicity of the sub-model structure and the uniqueness of the function, it can also realize parallel training, and effectively speed up the training iteration cycle of the pre-training model while ensuring the model prediction accuracy. In addition, the present invention also proposes a complete pre-training model application solution, which fully considers the complex data scenarios, prediction requirements and computing power limitations in the actual downstream application scenarios, and is compatible with a rich variety of downstream application scenarios by selecting different model structures and fine-tuning solutions, and has extremely strong versatility. The pre-training model of the present application supports the input and prediction output of single variables and multivariate variables, and can perform prediction tasks at multiple time scales such as ultra-short-term, short-term and medium- and long-term. The present invention extracts universal spatiotemporal features from massive data through a pre-training model, effectively improves the accuracy and generalization performance of source and load prediction, adapts to the needs of multiple scenarios such as planning and scheduling, and provides reliable technical support for the awareness of the operation risk situation of distribution networks with a high proportion of distributed resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A process for building a pre-trained model suitable for multi-spatiotemporal scale prediction of power system source and load data;

[0024] Figure 2 shows the structure of the spatiotemporal pattern hybrid expert sub-model;

[0025] Figure 3 is a schematic diagram of the attention score causal mask

[0026] Figure 4 shows the forecast information hybrid expert sub-model structure; DETAILED DESCRIPTION

[0027] The present invention will be further described below with reference to the accompanying drawings and implementation examples.

[0028] Example 1 provides a method for constructing a pre-trained model for power system source and load prediction.

[0029] The construction and deployment process of the pre-trained model is as follows Figure 1 As shown, it specifically includes the following steps:

[0030] Step 1: Construct the forecast information hybrid expert sub-model and the spatiotemporal pattern hybrid expert sub-model. This includes the following two parallel steps:

[0031] (1) Construct a spatiotemporal hybrid expert sub-model. The specific structure of the model is as follows: Figure 2 As shown. Assume that the input historical data is , B represents the number of input batches; S represents the length of the input historical sequence; C represents the number of input channels. The input data is first sliced and divided into N time series segments of the same length P along the S dimension. At this time, the input data becomes the sliced historical data :

[0032]

[0033] The dimension N should satisfy: The quantitative relationship of each data segment is marked to form Tag information ,in Used to mark which input channel the current sequence segment belongs to; Used to mark the input order of the current clip in the corresponding channel. Perform dimension changes and merge N and C dimensions. The dimension changes to The input embedding layer is mapped to the Transformer decoder (the dimension of the backbone network is D):

[0034]

[0035] In the formula is the mapping weight of the linear transformation of the embedding layer, is the bias coefficient. The feature mapping results This is then fed into the backbone network of the multi-layer decoder Transformer architecture.

[0036] Input Transformer architecture Obtained through three sets of independent linear mappings , and Matrix, the size of Split the matrix into NH heads along dimension D, where NH is the number of heads in the preset multi-head attention mechanism. The splitting process satisfies , where is the dimension of the attention head. , and The matrices are used , and Indicates that its dimensions become . Use the split matrix to perform attention calculation:

[0037]

[0038] is the calculated attention score. To ensure the causality of the calculation process and adapt to multi-channel data input, the attention score also needs to perform the operation of adding bias and mask according to the label information. Attention scores, each row and column has a corresponding number Label information of . Select a data point in the attention score , let the label information of the row be , the label information of its column is The logic for adding bias is:

[0039]

[0040] In the formula is the data point after adding the bias, It is a conditional item. When the condition in the brackets is met, it is 1, and when it is not met, it is 0. The logic of this formula is: when the channel dimensions of the row and column label information of the data point are equal, it means that the point comes from the attention score calculation results of two segments of the same channel, and add When the row and column label information of a data point is different, it means that the point comes from the attention score calculation results of two different channel segments, and add Through this processing, the attention mechanism can distinguish and learn different pattern features in the time dimension (i.e., the same channel input) and the spatial dimension (i.e., different channel input).

[0041] The logic for adding the mask is:

[0042]

[0043] In the formula is the data point after the mask operation. The logic of this formula is: when the time series dimensions of the row and column label information of the data point are not equal and the input sequence number of the column label information is greater than the input sequence number of the row channel, it means that the point is the attention score calculated by the future time series segment and the historical time series segment, which violates the causality (that is, the time series segment can only calculate the attention with the historical sequence with a smaller input sequence number than itself, but cannot calculate the attention with the future segment with a larger input sequence number than itself), so it needs to be assigned a value of 0. Otherwise, it is a reasonable result and the attention score is retained. Figure 3 As shown in the figure, the process of calculating the attention scores of the fragment (1,3) and all historical fragments is marked with solid arrows, which conforms to causality and therefore does not require masking; while the attention scores calculated for the fragment (1,3) and future fragments are marked with dotted lines, which do not conform to causality and need to be masked to zero.

[0044] Traversal For each element in , add bias and mask operations according to the above process. Perform normalization and introduce The weighted summation gives the attention output:

[0045]

[0046] In the formula is the attention output. The multi-head output results are merged and the size becomes Attention Output After residual connection and layer normalization, it is input into the feedforward neural network (FFN):

[0047]

[0048]

[0049] in, Output for attention The result after residual connection and layer normalization; is the output of the feedforward neural network; is the feature input to the decoder by the embedding layer, used for the attention output Residual connection, that is ;GELU is the activation function; and Represents the weight coefficient of the linear mapping inside and outside the FFN network, and Represents the bias coefficient of the linear mapping inside and outside the FFN network.

[0050] The output of the feedforward neural network is again processed by residual connection and layer normalization before output.

[0051] like Figure 2 As shown, the decoder structure adopts the same model structure of multi-layer stacking, and returns the output features of the final output backbone network after N-layer network processing. :

[0052]

[0053] The output features of the backbone network Input the multi-degree-of-freedom hybrid expert student T distribution output head, and use the routing network to calculate the weight g of each student T distribution output head. The process is expressed in the following formula:

[0054]

[0055] Where, represents the Student T distribution output head weight calculated by the routing network, is the routing network weight matrix, is the routing bias coefficient, It is the smoothing coefficient, which is set to 1 by default. The larger the smoothing coefficient is, the smaller the calculated weight difference is.

[0056] is the output feature of the backbone network Select the Student T distribution output head with the largest weight:

[0057]

[0058] Student's T distribution output head screening along the weights The last dimension is performed. is the output feature of the backbone network The corresponding optimal Student's T distribution output head number, The integer values correspond to the Student's T distribution output heads with low degrees of freedom, medium degrees of freedom, and high degrees of freedom, respectively. Contains The length is Data fragment, output head number according to the optimal Student T distribution Assign an output head to each data segment. Inside the output head, a separate linear mapping layer is used to map the input data segment to the parameters of the Student T distribution in the prediction space. Select a clip , let the fragment be assigned to the output header , then the formula for this distribution parameter mapping process is as follows:

[0059]

[0060]

[0061]

[0062] Where, , and Indicates output header The corresponding degrees of freedom, standard deviation, and mean of the Student's T distribution; , and Respectively represent the output head The corresponding weight matrix of the degrees of freedom, standard deviation and mean parameter mapping of the Student's T distribution, , and For output header The corresponding bias coefficient of the Student's T distribution's degrees of freedom, standard deviation, and mean parameter mapping; For output header The lower limit of the degrees of freedom of the Student's T distribution is used to set the degree of freedom range of the Student's T distribution output header.

[0063]

[0064] The three Student's T distribution output heads with different degrees of freedom share the same structural design, but are independent of each other in terms of model parameters. The degrees of freedom, standard deviation, and mean parameters are used to instantiate the Student's T distribution, i.e., the predictive distribution.

[0065]

[0066] Student's T distribution based on predictions implement Subsampling, you can get an array ,in, To predict the distribution Execute The result obtained by sampling. The probability prediction result in the form of quantiles can be obtained by counting the data distribution in the array; the median of each sampling moment of the array is selected as the deterministic prediction result output .

[0067] (2) Construct a forecast information hybrid expert sub-model.

[0068] Assume that the input forecast data is , B represents the number of input batches; S represents the length of the input historical sequence; C represents the number of input channels. The input data also needs to be sliced, and it is divided into N time series segments of the same length P along the S dimension. At this time, the input data becomes the sliced forecast data :

[0069]

[0070] The dimension N should satisfy: The data identification tag is used to distinguish the input data of different channels, including load (tag=01), photovoltaic (tag=02) and wind power (tag=03). The time series segments are mapped to a unified dimension, and the corresponding data identification bias coefficient is added to the time series segments according to the data type. , thereby incorporating data identification features into time series segments:

[0071]

[0072]

[0073] Where, , and are the bias coefficients under three data identifications, and are the linear mapping weight coefficient and bias coefficient of the feature mapping layer respectively; is the data identification bias coefficient. The forecast data that integrates the data identification features It is then input into the main routing network of the forecast information hybrid expert sub-model, and the main routing network selects the corresponding expert model group based on the data identifier. This process is shown as follows:

[0074]

[0075]

[0076] Where, Score the expert model, and The mapping matrix and bias weights of the main routing network; , indicating the number of the expert model, there are expert models, and the optimal expert model number selected after the sub-routing network operation is .

[0077] Subsequently, the sub-routing network within the expert model group will further select the optimal expert model for the input Perform operations and output the Gaussian distribution mean of the prediction space and standard deviation .

[0078]

[0079]

[0080] Among them, an expert model includes two FFN networks with independent parameters, which are used to calculate the mean and standard deviation of the Gaussian distribution respectively. and Represents the optimal expert model The two weight matrices of the FFN network used to calculate the standard deviation, and Represents the optimal expert model Two bias coefficients of the FFN network used to calculate the standard deviation; and Represents the optimal expert model The two weight matrices of the FFN network used to calculate the mean of the Gaussian distribution are: and Represents the optimal expert model The two bias coefficients of the FFN network used to calculate the mean of the Gaussian distribution.

[0081] This instantiation of a Gaussian distribution is done using its mean and standard deviation:

[0082]

[0083] Gaussian distribution based on predictions Execution Subsampling, you can get an array ,in, To predict the distribution Execute The result obtained by sampling. The probability prediction result in the form of quantiles can be obtained by counting the data distribution in the array; the median of each sampling moment of the array is selected as the deterministic prediction result output .

[0084] Step 2: Based on the model constructed in Step 1, further construct a pre-training dataset. Normalize the open-source time series dataset to a sequence with a mean of 0 and a variance of 1. In the power system dataset, align the weather forecast data (wind speed, wind direction, irradiance, temperature, and humidity) with the timestamps of the corresponding photovoltaic, wind power, and load data.

[0085] For the spatiotemporal pattern hybrid expert sub-model, 70% of open source data sets and 30% of power historical data were selected when constructing the pre-training data set, which was divided in chronological order. The first 80% was the training set, the middle 10% was the validation set, and the last 10% was the test set.

[0086] For the forecast information hybrid expert sub-model, the pre-training dataset was constructed using only paired samples of forecast and actual values from the power system data. Sampling was stratified by site, ensuring a 7:2:1 ratio of site data in the training, validation, and test sets. In the photovoltaic forecasting scenario, input features include forecast information such as irradiance and temperature over the forecast period. In the wind power forecasting scenario, input features include forecast information such as hub high wind speed and wind direction over the forecast period. In the load forecasting scenario, input features also include forecast information such as temperature, holiday identification, weather, and humidity.

[0087] Step 3: Based on the pre-training dataset constructed in Step 2, complete model parameter initialization and pre-training. The spatiotemporal pattern hybrid expert sub-model uses a 24-layer Transformer decoder, each layer containing 8 attention heads. The model backbone dimension is 1024, and the attention head dimension is 128. The hybrid expert distribution output head uses three sets of Student-T distribution output heads with different degrees of freedom. Each Student-T distribution output head uses three independent linear mapping networks to obtain the distribution parameters. The AdamW optimizer is used during training, with an initial learning rate of 5e-5, a learning rate decay coefficient of 0.01, and a cosine annealing learning rate scheduling strategy. The negative log-likelihood is used as the loss function for model training.

[0088] The forecast information hybrid expert sub-model includes 54 FFN expert models, divided equally into three expert model groups, responsible for photovoltaic, wind power, and load forecasting, respectively. Training is divided into two phases. In the first phase, the routing network parameters are fixed, and only the expert models in each field are trained to achieve predictive performance in their respective fields. A learning rate of 1e-4 is set. In the second phase, the routing network is introduced into the training, and the entire hybrid expert model is jointly optimized. The learning rate is set to 3e-5. The AdamW optimizer is also used during training, and the input data batches can be selected from the range of [32-4096] based on the dataset size and computing power constraints.

[0089] Based on the above pre-trained model, power system source and load forecasting can be achieved. The available data obtained from the specific application scenario is input into the fine-tuned pre-trained model to perform prediction and obtain the prediction results.

[0090] Based on the available data of the downstream application scenario, the final model structure and fine-tuning scheme for deployment can be confirmed, which includes three types in total:

[0091] (1) Fine-tune and deploy the forecast information hybrid expert sub-model. Use the forecast-actual value paired data of the target scenario to build a fine-tuning dataset. Load the pre-trained model parameters after training, freeze the forecast hybrid expert model parameters, and only fine-tune the main routing network and sub-routing network. The trainable parameters are The base learning rate is set to 1e-5, and it decreases by 5% per epoch during fine-tuning. If the negative log-likelihood loss of the validation set does not decrease for 5 consecutive epochs, fine-tuning is stopped.

[0092] (2) Fine-tune and deploy the spatiotemporal pattern hybrid expert sub-model. During the fine-tuning process, the trainable parameters are Specifically, the multi-degree-of-freedom hybrid expert Student-T distribution output head and the last three layers of the Transformer network are frozen. The remaining model parameters are loaded. The pre-trained model parameters are loaded after training, and a fine-tuning dataset is constructed using actual historical data from downstream. Depending on the validation set performance during fine-tuning, redundant Student-T distribution output heads can be disabled. The base learning rate is set to 1e-5, decreasing by 5% per epoch during fine-tuning. Fine-tuning is terminated if the negative log-likelihood loss on the validation set does not decrease for five consecutive epochs.

[0093] (3) Integrate fine-tuning and deploy dual models. Build a hybrid architecture based on dual-model prediction and perform joint fine-tuning. Let the trainable parameters of the spatiotemporal pattern hybrid expert sub-model be , the trainable parameters of the forecast information hybrid expert sub-model are The prediction output of the ensemble model is expressed as:

[0094]

[0095]

[0096]

[0097] Where, predict dynamic weights for a trainable ensemble, and These are the prediction results of the spatiotemporal pattern hybrid expert sub-model and the forecast information hybrid expert sub-model, respectively. The prediction output of the ensemble model is as follows: The ensemble fine-tuning is divided into two stages: the first stage uses 30% of the training samples and freezes the weights. and , only adjust the dynamic weights of the ensemble predictions In the second stage, 70% of the training samples are used to jointly optimize all model parameters, and the loss function is set to:

[0098]

[0099] In the formula and are the negative log-likelihood loss functions of the spatiotemporal pattern hybrid expert sub-model and the forecast information hybrid expert sub-model, respectively. Represents the KL divergence operation, which is used to align the prediction distributions of the two sub-models. The attention coefficient for integrated fine-tuning is gradually increased from 0.3 to 1.0 during the fine-tuning process, guiding the attention of fine-tuning parameter updates to gradually shift from the prediction effect of the sub-model to the prediction effect of the integrated model.

[0100] Furthermore, during the model parameter loading phase, the number of model layers and experts can be reduced to accommodate limited hardware resources. Model validation testing is performed, with probabilistic evaluation metrics such as the prediction error band coverage and average interval width of the fine-tuned model calculated in real time. Once the fine-tuned model meets the accuracy requirements of downstream applications for three consecutive days, it can be deployed. Conversely, if the model fails to meet accuracy requirements during continuous testing, a model rollback mechanism is activated to adjust fine-tuning parameters and re-train.

[0101] Example 2

[0102] The present invention also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the pre-training model construction method for power system source and load prediction.

[0103] Example 3

[0104] The present invention also provides a computer-readable storage medium having computer instructions stored thereon, wherein the computer instructions are used to enable a computer to execute the method for constructing a pre-training model for power system source-load prediction.

[0105] The above description of the specific implementation methods of the present invention in conjunction with the accompanying drawings is not intended to limit the scope of protection of the present invention. All equivalent models or equivalent algorithm processes made using the contents of the present invention specification and accompanying drawings, which are directly or indirectly applied to other related technical fields, are within the scope of patent protection of the present invention.

Claims

1. A method for constructing a pre-training model for power system source and load prediction, characterized in that: The following steps are involved: A spatiotemporal pattern hybrid expert sub-model and a forecast information hybrid expert sub-model are constructed respectively; the spatiotemporal pattern hybrid expert sub-model adopts a multi-degree-of-freedom hybrid expert Student T distribution output head, and the multi-degree-of-freedom hybrid expert Student T distribution output head includes a high-degree-of-freedom Student T distribution output head, a medium-degree-of-freedom Student T distribution output head and a low-degree-of-freedom Student T distribution output head. The high-degree-of-freedom Student T distribution output head is used to fit the distribution characteristics of load-type stable data; the medium-degree-of-freedom Student T distribution output head is used to fit the scenario with periodic fluctuations and a lighter distribution tail; the low-degree-of-freedom Student T distribution output head is used to fit the thick-tail characteristics with frequent extreme values; the forecast information hybrid expert sub-model adopts a two-stage routing network design. In the first stage, the main routing network distributes the input data to the corresponding expert model group according to the data type. In the second stage, the sub-routing network within the expert model group further selects the optimal expert model to process the input data; Integrate open source time series datasets with real power system source and load data to create pre-training datasets; Pre-training the spatiotemporal pattern hybrid expert sub-model and the forecast information hybrid expert sub-model based on the pre-training data set; After the training is completed, save the model parameter file formed during the pre-training process to obtain the pre-trained model; The main routing network of the forecast information hybrid expert sub-model selects the forecast data X that has the data identification feature according to the data identification. F,tag The corresponding expert model group; the process is expressed as follows: g sub =softmax(X F,tag ·W sub +b sub ); Where g sub Score the expert model, W sub with b sub is the mapping matrix and bias weight of the sub-routing network; m=1,2,...,M, represents the number of the expert model. There are M expert models in the expert model group. The optimal expert model number selected after the sub-routing network operation is Subsequently, the sub-routing network within the expert model group further selects the optimal expert model for the input X F,tag To perform calculations, each expert model includes two FFN networks with independent parameters, which are used to calculate the Gaussian distribution mean of the prediction space. and standard deviation Where, and Represents the optimal expert model The two weight matrices of the FFN network used to calculate the standard deviation, and Represents the optimal expert model Two bias coefficients of the FFN network used to calculate the standard deviation; and Represents the optimal expert model The two weight matrices of the FFN network used to calculate the mean of the Gaussian distribution are: and Represents the optimal expert model The two bias coefficients of the FFN network used to calculate the mean of the Gaussian distribution.

2. The method for constructing a pre-training model for power system source-load prediction according to claim 1, characterized in that: The pre-training data set includes a pre-training data set of a spatiotemporal pattern hybrid expert sub-model and a pre-training data set of a forecast information hybrid expert sub-model; the pre-training data set of the spatiotemporal pattern hybrid expert sub-model specifically includes historical data extracted from an open source time series data set and real source and load data of the power system; the pre-training data set of the forecast information hybrid expert sub-model includes forecast data and actual data extracted from real source and load data of the power system.

3. The method for constructing a pre-training model for power system source-load prediction according to claim 1, characterized in that: The forecast data X fused with data identification features F,tag The construction method is: Assume that the input forecast data is X F ∈R B×S×C , B represents the number of batches of input; S represents the length of the historical sequence of input; C represents the number of channels of input; X is converted along the S dimension F Divide into N time series segments of the same length P, and obtain the sliced forecast data X F,patch : X F,patch ∈R B×N×P×C ; Among them, the dimension N satisfies: N = S / P; Introduce data identification tags to distinguish different types of input data, including load, photovoltaic and wind power; map the time series segments to a unified dimension, and add corresponding data identification bias coefficients E to the time series segments according to the data type. tag , thereby integrating data identification features into the time series segment and obtaining forecast data X fused with data identification features F,tag : X F,tag =X F,patch ·W e +b e +E tag ; Among them, W e and b e are the linear mapping weight coefficient and bias coefficient of the feature mapping layer respectively; The forecast information hybrid expert sub-model realizes Gaussian distribution prediction based on mean and standard deviation: Where, P F is the predicted Gaussian distribution; Based on the predicted Gaussian distribution P F Execute n sample Subsample, get array in, For the prediction distribution P F Execute the nth sample The result obtained by sampling times; the data distribution in the statistical array can be used to obtain the probability prediction result in the form of quantiles; the median of each sampling moment of the array is selected as the deterministic prediction result.

4. A method for predicting power system source and load, characterized in that: The prediction method is implemented using a pre-trained model trained based on the method according to any one of claims 1 to 3; the pre-trained model structure and fine-tuning scheme ultimately used for deployment are determined based on available data for a specific application scenario; The available data obtained from the application scenario is input into the fine-tuned pre-trained model for prediction to obtain the prediction results.

5. The power system source load prediction method according to claim 4, characterized in that: The final pre-trained model structure and fine-tuning solution for deployment are determined based on the available data of the specific application scenario. There are three specific cases: When the application scenario can only provide historical data for model prediction, the spatiotemporal pattern hybrid expert sub-model is selected, its pre-trained parameters are loaded, and the model is fine-tuned using the historical data of the target scenario; When there is no available historical data or the historical data is too little for the application scenario, the forecast information hybrid expert sub-model is selected, its pre-trained parameters are loaded, and the model is fine-tuned using the target scenario prediction data; When both historical data and forecast data are available and sufficient, the combined structure of the spatiotemporal pattern hybrid expert sub-model and the forecast information hybrid expert sub-model is selected to achieve the highest prediction accuracy, and the integrated fine-tuning technology of the pre-trained model is used to complete the scale alignment and complementary advantages of the two sub-models.

6. The power system source load prediction method according to claim 5, characterized in that: When the application scenario can only provide historical data for model prediction, the spatiotemporal pattern hybrid expert sub-model is selected, its pre-trained parameters are loaded, and the model is fine-tuned using the target scenario historical data; the specific method is: The spatiotemporal mode hybrid expert sub-model uses the Transformer decoder as the model backbone structure; during the fine-tuning process, the training parameters include the multi-degree-of-freedom hybrid expert Student T distribution output head and the last three layers of the Transformer network, and the other parameters of the spatiotemporal mode hybrid expert sub-model are frozen; Loading the parameter file of the spatiotemporal pattern hybrid expert sub-model, and constructing a fine-tuning dataset using the target scene historical data; Based on the validation set test results during fine-tuning, we choose to disable redundant Student T distribution output heads. The base learning rate is set to 1e-5, and it decreases by 5% per epoch during fine-tuning. If the negative log-likelihood loss of the validation set does not decrease for five consecutive epochs, fine-tuning is stopped.

7. The power system source load prediction method according to claim 5, characterized in that: When there is no available historical data or the historical data is relatively small for the application scenario, the forecast information hybrid expert sub-model is selected, its pre-trained parameters are loaded, and the model is fine-tuned using the target scenario forecast data; the specific method is as follows: Construct paired samples of forecast data and actual data for the target scenario, and use the paired samples to construct a fine-tuning dataset; Loading the parameter file of the forecast information hybrid expert sub-model, freezing the parameters of the forecast information hybrid expert sub-model, and only fine-tuning the main routing network and the sub-routing network; The base learning rate is set to 1e-5, and it decreases by 5% per epoch during fine-tuning. If the negative log-likelihood loss of the validation set does not decrease for five consecutive epochs, fine-tuning is stopped.

8. The power system source load prediction method according to claim 5, characterized in that: When both historical data and forecast data are available and sufficient, a combination of a spatiotemporal pattern hybrid expert sub-model and a forecast information hybrid expert sub-model is selected to achieve the highest prediction accuracy. The integrated fine-tuning technology of the pre-trained model is used to achieve scale alignment and complementary advantages of the two sub-models. The specific method is as follows: Assume that the trainable parameter of the spatiotemporal pattern mixture expert sub-model is θ H , the trainable parameter of the forecast information hybrid expert sub-model is θ F ; The prediction output of the integrated model composed of two sub-models is Expressed as: Where, α E Predict dynamic weights for trainable ensembles; and These are the prediction results of the spatiotemporal pattern hybrid expert sub-model and the forecast information hybrid expert sub-model respectively; The ensemble fine-tuning is divided into two stages: the first stage uses 30% of the training samples and freezes the weights θ H and θ F , only the dynamic weight α of the integrated prediction is adjusted E In the second stage, 70% of the training samples are used to jointly optimize all parameters of the model, and the loss function is set to: L total =L H +L F +λ·KL(P F ||P H ); Where, L H and L F are the negative log-likelihood loss functions of the spatiotemporal pattern hybrid expert sub-model and the forecast information hybrid expert sub-model, respectively; KL(·) represents the KL divergence operation, which is used to align the prediction distributions of the two sub-models, and λ is the attention coefficient of the integrated fine-tuning; P H is the predicted Student's T distribution.

9. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 3.

10. A computer-readable storage medium having computer instructions stored thereon, characterized in that: The computer instructions are used to enable a computer to execute the steps of the method according to any one of claims 1 to 3.

Images