Large electric power model pre-training method, device, equipment, medium and program product

Abstract

The invention relates to an electric power large model pre-training method and device, equipment, a medium and a program product. The method comprises the following steps: acquiring a pre-training corpus containing multi-source power time sequence data; based on the pre-training corpus, constructing a pre-training task set comprising a plurality of pre-training tasks; sampling from a pre-training corpus to obtain pre-training sample data of a pre-training task set; the method comprises the following steps: performing multi-task learning pre-training on a time sequence backbone network according to each pre-training task by using pre-training sample data to obtain a pre-trained power time sequence large model; by adopting the method, the large power time sequence model with strong generalization capability can be obtained, and the prediction performance requirements under different data distribution scenes can be better met.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, medium, and program product for pre-training a large power model. Background Technology

[0002] With the rapid development of smart grids and the energy internet, massive amounts of time-series data have been generated in the power system. Accurate forecasting of this data is crucial for grid security, economic dispatch, and market transactions.

[0003] Currently, power time-series forecasting methods mainly include statistical and machine learning methods, and deep learning methods. These methods are usually trained on specific datasets and are only suitable for relatively narrow scenarios. When transferred to other scenarios with different data distributions (such as different climate zones or different user groups), their performance will significantly decrease. Summary of the Invention

[0004] Therefore, it is necessary to provide a power large-scale model pre-training method, device, equipment, medium, and program product with strong generalization ability to address the above-mentioned technical problems and adapt to the prediction performance requirements under different data distribution scenarios.

[0005] Firstly, this application provides a method for pre-training a large-scale power model, including:

[0006] Obtain a pre-trained corpus containing multi-source power time-series data;

[0007] Based on the pre-training corpus, a pre-training task set containing multiple pre-training tasks is constructed;

[0008] The pre-training sample data of the pre-training task set is obtained by sampling from the pre-training corpus;

[0009] By using the pre-trained sample data, the time-series backbone network is pre-trained through multi-task learning according to each pre-training task to obtain a pre-trained large-scale power time-series model.

[0010] In one embodiment, constructing a pre-training task set containing multiple pre-training tasks based on the pre-training corpus includes:

[0011] Electricity time-series data were sampled from the pre-trained corpus;

[0012] According to the preset task objectives, the power time-series data is converted into task time-series data;

[0013] The task time series data is integrated to determine the task parameters for each of the various task types in the pre-training task set, resulting in multiple pre-training tasks.

[0014] In one embodiment, the step of converting the power time-series data into task time-series data according to a preset task objective includes:

[0015] When the preset task objective includes a generative task objective, the power time series data is subjected to mask reconstruction processing to obtain task time series data;

[0016] When the preset task objective includes a discriminative task objective, positive and negative sample data of the power time series data are extracted from the pre-trained corpus, and the positive and negative sample data and the power time series data are combined to obtain task time series data.

[0017] In one embodiment, the step of using the pre-trained sample data to perform multi-task learning pre-training on the time-series backbone network according to each of the pre-training tasks to obtain a pre-trained large-scale power time-series model includes:

[0018] The pre-trained sample data is input into the temporal backbone network, and the pre-trained sample data is subjected to variable segmentation processing. Under the guidance of each pre-training task, the target sequence perception vector corresponding to each pre-training task is output.

[0019] The task loss for each of the pre-training tasks is determined based on the perception vectors of each of the target sequences.

[0020] If the statistical values ​​of the losses of each task satisfy the pre-training stopping condition, a pre-trained large-scale power time series model is obtained.

[0021] In one embodiment, the method further includes:

[0022] If the number of pre-training iterations of the temporal backbone network meets the task weight update condition, the task loss of each pre-training task is obtained.

[0023] Based on the changing trends of the loss of each task, the task weights of at least one pre-trained task are updated.

[0024] In one embodiment, the method further includes:

[0025] Obtain training sample data for the target scene;

[0026] Based on the training sample data, the model components in the pre-trained power time series large model are subjected to hierarchical training and unfreezing processing until the pre-trained power time series large model converges, thereby obtaining a power prediction model for the target scenario.

[0027] Secondly, this application also provides a pre-training device for a large power model, comprising:

[0028] The corpus acquisition module is used to acquire a pre-trained corpus containing multi-source power time-series data.

[0029] The task set construction module is used to construct a pre-training task set containing multiple pre-training tasks based on the pre-training corpus.

[0030] The sample data sampling module is used to sample the pre-training sample data of the pre-training task set from the pre-training corpus.

[0031] The model pre-training module is used to perform multi-task learning pre-training on the time-series backbone network according to each of the pre-training tasks by using the pre-training sample data, so as to obtain a pre-trained large-scale power time-series model.

[0032] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0033] Obtain a pre-trained corpus containing multi-source power time-series data;

[0034] Based on the pre-training corpus, a pre-training task set containing multiple pre-training tasks is constructed;

[0035] The pre-training sample data of the pre-training task set is obtained by sampling from the pre-training corpus;

[0036] By using the pre-trained sample data, the time-series backbone network is pre-trained through multi-task learning according to each pre-training task to obtain a pre-trained large-scale power time-series model.

[0037] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0038] Obtain a pre-trained corpus containing multi-source power time-series data;

[0039] Based on the pre-training corpus, a pre-training task set containing multiple pre-training tasks is constructed;

[0040] The pre-training sample data of the pre-training task set is obtained by sampling from the pre-training corpus;

[0041] By using the pre-trained sample data, the time-series backbone network is pre-trained through multi-task learning according to each pre-training task to obtain a pre-trained large-scale power time-series model.

[0042] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0043] Obtain a pre-trained corpus containing multi-source power time-series data;

[0044] Based on the pre-training corpus, a pre-training task set containing multiple pre-training tasks is constructed;

[0045] The pre-training sample data of the pre-training task set is obtained by sampling from the pre-training corpus;

[0046] By using the pre-trained sample data, the time-series backbone network is pre-trained through multi-task learning according to each pre-training task to obtain a pre-trained large-scale power time-series model.

[0047] The aforementioned power large-scale model pre-training method, device, equipment, medium, and program products acquire a pre-training corpus containing multi-source power time-series data; based on the pre-training corpus, construct a pre-training task set containing multiple pre-training tasks; sample pre-training sample data of the pre-training task set from the pre-training corpus; and use the pre-training sample data to perform multi-task learning pre-training on the time-series backbone network according to each pre-training task to obtain a pre-trained power time-series large-scale model. By performing multi-task pre-training on the pre-training corpus containing multi-source power time-series data, the time-series backbone network can learn the general dynamic laws of the power system, rather than the overfitting pattern of a specific scenario. Therefore, it can exhibit extremely strong generalization ability when transferred to new scenarios. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a diagram illustrating the application environment of a power large model pre-training method in one embodiment;

[0050] Figure 2 This is a flowchart illustrating a pre-training method for a large power model in one embodiment;

[0051] Figure 3 This is a flowchart illustrating the process of constructing a pre-training task set based on a pre-training corpus in one embodiment.

[0052] Figure 4 This is a flowchart illustrating the process of performing multi-task learning pre-training on a time-series backbone network based on a pre-training corpus and pre-training task sets to obtain a pre-trained large-scale power time-series model in one embodiment.

[0053] Figure 5 This is a flowchart illustrating the pre-training method for a large power model in another embodiment;

[0054] Figure 6 This is a flowchart illustrating the pre-training method for a large power model in yet another embodiment;

[0055] Figure 7 This is a structural block diagram of a power large model pre-training device in one embodiment;

[0056] Figure 8 This is an internal structural diagram of a computer device in one embodiment;

[0057] Figure 9 This is a diagram of the internal structure of a computer device in another embodiment. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0059] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0060] Current power time-series forecasting methods mainly include traditional statistical and machine learning methods, as well as deep learning methods. These methods have certain drawbacks. First, they lack sufficient data utilization; models are typically trained only on data from a single task or region, failing to fully leverage the general patterns and knowledge inherent in the widely available and diverse power time-series data. Second, their generalization ability is weak; a model trained on one dataset will show significant performance degradation when transferred to a scenario with slightly different data distribution (such as different climate zones or different user groups), requiring extensive fine-tuning and retraining. Finally, they perform poorly in small-sample scenarios; for newly built power plants, new grid-connected users, and other small-sample scenarios lacking historical data, it is difficult to build effective prediction models.

[0061] Based on this, the embodiments of this application utilize a pre-trained corpus containing multi-source power time-series data to perform multi-task learning pre-training on the time-series backbone network. During the multi-task learning process, the time-series backbone network can learn the general rules contained in the power time-series data. When it is necessary to transfer to a new scenario or when sample data is scarce, a power prediction model with good prediction performance can be obtained by fine-tuning the pre-trained power time-series large model using a small amount of labeled data.

[0062] The power large model pre-training method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. Terminal 102 acquires a pre-training corpus containing multi-source power time-series data; based on the pre-training corpus, it constructs a pre-training task set containing multiple pre-training tasks; it samples pre-training sample data of the pre-training task set from the pre-training corpus; using the pre-training sample data, it performs multi-task learning pre-training on the time-series backbone network according to each pre-training task to obtain a pre-trained large-scale power time-series model. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, etc. Server 104 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0063] In one exemplary embodiment, such as Figure 2 As shown, a pre-training method for a large power model is provided, which is then applied to... Figure 1 The following steps are used as an example of the terminal in the example, including steps 202 to 208.

[0064] Step 202: Obtain a pre-trained corpus containing multi-source power time-series data.

[0065] Power time-series data refers to power data that carries time information and can include various data collected through different data acquisition components of the power system. Multi-source power time-series data indicates that the data source is multiple. For example, multi-source power time-series data can include load data, wind power output data, photovoltaic power output data, market electricity price data, and meteorological data, which can further include temperature, humidity, wind speed, and irradiance. A pre-training corpus is a database that stores multi-source power time-series data used for pre-training.

[0066] For example, raw power data collected by a data acquisition unit is acquired, preprocessed to obtain multi-source power time-series data, and a pre-training corpus is constructed based on the multi-source power time-series data. Preprocessing of the raw power data may include data cleaning, missing value imputation, data normalization, and time-series alignment. For raw power data from different data sources, preprocessing operations can be performed independently to preserve sequence differences.

[0067] Step 204: Based on the pre-trained corpus, construct a pre-training task set containing multiple pre-training tasks.

[0068] A pre-training task set is a collection of pre-training tasks, which can include multiple tasks for pre-training a large model. The purpose of pre-training is to enable the large model to learn the general laws of power systems. Based on the actual application requirements of the large model, the task objectives of each pre-training task can be pre-defined. Through pre-training with different task objectives, the large model can acquire the ability to complete multiple tasks.

[0069] For example, a pre-training task identifier is obtained, and the power time-series data in the pre-training corpus is processed according to the pre-training task identifier to obtain pre-training task data corresponding to the pre-training task identifier. The set of pre-training task data is used as the pre-training task set. The pre-training task identifier can represent the task objective of different pre-training tasks. Based on the task objective, the input and output model elements of the pre-training task are constructed to form the pre-training task data.

[0070] Step 206: Sample the pre-training sample data of the pre-training task set from the pre-training corpus.

[0071] Pre-training sample data refers to the sample data used as input to the temporal backbone network for multi-task learning pre-training. For example, the data in the pre-training corpus can be divided to define the power time-series data already used to construct the pre-training task set. During the sampling process of pre-training sample data, power time-series data not used to construct the pre-training task set can be sampled, or only a portion of the power time-series data already used to construct the pre-training task set can be sampled. This avoids excessive overlap between the data in the pre-training task set and the data from the pre-training process, which could negatively impact training performance.

[0072] Step 208: Using pre-trained sample data, the time-series backbone network is pre-trained using multi-task learning according to each pre-training task to obtain a pre-trained large-scale power time-series model.

[0073] A temporal backbone network is a network structure used to process time-series data and output time-series prediction results. Multi-task learning refers to a training method that utilizes useful information contained in multiple learning tasks to help each task obtain more accurate results. A pre-trained power time-series large model is a large model that has undergone pre-training. Through pre-training using multi-task learning, the pre-trained power time-series large model can learn general patterns in the power system. After fine-tuning for downstream tasks, the pre-trained power time-series large model can be transferred and applied to new scenarios or scenarios with scarce data and small sample sizes.

[0074] For example, for each pre-training task in the pre-training task set, a corresponding output layer is constructed in the temporal backbone network. Input temporal data is extracted from the pre-training corpus and input into the temporal backbone network. Multiple output layers output the prediction data corresponding to each pre-training task. The loss of the prediction data is calculated, and the loss is used to perform pre-training again until the loss converges or the number of iterations reaches the preset number, thus completing the pre-training and obtaining the pre-trained power time series large model.

[0075] In the aforementioned pre-training method for the large-scale power model, a pre-training corpus containing multi-source power time-series data is obtained; based on the pre-training corpus, a pre-training task set containing multiple pre-training tasks is constructed; pre-training sample data of the pre-training task set is sampled from the pre-training corpus; and multi-task learning pre-training of the time-series backbone network is performed using the pre-training sample data according to each pre-training task to obtain the pre-trained large-scale power time-series model. By performing multi-task pre-training on the pre-training corpus containing multi-source power time-series data, the time-series backbone network can learn the general dynamic laws of the power system, rather than the overfitting pattern of a specific scenario. Therefore, it can exhibit extremely strong generalization ability when transferred to new scenarios.

[0076] In one exemplary embodiment, such as Figure 3 As shown, based on the pre-trained corpus, a pre-training task set containing multiple pre-training tasks is constructed, including steps 302 to 306.

[0077] Step 302: Sample power time series data from the pre-training corpus.

[0078] The sampling step size for power time-series data can be set according to actual conditions. With a fixed sampling step size, power time-series data has a fixed time span. By setting different sampling step sizes, power time-series data with different time spans can be obtained, resulting in different requirements for the time-series backbone network's ability to process sequence data during pre-training. For pre-training tasks with different objectives, inputting power time-series data with the same sampling step size into the time-series backbone network will also produce different prediction results.

[0079] Step 304: Based on the preset task objectives, convert the power time series data into task time series data.

[0080] Preset task objectives refer to the pre-defined objectives of the pre-training tasks. Task time-series data refers to the time-series data in the pre-training task set. Pre-training elements related to the pre-training process, such as input data, output data, and loss functions, can be set in the preset task objectives. Through data processing methods matched to the preset task objectives, power time-series data can be transformed into task time-series data.

[0081] For example, the preset task objectives include classification tasks and regression tasks. Power time-series data are transformed into task time-series data based on classification task rules matching the classification task and regression task rules matching the regression task, respectively. Classification tasks can include binary classification and multi-objective classification. Different classification task rules can be set based on different types of classification tasks to perform targeted transformations of the power time-series data. For regression tasks, the sequence length of the predicted result after regression and the length of the input sequence can be set in the regression task rules.

[0082] In one embodiment, converting power time-series data into task time-series data according to a preset task objective includes: if the preset task objective includes a generative task objective, performing mask reconstruction processing on the power time-series data to obtain task time-series data; if the preset task objective includes a discriminative task objective, extracting positive and negative sample data of the power time-series data from a pre-training corpus, and combining the positive and negative sample data with the power time-series data to obtain task time-series data.

[0083] Generative task objectives refer to tasks that generate new data based on existing input data. Discriminative task objectives refer to tasks that determine which specific category a given input belongs to. It can be understood that preset task objectives can include all tasks that require pre-training in a multi-task learning process. For each power time series data set, corresponding transformations can be performed on all types of task objectives included in the preset objectives to obtain various transformed data.

[0084] For generative tasks, a masking strategy can be applied for masked reconstruction, masking data at a specific time point in the power time series data to generate masked task time series data for reconstruction. The masked time point can be randomly selected or a specific time point can be selected each time. For discriminative tasks, positive and negative samples can be selected for the power time series data based on metadata or sample selection rules to form triples for comparative learning. Positive samples can be other sequences with the same metadata as the anchor sequence, such as data from the same substation, the same season, or the same workday. Negative samples can be sequences with different metadata, such as data from different regions or different seasons. Generative tasks require the model to pay attention to details and local context to accurately recover the data. Discriminative tasks require the model to grasp the overall, high-level semantic similarity.

[0085] The masked task time-series data can be input into the shared encoder in the time-series backbone network. Then, the loss is calculated using both the reconstruction head and the contrast head. The reconstruction head is used for generative tasks, and the contrast head is used for discriminative tasks. The total loss = λ1 * L_reconstruction + λ2 * L_contrastive, where λ1 and λ2 are hyperparameters used to balance the importance of the two tasks.

[0086] Step 306: Integrate the task time series data, determine the task parameters of each task type in the pre-training task set, and obtain multiple pre-training tasks.

[0087] Task parameters refer to model parameters related to the pre-training task, such as the length of the output sequence and the loss function used in the task. The settings of task parameters can be different for different types of pre-training tasks. Task time series data can be mapped to task types according to the preset task objectives, and then task time series data of the same task type can be integrated to obtain the pre-training task.

[0088] In this embodiment, by converting power time-series data to obtain a pre-trained task set, task objective guidance can be provided for subsequent multi-task learning.

[0089] In one exemplary embodiment, such as Figure 4 As shown, based on the pre-training corpus and the pre-training tasks in the pre-training task set, the temporal backbone network is pre-trained using multi-task learning to obtain a pre-trained large-scale power time series model, including steps 402 to 406.

[0090] Step 402: Input the pre-training sample data into the temporal backbone network, perform variable segmentation on the pre-training sample data, and output the target sequence perception vector corresponding to each pre-training task under the guidance of each pre-training task.

[0091] This embodiment employs a PatchTST (Patch Time Series Transformer) architecture for multi-task learning pre-training of a time-series backbone network. The network outputs a target sequence perception vector corresponding to each pre-training task through its output layer. The pre-training sample data can contain multiple variables, represented as a multivariate time series. A multivariate time series X∈R^(L x C) is input into the PatchTST architecture, where L is the sequence length and C is the number of variables. Each variable is independently segmented into overlapping patches (subsequences). Each patch can be mapped to a set dimension through a linear projection layer, resulting in Patch_Embedding∈R^(N x d_model), where N is the number of patches (N < L) and d_model is the set dimension. The subsequences are then encoded, including positional encoding and variable encoding. Variable encoding adds a learnable variable identifier embedding to each variable.

[0092] The PatchTST architecture includes a Transformer encoder consisting of multiple identical stacked layers. Each layer of the encoder contains a multi-head attention mechanism module and a feedforward network, allowing the model to learn the dependencies between different patches.

[0093] For input multivariate time series, a target sequence perceptual vector can be output, which is the context-aware representation Z∈ R^(N x d_model) of the entire input sequence. During processing, the multivariate sequence is treated as multiple independent univariate sequences. Multiple pre-training tasks can share the same Transformer encoder. At the end of the PatchTST architecture, each pre-training task corresponds to a different prediction head, outputting a target sequence perceptual vector that conforms to the task objective. This reduces model parameters and forces the model to focus on learning patterns in the time dimension, while the correlation between variables is implicitly learned through subsequent attention mechanisms. The sub-sequence processing effectively reduces sequence length and gives the model some noise resistance.

[0094] Step 404: Determine the task loss for each pre-training task based on the perception vector of each target sequence.

[0095] For each pre-training task, the task loss can be determined by comparing the target sequence perceptual vector with the task's defined objective. The task's defined objective refers to the expected model output, which, for different pre-training tasks, can be the label of the pre-training sample data or a pre-defined target value. Each pre-training task can set a loss function to calculate the task loss. By substituting the target sequence perceptual vector and the expected model output into the loss function, the task loss can be calculated.

[0096] Step 406: If the statistical values ​​of the loss of each task meet the pre-training stopping condition, continue until the pre-training stopping condition is met to obtain the pre-trained power time series large model.

[0097] The pre-training stopping condition can be set to the statistical value of the task loss being less than a preset loss threshold. Each pre-training iteration samples different pre-training data from the pre-training corpus, ensuring the samples contain a sufficient number of variables. By repeatedly sampling pre-training data from the corpus and performing iterative pre-training, the temporal backbone network can learn the general patterns of power system operation. The pre-training iteration process can be stopped when the pre-training stopping condition is met.

[0098] In one embodiment, during the multi-task learning pre-training process of the temporal backbone network, the method further includes: obtaining the task loss of each pre-training task when the number of pre-training cycles of the temporal backbone network meets the task weight update condition; and updating the task weight of at least one pre-training task according to the changing trend of the task loss.

[0099] Task weight update conditions are the conditions set for updating the weights of pre-training tasks. Setting different task weights for different pre-training tasks allows each task to learn fully during pre-training, balances the computing resources allocated to different pre-training tasks, or strengthens the learning of difficult tasks, enhancing their predictive performance. The trend of task loss can characterize whether the loss of the pre-training task increases or decreases at different times. In one feasible implementation, when the task loss decreases, the task weight of the corresponding pre-training task can be reduced; when the task loss increases, the task weight of the corresponding pre-training task can be increased.

[0100] In one embodiment, task weights are used to determine the statistical value of the loss for each task. The task losses of each pre-training task are then weighted and summed according to their respective task weights to obtain the total loss of the temporal backbone network. During multi-task collaborative pre-training, the task weights of pre-training tasks with fast loss convergence are reduced, while the task weights of pre-training tasks with slow loss convergence are increased.

[0101] For example, in multi-task collaborative pre-training, a dynamic weight balancing algorithm can be used. Initially, all pre-training tasks are assigned equal weights, with a total pre-training step count of 1,000,000. Weights are automatically adjusted every 1,000 training steps based on task loss information. During weight adjustment, the AdamW optimizer updates all parameters to improve training efficiency. Pre-training tasks can include time-series mask reconstruction, time-series contrast learning, periodic pattern recognition, and external factor association. In the time-series mask reconstruction task, 15% of the time-series data points are randomly masked, and mean squared error is used as the loss function. In the time-series contrast learning task, for a load curve, two enhanced views are generated by adding Gaussian noise and a small time offset, forming a positive sample pair. A load curve and a wind power curve form a negative sample pair, using the InfoNCE loss function. In the periodic pattern recognition task, data from the past week is input, and the model predicts the load curve profile for the next 24 hours. In the external factor association task, temperature sequences and date types are embedded and concatenated with power time-series data, allowing the model to predict the overall load change trend for the next hour.

[0102] In this embodiment, multi-task learning pre-training is performed using a temporal backbone network based on the PatchTST architecture, which can reduce sequence length and improve pre-training efficiency.

[0103] In one exemplary embodiment, such as Figure 5 As shown, after performing multi-task learning pre-training on the temporal backbone network based on the pre-training corpus and pre-training tasks in the pre-training task set, the method further includes steps 502 to 504.

[0104] Step 502: Obtain training sample data for the target scene.

[0105] The target scenario refers to the application scenario in which the pre-trained power time series model will be transferred. Training sample data refers to the sample data used for model training within the target scenario. The target scenario can be a specific scenario within a power system, such as a scenario for predicting electricity consumption in a particular region. In this scenario, the training sample data can include historical meteorological data, historical electricity consumption data, etc., for that region.

[0106] Step 504: Based on the training sample data, perform layered training and unfreezing processing on the model components in the pre-trained power time series large model until the pre-trained power time series large model converges, and obtain the power prediction model for the target scenario.

[0107] This process involves fine-tuning a pre-trained large-scale power time-series model for downstream tasks to obtain a power prediction model for a target scenario. After completing the pre-training process, all weights of the shared Transformer backbone network and the weights of the multi-task pre-training heads are saved. The code for all model components, such as the Transformer backbone network, input encoding layer, and attention mechanism, is defined, and all hyperparameters of the model are recorded in a configuration file. A variable mapping table stores the mapping relationship between variable names and their corresponding variable identifiers.

[0108] During the application migration process, the weights and configuration of a pre-trained large-scale power time-series model can be loaded first. Based on the task type of the target scenario, a prediction head can be added to its output. Training sample data can include historical power grid data from the past 168 hours, including variables such as total load, temperature, wind speed, and solar radiation intensity. A layered unfreezing approach can be used: first train the prediction head for 10 epochs, then unfreeze the last two layers of the time-series backbone network, train for another 10 epochs, and finally unfreeze the entire network and train until convergence. Using the AdamW optimizer, the prediction head learning rate can be set to 1e. -3 The backbone network learning rate can be set to 5e. -5 The MSE loss function was used. The model performance was evaluated on a reserved test set. Once the performance met the target, the entire fine-tuned model was saved as a power prediction model for online prediction of the production environment.

[0109] In this embodiment, by fine-tuning the pre-trained power time series model for downstream tasks, only a small amount of labeled data is needed to achieve the same or even better results as training with a large amount of data.

[0110] In one exemplary embodiment, such as Figure 6 As shown, the method includes the following steps S1 to S6.

[0111] S1. Construct a pre-training corpus of multi-source heterogeneous power time series data. Massive amounts of power time series data are collected from multiple data sources and preprocessed uniformly to form a pre-training corpus. Data sources include load data, wind power output data, photovoltaic power output data, market electricity price data, and meteorological data, including temperature, humidity, wind speed, and irradiance. Preprocessing includes data cleaning, missing value imputation, data normalization, and time series alignment.

[0112] S2. Construct a pre-training task set that includes masked sequence reconstruction and temporal contrastive learning. For each sequence sample in the pre-training corpus, the following two transformations can be performed: apply a masking strategy to generate a masked sequence for the reconstruction task; select positive and negative samples based on metadata or rules to form triples for contrastive learning; positive samples can be other sequences with the same metadata as the anchor sequence, such as: the same substation, the same season, or the same workday; negative samples are sequences with different metadata, such as: different regions, summer vs. winter. Input the masked sequence into the shared encoder, and then calculate the loss using the reconstruction head and the contrast head respectively.

[0113] S3. Construct a general temporal backbone network based on Transformer. Input a multivariate temporal sequence X ∈ R^(L x C) into the PatchTST architecture, where L is the sequence length and C is the number of variables. Independently segment each variable into overlapping patches. Each patch is mapped to the d_model dimension through a linear projection layer, resulting in Patch_Embedding ∈ R^(N x d_model), where N is the number of patches (N < L). Input encoding includes positional encoding and variable encoding. Variable encoding adds a learnable variable identifier embedding to each variable. The Transformer encoder, composed of multiple identical stacked layers, contains a multi-head self-attention module and a feedforward network in each layer. The output is a context-aware representation Z ∈ R^(N x d_model) of the entire input sequence. Treat the multivariate sequence as multiple independent univariate sequences sharing the same Transformer encoder.

[0114] S4. Multi-task Collaborative Pre-training. The AdamW optimizer is used to pre-train the model on the pre-training corpus built in S1. A dynamic weight balancing algorithm is employed, initially assigning equal weights to the four pre-training tasks, and then automatically adjusting the weights every 1000 training steps based on the changes in the loss of each task, for a total of 1,000,000 pre-training steps. The AdamW optimizer is used to update all parameters to improve training efficiency. The pre-training task set includes a time-series mask reconstruction task, a time-series contrast learning task, a periodic pattern recognition task, and an external factor association task. The time-series mask reconstruction task randomly masks 15% of the time-series data points, using mean squared error as the loss function. The time-series contrast learning task generates two enhanced views of a load curve by adding Gaussian noise and a small time offset, forming a positive sample pair. A load curve and a wind power curve form a negative sample pair. The InfoNCE loss function is used. The periodic pattern recognition task takes data from the past week as input and asks the model to predict the outline of the load curve for the next 24 hours. External factor correlation task: After embedding temperature series and date types, it is concatenated with power time series data to enable the model to predict the overall load change trend in the next hour.

[0115] S5. Generate a pre-trained large-scale power time-series model. Save the pre-trained weight parameters, sharing all weights of the Transformer backbone network and the weights of the multi-task pre-training heads. Define the Python code for all components, including the Transformer backbone network, input encoding layer, and attention mechanism. A configuration file records all hyperparameters of the model. Data preprocessing and Tokenizer module: Standardizer: Stores statistics such as the mean and standard deviation used for data standardization; Variable mapping table: Stores the mapping relationship between variable names and their corresponding variable identifiers. Encapsulate this into a Tokenizer class, providing `encode` or `preprocess` methods to automatically convert the raw time-series data into the input format required by the model. Ensure that the input data maintains a consistent distribution with the pre-training data when applied downstream, clearly defining the scope of knowledge learned by the model during pre-training, and making it suitable for downstream tasks.

[0116] S6. Fine-tune downstream tasks. Load the weights and configuration of PowerTST-base, and add a prediction head to its output: Global Average Pooling -> Linear(d_model, 24). Input historical grid data from the past 168 hours, including variables such as total load, temperature, wind speed, and solar radiation intensity. Label the ground truth total load for the next 24 hours, and process the data using the normalizer and time encoder included with PowerTST-base. First, train only the prediction head for 10 epochs, then unfreeze the last 2 layers of the backbone network, train for another 10 epochs, and finally unfreeze the entire network and train until convergence; use AdamW, and set the prediction head learning rate to 1e. -3 The backbone network learning rate is set to 5e. -5 Use MSE loss. Evaluate the model performance on a reserved test set. Once the performance meets the target, save the entire fine-tuned model for online prediction in the production environment.

[0117] The model in this embodiment can start with general knowledge from hundreds of millions of data points, and can achieve better performance on specific tasks than training a model from scratch. The computational resources and data volume required for fine-tuning are greatly reduced, the training speed is extremely fast, and the same pre-trained large model can be quickly adapted to hundreds or thousands of different downstream tasks, which greatly improves the reusability and deployment efficiency of the technology.

[0118] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0119] Based on the same inventive concept, this application also provides a power large model pre-training device for implementing the aforementioned power large model pre-training method. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more power large model pre-training device embodiments provided below can be found in the limitations of the power large model pre-training method described above, and will not be repeated here.

[0120] In one exemplary embodiment, such as Figure 7 As shown, a pre-training device 700 for a large power model is provided, comprising: a corpus acquisition module 701, a task set construction module 702, a sample data sampling module 703, and a model pre-training module 704, wherein:

[0121] The corpus acquisition module 701 is used to acquire a pre-trained corpus containing multi-source power time-series data.

[0122] Task set construction module 702 is used to construct a pre-trained task set based on a pre-trained corpus.

[0123] The sample data sampling module 703 is used to sample pre-training sample data of the pre-training task set from the pre-training corpus.

[0124] The model pre-training module 704 is used to perform multi-task learning pre-training on the time series backbone network based on the pre-training corpus and the pre-training task set, so as to obtain a pre-trained large-scale power time series model.

[0125] In an exemplary embodiment, the task set construction module 702 is further configured to: sample power time-series data from a pre-training corpus; convert the power time-series data into task time-series data according to a preset task objective; integrate the task time-series data, determine the task parameters of various task types in the pre-training task set, and obtain multiple pre-training tasks.

[0126] In an exemplary embodiment, the task set construction module 702 is further configured to: perform mask reconstruction processing on the power time series data to obtain task time series data when the preset task objective includes a generative task objective; and extract positive and negative sample data of the power time series data from the pre-trained corpus when the preset task objective includes a discriminative task objective, and combine the positive and negative sample data with the power time series data to obtain task time series data.

[0127] In an exemplary embodiment, the model pre-training module 704 is further configured to: input pre-training sample data into the time-series backbone network, perform variable segmentation processing on the pre-training sample data, and output target sequence perception vectors corresponding to each pre-training task under the guidance of each pre-training task; determine the task loss of each pre-training task based on each target sequence perception vector; and obtain the pre-trained power time-series large model when the statistical value of each task loss meets the pre-training stopping condition.

[0128] In an exemplary embodiment, the model pre-training module 704 is further configured to: obtain the task loss of each pre-training task when the number of pre-training iterations of the temporal backbone network meets the task weight update condition; and update the task weight of at least one pre-training task according to the changing trend of the task loss.

[0129] In an exemplary embodiment, the power large model pre-training device 700 further includes a downstream task fine-tuning module, used to: acquire training sample data of the target scenario; and based on the training sample data, perform layered training unfreezing processing on the model components in the pre-trained power time series large model until the pre-trained power time series large model converges to obtain a power prediction model for the target scenario.

[0130] Each module in the aforementioned large-scale power model pre-training device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0131] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores multi-source power time-series data. The I / O interfaces are used for information exchange between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a pre-training method for a large-scale power model.

[0132] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 9As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a pre-training method for a large-scale power model. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0133] Those skilled in the art will understand that Figure 8 and Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0134] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the embodiments of the methods described above.

[0135] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the embodiments of the above methods.

[0136] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the embodiments of the methods described above.

[0137] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0138] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0139] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A pre-training method for a large-scale power model, characterized in that, The method includes: Obtain a pre-trained corpus containing multi-source power time-series data; Based on the pre-training corpus, a pre-training task set containing multiple pre-training tasks is constructed; The pre-training sample data of the pre-training task set is obtained by sampling from the pre-training corpus; By using the pre-trained sample data, the time-series backbone network is pre-trained through multi-task learning according to each pre-training task to obtain a pre-trained large-scale power time-series model.

2. The method according to claim 1, characterized in that, The step of constructing a pre-training task set containing multiple pre-training tasks based on the pre-training corpus includes: Electricity time-series data were sampled from the pre-trained corpus; According to the preset task objectives, the power time-series data is converted into task time-series data; The task time series data is integrated to determine the task parameters for each of the various task types in the pre-training task set, resulting in multiple pre-training tasks.

3. The method according to claim 2, characterized in that, The step of converting the power time-series data into task time-series data according to the preset task objective includes: When the preset task objective includes a generative task objective, the power time series data is subjected to mask reconstruction processing to obtain task time series data; When the preset task objective includes a discriminative task objective, positive and negative sample data of the power time series data are extracted from the pre-trained corpus, and the positive and negative sample data and the power time series data are combined to obtain task time series data.

4. The method according to claim 1, characterized in that, The step of using the pre-trained sample data to perform multi-task learning pre-training on the time-series backbone network according to each pre-training task to obtain a pre-trained large-scale power time-series model includes: The pre-trained sample data is input into the temporal backbone network, and the pre-trained sample data is subjected to variable segmentation processing. Under the guidance of each pre-training task, the target sequence perception vector corresponding to each pre-training task is output. The task loss for each of the pre-training tasks is determined based on the perception vectors of each of the target sequences. If the statistical values ​​of the losses of each task satisfy the pre-training stopping condition, a pre-trained large-scale power time series model is obtained.

5. The method according to claim 4, characterized in that, The method further includes: If the number of pre-training iterations of the temporal backbone network meets the task weight update condition, the task loss of each pre-training task is obtained. Based on the changing trends of the loss of each task, the task weights of at least one pre-trained task are updated.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: Obtain training sample data for the target scene; Based on the training sample data, the model components in the pre-trained power time series large model are subjected to hierarchical training and unfreezing processing until the pre-trained power time series large model converges, thereby obtaining a power prediction model for the target scenario.

7. A pre-training device for a large-scale power model, characterized in that, The device includes: The corpus acquisition module is used to acquire a pre-trained corpus containing multi-source power time-series data. The task set construction module is used to construct a pre-training task set containing multiple pre-training tasks based on the pre-training corpus. The sample data sampling module is used to sample the pre-training sample data of the pre-training task set from the pre-training corpus. The model pre-training module is used to perform multi-task learning pre-training on the time-series backbone network according to each of the pre-training tasks by using the pre-training sample data, so as to obtain a pre-trained large-scale power time-series model.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images