A method and system for predicting cold start of electric load based on pre-training and fine-tuning

Abstract

The present invention relates to a method and system for predicting cold start of electric load based on pre-training and fine-tuning, belonging to the field of data mining. The present invention first collects customer load data, and randomly divides the training set, validation set and test set, divides each customer load data into small segments, and normalizes the segmented customer load data, and uses PatchTST to pre-train the base model on all segments; then uses K-Shape to cluster the segments into K power consumption pattern clusters, and uses these clusters to fine-tune the base model respectively, and obtains the prediction model of each power consumption pattern; finally, uses the power consumption pattern prediction model that is most similar to the historical load segment of the new user to predict the electric load. The present invention can achieve a prediction accuracy close to that of a model trained using historical load data without using the historical load data of the target customer, which solves the problem of cold start of electric load very well.

Description

Technical Field

[0001] The present invention relates to a method and system for predicting cold start of electric load based on pre-training and fine-tuning, belonging to the technical field of data mining. Background Art

[0002] Electricity load cold start prediction is the process of predicting the future electricity demand of users who lack sufficient historical data. Currently, the most direct way to solve this problem is to select the user model that is most similar to the cold start user for prediction, but it is difficult to find users with exactly the same electricity usage pattern as the target user. Another approach is to use a deep neural network architecture based on forward and backward residual connections and a very deep fully connected layer stack. Through the universal feature representation learned from a large amount of data, it can be easily migrated to new time series datasets without extensive modification or adjustment of the model structure. Although this method has shown strong ability to capture the universal variation pattern of electricity load, it may to some extent ignore the subtle and critical features unique to different electricity usage patterns or scenarios.

[0003] Therefore, the present invention first divides the training data into equal-length time series segments and uses the PatchTST model to learn universal load characteristics. Subsequently, the K-Shape algorithm is used to identify clusters of similar load variation patterns, i.e., specific power usage patterns. Next, the base model is fine-tuned, adaptively learning each power usage pattern cluster to generate a customized prediction model. During prediction, the corresponding model is selected based on the user's power usage pattern, leveraging the versatility of big data while accurately capturing specific patterns, significantly improving prediction accuracy and practicality. Summary of the Invention

[0004] The present invention provides a method and system for cold start prediction of electric load based on pre-training and fine-tuning, which is used to solve the problem of cold start prediction of electric load. When making predictions, the present invention selects the corresponding model according to the user's electricity usage pattern, which not only utilizes the versatility of big data, but also accurately captures specific patterns, significantly improving the prediction accuracy and practicality.

[0005] The technical solution of the present invention is: a method for predicting cold start of power load based on pre-training and fine-tuning, the method comprising:

[0006] Step 1: Collect customer load data and randomly divide it into training, validation, and test sets. Each customer load data is divided into small segments and then normalized.

[0007] Based on the Electricity Load Diagrams 2011-2014 dataset, the first 60% of the electricity load data of 80 users numbered 0 to 79 were divided into four training sets: users numbered 0 to 19 as Train-1, users numbered 20 to 39 as Train-2, users numbered 40 to 59 as Train-3, and users numbered 60 to 79 as Train-4. The first 60% to 80% of the electricity load data of the 40 users numbered 80 to 119 were used as the validation set, and the last 20% of the electricity load data of the last 20 users were used as the test set.

[0008] Step 2: Use the payload fragment to train the base model on PatchTST.

[0009] Step 3: Based on the base model, fine-tune the model for each power consumption pattern cluster (i.e., similar load segment cluster);

[0010] Step 4: Perform base model inference and prediction in the cold start scenario.

[0011] Furthermore, in Step 1, the processing is summarized as follows:

[0012]

[0013] z is the normalized result, x is the input sequence, μ is the mean of the input sequence, and σ is the standard deviation of the input sequence.

[0014] Furthermore, the Step 2 includes:

[0015] Step 2.1. The t-th load segment x of the i-th user i t Split into overlapping batches (Patch); in the input load segment x i t After splitting into batches, combine the split batches into matrices P is the length of the batch, L is the look-back window length, and N is the number of batches. S is the stride (i.e., the non-overlapping area between two consecutive batches);

[0016] Step 2.2, through a trainable linear projection matrix and a learnable position information encoding matrix D represents the latent space dimension, which maps the input batch to the latent representation space h d , the latent representation space h d Expressed as:

[0017] h d=W p X p +W pos ;

[0018] Step 2.3, in the h-th head of the Transformer multi-head attention mechanism, h d Convert to query matrix Bond Matrix Sum Matrix in Then get the attention value through the scaled dot product The value of attention O h The transpose (O h ) T Expressed as:

[0019]

[0020] in, Represent the weight matrices that map the input vector to the query, key, and value spaces, respectively. d k represents the dimension of the hidden layer;

[0021] Step 2.4, the multi-head attention mechanism also includes a batch normalization layer and a feedforward network with residual connections; input load segment The characterization of The result is predicted by a flattening layer with a linear head. Expressed as:

[0022]

[0023] Where T is the prediction step length, represents the load value at point t+L of the i-th load segment, represents the load value of the t+L+T-1th point in the i-th load segment;

[0024] Step 2.5. Use mean square error loss to measure the difference between the predicted value and the actual value to obtain the target loss for base model training. Target loss Expressed as:

[0025]

[0026] Where j is the jth point of the load segment, represents the actual load value of the jth point in the i-th load segment, Indicates the load value of the jth point in the i-th load segment.

[0027] Furthermore, the Step 3 includes:

[0028] Step 3.1 First, use cross-correlation to determine two power load segments a = (a1, ..., a m ) and b=(b1,...,b m ), to achieve displacement invariance, b remains stationary while a slides along b. For each offset s of a, the inner product of a and b in the current overlapping part is calculated. The value of this inner product reflects the similarity of the two segments under the current offset s. The offset of a is expressed as follows:

[0029]

[0030] Step 3.2 Calculate the cross-correlation between the two segments. The s value range is [-m, m]. Different s values ​​will calculate the cross-correlation according to the following formula, and a total of 2m+1 values ​​will be obtained;

[0031]

[0032] Among them, R s (a,b) represents the cross-correlation between a and b when the offset is s, b j is the load value of the jth point of b, a j+s is the load value of the j+sth point a, R -s (a, b) When the offset s is negative, the offset is converted to a positive value and then the cross-correlation is calculated;

[0033] Step 3.3 Calculate the shape-based distance (SBD) between segments a and b by dividing the maximum cross-correlation value at all offsets by the geometric mean of the autocorrelation of a single segment. The shape-based distance between a and b is expressed as:

[0034]

[0035] Where SBD(a,b) represents the shape-based distance between a and b, R0(a,a) and R0(b,b) are the correlations between the sequence itself and its unshifted version, respectively.

[0036] Step 3.4 Based on the shape-based distance between a and b, the centroid of the kth electricity consumption pattern cluster is calculated using the following optimization method:

[0037]

[0038] Among them, μ k represents the center of mass, represents the optimal value of the centroid, represents the set of payload segments of the kth cluster.

[0039] Furthermore, the Step 4 includes:

[0040] Step 4.1: For new user load segments, assign them to corresponding power consumption pattern clusters using the K-Shape algorithm.

[0041] Step 4.2: Use the corresponding fine-tuned model for prediction;

[0042] Let the new data segment be x new , calculated by K-Shape, the cluster to which it belongs is Then the corresponding prediction model is used for prediction:

[0043]

[0044] Among them, θ k represents the prediction model parameters for the kth power consumption mode, f(;) represents the prediction model, Indicates the prediction result.

[0045] The present invention also provides a system for cold start prediction of electric load based on pre-training and fine-tuning, the system comprising: a module for executing the above-mentioned method for cold start prediction of electric load based on pre-training and fine-tuning.

[0046] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the above-mentioned method for predicting cold start of electric load based on pre-training and fine-tuning is implemented.

[0047] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements the above-mentioned pre-training-fine-tuning based power load cold start prediction method.

[0048] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the above-mentioned pre-training-fine-tuning based power load cold start prediction method.

[0049] The beneficial effects of the present invention are:

[0050] 1. This method cuts the training data into equal time series segments and uses the PatchTST model to learn universal load characteristics. Subsequently, the K-Shape algorithm is used to identify clusters of similar load variation patterns, i.e., specific power consumption patterns. Next, the base model is fine-tuned to perform adaptive learning for each power consumption pattern cluster, generating a customized prediction model.

[0051] 2. When making predictions, the present invention selects the corresponding model based on the user's electricity usage pattern, which not only takes advantage of the versatility of big data but also accurately captures specific patterns, significantly improving the accuracy and practicality of predictions;

[0052] 3. On the self-constructed dataset, the performance of the proposed model is significantly improved. Compared with the model trained directly on the test users, the performance difference of the model is less than 1%;

[0053] 4. The present invention can achieve a prediction accuracy close to that of a model trained using historical load data without using the target customer's historical load data, thus effectively solving the problem of cold start prediction of power load. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is the overall flow chart of the present invention;

[0055] Figure 2 It is a model structure diagram of the present invention. DETAILED DESCRIPTION

[0056] Example 1: Figure 1-2 As shown, a power load cold start prediction method based on pre-training and fine-tuning includes:

[0057] Step 1: Collect customer load data and randomly divide it into training, validation, and test sets. Each customer load data is divided into small segments and then normalized.

[0058] Based on the Electricity Load Diagrams 2011-2014 dataset, the first 60% of the electricity load data of 80 users numbered 0 to 79 were divided into four training sets: users numbered 0 to 19 as Train-1, users numbered 20 to 39 as Train-2, users numbered 40 to 59 as Train-3, and users numbered 60 to 79 as Train-4. The first 60% to 80% of the electricity load data of the 40 users numbered 80 to 119 were used as the validation set, and the last 20% of the electricity load data of the last 20 users were used as the test set.

[0059] Furthermore, in Step 1, the processing is summarized as follows:

[0060]

[0061] z is the normalized result, x is the input sequence, μ is the mean of the input sequence, and σ is the standard deviation of the input sequence.

[0062] Step 2: Use the payload fragment to train the base model on PatchTST.

[0063] Further, the Step 2 includes:

[0064] Step 2.1. The t-th load segment x of the i-th user i t is divided into overlapping batches (Patch); in the input load segment After splitting into batches, combine the split batches into matrices P is the length of the batch, L is the look-back window length, and N is the number of batches. S is the stride (i.e., the non-overlapping area between two consecutive batches);

[0065] Step 2.2, through a trainable linear projection matrix and a learnable position information encoding matrix D represents the latent space dimension, which maps the input batch to the latent representation space h d , the latent representation space h d Expressed as:

[0066] h d =W p X p +W pos ;

[0067] Step 2.3, in the h-th head of the Transformer multi-head attention mechanism, h d Convert to query matrix Bond Matrix Sum Matrix in Then get the attention value through the scaled dot product The value of attention O h The transpose (O h ) T Expressed as:

[0068]

[0069] in, Represent the weight matrices that map the input vector to the query, key, and value spaces, respectively. d k represents the dimension of the hidden layer;

[0070] Step 2.4, the multi-head attention mechanism also includes a batch normalization layer and a feedforward network with residual connections; input load segment The characterization of The result is predicted by a flattening layer with a linear head. Expressed as:

[0071]

[0072] Where T is the prediction step length, represents the load value at point t+L of the i-th load segment, represents the load value of the t+L+T-1th point in the i-th load segment;

[0073] Step 2.5. Use the mean squared error (MSE) loss to measure the difference between the predicted value and the actual value to obtain the target loss for base model training. Target loss Expressed as:

[0074]

[0075] Where j is the jth point of the load segment, represents the actual load value of the jth point in the i-th load segment, Indicates the load value of the jth point in the i-th load segment.

[0076] Step 3: Based on the base model, fine-tune the model for each power consumption pattern cluster (i.e., similar load segment cluster);

[0077] Furthermore, the Step 3 includes:

[0078] Step 3.1 First, use cross-correlation to determine two power load segments a = (a1, ..., a m ) and b=(b1,...,b m ), to achieve displacement invariance, b remains stationary, while a slides along b. For each offset s of a, the inner product of a and b in the current overlapping part is calculated; the offset of a is expressed as follows:

[0079]

[0080] Step 3.2 Calculate the cross-correlation between the two segments. The s value range is [-m, m]. Different s values ​​will calculate the cross-correlation according to the following formula, and a total of 2m+1 values ​​will be obtained;

[0081]

[0082] Among them, R s (a,b) represents the cross-correlation between a and b when the offset is s, b jis the load value of the jth point of b, a j+s is the load value of the j+sth point a, R -s (a, b) When the offset s is negative, the offset is converted to a positive value and then the cross-correlation is calculated;

[0083] Step 3.3 Calculate the shape-based distance (SBD) between segments a and b by dividing the maximum cross-correlation value at all offsets by the geometric mean of the autocorrelation of a single segment. The shape-based distance between a and b is expressed as:

[0084]

[0085] Where SBD(a,b) represents the shape-based distance between a and b, R0(a,a) and R0(b,b) are the correlations between the sequence itself and its unshifted version, respectively.

[0086] Step 3.4 Based on the shape-based distance between a and b, the centroid of the kth electricity consumption pattern cluster is calculated using the following optimization method:

[0087]

[0088] Among them, μ k represents the center of mass, represents the optimal value of the centroid, represents the set of payload segments of the kth cluster.

[0089] Step 4: Perform base model inference and prediction in the cold start scenario.

[0090] Furthermore, the Step 4 includes:

[0091] Step 4.1: For new user load segments, assign them to corresponding power consumption pattern clusters using the K-Shape algorithm.

[0092] Step 4.2: Use the corresponding fine-tuned model for prediction;

[0093] Let the new data segment be x new , calculated by K-Shape, the cluster to which it belongs is Then the corresponding prediction model is used for prediction:

[0094]

[0095] Among them, θ k represents the prediction model parameters for the kth power consumption mode, f(;) represents the prediction model, Indicates the prediction result.

[0096] The present invention also provides a system for cold start prediction of electric load based on pre-training and fine-tuning, the system comprising:

[0097] The segmentation module is used to collect customer load data and randomly divide it into training sets, validation sets, and test sets, dividing each customer load data into small segments and normalizing the segmented customer load data;

[0098] Base module, used to train the base model on PatchTST using payload segments;

[0099] A fine-tuning module is used to fine-tune the model for each power consumption pattern cluster (i.e., similar load segment cluster) based on the base model;

[0100] The inference and prediction module is used to perform inference and prediction of the base model in cold start scenarios.

[0101] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the above-mentioned method for predicting cold start of electric load based on pre-training and fine-tuning is implemented.

[0102] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements the above-mentioned pre-training-fine-tuning based power load cold start prediction method.

[0103] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the above-mentioned pre-training-fine-tuning based power load cold start prediction method.

[0104] In order to illustrate the prediction effect of the present invention, a baseline system is used for comparison with the present invention. Table 1 shows the performance comparison experimental results with the baseline model.

[0105] Table 1 shows the results of the comparative experiment between the baseline system and this method.

[0106]

[0107]

[0108] Table 1 details a series of comparative experimental results of our method and baseline models in cross-user prediction tasks. Using Train-1 as the training data, our method significantly outperforms PatchTST-K and N-BEATS in long-term prediction (prediction step length greater than 48), demonstrating its effectiveness in capturing long-term dependencies and adapting to diverse user characteristics. Notably, the PatchTST-T model performs particularly well in short-term prediction (prediction step length ≤ 48). This is primarily attributed to its direct use of the test user's historical data for training, enabling it to accurately capture the user's short-term behavioral characteristics. However, this training approach has significant limitations in practical applications because it relies on the availability of test data, which is impractical for cold-start prediction. Therefore, the performance of PatchTST-T can be considered an ideal upper limit or performance benchmark for cold-start prediction tasks. Specifically, when the prediction step size is 12 to 36, the MSE and MAE of our method are lower than those of PatchTST-K and N-BEATS, but higher than those of PatchTST-T. In particular, when the prediction step size is 36, the MSE and MAE of our method are 0.1358 and 0.2322, respectively, which are 21.5% and 16.0% lower than PatchTST-K, 33.6% and 25.8% lower than N-BEATS, but 11.5% and 5.8% higher than PatchTST-T. When the prediction step size is increased to 48, the prediction loss of each model increases, but the increase in the loss of our method is smaller, with an MSE of 0.1417, 0.5% higher than PatchTST-T, and a MAE of 0.2369, 0.9% lower than PatchTST-T. This indicates that at this prediction step size, our method has more error than PatchTST-T overall, but it has better prediction of extreme values. When the prediction step size is greater than 48, the proposed method achieves lower MSE and MAE than other models. In particular, when the prediction step size is 192, the proposed method achieves MSE and MAE of 0.1779 and 0.2639, respectively. This is 3.8% and 3.3% lower than PatchTST-T, 18.0% and 15.4% lower than PatchTST-K, and 43.8% and 33.9% lower than N-BEATS. These results demonstrate that the proposed method excels in new user load forecasting, especially in long-term forecasting scenarios. This is primarily attributed to the fact that the pre-training and fine-tuning strategy employed in this paper effectively captures the common features of power load data and learns features across various power consumption patterns. In contrast, PatchTST-T performs better in short-term forecasting, but its predictive ability gradually declines as the prediction step size increases, especially when the prediction step size exceeds 48, where it struggles to maintain its leading edge.PatchTST-K's performance is highly dependent on the similarity between training and test users. Even when selecting the most similar users, differences in load characteristics still exist, resulting in poor prediction results. N-BEATS outperforms PatchTST-K in short-term predictions, but its performance decreases significantly as the prediction step size increases. The N-BEATS model is less capable of modeling long-term relationships in power load.

[0109] When training on Train-2, Train-3, and Train-4, the results were similar to those on Train-1, indicating that differences in the training set have little impact on model performance, demonstrating the robustness of this method. In summary, the pre-training-fine-tuning load forecasting method can simultaneously capture common load characteristics and characteristics of individual power consumption patterns, thereby better addressing the cold start problem of power loads.

[0110] The specific embodiments of the present invention are described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Various changes can be made within the knowledge of ordinary technicians in this field without departing from the scope of the present invention.

Claims

1. A method for predicting cold start of power load based on pre-training and fine-tuning, characterized by: The method comprises: Step 1: Collect customer load data and randomly divide it into training, validation, and test sets. Each customer load data is divided into small segments and then normalized. Step 2: Use the payload fragment to train the base model on PatchTST. Step 3: Fine-tune the model for each power consumption pattern cluster based on the base model; Step 4: Perform base model inference and prediction in the cold start scenario; Step 2 includes: Step 2.1, User's Load Segments Split into overlapping batches; in the input load segment After splitting into batches, combine the split batches into matrices , is the length of the batch, L is the length of the look-back window, is the number of batches, , is the stride; Step 2.2, through a trainable linear projection matrix and a learnable position information encoding matrix , D represents the latent space dimension, mapping the input batch to the latent representation space , latent representation space Expressed as: ； Step 2.3, in the Transformer multi-head attention mechanism In the head, Convert to query matrix , key matrix Sum Matrix ,in 、 ; Then get the attention value through the scaled dot product , the value of attention Transpose Expressed as: ； in, 、 、 Represent the weight matrices that map the input vector to the query, key, and value spaces, respectively. 、 、 ; represents the dimension of the hidden layer; Step 2.

4. Input load segment The characterization of , the result is predicted by a flattening layer with a linear head, and the prediction result Expressed as: ； Where T is the prediction step length, represents the load value at point t+L of the i-th load segment, represents the load value of the t+L+T-1th point in the i-th load segment; Step 2.

5. Use mean square error loss to measure the difference between the predicted value and the actual value to obtain the target loss for base model training. , target loss Expressed as: ； Where j is the jth point of the load segment, represents the actual load value of the jth point in the i-th load segment, Indicates the load value of the jth point in the i-th load segment.

2. The method for predicting cold start of power load based on pre-training and fine-tuning according to claim 1, characterized in that: In Step 1, the following processing is performed: ； z is the result after normalization, x is the input sequence, is the mean of the input sequence, is the standard deviation of the input sequence.

3. The method for predicting cold start of power load based on pre-training and fine-tuning according to claim 1, characterized in that: Step 3 includes: Step 3.1 First, use cross-correlation to determine two power load segments of length m and The similarity, Stay still, and Then slide along b, for For each offset s, the inner product of a and b in the current overlapping part is calculated; The offset is expressed as follows: ； Step 3.2 Calculate the cross-correlation between the two segments; ； in, express and The cross-correlation at an offset of s is, for The load value of the j-th point, for The load value of the j+sth point, When the offset s is negative, convert the offset into a positive value and then calculate the cross-correlation; Step 3.3 Calculation fragment and , the shape-based distance between two segments is the maximum cross-correlation value at all offsets divided by the geometric mean of the autocorrelations of the individual segments, and The shape-based distance between them is expressed as: ； in, express and The shape-based distance between 、 are the correlations of the sequence itself and its unshifted version, respectively; Step 3.4 and Based on the shape-based distance between them, the centroid of the k-th electricity consumption pattern cluster is calculated using the following optimization method: ； in, represents the center of mass, represents the optimal value of the centroid, represents the set of payload segments of the kth cluster.

4. The method for predicting cold start of power load based on pre-training and fine-tuning according to claim 1, characterized in that: Step 4 includes: Step 4.1: For new user load segments, assign them to corresponding power consumption pattern clusters using the K-Shape algorithm. Step 4.2: Use the corresponding fine-tuned model for prediction; Let the new data segment be , calculated by K-Shape, the cluster to which it belongs is , then the corresponding prediction model is used for prediction: ； in, represents the prediction model parameters for the kth electricity consumption pattern, (;) indicates the prediction model, Indicates the prediction result.

5. A system for cold start prediction of power load based on pre-training and fine-tuning, characterized in that: The system includes: a module for executing the power load cold start prediction method based on pre-training and fine-tuning as described in any one of claims 1 to 4.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method for predicting cold start of electric load based on pre-training and fine-tuning as described in any one of claims 1 to 4 is implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for predicting cold start of electric load based on pre-training and fine-tuning as claimed in any one of claims 1 to 4 is implemented.

8. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for predicting cold start of electric load based on pre-training and fine-tuning as claimed in any one of claims 1 to 4 is implemented.

Images