Power time sequence prediction model training method and device based on dynamic learning rate adjustment

Abstract

The invention relates to a power time sequence prediction model training method and device based on dynamic learning rate adjustment. The method comprises the steps of obtaining a power time sequence sample set corresponding to a target power prediction task and a pre-training model matched with the target power prediction task; training the pre-training model by using the power time sequence sample set, collecting respective model performance parameters of each training period, and dynamically adjusting the learning rate of the pre-training model according to the model performance parameters; dynamically adjusting the learning rate of the pre-training model based on the model performance parameters respectively corresponding to at least two continuous training periods; and obtaining a power time sequence prediction model corresponding to the target power prediction task under the condition that the pre-training model under the current learning rate meets a training ending condition. By adopting the method, the accuracy of model training can be improved.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and power system technology, and in particular to a training method and apparatus for a power time series prediction model based on dynamic learning rate adjustment. Background Technology

[0002] Accurate time-series data prediction technology plays a crucial role in tasks such as power system dispatching, operation, and load management. With the development of deep learning and large model technology, more and more research is attempting to apply pre-trained large models to time-series prediction tasks in the power field to improve the model's ability to process complex time-series data.

[0003] However, when building power time series prediction models based on pre-trained large models, there is a problem of low model training accuracy. Summary of the Invention

[0004] Therefore, it is necessary to provide a method and apparatus for training a power time series prediction model based on dynamic learning rate adjustment, which can improve the accuracy of model training, in order to address the above-mentioned technical problems.

[0005] In a first aspect, this application provides a method for training a power time-series prediction model based on dynamic learning rate adjustment, comprising: acquiring a power time-series sample set corresponding to a target power prediction task and a pre-trained model matching the target power prediction task; training the pre-trained model using the power time-series sample set, collecting model performance parameters for each training cycle, and dynamically adjusting the learning rate of the pre-trained model according to the model performance parameters; dynamically adjusting the learning rate of the pre-trained model based on the model performance parameters corresponding to at least two consecutive training cycles; and obtaining a power time-series prediction model corresponding to the target power prediction task when the pre-trained model at the current learning rate meets the training termination condition.

[0006] The aforementioned power time-series prediction model training method based on dynamic learning rate adjustment obtains a power time-series sample set corresponding to the target power prediction task and a pre-trained model matching the target power prediction task. It then trains the pre-trained model using the power time-series sample set, collects model performance parameters for each training cycle, and dynamically adjusts the learning rate of the pre-trained model based on these parameters. Furthermore, it dynamically adjusts the learning rate based on the model performance parameters corresponding to at least two consecutive training cycles. Finally, when the pre-trained model at the current learning rate meets the training termination condition, the power time-series prediction model corresponding to the target power prediction task is obtained. Therefore, this application, when training the pre-trained model using a power time-series sample set, adjusts the learning rate by real-time sensing the model performance feedback of a single training cycle and further dynamically adjusts it based on the performance trends of multiple consecutive training cycles, thereby achieving dual dynamic optimization of the learning rate and improving the accuracy of model training. Therefore, compared to the traditional approach using a fixed learning rate, this application overcomes the problem of low model training accuracy caused by the inability of the fixed learning rate strategy to adapt to changes in the training stage, thus contributing to improved accuracy in power time-series prediction tasks. Attached Figure Description

[0007] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0008] Figure 1 This is an application environment diagram of a power time series prediction model training method based on dynamic learning rate adjustment in one embodiment;

[0009] Figure 2 This is a flowchart illustrating a power time-series prediction model training method based on dynamic learning rate adjustment in one embodiment.

[0010] Figure 3 This is a schematic diagram of the structure of a pre-trained model in one embodiment;

[0011] Figure 4 This is a flowchart illustrating a power time-series prediction model training method based on dynamic learning rate adjustment in another embodiment.

[0012] Figure 5 This is a structural block diagram of a power time series prediction model training device based on dynamic learning rate adjustment in one embodiment;

[0013] Figure 6This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0015] Power data is characterized by strong nonlinearity, high noise, strong seasonality, and diverse data sources, making it difficult for general-purpose large models to directly adapt to the specific needs of the power sector. Fine-tuning techniques are necessary for adaptation. Current technologies for power time series forecasting primarily employ traditional time series analysis methods, such as the Autoregressive Integrated Moving Average (ARIMA) model, machine learning algorithms (such as Support Vector Machines and Random Forests), and some small deep learning-based models, such as Long Short-Term Memory Networks (LSTM) and Gated Recurrent Units (GRUs). While these methods improve prediction accuracy to some extent, they still present numerous challenges when dealing with large-scale, heterogeneous power data, such as insufficient generalization ability, limited feature extraction capabilities, and difficulty adapting to various prediction scenarios. For example, due to the small sample size and high dimensionality of power data, directly applying large models faces the following major problems: small sample learning issues, such as the difficulty and high cost of obtaining labeled data in the power sector, resulting in a limited amount of usable data for specific prediction tasks, thus restricting the effective training of large models. Model adaptability issues include the complexity of power system operating environments and significant differences in data distribution across various scenarios, making it difficult for traditional fine-tuning methods to maintain prediction stability across multiple power scenarios. Overfitting and underfitting are also problematic; complex models with limited sample sizes are prone to overfitting, while excessive regularization or improper parameter settings can lead to underfitting, affecting prediction accuracy. Finally, there is the issue of lag in response to dynamic changes. Existing fine-tuning strategies are mostly static, lacking the ability to dynamically respond to model states during training, making it difficult to capture changes in model performance in a timely manner, thus reducing training efficiency and final performance.

[0016] Furthermore, large models (such as Transformer and its variants) have demonstrated powerful modeling and transfer capabilities in fields such as Natural Language Processing (NLP) and Computer Vision (CV), attracting attention in power time series forecasting. For example, when building power time series forecasting models based on pre-trained large models, the use of a fixed learning rate can lead to extremely slow initial convergence if the fixed learning rate is set too small, prolonging the training period and potentially preventing sufficient convergence under limited training resources, resulting in underfitting and low accuracy. Conversely, if the fixed learning rate is set too large, the model parameters will oscillate continuously around the optimal solution, causing drastic fluctuations in training loss and validation error, also resulting in low model training accuracy.

[0017] In view of this, this application provides a training method for power time series prediction models based on dynamic learning rate adjustment, which can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. Specifically, server 104 obtains the power time-series sample set corresponding to the target power prediction task and a pre-trained model matching the target power prediction task from terminal 102. It then trains the pre-trained model using the power time-series sample set, collects the model performance parameters for each training cycle, and dynamically adjusts the learning rate of the pre-trained model based on the model performance parameters. Furthermore, it dynamically adjusts the learning rate of the pre-trained model based on the model performance parameters corresponding to at least two consecutive training cycles. When the pre-trained model at the current learning rate meets the training termination condition, the power time-series prediction model corresponding to the target power prediction task is obtained.

[0018] The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, etc. The server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0019] In one embodiment, such as Figure 2 As shown, a training method for a power time-series prediction model based on dynamic learning rate adjustment is provided. Taking the application of this method to server 104 as an example, it includes:

[0020] S202, obtain the power time series sample set corresponding to the target power prediction task, and the pre-trained model that matches the target power prediction task.

[0021] Among them, the target power forecasting task can be a task that forecasts the operating status of the power system, a task that forecasts the load of the power system, or other tasks related to the power system.

[0022] For example, the electricity time-series sample set includes multiple time features, as well as meteorological and load features under each time feature. Time features can be represented in sequence form, including discrete time signals such as hour, day of the week, date, and holiday identifiers, used to represent the periodic changes in electricity demand. Meteorological features can be represented as continuous variables, including external environmental parameters affecting electricity load such as temperature, humidity, wind speed, air pressure, and light intensity, which can be obtained through meteorological monitoring systems or public meteorological feature interfaces. Load features are historical electricity consumption data, which can include historical user volume and historical user power, used to reflect user electricity consumption behavior at different times, and can be collected through power dispatching systems or smart meter data platforms.

[0023] S204 uses a power time series sample set to train the pre-trained model, collects the model performance parameters for each training cycle, and dynamically adjusts the learning rate of the pre-trained model based on the model performance parameters.

[0024] The pre-trained model refers to a model that has been pre-trained on a wide range of power time-series data and possesses good time-series feature extraction capabilities. The initial learning rate of the pre-trained model can be a pre-set learning rate or a learning rate determined in real time.

[0025] In one optional implementation, the initial learning rate of the pre-trained model can be determined based on Bayesian optimization. Bayesian optimization, based on prior distribution and numerical evaluation results of the objective function, automatically searches for the initial learning rate, batch size, and regularization weight combination that minimizes the validation loss over multiple iterations. Specifically, in the first iteration, several parameter combinations are randomly selected for model training, and the validation set loss value corresponding to each parameter combination is recorded. The parameter combinations include the learning rate, batch size, and regularization weights. The Gaussian distribution trend of the pre-trained model's parameter space is estimated based on the validation set loss value. A new parameter combination is predicted based on this distribution trend, and the validation set loss value corresponding to this new parameter combination is less than the preset loss value. After multiple updates, the learning rate in the optimal parameter combination is determined as the initial learning rate. That is, the parameters in the optimal parameter combination serve as the starting point for training the pre-trained model.

[0026] The model performance parameters are performance metrics used to characterize the current training state of the model, and may include training loss, validation loss, and gradient rate of change. Training loss represents the model's average error on the training data, validation loss represents the model's prediction error on the validation dataset, and the gradient rate of change reflects the magnitude of model parameter updates in adjacent training batches. The power time-series sample set includes both training and validation datasets.

[0027] S206 dynamically adjusts the learning rate of the pre-trained model based on the model performance parameters corresponding to at least two consecutive training cycles.

[0028] S208. If the pre-trained model at the current learning rate meets the training termination condition, the power time series prediction model corresponding to the target power prediction task is obtained.

[0029] In an optional implementation, the pre-trained model at the current learning rate is determined to meet the training termination condition if the training loss of the pre-trained model at the current learning rate is less than a set loss threshold. Alternatively, the pre-trained model at the current learning rate is determined to meet the training termination condition if the number of training cycles corresponding to the pre-trained model at the current learning rate reaches a preset number.

[0030] The method described in the above embodiments involves acquiring a power time-series sample set corresponding to the target power prediction task and a pre-trained model matching the target power prediction task. The pre-trained model is trained using the power time-series sample set, and model performance parameters are collected for each training cycle. The learning rate of the pre-trained model is dynamically adjusted based on these performance parameters, and further adjusted based on the model performance parameters corresponding to at least two consecutive training cycles. Finally, when the pre-trained model at the current learning rate meets the training termination condition, a power time-series prediction model corresponding to the target power prediction task is obtained. Therefore, this application, when training the pre-trained model using a power time-series sample set, adjusts the learning rate by real-time sensing the model performance feedback of a single training cycle and further dynamically adjusts it by combining the performance trends of multiple consecutive training cycles, thereby achieving dual dynamic optimization of the learning rate and improving the accuracy of model training. Therefore, compared to the traditional approach using a fixed learning rate, this application overcomes the problem of low model training accuracy caused by the inability of the fixed learning rate strategy to adapt to changes in the training stage, thus contributing to improved accuracy in power time-series prediction tasks.

[0031] In one embodiment, if the model performance parameters include training loss, then the learning rate of the pre-trained model is dynamically adjusted based on the model performance parameters corresponding to at least two consecutive training epochs. This includes: increasing the learning rate of the pre-trained model based on a first preset coefficient range if the decrease in training loss for M consecutive training epochs is less than an amplitude threshold. Here, M can be 5, the amplitude threshold can be 1%, and the lower limit of the first preset coefficient range is 10%, and the upper limit is 20%.

[0032] In an optional implementation, any coefficient is determined from a first preset coefficient range; the product between the chosen coefficient and the learning rate of the pre-trained model is determined; the sum of the pre-trained model's learning rate and the product is determined as the reduced learning rate. In other words, the learning rate of the pre-trained model is increased by any coefficient, for example, by 15%.

[0033] It's easy to understand that if the decrease in training loss over M consecutive training epochs is less than the magnitude threshold, it indicates that the rate of decrease in training loss is less than the rate of decrease threshold, suggesting that the model is converging slowly or is undertrained. The rate of decrease threshold can be calculated based on the average rate of decrease in loss during the pre-training phase of the pre-trained model.

[0034] In an optional implementation, if the decrease in training loss over M consecutive training epochs is less than a threshold, the batch size of the pre-trained model is adjusted. Specifically, while keeping the model's memory usage stable, the number of input samples is increased by 5% to 10% each time to increase the stability of gradient estimation and sample diversity, thereby improving the model's learning efficiency and parameter convergence speed.

[0035] In one embodiment, if the model performance parameters include training loss, then the learning rate of the pre-trained model is dynamically adjusted based on the model performance parameters corresponding to at least two consecutive training cycles, including: increasing the learning rate of the pre-trained model based on a third preset coefficient range when the training loss shows a decreasing trend over N consecutive training cycles and the decreasing range is within a second preset coefficient range; the upper limit of the second preset coefficient range is less than the lower limit of the third preset coefficient range, and the upper limit of the third preset coefficient range is less than or equal to the lower limit of the first preset coefficient range.

[0036] Wherein, N is less than M and greater than or equal to 2, for example, N can be 2 or 3; the lower limit of the second preset range can be 1% and the upper limit can be 3%; the lower limit of the third preset range can be 5% and the upper limit can be 10%. In the embodiments related to this application, M, S and N can also be set to other values.

[0037] In the optional implementation, if the training loss in two consecutive training cycles shows a decreasing trend and the decrease is within the range of the second preset coefficient, the learning rate of the pre-trained model is increased based on the range of the third preset coefficient.

[0038] In an optional implementation, the learning rate of the pre-trained model is increased based on a third preset coefficient range. This includes: determining a third coefficient from the third preset coefficient range; determining the product of the third coefficient and the learning rate of the pre-trained model; and summing the product with the learning rate of the pre-trained model to determine the adjusted learning rate. For example, if the third coefficient is 5%, the increase in the learning rate is 5% of the learning rate of the pre-trained model. Thus, by increasing the learning rate, model training efficiency can be improved and model convergence speed can be accelerated.

[0039] In one embodiment, the model performance parameters include validation loss and training loss. Then, based on the model performance parameters corresponding to at least two consecutive training cycles, the learning rate of the pre-trained model is dynamically adjusted, including: when the training loss corresponding to N consecutive training cycles shows a decreasing trend and the validation loss corresponding to two consecutive training cycles shows an increasing trend, the learning rate of the pre-trained model is reduced based on a fourth preset coefficient range.

[0040] The third preset coefficient range has a lower limit of 20% and an upper limit of 30%. It's easy to understand that if the training loss shows a decreasing trend over N consecutive training epochs, and the validation loss shows an increasing trend over two consecutive training epochs, it can be determined that the model is overfitting. Therefore, by reducing the learning rate, the risk of the model parameters overfitting the training data can be reduced.

[0041] In an optional implementation, reducing the learning rate of the pre-trained model based on a fourth preset coefficient range includes: determining a fourth coefficient from the fourth preset coefficient range; determining the product of the fourth coefficient and the learning rate of the pre-trained model; and determining the difference between the learning rate of the pre-trained model and the product as the adjusted learning rate, i.e., the reduction is equal to the fourth coefficient of the learning rate of the pre-trained model. Therefore, by increasing the learning rate, model training efficiency can be improved and model convergence speed can be accelerated.

[0042] In an optional implementation, the fourth coefficient can be any coefficient within a fourth preset range. Alternatively, the fourth coefficient can be determined from the fourth preset range based on the rate of increase of the verification loss.

[0043] In an optional implementation, if the training loss shows a decreasing trend over N consecutive training epochs, and the validation loss shows an increasing trend over two consecutive training epochs, the regularization parameter of the pre-trained model is increased. Specifically, the regularization parameter is increased to a preset multiple of its initial value to reduce the risk of the model parameters overfitting the training data. The preset multiple can be any multiple between 1.5 and 2.

[0044] In one embodiment, the model performance parameters include validation loss. Based on the model performance parameters corresponding to at least two consecutive training epochs, the learning rate of the pre-trained model is dynamically adjusted, including: if the validation loss shows an upward trend over S consecutive training epochs, and the average increase in each training epoch reaches any coefficient within the fifth preset coefficient range of the previous training epoch, the learning rate of the pre-trained model is reduced based on the fourth preset coefficient range; the lower limit of the fourth preset coefficient range is greater than the upper limit of the fifth preset coefficient range.

[0045] Where S is greater than or equal to 2 and less than M, for example, S can be any value from 2 to 4. The lower limit of the fifth preset range can be 5%, and the upper limit can be 15%. For example, if the learning rate of the pre-trained model is 0.01, the learning rate can be adjusted to any value from 0.007 to 0.008.

[0046] In an optional implementation, the learning rate of the pre-trained model is reduced based on a fourth preset coefficient range, including: determining a target coefficient from the fourth preset coefficient range; determining the product of the target coefficient and the learning rate of the pre-trained model; and determining the difference between the learning rate of the pre-trained model and the product as the adjusted learning rate. In other words, the reduction is equal to the target coefficient of the current learning rate.

[0047] The target coefficient can be any value within the fourth preset coefficient range. Alternatively, the target coefficient can be determined from the fourth preset coefficient range based on the rate of increase of the verification loss. Specifically, if the rate of increase is less than the lower limit of the warning range, the lower limit of the fourth preset coefficient range is determined as the target coefficient; if the rate of increase is greater than or equal to the upper limit of the warning range, the upper limit of the fourth preset coefficient range is determined as the target coefficient.

[0048] It is easy to understand that if the validation loss shows an upward trend for S consecutive training cycles, and the average increase in each training cycle reaches any coefficient within the fifth preset coefficient range of the previous training cycle, it can be determined that the model has entered an overfitting state. Therefore, by reducing the learning rate, the overfitting of the model to the training data can be reduced, which helps to improve the accuracy of model training.

[0049] In an optional implementation, if the validation loss shows an increasing trend over S consecutive training epochs, and the average increase in each training epoch reaches any coefficient within the fifth preset coefficient range of the previous training epoch, the regularization parameter of the pre-trained model is increased. Specifically, the regularization parameter is increased to a preset multiple of its initial value to enhance the regularization strength and reduce overfitting of the model to the training data. The preset multiple can be any multiple between 1.5 and 2.

[0050] In one embodiment, the model performance parameters include validation loss; dynamically adjusting the learning rate of the pre-trained model based on the model performance parameters corresponding to at least two consecutive training epochs includes: reducing the learning rate of the pre-trained model based on a second preset value when the change value of the validation loss corresponding to M consecutive training epochs is less than a first preset value; the second preset value is greater than the first preset value.

[0051] The change in validation loss refers to the difference between the validation loss in the current training period and the validation loss in the previous training period. The first preset value can be 0.5%, and the second preset value can be 80%.

[0052] It is easy to understand that if the change in validation loss is less than the first preset value for M consecutive training cycles, it means that the model’s performance on the validation set has not improved. Therefore, by reducing the learning rate, the model can be prompted to jump out of the local optimum and re-explore a better parameter space.

[0053] In an optional implementation, if the change in validation loss over ten consecutive training epochs is less than a first preset value, the learning rate of the pre-trained model is reduced based on a fourth preset value. The fourth preset value is less than the second preset value but greater than the first preset value. For example, the fourth preset value is 60%. In other words, the preset value can be determined based on the duration of loss stagnation, which characterizes the number of training epochs. For instance, if stagnation lasts for 5 training epochs, the learning rate is reduced to 80% of its original value; if stagnation lasts for 10 epochs, the learning rate is reduced to 60% of its original value, prompting the model to escape local optima and re-explore a better parameter space.

[0054] In one embodiment, if the model performance parameters include model parameter gradients, then the learning rate of the pre-trained model is dynamically adjusted based on the model performance parameters, including: if the gradient variances between multiple model parameter gradients are all greater than a variance threshold, the learning rate of the pre-trained model is reduced based on a fourth preset coefficient range.

[0055] Gradient variance measures the degree of variability between the gradients of different parameters. The variance threshold can be a fixed threshold or determined during the model initialization phase using the average gradient variance of the first five training epochs, specifically twice the average gradient variance during the initialization phase.

[0056] It is easy to understand that when the gradient variance between multiple model parameter gradients is greater than the variance threshold, it indicates that the model gradient fluctuates too much and there may be a risk of gradient explosion. In this case, by reducing the learning rate of the pre-trained model, for example, reducing the learning rate to any value between 20% and 30%, the parameter update step size can be reduced, making the model parameter changes more stable and preventing gradient explosion or oscillation.

[0057] In one embodiment, if the model performance parameters include model parameter gradients, then dynamically adjusting the learning rate of the pre-trained model based on the model performance parameters includes: reducing the learning rate of the pre-trained model based on a third preset value when the gradient change rate between model parameter gradients corresponding to adjacent training cycles is less than a first preset threshold.

[0058] The first preset threshold can be 0.5%, and the third preset value can be 10%. It's easy to understand that if the rate of change of the gradient between the model parameters in adjacent training cycles is less than the first preset threshold, it indicates that the rate of change of the gradient has stabilized over multiple training cycles, and the model can be determined to have entered the convergence phase. Therefore, by no longer increasing the learning rate but gradually decreasing it to 10% of the previous learning rate, stable convergence can be achieved. In the embodiments related to this application, the specific values ​​of the threshold, preset value, and preset coefficient range can all be determined by conducting numerous experiments on representative power time-series prediction tasks and statistically analyzing the results, in order to achieve better convergence results in different scenarios.

[0059] While monitoring gradient fluctuations, the gradient change trend is also recorded for multiple consecutive training cycles to determine whether to adjust the learning rate based on the gradient change trend.

[0060] In one embodiment, when the gradient variance shows an increasing trend over N consecutive training epochs, the learning rate of the pre-trained model is reduced by 10% to prevent further amplification of fluctuations. Further:

[0061] If, when reducing the learning rate of the pre-trained model, the gradient variance is greater than the variance threshold for M consecutive training epochs, the learning rate is further reduced until the fluctuations return to the normal range, i.e., the gradient variance is less than or equal to the variance threshold. If the gradient variance is less than 70% of the variance threshold for M consecutive training epochs, it indicates that the model has entered a stable convergence phase, and the learning rate is restored to the pre-adjustment level to maintain training efficiency.

[0062] When the gradient variance remains consistently high and fails to stabilize even after multiple reductions in the learning rate, it is determined that the model training process may have entered a poor convergence state. In this case, a re-initialization mechanism is triggered, recalculating the variance threshold and resetting the learning rate to 50% of the initial learning rate to re-establish a stable training step size. This mechanism executes cyclically throughout the entire training cycle, automatically adjusting the learning rate based on model gradient fluctuations at different stages, avoiding loss function oscillations or gradient explosion problems caused by excessive parameter updates.

[0063] In one embodiment, the model performance parameters include training loss, and the learning rate of the pre-trained model is dynamically adjusted according to the model performance parameters, including: when the decrease ratio corresponding to the training loss in N consecutive training cycles is less than a second preset threshold, the learning rate of the pre-trained model is increased based on a sixth preset coefficient range.

[0064] The decrease percentage refers to the percentage decrease in the current training cycle compared to the previous training cycle. The upper limit of the sixth preset coefficient range can be 20%, and the lower limit can be 10%. The second preset threshold can be a fixed threshold. Alternatively, the second preset threshold can be obtained statistically during the pre-training phase, for example, set to 20% of the average decrease percentage in the early stages of training. For example, if the average decrease in training loss per round in the early stages of training is 5%, then the second preset threshold can be set to 1%. Then, when the decrease percentage of training loss is less than 1% for three consecutive rounds, the learning rate is automatically increased by any value between 10% and 20%.

[0065] Using the method described above, if the decrease rate of training loss in N consecutive training cycles is less than the second preset threshold, it can be determined that the model training convergence speed is too slow. In this case, by automatically increasing the learning rate, the parameter update process can be accelerated and the training efficiency can be improved.

[0066] In one embodiment, the power time-series sample set includes multiple time features, as well as meteorological and load features under each time feature; the pre-trained large model includes multiple attention modules, and the time features correspond to the attention modules; the method further includes: for each time feature, using the attention module corresponding to the time feature, based on the meteorological and load features under the time feature, to obtain the correlation score corresponding to the time feature; normalizing the correlation score corresponding to the time feature to obtain the attention score corresponding to the time feature; and dynamically adjusting the learning rate of the pre-trained model according to the multiple attention scores corresponding to the training period in each training cycle.

[0067] Among them, multiple attention modules are used to model different dimensional features of the power time series sample set in parallel in multiple subspaces. The parallelism of multiple attention modules ensures that the model can capture short-term fluctuation features and long-term trend features at the same time. Thus, by ensuring that the model can capture time dependence and coupling relationship between variables in parallel on multiple feature dimensions, the model's ability to extract time correlation and coupling relationship between key variables is enhanced.

[0068] To ensure the stable operation of the attention modules, the weight distribution of each attention module is initialized uniformly in the early stage of training and automatically optimized based on the gradient update results during dynamic training.

[0069] In the optional implementation, the number of attention modules for each subspace can be determined based on the complexity of the power time-series sample set and the number of training samples in the set. For example, when the number of meteorological and load features under the time characteristics is large or includes multiple physical variables, the number of attention modules can be set to any number between 8 and 12 to achieve better feature extraction results; when the number of training samples is small or the feature similarity is high, the number of attention modules can be set to any number between 4 and 6 to reduce computational complexity.

[0070] In an optional implementation, an attention mechanism corresponding to time features is used to obtain the association score corresponding to the time features based on meteorological and load features under the time features. This includes: combining meteorological and load features under the time features to obtain the feature vector corresponding to the time features; determining the difference between the feature vector corresponding to the time features and the feature vectors corresponding to other time features besides the time features; weighted summing of the difference results corresponding to the time features to obtain the association score corresponding to the time features; combining the association scores corresponding to each time feature according to the time order to obtain the association score sequence; and obtaining the attention score corresponding to the time features after normalizing the association score sequence. The sum of the scores in the normalized association score sequence is 1.

[0071] By normalizing the associated score sequence using the method described in the above embodiments, the model can automatically focus on time segments with high feature similarity and more important change patterns, thereby improving the accuracy of overall feature modeling and further enhancing the accuracy of model training.

[0072] The difference results between feature vectors can characterize the magnitude of the difference, the consistency of direction, and the trend of change over time. The higher the association score corresponding to a time feature, the more important the feature vector corresponding to that time feature is in the entire sequence; the lower the association score, the smaller its influence.

[0073] It should be noted that the model generates an attention score distribution for all time features in each training cycle (composed of attention scores for each of the multiple time features), which serves as the basis for subsequent judgments on whether attention is over-concentrated or scattered.

[0074] In the optional implementation, for each time feature, after the meteorological and load features under the time feature have been standardized and encoded, the attention module corresponding to the time feature is used to obtain the association score corresponding to the time feature based on the processed meteorological and load features under the time feature.

[0075] In an optional implementation, the learning rate of the pre-trained model is dynamically adjusted based on multiple attention scores corresponding to the training cycle. This includes: reducing the learning rate of the pre-trained model based on a fifth preset value when the distribution trend among the multiple attention scores shows a concentrated distribution trend; and increasing the learning rate of the pre-trained model based on a seventh preset coefficient range when the distribution trend among the multiple attention scores shows a uniform distribution trend. The second preset value can be 20%, and the lower limit of the seventh preset coefficient range can be 10%, and the upper limit can be 15%.

[0076] In an optional implementation, if the distribution trend among multiple attention scores shows a concentrated distribution trend, the regularization parameter of the pre-trained model is increased.

[0077] Using the method described above, when the distribution trend among multiple attention scores shows a concentrated distribution trend, it indicates that some feature weights are too high while other feature weights are close to zero. This suggests that the model may be ignoring some feature information. Reducing the learning rate by 20% helps the model rebalance the feature weight distribution. Conversely, when the distribution trend among multiple attention scores shows a uniform distribution trend, it indicates that the model has failed to effectively distinguish key features. In this case, increasing the learning rate by any value between 10% and 15% strengthens the model's focus on significant features. This allows the model to achieve stable and balanced feature learning in the multi-dimensional space of multi-head attention, thus improving the accuracy of model training.

[0078] In one embodiment, the pre-trained large model further includes an input fusion layer, an output layer, and a feature fusion module. The feature fusion module includes a weighted fusion layer and a deep fusion layer connected in sequence. The input fusion layer is connected to the output layer through multiple attention modules, and the output layer is connected to the feature fusion module. The output layer is used to concatenate the feature mapping results output by multiple attention modules. The method further includes: obtaining the inter-layer output difference corresponding to the deep fusion layer while the input fusion layer processes the concatenation result; and dynamically adjusting the learning rate of the pre-trained model according to the inter-layer output difference.

[0079] The input fusion layer standardizes the preprocessed power time-series dataset, unifying all input values ​​to between 0 and 1. This ensures consistent scaling across different feature dimensions in the model calculations, unifying the processing of features from different sources and with different dimensions. This guarantees that time features, meteorological features, and load features meet the requirements of subsequent model calculations in terms of numerical range, scale, and arrangement. The data processed by the input fusion layer is then fed into multiple attention modules for further processing. Preprocessing includes data cleaning, missing value imputation, and normalization.

[0080] The splicing result reflects the important dependencies learned by the model in the fused feature space. The splicing result is further integrated by the feature fusion module, enabling the model to identify the impact of key variables on the target load at the overall level.

[0081] The feature fusion module concatenates and weights time features, meteorological features, and load features at the feature level to achieve a comprehensive expression of multi-dimensional information, enabling the model to identify the overall variation patterns of power load. Specifically, the feature fusion module includes a multi-layer fusion structure for fusing time features, meteorological features, and load features, and assigns different fusion weights according to the degree of influence of each feature on the prediction task.

[0082] In optional implementations, to address the problem of heterogeneous sources and significant differences in the intensity of influence of multidimensional features in power time series forecasting, the feature fusion module may include a weighted fusion layer and a deep fusion layer connected in sequence to ensure that time features, meteorological features and load features can be fully integrated and effectively expressed at different levels.

[0083] The weighted fusion layer assigns different fusion weights based on the importance of each feature in the prediction task. The initial weight values ​​are determined through multiple rounds of cross-validation on the sample set. Specifically, the validation errors when using time features, meteorological features, and load features for prediction are calculated separately, and the reciprocal of these errors is used as the initial weight allocation ratio. After the initial weights are set, the weights are dynamically fine-tuned based on the prediction results on the validation set in the first ten epochs of model training. If a feature group (i.e., the feature group consisting of time features, meteorological features, and load features) contributes significantly to the prediction results (i.e., the validation error increases by more than 5% after removing the feature alone), its corresponding weight is automatically increased by 10%. If a feature group contributes little (i.e., the validation error changes by less than 1% after removal), its weight is automatically decreased by 5%. This process iterates until the weight changes converge to within 1%, ensuring that the final weights reflect the actual impact of each feature on the prediction task.

[0084] The deep fusion layer inputs the weighted features from the weighted fusion layer into a multi-layer structure for nonlinear fusion. Each layer's nodes are automatically assigned input connections based on the feature weights. The first layer focuses on local feature fusion, such as the short-term dependency between time and load. The second layer extracts medium-term dependencies, focusing on the delayed impact of weather changes on the load curve. The third layer performs global feature fusion, achieving a holistic interactive representation of time, weather, and load features. After each layer's fusion is complete, the learning rate is dynamically adjusted based on the calculated inter-layer output differences.

[0085] In an optional implementation, the learning rate of the pre-trained model is dynamically adjusted based on the inter-layer output differences. This includes reducing the learning rate of the pre-trained model by 15% when the decrease in inter-layer output differences is less than a third preset threshold. This prompts the model to more finely adjust the fusion relationship between features, thereby improving the model's convergence stability and prediction accuracy. The third preset threshold can be 0.5%.

[0086] Inter-layer output difference refers to the degree of difference in the overall structure, numerical trend, and feature distribution of the output results of different fusion layers after processing the same batch of input features. In optional implementations, this difference can be calculated by comparing multiple metrics such as the average value, variance, magnitude of change, and similarity of the outputs of each layer in the deep fusion layer. Similarity measures whether different layers pay consistent attention to the same feature, while magnitude of change reflects the sensitivity of layers to input perturbations.

[0087] It's easy to understand that when the output differences between layers are small, it indicates that the multi-layer fusion results are becoming consistent, and the model is stable in terms of feature fusion. When the differences between layers increase significantly, it indicates that the processing of features by different layers has shifted, and the model may enter an unstable state. Therefore, judging whether further adjustment of the learning rate is needed based on the relationship between this difference and the third preset threshold can keep the layers coordinated during the feature fusion process.

[0088] In an optional implementation, data augmentation is performed on the preprocessed power time-series dataset to obtain the training samples required for each training cycle. The data augmentation process includes, in sequence, noise addition, temporal perturbation, random truncation, and sliding window sampling.

[0089] The noise addition process simulates the impact of measurement errors and external disturbances on the power time-series sample set. Specifically, a small amount of random noise is added to the values ​​of each time feature, with the noise amplitude set to 1% to 3% of the original data standard deviation. This amplitude is determined during model initialization and fixed after calculating the average error range on five different samples to ensure that the noise does not change the overall trend of the data. The time perturbation process enhances the model's adaptability to changes in time dependence. Specifically, one to two time steps are randomly shifted on the time feature index, causing a slight change in the time structure of the input samples, thereby training the model to recognize non-strictly aligned time-series features. The perturbation range is determined based on the data sampling interval; for example, when the data is sampled in hours, the time perturbation range is set to one to two hours. The random truncation process breaks the model's dependence on fixed-length sequences. Specifically, the starting position of the sequence is randomly selected each time training samples are generated, and 70% to 90% of the original sequence length is retained to form an incomplete sample sequence. This method allows the model to maintain stable predictive performance even when faced with missing or incomplete data. Sliding window sampling is used to expand the number of training samples. Specifically, a sliding window is used to extract subsequences on the power time series sample set with a fixed step size. The step size of each window slide is set to 10% to 20% of the original sequence length. Each subsequence is used as an independent sample to input into the model for training, thereby improving data diversity. This method is particularly suitable for scenarios with small sample power data and can significantly increase the number of effective training samples for the model.

[0090] After performing the above data augmentation operations, the distribution characteristics of the augmented data in each training cycle are recorded, and the prediction error difference between the augmented samples and the original samples is compared during the validation phase. Specifically, when the validation error of the augmented samples is less than 5% of the error of the original samples, it indicates that the augmentation operation has not damaged the data structure and can continue to be used; if the error increases by more than 10%, the perturbation or noise increase is automatically reduced to maintain the rationality and effectiveness of the data augmentation.

[0091] To improve the model's generalization ability and robustness in power time series prediction tasks, a data augmentation step is introduced during the training phase using the above-mentioned implementation method. By introducing controlled perturbations and randomization operations into the power time series sample set, multiple training samples with different distribution characteristics are constructed, enabling the model to stably learn the main features in a changing time series environment. This improves the model's robustness, alleviates overfitting, and reduces the risk of overfitting.

[0092] It is easy to understand that after completing the entire training process, the output is a power time-series forecasting model optimized by dynamic learning rate adjustment. During the forecasting phase, the model takes new time features, meteorological features, and historical load data into the same format. After processing by the multi-head attention layer and feature fusion module, it can output load forecast values ​​for future time periods. Therefore, the load forecast values ​​based on this output can be used for load forecasting or operation status analysis in power dispatch centers.

[0093] Based on the above, this application provides a dynamic learning rate adjustment strategy, employing three triggering methods corresponding to different training scenarios. The first method is based on changes in training and validation losses, used to control the decrease and increase of the learning rate when the validation loss continuously rises or decreases slowly. The second method is based on the attention score distribution, used to identify whether the model exhibits excessive focus or excessive dispersion of attention, thereby fine-tuning the learning rate to balance the attention areas of multi-head attention. The third method is based on inter-layer output differences, used to correct the learning rate to enhance inter-layer consistency when the cumulative deviation of outputs from different fusion layers exceeds a set range. When any of these triggering conditions is met, the corresponding learning rate adjustment action can be executed, allowing the learning rate to be specifically corrected according to different unstable performances during training. This ensures the training stability and convergence efficiency of the model in multi-feature fusion scenarios, further improving the accuracy of model training.

[0094] Based on the above, the core basis for dynamic adjustment of the learning rate is the model's performance indicators in the current training cycle, mainly including training loss, validation loss and their downward trend. By continuously evaluating changes in the objective function value, the learning rate is adaptively updated, thereby improving the stability and convergence efficiency of the training process.

[0095] In the optional implementation, the learning rate of the pre-trained model is automatically reduced when the validation loss does not decrease over several consecutive training epochs; and the learning rate of the pre-trained model is automatically increased when the rate of decrease of the training loss is below a threshold.

[0096] In an optional implementation, after each learning rate adjustment is triggered, the adjustment result is recorded in the training log, and the starting point for the next monitoring cycle is updated synchronously to avoid repeated triggering. If the loss curve does not improve after two consecutive adjustments, the system enters a stabilization phase, maintaining the learning rate at the current level and extending the monitoring window to 10 epochs to further observe the model's stability trend. The monitoring window is related to the training cycle and records model performance parameters for P consecutive training epochs. When all P performance parameters recorded within the monitoring window meet the corresponding conditions, a dynamic adjustment of the learning rate of the pre-trained model is triggered. P can be any value from 2 to 5.

[0097] In the optional implementation, when the loss decreases significantly in the early stages of model training, a large learning rate is maintained to quickly approach the optimal region; when the loss decreases stagnates or fluctuates, the learning rate is reduced to avoid oscillations and to finely search for the optimal parameters.

[0098] By adopting the above-mentioned method of dynamically adjusting the learning rate, adaptive learning rate control can be achieved based on the model's training performance, enabling the model to maintain efficient convergence at different training stages and ensuring the model's stability and efficient convergence during the training process.

[0099] In one embodiment, if the pre-trained model at the current learning rate meets the training termination condition, a power time-series prediction model corresponding to the target power prediction task is obtained, including: dynamically adjusting the regularization parameter of the pre-trained model based on the model performance parameters corresponding to at least two consecutive training cycles; and obtaining the power time-series prediction model corresponding to the target power prediction task if the pre-trained model at the current learning rate and the current regularization parameter meets the training termination condition.

[0100] The regularization parameters can include the L2 regularization coefficient and the dropout rate. In the initial training phase, the L2 regularization coefficient can be set to 0.0005, and the dropout rate to 0.1.

[0101] In one embodiment, the model performance parameters include validation loss and training loss. Based on the model performance parameters corresponding to at least two consecutive training epochs, the regularization parameter of the pre-trained model is dynamically adjusted, including: if the difference between the validation loss and the training loss is greater than a set value within N consecutive training epochs, the regularization parameter of the pre-trained model is increased; if the trend of the validation loss corresponding to M consecutive training epochs does not match the downward trend when the regularization parameter of the pre-trained model is increased, the regularization parameter of the pre-trained model is further increased.

[0102] In the optional implementation, the regularization parameters include the L2 regularization coefficient and the dropout rate. Increasing the regularization parameters of the pre-trained model includes increasing the L2 regularization coefficient by a sixth preset value and increasing the dropout rate by a seventh preset value. These preset values ​​can be 5%, 20%, and 10%, or other values.

[0103] It is easy to understand that if the difference between the validation loss and the training loss in N consecutive training cycles is greater than the third preset threshold, it indicates that the model is overfitting. Therefore, by increasing the regularization parameter, the model can be more constrained by the parameter weights and the proportion of random dropouts can be increased, thereby reducing the model complexity and suppressing feature redundancy.

[0104] In the optional implementation, if the trend of the validation loss corresponding to M consecutive training cycles does not match the downward trend, the regularization parameter of the pre-trained model is further increased, including: if the trend of the validation loss corresponding to M consecutive training cycles does not match the downward trend, the L2 regularization coefficient is increased by a seventh preset value, and the dropout rate is increased to the adjustment upper limit, which can be 30%.

[0105] It is easy to understand that if the trend of the validation loss corresponding to M consecutive training cycles does not match the downward trend, that is, the overfitting trend has not been alleviated after five consecutive rounds, the regularization suppression ability can be further strengthened by continuing to increase the regularization parameter.

[0106] The method described above is used to suppress overfitting during the training process of power time series prediction. By introducing the L2 regularization coefficient and dropout rate as dynamically adjustable regularization parameters, the values ​​of these two types of parameters are automatically adjusted according to the degree of overfitting in different training stages. This helps to maintain the generalization performance and convergence stability of the model, and further improves the accuracy of model training.

[0107] In one embodiment, if the difference between the validation loss and the training loss for N consecutive training epochs is less than an eighth preset value when the regularization parameter of the pre-trained model is increased, then the regularization parameter of the pre-trained model is decreased. The eighth preset value can be 1%.

[0108] In an optional implementation, the regularization parameters of the pre-trained model are reduced, including: reducing the L2 regularization coefficients of the pre-trained model based on a fourth preset coefficient range, and reducing the dropout rate based on a first preset coefficient range. The lower limit of the fourth preset coefficient range is greater than or equal to the upper limit of the first preset coefficient range. For example, the upper limit of the fourth preset coefficient range is 30%, and the lower limit is 20%; the upper limit of the first preset coefficient range is 20%, and the lower limit is 10%. Thus, the L2 regularization coefficients can be reduced by any value between 20% and 30%, and the dropout rate can be reduced by any value between 10% and 20%.

[0109] Using the method described in the above embodiment, if the difference between the validation loss and the training loss for N consecutive training cycles is less than the eighth preset value when the regularization parameter of the pre-trained model is increased, it indicates that the training loss and the validation loss are basically consistent. Therefore, it can be determined that the model fit is reasonable. By gradually weakening the regularization strength, i.e. reducing the regularization parameter, the model can have a stronger feature expression ability in the later stage of convergence, thereby obtaining higher prediction accuracy in the testing stage.

[0110] In one embodiment, if the training loss is too high and the rate of decrease is lower than the eighth preset value when the regularization parameter of the pre-trained model is increased, the model can be considered undertrained. Under the premise of keeping the current learning rate unchanged, the learning ability of the model can be enhanced by reducing the regularization coefficient and the dropout rate.

[0111] Understandably, after each training cycle, the change curve of the regularization parameter is recorded, and the strategy is dynamically adjusted in conjunction with the performance metrics, updating synchronously with the learning rate change curve. The adjustment of the L2 regularization coefficient and the dropout rate are both based on fixed ratio rules, not relying on randomness, thus ensuring the determinism and reproducibility of the adjustment results.

[0112] In one embodiment, training a pre-trained model using a power time-series sample set includes: determining the optimal hyperparameter combination as the initial hyperparameter combination; and training the pre-trained model using the power time-series sample set, given the initial hyperparameter combination is configured for the pre-trained model. For example, a Gaussian process can be used as a surrogate model, and by iteratively sampling and predicting the possible distribution of the objective function, a hyperparameter combination close to the global optimum can be obtained with fewer actual training iterations.

[0113] Specifically, the hyperparameter combination is first defined, including key hyperparameters such as learning rate, batch size, regularization coefficient, and momentum parameter. The value range of each hyperparameter is preset based on historical experience and hardware resources. For example, the learning rate ranges from 0.0001 to 0.01, the batch size ranges from 16 to 128, and the regularization coefficient ranges from 0.0001 to 0.001.

[0114] At the start of the search, five sets of hyperparameter combinations are randomly sampled. Simplified models are trained under each of these combinations, and the validation loss value for each set is recorded. Then, a Gaussian process surrogate model is constructed using these sampling results. The Gaussian process generates the predicted mean and uncertainty distribution across the entire parameter space by recording the relationship between each set of parameters and the corresponding validation loss, thereby estimating the possible range of model performance in the untested region. In each iteration, based on the current surrogate model, the expected improvement value for each candidate parameter combination is calculated. The expected improvement value represents the potential performance improvement that a particular candidate point can bring, given the known best result.

[0115] The specific operation is as follows: when the mean validation loss of a candidate parameter combination is significantly lower than the current best result and the uncertainty is small, it indicates a high expected improvement value, and this candidate parameter combination is prioritized for the next round of actual training and validation. When multiple candidate parameter combinations have similar mean validation losses, the combination with higher prediction uncertainty is selected for exploration to prevent getting trapped in local optima. After each iteration, the new validation loss results are added to the dataset, and the Gaussian process surrogate model is updated, gradually making the model's understanding of the entire parameter space more accurate.

[0116] The optimization process continues until the improvement in validation loss over three consecutive iterations is less than one percent. Finally, the set of hyperparameters with the lowest validation loss is identified as the optimal hyperparameter combination, which can then be used as the initial parameters for the pre-trained model. Using this Gaussian process-based surrogate search method, the entire hyperparameter optimization converges to near-optimal results in approximately twenty iterations, reducing computation by about 80% compared to traditional grid search.

[0117] In each round of experimental training for candidate parameter combinations, the actual validation loss value corresponding to that combination is recorded, and the correspondence between the candidate parameter combinations and their corresponding validation loss values ​​is continuously accumulated in the sample database. A Gaussian process surrogate model is constructed using the data in the sample database. This surrogate model takes the changing trend of the parameter combinations as input and the changing pattern of the validation loss as output, enabling it to predict loss values ​​for unexperimented regions in the entire parameter space and provide the mean validation loss and the range of prediction uncertainty. The mean validation loss reflects the model's central judgment on the possible validation loss of the candidate parameter combination, while the prediction uncertainty measures the confidence level in that judgment. During each round of parameter selection, the expected improvement value for each candidate parameter combination is calculated based on the current surrogate model; that is, the potential performance improvement that combination can bring based on the existing optimal results. The current optimal result of a candidate parameter combination is determined by the lowest validation loss obtained in all historical trials. When the predicted validation loss of a candidate parameter combination is lower than the best loss recorded in the database and the prediction uncertainty is small, it can be determined that its expected improvement value is high, and this parameter set is preferentially selected for the next round of training and validation. If multiple candidate parameter combinations have similar prediction results, the exploration continues from the combination with higher prediction uncertainty to prevent getting trapped in local optima. As new validation losses are continuously added to the database, the surrogate model is repeatedly updated, allowing for a progressively refined understanding of the parameter space. Finally, when the improvement in validation loss over three consecutive updates is less than a preset threshold, the parameter combination with the lowest validation loss can be selected from all candidate parameter combinations as the optimal hyperparameter combination. This optimal hyperparameter combination can then be used as the initial parameters for the pre-trained model.

[0118] Based on the above, besides dynamically adjusting the learning rate according to model performance parameters, the learning rate can also be non-linearly and periodically decreased during training using cosine annealing or periodic restart methods to improve model convergence efficiency and generalization ability. Therefore, by introducing a dynamic learning rate adjustment strategy based on cosine annealing and periodic restart, the model achieves non-linear and periodic learning rate changes during training, thus maintaining convergence speed while avoiding getting trapped in local optima, thereby improving overall prediction performance and generalization ability.

[0119] At the beginning of training, the pre-trained model is configured with an initial learning rate, a minimum learning rate, a maximum number of training epochs per training cycle, and a maximum number of restarts. The initial learning rate is determined based on the Bayesian optimization search results, and the minimum learning rate is set to 10% to 20% of the initial learning rate. A complete training process is divided into several training cycles, each containing a fixed number of training epochs. Taking a single training cycle containing fifty epochs as an example, the system gradually decreases the learning rate within each training cycle according to the decreasing trend of a cosine function curve. The learning rate reaches its maximum value at the beginning of the cycle and gradually decreases to the preset minimum value at the end of the cycle, achieving a smooth, non-linear decay.

[0120] The aforementioned training process incorporates a Bayesian optimization strategy based on a Gaussian process surrogate model. This strategy records, fits, and predicts the validation loss of different parameter combinations during experimental training, gradually establishing a correspondence between parameter combinations and performance, and providing the predicted effect and confidence range of each parameter in the unexperimented region. After multiple rounds of trials, the set with the lowest validation loss from all candidate parameter combinations is selected as the optimal result. Since the learning rate is a key parameter affecting the model's convergence speed and training stability, the candidate learning rates are evaluated multiple times using the aforementioned Bayesian optimization process at the beginning of training. The validation loss corresponding to each learning rate is included in the search range, and the prediction-experimentation-update cycle is repeated until a set of learning rate values ​​with the lowest validation loss is obtained. This optimal learning rate is the Bayesian optimization search result and is used as the learning rate setting in the initial training phase.

[0121] In an optional implementation, a periodic restart is performed at the end of each training cycle to restore the learning rate to its initial value and proceed to the next training cycle. After each training cycle, the learning rate can be dynamically adjusted based on the model's performance parameters. After each restart, the cycle length can be selectively shortened or the initial learning rate can be reduced to allow for faster and more stable convergence in subsequent training phases. If the model's validation loss does not decrease significantly (i.e., the decrease is less than one percent) over two consecutive training cycles, an automatic cycle extension mechanism is triggered, increasing the number of training rounds in the next training cycle by twenty percent to allow the model to converge more fully within the current parameter range.

[0122] Throughout the training process, the training loss, validation loss, and gradient changes are monitored in real time, and the cosine curve descent rate is adjusted based on the model's state after each training cycle. When slow model convergence or loss curve oscillations are detected, the cosine curve descent rate is reduced to make the learning rate decay process smoother, preventing learning stagnation caused by excessively rapid descent. When the validation loss continuously decreases and stabilizes, the system maintains the current policy to ensure the model continues to optimize along a smooth trajectory.

[0123] As discussed above, by introducing a dynamic strategy combining cosine annealing and periodic restarts, the model can undergo a complete process from rapid exploration to fine-grained convergence in each training cycle. During the rapid exploration phase, a larger learning rate helps the model escape local optima, while the fine-grained convergence phase achieves high-precision optimization within local regions by gradually decreasing the learning rate. The periodic restart mechanism ensures that the model can re-explore different parameter space regions multiple times, thereby obtaining more comprehensive optimization results and improving overall robustness and generalization ability.

[0124] The above describes the method for constructing a power time-series forecasting model. In practical applications, to improve the reliability and interpretability of this model, an uncertainty estimation module can be added to the model output stage. This ensures that the output includes not only the master forecast value but also the confidence interval, reflecting the credibility of the model's predictions. In other words, the model's output includes both the master forecast value and the confidence interval. Introducing the uncertainty estimation module enhances the model's risk perception capability and result interpretability in real-world power scenarios. The uncertainty estimation module comprehensively calculates the variance and historical residual characteristics of the model's prediction distribution, dynamically generating prediction confidence intervals, thereby helping the power system assess prediction risks and error boundaries during actual operation.

[0125] In practice, after completing the forward prediction, the model generates two types of information for each time feature of the output layer: one is the master prediction value, which is the load prediction result output by the model under the current input features; the other is the confidence interval estimate, which describes the uncertainty range of the prediction result. During the training phase, the variance of the prediction error is modeled by introducing an uncertainty estimation term into the loss function. At the end of each training epoch, the prediction residual of the current batch is calculated, which is the difference between the actual load value and the predicted value, and its variance trend over time is statistically analyzed. This variance value is used to reflect the prediction stability of the model in different time periods.

[0126] During the prediction phase, the confidence interval for each prediction time point is determined by combining historical residual variance with the current model output gradient information. When the residual variance is small and the gradient change is stable, the system considers the prediction stable, and the confidence interval is set to 5% to 10% above and below the master prediction value. When the residual variance is large or the gradient fluctuates significantly, the prediction is considered to have high uncertainty, and the confidence interval is expanded to 15% to 20% above and below the master prediction value. This percentage is determined by cross-validation during the model initialization phase to ensure that the prediction error distribution is reasonably reflected under different data scenarios.

[0127] Historical residuals refer to the differences between predicted and actual values ​​continuously recorded during the model's training and validation processes. After each round of training and validation, the error at each prediction time point in that round is saved, forming an error sequence accumulated over multiple training rounds. Historical residual variance is the degree of dispersion calculated based on this error sequence, reflecting the overall error fluctuation of the model across multiple training phases. This variance is calculated by first calculating the average deviation of all historical residuals, and then combining the differences between each residual and the average deviation to obtain the overall fluctuation range. A smaller variance indicates more stable prediction errors during past training; a larger variance indicates significant error fluctuations and model instability. Since the differences between predicted and actual values ​​are recorded across multiple training rounds, historical residual variance is a statistic generated from this accumulated data. It is used in conjunction with the gradient change information of the current model output during the prediction phase to determine the confidence range for each prediction time point. By introducing this metric, error patterns from past training processes can be used to assess the uncertainty of current predictions, thereby improving the overall stability and reliability of predictions.

[0128] Furthermore, the uncertainty estimation module is calibrated after each training cycle. The variance estimation parameters are dynamically adjusted by comparing the distribution differences between the predicted and actual values ​​on the validation set. When the proportion of predicted values ​​falling within the confidence interval on the validation set is less than 95%, the confidence interval is automatically widened by 5% to improve prediction coverage; when the proportion is greater than 98%, the confidence interval is narrowed by 5% to improve prediction resolution. This calibration process ensures that the output confidence interval is both reliable and reasonably tight.

[0129] Therefore, the model's final output consists of two parts: the master forecast reflects the central trend of power load, used for dispatching and planning decisions; and the confidence interval reflects the range of uncertainty in the forecast, used for risk assessment and safety margin management. Thus, through the power time-series forecasting model, potential risks can be warned of using confidence interval information in scenarios of drastic power load fluctuations or significant changes in weather conditions, providing dispatchers with a reference for decision-making.

[0130] In summary, addressing the characteristics of power sector data such as small sample size, high noise, and heterogeneous multi-source data, as well as the problems of lag and insufficient generalization ability in existing fine-tuning strategies, this application presents a highly efficient fine-tuning method that integrates multiple attention modules, feature fusion structures, dynamic learning rate control, and Bayesian optimization. This method adaptively adjusts the learning rate and regularization parameters by monitoring training state indicators in real time, effectively solving the overfitting or underfitting problems caused by static parameter settings in traditional fine-tuning processes, and significantly improving the model's ability to model complex time-series features of the power sector. Simultaneously, by combining the Bayesian optimization algorithm to search for initial hyperparameter combinations, it not only improves training efficiency but also enhances the model's adaptability and prediction accuracy under different power scenarios, achieving accurate prediction of power operation status or load under small sample conditions, and possessing strong engineering practical value and promotion potential.

[0131] In summary, such as Figure 3 The diagram illustrates the structure of a pre-trained model. The pre-trained model includes an input fusion layer, multiple attention modules, an output layer, and a feature fusion module connected sequentially.

[0132] The input fusion layer standardizes the preprocessed power time-series dataset, unifying all input values ​​to between 0 and 1. This ensures consistent scaling across different feature dimensions in the model calculations, unifying the processing of features from different sources and with different dimensions. This guarantees that time features, meteorological features, and load features meet the requirements of subsequent model calculations in terms of numerical range, scale, and arrangement. The data processed by the input fusion layer is then fed into multiple attention modules for further processing. The output layer concatenates the feature mapping results from multiple attention modules. The concatenated results are further integrated by the feature fusion module, enabling the model to identify the impact of key variables on the target load at a holistic level. The feature fusion module consists of a weighted fusion layer and a deep fusion layer connected sequentially.

[0133] Combination Figure 3 The structure shown is as follows: Figure 4 As shown, a training method for a power time-series prediction model based on dynamic learning rate adjustment is provided, and this method is applied to... Figure 1 Taking server 104 as an example, it includes:

[0134] S402, Obtain the power time series sample set corresponding to the target power forecasting task; the power time series sample set includes multiple time features, as well as meteorological features and load features under each time feature.

[0135] S404, Obtain the pre-trained model matching the target power prediction task. The pre-trained model includes an input fusion layer, multiple attention modules, an output layer, and a feature fusion module connected in sequence.

[0136] S406 uses a power time-series sample set to train the pre-trained model, collects the model performance parameters for each training cycle, and dynamically adjusts the learning rate of the pre-trained model based on the model performance parameters. The model performance parameters include at least one of validation loss, training loss, and model parameter gradients.

[0137] S408 dynamically adjusts the learning rate and regularization parameters of the pre-trained model based on the model performance parameters corresponding to at least two consecutive training cycles.

[0138] S410, under the current learning rate and current regularization parameters, if the pre-trained model meets the training termination condition, the power time series prediction model corresponding to the target power prediction task is obtained.

[0139] The contents of S402 to S410 can be referred to the aforementioned content description, and will not be repeated here.

[0140] Based on the above, it can be seen that the method provided in this application achieves accurate prediction of power operation status or load under small sample conditions, and has strong engineering practical value and promotion potential.

[0141] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0142] Based on the same inventive concept, this application also provides a power time series prediction model training device based on dynamic learning rate adjustment for implementing the power time series prediction model training method based on dynamic learning rate adjustment described above. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more power time series prediction model training device embodiments based on dynamic learning rate adjustment provided below can be found in the limitations of the power time series prediction model training method based on dynamic learning rate adjustment described above, and will not be repeated here.

[0143] In one embodiment, such as Figure 5 As shown, a power time-series prediction model training device based on dynamic learning rate adjustment is provided, including: an acquisition module 502, a first adjustment module 504, a second adjustment module 506, and a processing module 508. The acquisition module 502 is used to acquire a power time-series sample set corresponding to the target power prediction task, and a pre-trained model matching the target power prediction task; the first adjustment module 504 is used to train the pre-trained model using the power time-series sample set, collect the model performance parameters for each training cycle, and dynamically adjust the learning rate of the pre-trained model according to the model performance parameters; the second adjustment module 506 is used to dynamically adjust the learning rate of the pre-trained model based on the model performance parameters corresponding to at least two consecutive training cycles; and the processing module 508 is used to obtain the power time-series prediction model corresponding to the target power prediction task when the pre-trained model at the current learning rate meets the training termination condition.

[0144] In one embodiment, the model performance parameters include training loss; the second adjustment module 506 is further configured to perform any of the following: First, when the decrease in training loss for M consecutive training epochs is less than the magnitude threshold, increase the learning rate of the pre-trained model based on a first preset coefficient range; Second, when the training loss for N consecutive training epochs shows a decreasing trend and the decrease is within the second preset coefficient range, increase the learning rate of the pre-trained model based on a third preset coefficient range; the upper limit of the second preset coefficient range is less than the lower limit of the third preset coefficient range, and the upper limit of the third preset coefficient range is less than or equal to the lower limit of the first preset coefficient range.

[0145] In one embodiment, the model performance parameters include validation loss and training loss; the second adjustment module 506 is further configured to: reduce the learning rate of the pre-trained model based on a fourth preset coefficient range when the training loss for N consecutive training cycles shows a decreasing trend and the validation loss for two consecutive training cycles shows an increasing trend.

[0146] In one embodiment, the model performance parameters include validation loss; the second adjustment module 506 is further configured to perform any of the following: First, if the validation loss shows an upward trend for S consecutive training epochs, and the average increase in each training epoch reaches any coefficient in the fifth preset coefficient range of the previous training epoch, reduce the learning rate of the pre-trained model based on the fourth preset coefficient range; the lower limit of the fourth preset coefficient range is greater than the upper limit of the fifth preset coefficient range; Second, if the change value of the validation loss for M consecutive training epochs is less than the first preset value, reduce the learning rate of the pre-trained model based on the second preset value; the second preset value is greater than the first preset value.

[0147] In one embodiment, the model performance parameters include model parameter gradients; the first adjustment module 504 is further configured to perform any of the following: first, when the gradient variance between multiple model parameter gradients is greater than a variance threshold, reduce the learning rate of the pre-trained model based on a fourth preset coefficient range; second, when the gradient change rate between model parameter gradients corresponding to adjacent training periods is less than a first preset threshold, reduce the learning rate of the pre-trained model based on a third preset value.

[0148] In one embodiment, the power time series sample set includes multiple time features, as well as meteorological and load features under each time feature; the pre-trained large model includes multiple attention modules, and the time features correspond to the attention modules; the processing module 508 is further configured to: for each time feature, use the attention module corresponding to the time feature to obtain the correlation score corresponding to the time feature based on the meteorological and load features under the time feature; normalize the correlation score corresponding to the time feature to obtain the attention score corresponding to the time feature; and dynamically adjust the learning rate of the pre-trained model according to the multiple attention scores corresponding to the training period in each training period.

[0149] In one embodiment, the pre-trained large model further includes an input fusion layer, an output layer, and a feature fusion module. The feature fusion module includes a weighted fusion layer and a deep fusion layer connected in sequence. The input fusion layer is connected to the output layer through multiple attention modules, and the output layer is connected to the feature fusion module. The output layer is used to concatenate the feature mapping results output by multiple attention modules. The first adjustment module 504 is further used to: obtain the inter-layer output difference corresponding to the deep fusion layer after the concatenated result has passed through the weighted fusion layer and the deep fusion layer; and dynamically adjust the learning rate of the pre-trained model according to the inter-layer output difference.

[0150] In one embodiment, the processing module 508 is further configured to: dynamically adjust the regularization parameters of the pre-trained model based on the model performance parameters corresponding to at least two consecutive training cycles; and obtain the power time series prediction model corresponding to the target power prediction task when the pre-trained model under the current learning rate and the current regularization parameters meets the training termination condition.

[0151] In one embodiment, the model performance parameters include validation loss and training loss; the second adjustment module 506 is further configured to: increase the regularization parameter of the pre-trained model if the difference between the validation loss and the training loss is greater than a set value in N consecutive training cycles; and if the trend of the validation loss in M ​​consecutive training cycles does not match the downward trend when the regularization parameter of the pre-trained model is increased, then continue to increase the regularization parameter of the pre-trained model.

[0152] The modules in the aforementioned power time-series prediction model training device based on dynamic learning rate adjustment can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0153] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores data from the training process of a power time-series prediction model based on dynamic learning rate adjustment. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for training a power time-series prediction model based on dynamic learning rate adjustment.

[0154] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0155] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0156] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0157] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0158] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0159] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0160] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this application. The above embodiments only illustrate several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of this application. It should be noted that for those skilled in the art, several modifications and improvements can be made without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A training method for a power time-series prediction model based on dynamic learning rate adjustment, characterized in that, include: Obtain the power time series sample set corresponding to the target power prediction task, and the pre-trained model that matches the target power prediction task; The pre-trained model is trained using the power time series sample set, and the model performance parameters for each training cycle are collected. The learning rate of the pre-trained model is then dynamically adjusted based on the model performance parameters. The learning rate of the pre-trained model is dynamically adjusted based on the model performance parameters corresponding to at least two consecutive training cycles. If the pre-trained model at the current learning rate meets the training termination condition, the power time series prediction model corresponding to the target power prediction task is obtained.

2. The method according to claim 1, characterized in that, The model performance parameters include training loss; the dynamic adjustment of the learning rate of the pre-trained model based on the model performance parameters corresponding to at least two consecutive training epochs includes any one of the following: First item: If the decrease in training loss for M consecutive training cycles is less than the magnitude threshold, the learning rate of the pre-trained model is increased based on a first preset coefficient range. Second item: If the training loss shows a decreasing trend over N consecutive training cycles, and the decrease is within the range of the second preset coefficient, the learning rate of the pre-trained model is increased based on the range of the third preset coefficient. N is less than M, the upper limit of the second preset coefficient range is less than the lower limit of the third preset coefficient range, and the upper limit of the third preset coefficient range is less than or equal to the lower limit of the first preset coefficient range.

3. The method according to claim 1, characterized in that, The model performance parameters include validation loss and training loss; dynamically adjusting the learning rate of the pre-trained model based on the model performance parameters corresponding to at least two consecutive training epochs includes: If the training loss shows a decreasing trend over N consecutive training cycles and the validation loss shows an increasing trend over two consecutive training cycles, the learning rate of the pre-trained model is reduced based on the fourth preset coefficient range.

4. The method according to claim 1, characterized in that, The model performance parameters include validation loss; the dynamic adjustment of the learning rate of the pre-trained model based on the model performance parameters corresponding to at least two consecutive training epochs includes any one of the following: First item: If the validation loss shows an upward trend for S consecutive training cycles, and the average increase in each training cycle reaches any coefficient in the fifth preset coefficient range of the previous training cycle, the learning rate of the pre-trained model is reduced based on the fourth preset coefficient range. The lower limit of the fourth preset coefficient range is greater than the upper limit of the fifth preset coefficient range; Second item: If the change in validation loss for M consecutive training epochs is less than a first preset value, the learning rate of the pre-trained model is reduced based on a second preset value; the second preset value is greater than the first preset value.

5. The method according to claim 1, characterized in that, The model performance parameters include model parameter gradients; dynamically adjusting the learning rate of the pre-trained model based on the model performance parameters includes any one of the following: First item: If the gradient variance between multiple model parameter gradients is greater than the variance threshold, the learning rate of the pre-trained model is reduced based on the fourth preset coefficient range. Second item: If the rate of change of the gradient between the model parameters in adjacent training cycles is less than a first preset threshold, the learning rate of the pre-trained model is reduced based on a third preset value.

6. The method according to claim 1, characterized in that, The power time-series sample set includes multiple time features, as well as meteorological and load features for each time feature; the pre-trained large model includes multiple attention modules, and the time features correspond to the attention modules; the method further includes: For each of the time features, the attention module corresponding to the time feature is used to obtain the correlation score corresponding to the time feature based on the meteorological features and load features under the time feature; The correlation scores corresponding to the time features are normalized to obtain the attention scores corresponding to the time features. Within each training cycle, the learning rate of the pre-trained model is dynamically adjusted based on multiple attention scores corresponding to the training cycle.

7. The method according to claim 6, characterized in that, The pre-trained large model further includes an input fusion layer, an output layer, and a feature fusion module. The feature fusion module includes a weighted fusion layer and a deep fusion layer connected in sequence. The input fusion layer is connected to the output layer through the multiple attention modules. The output layer is connected to the feature fusion module. The output layer is used to concatenate the feature mapping results output by the multiple attention modules. The method further includes: After the splicing result passes through the weighted fusion layer and the deep fusion layer, the inter-layer output difference corresponding to the deep fusion layer is obtained; The learning rate of the pre-trained model is dynamically adjusted based on the differences in output between the layers.

8. The method according to any one of claims 1 to 7, characterized in that, When the pre-trained model at the current learning rate meets the training termination condition, the power time-series prediction model corresponding to the target power prediction task is obtained, including: The regularization parameters of the pre-trained model are dynamically adjusted based on the model performance parameters corresponding to at least two consecutive training cycles. If the pre-trained model under the current learning rate and current regularization parameters satisfies the training termination condition, the power time series prediction model corresponding to the target power prediction task is obtained.

9. The method according to claim 8, characterized in that, The model performance parameters include validation loss and training loss; the dynamic adjustment of the regularization parameters of the pre-trained model based on the model performance parameters corresponding to at least two consecutive training epochs includes: If the difference between the validation loss and the training loss is greater than a set value in N consecutive training cycles, the regularization parameter of the pre-trained model is increased. If the trend of the validation loss over M consecutive training epochs does not match the downward trend when the regularization parameter of the pre-trained model is increased, then the regularization parameter of the pre-trained model is further increased.

10. A training device for a power time-series prediction model based on dynamic learning rate adjustment, characterized in that, The device includes: The acquisition module is used to acquire the power time series sample set corresponding to the target power prediction task, as well as the pre-trained model that matches the target power prediction task; The first adjustment module is used to train the pre-trained model using the power time series sample set, collect the model performance parameters for each training cycle, and dynamically adjust the learning rate of the pre-trained model according to the model performance parameters. The second adjustment module is used to dynamically adjust the learning rate of the pre-trained model based on the model performance parameters corresponding to at least two consecutive training cycles. The processing module is used to obtain the power time series prediction model corresponding to the target power prediction task when the pre-trained model at the current learning rate meets the training termination condition.

Images