Multi-edge collaborative load forecasting method based on personalized federated deep learning

Abstract

This invention provides a multi-edge collaborative load prediction method based on personalized federated deep learning, comprising the following steps: Step 1: The parameter server selects clients to participate in this round of federated aggregation and sends work requests to the clients; Step 2: The selected clients decide whether to participate in this round of federated aggregation based on the current status of the edge servers and return an accept / reject response to the parameter server; Step 3: The parameter server distributes the global model and global control parameters to the clients participating in the federated aggregation; Step 4: The clients preprocess the historical load data on their edge servers, including data cleaning, resampling, normalization, and denoising. This technical solution integrates multi-edge collaboration with personalized federated deep learning, and by solving key problems such as high noise, high variability, insufficient data volume, and poor model generalization ability of load data, it adaptively achieves efficient and accurate prediction of edge load.

Description

Technical Field

[0001] This invention relates to the field of Internet of Things (IoT) technology, and in particular to a multi-edge collaborative load prediction method based on personalized federated deep learning. Background Technology

[0002] As an emerging computing paradigm in the Internet of Things (IoT) era, edge computing effectively reduces the response time of IoT applications (such as autonomous driving, AR / VR, and smart cities) and improves user experience by deploying computing and storage resources closer to the network edge. By 2025, the total number of IoT-connected devices installed globally is projected to reach 30.9 billion. Uploading all the operational data and computing tasks of IoT applications on terminal devices to remote cloud data centers would undoubtedly consume a large amount of bandwidth resources and place a huge processing burden on cloud data centers. Simultaneously, long-distance data transmission between cloud data centers and terminal devices would also cause severe response latency. In contrast, edge computing can significantly reduce the latency of data transmission and processing tasks, thereby reducing system costs and improving service quality to some extent. Therefore, the emergence of edge computing can effectively support the high performance demands of a large number of emerging IoT applications.

[0003] As a crucial technology supporting edge computing, load forecasting better supports the pre-configuration and allocation of resources in edge systems, thereby achieving more efficient and rational resource supply. For example, when a large number of service requests arrive at an edge server simultaneously, insufficient resource allocation increases response time. Conversely, when only a small number of service requests arrive at the edge server over a prolonged period, over-allocation of resources leads to frequent idle periods, resulting in resource waste. Predicting edge load changes over a future period and adjusting resource supply accordingly can better guarantee Service Level Agreements (SLAs) and effectively improve the operational efficiency of edge systems. For instance, edge systems can use load forecasting results to guide the allocation and migration of virtual machines, alleviating issues such as server overload or network congestion. Furthermore, load forecasting can effectively improve the resource utilization of edge systems and reduce operating costs.

[0004] Existing research on load forecasting largely focuses on cloud environments, typically employing regression, heuristic, or classical neural networks. Classical neural networks can achieve accurate predictions for loads with clear changing trends, but they usually only contain shallow network structures, such as Multilayer Perceptrons (MLPs) and Radial Basis Functions (RBFs). Therefore, when facing highly variable edge loads, classical neural networks cannot achieve high prediction accuracy because they cannot effectively capture the patterns of edge load changes. To address this issue, Recurrent Neural Networks (RNNs) have been proposed and applied to load forecasting, demonstrating good performance in time-series data modeling and processing. However, due to the vanishing gradient problem, classical RNNs struggle to learn long-term memory dependencies. To solve this problem, some improved variants of RNNs have been proposed, such as Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRUs), which exhibit excellent learning capabilities for long-term memory. Furthermore, due to noise interference in the raw data, nonlinear models often suffer from overfitting and poor generalization, which significantly affects the model's prediction accuracy. Unlike cloud environments, server deployments in edge environments are typically smaller in scale, more discretely distributed, and have limited historical load data storage, posing the following significant challenges to edge load prediction.

[0005] (1) High variability and high noise in load. Due to the dynamic nature of the edge environment, the diversity of user requests, and the elastic supply of resources, the load of different edge servers is usually highly variable and unevenly distributed over a period of time. For example, the maximum difference in load between edge servers across sites is 19.8 times, and the maximum difference in load between edge servers within the same site is 14.3 times. In addition, the complex edge environment also introduces a lot of noise into the load data. This high variability and high noise pose a great challenge to accurate and efficient load prediction.

[0006] (2) Limited historical load data in a single edge server. Deep learning-based load prediction models typically require training on a large amount of historical load data to support high accuracy. However, compared to cloud data centers, a single edge server usually has less historical load data, making it difficult to obtain a high-accuracy prediction model by independently training local load data.

[0007] (3) Poor model generalization ability. The load status of different edge servers is constantly changing dynamically due to their deployment environment and application scenarios, which leads to significant differences in the distribution of load data among different edge servers. Under such highly differentiated data samples, the generalization ability of the load prediction model trained by a single edge server is severely insufficient and cannot adapt well to different edge environments. Summary of the Invention

[0008] In view of this, the purpose of this invention is to provide a multi-edge collaborative load prediction method based on personalized federated deep learning, which integrates multi-edge collaboration and personalized federated deep learning. By solving key problems such as high noise, high variability, insufficient data volume, and poor model generalization ability of load data, it adaptively achieves efficient and accurate prediction of edge load.

[0009] To achieve the above objectives, the present invention adopts the following technical solution: a multi-edge collaborative load prediction method based on personalized federated deep learning, comprising the following steps:

[0010] Step 1: The parameter server selects clients to participate in this round of federated aggregation and sends work requests to the clients;

[0011] Step 2: The selected client decides whether to participate in this round of federated aggregation based on the current status of the edge server, and returns an accept / reject response to the parameter server;

[0012] Step 3: The parameter server distributes the global model w and global control parameters c to the clients participating in the federated aggregation;

[0013] Step 4: The client preprocesses the historical load data on its edge server, including data cleaning, resampling, normalization and noise reduction, in order to extract important features from the original data;

[0014] Step 5: Based on the global model w, global control parameter c, and local control parameter c k The client updates its local load prediction model w after several rounds of training, validation, and testing. k and local control parameter c k ;

[0015] Step 6: The trained model can be used to predict future edge load, which will help edge service providers to develop appropriate resource scheduling schemes to achieve load balancing among edge servers.

[0016] Step 7: The parameter server waits for all clients participating in the federated aggregation to complete their local training and upload their local models and local control parameters;

[0017] Step 8: The parameter server aggregates all received local models and updates the global model, then begins the next round of federated learning training.

[0018] In a preferred embodiment, the objective function of the MC-2PF general framework for multi-edge collaborative load prediction based on personalized federated deep learning is defined as:

[0019]

[0020] Among them, w k This is the local model for client k.

[0021] In a preferred embodiment, the MC-2PF model uses an RNN-based method as the client model of the federation for time series prediction; through data preprocessing and model training, the RNN-based method captures the temporal dependencies in the sequence and uses past observations to predict future changes; based on historical memory and current input, the future prediction output is defined as:

[0022]

[0023] Among them, s t x t and y t Let represent the hidden state, input, and output at time t, respectively.

[0024] In a preferred embodiment, the metrics for evaluating the performance of the MC-2PF include:

[0025] (1) MSE is the mean squared error, which is defined as:

[0026]

[0027] Among them, y t Let be the actual load value at time t. Let L be the predicted load value at time t, and L be the prediction length.

[0028] (2) MAE is the mean absolute error, which is defined as:

[0029]

[0030] (3)R 2 The coefficient of determination reflects the model's fit and is defined as:

[0031]

[0032] in, R 2 The value of is in the interval [-∞, 1], when R 2The closer the value is to 1, the better the model fits.

[0033] In a preferred embodiment, the workflow of the MC-2PF model includes: first, the parameter server initializes the global model w, global control parameters c, and the number of clients K; each client maintains its local control parameters c. k c and c k All are initialized to 0 to ensure For each communication round, the parameter server randomly selects max(C·K,1) clients from all K clients according to the selection ratio C to participate in federated aggregation and form a client set S. The function max(,) is used to avoid training interruption due to no clients being selected. Then, the parameter server distributes the global model and global control parameters (w,c) to the clients in S.

[0034] In a preferred embodiment, the workflow of the MC-2PF model includes: Next, the client k (k∈S) receives (w,c) and updates its local model to w. k ←w; Specifically, each client retrieves historical load data locally for training its local model; this load data contains runtime information about the edge server; CPU utilization, which best reflects the performance of the edge system, is used as the primary predictive metric; for an edge server, x t The CPU utilization at time t is represented by X = {x1, x2, ..., xt}. The historical load sequence of this edge server can be denoted as X = {x1, x2, ..., xt}. n During data preprocessing, the data is replaced by the interval average and compressed through resampling; it is then normalized to accelerate the convergence of the local model; after normalization, the original data is mapped to the interval [0,1]; and is defined as:

[0035]

[0036] Where X represents the original load data; X max and X min These represent the maximum and minimum CPU utilization rates in the original load data, respectively.

[0037] Design a Savitzky-Golay-based filter to preprocess the raw load data, and define a filter based on... subsequence of:

[0038]

[0039] in, q is S q The center point, S q The length is (2w+1), where w is half the length of the window.

[0040] In a preferred embodiment, the workflow of the MC-2PF model includes: Next, for S... q Fit each element in:

[0041]

[0042] in, x and x′ are the elements to be fitted and the fitted elements, respectively; a is the fitting parameter, and γ is the polynomial degree;

[0043] By solving a system of γ-variable linear equations, the elements within the window are fitted; specifically, through S... q The elements in the equation form (2w+1) equations, which constitute a system of γ-variable linear equations. The fitting parameter a can then be determined using the least squares method. The proposed SG-based filter operates in a sliding window manner until all the load data has been processed.

[0044] In a preferred embodiment, the workflow of the MC-2PF model includes: next, dividing the preprocessed load X′ into multiple batches as local training data for the client; each client performing E rounds of local model training; the local model w k The update definition is as follows:

[0045]

[0046] Among them, the difference term (c) is introduced. w -c) to estimate the client's offset and correct the direction of local updates in a timely manner;

[0047] After round E of local updates, the local control parameters are updated; the process is defined as follows:

[0048]

[0049] Where η is the learning rate; and the local control parameter c k The update comprehensively considers the global control parameter c and the difference between the global model and the updated local model, denoted as (ww k ).

[0050] In a preferred embodiment, the workflow of the MC-2PF model includes: after completing local training, client k uploads the updated w. k and c k The parameter server receives all model updates uploaded by clients; after receiving these updates, the parameter server updates the global model w and the global control parameters c and begins the next round of FL training; the process is defined as follows:

[0051]

[0052] In a preferred embodiment, control parameters c and c are designed. k The convergence directions of the global and local models are estimated; by using c, each client obtains relevant information about the update directions of the global model and other local models during its local model update process; as shown in formula (9), a correction term (c) is added during the local model update process. k -c); The locally updated movement will be corrected back to the vicinity of the ideal convergence direction, thereby achieving better convergence results; After the client completes the local model update, according to formulas (10) and (11), the local control parameter c of each client is adjusted. k Update the global control parameter c; k The update estimate of c is the convergence direction of the local model, and the update estimate of c is the convergence direction of the global model, which also contains information about the update directions of all clients.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] 1. This paper proposes a general framework for load forecasting based on multi-edge collaboration for the first time. In particular, the proposed MC-2PF innovatively integrates multi-edge collaboration with personalized federated deep learning. By addressing key issues such as high noise, high variability, insufficient data volume, and poor model generalization ability of load data, it adaptively achieves efficient and accurate forecasting of edge loads.

[0055] 2. A set of model control parameters was designed on both the cloud server and each edge server to assist in creating a personalized prediction model for each edge server. The designed control parameters intuitively estimate and correct the update directions of the local and global models, overcoming the client-side bias phenomenon caused by large differences in data distribution in classical federated learning. In addition, a Savitzky-Golay (SG) based filter was developed to smooth the raw load data, reducing noise interference and mitigating model overfitting.

[0056] 3. The effectiveness of the proposed model control parameters in guiding the learning direction of each edge load prediction model during personalized federated learning training is theoretically analyzed, and the generalization bound of the proposed MC-2PF is proved. In particular, a practical case demonstrates the workflow of MC-2PF applied to an RNN-based prediction model, verifying its feasibility and generality.

[0057] 4. Extensive experiments using real-world testbeds and edge load datasets validated the effectiveness and practicality of the proposed MC-2PF. Results show that MC-2PF can be applied to the training of various advanced RNN-based prediction models (including LSTM, GRU, BiLSTM, and S2S) with high-precision prediction performance. Compared to single-edge training frameworks applied to different RNN-based prediction models, MC-2PF exhibits stronger adaptability and achieves higher prediction accuracy. Compared to advanced federated learning frameworks, MC-2PF can further improve prediction accuracy and convergence speed under different prediction lengths. Attached Figure Description

[0058] Figure 1 This is a schematic diagram of an overview of the MC-2PF according to a preferred embodiment of the present invention;

[0059] Figure 2 This is a schematic diagram illustrating how the control parameters improve model convergence in a preferred embodiment of the present invention.

[0060] Figure 3 This is a schematic diagram of the basic structure of the LSTM unit in a preferred embodiment of the present invention;

[0061] Figure 4 This is a schematic diagram comparing load data before and after using SG filtering in a preferred embodiment of the present invention; wherein, (a) is w=7, p=5, (b) is w=15, p=3, and (c) is w=30, p=2;

[0062] Figure 5 The diagram illustrates the prediction performance of SG filters with different settings in different scenarios according to a preferred embodiment of the present invention; wherein, (a) is minute-level prediction, (b) is hour-level prediction, and (c) is day-level prediction.

[0063] Figure 6 This is a schematic diagram comparing the convergence of models on different edge servers in a preferred embodiment of the present invention under FedAvg and MC-2PF; where (a) is client 1, (b) is client 2, (c) is client 3, and (d) is client 4.

[0064] Figure 7 The diagram illustrates the performance of the S2S-GRU with MC-2PF in a preferred embodiment of the present invention at different prediction lengths; where (a) represents minute-level prediction, (b) represents hour-level prediction, and (c) represents day-level prediction.

[0065] Figure 8The diagram illustrates the performance of the S2S-GRU with MC-2PF in a preferred embodiment of the present invention at different prediction lengths; where (a) represents minute-level prediction, (b) represents hour-level prediction, and (c) represents day-level prediction.

[0066] Figure 9 The diagram illustrates the performance of the S2S-GRU with MC-2PF in a preferred embodiment of the present invention for different types of loads under minute-level prediction lengths; where (a) represents a highly periodic load (type 1), (b) represents a highly periodic load (type 2), (c) represents a highly random load (type 1), and (d) represents a highly random load (type 2). Detailed Implementation

[0067] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0068] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0069] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations according to this application; as used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise; furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0070] A multi-edge collaborative load prediction method based on personalized federated deep learning, reference Figure 1-9 To better meet user service requests, edge systems need to be able to efficiently schedule computing and storage resources based on current and future load conditions to achieve system load balancing. However, due to the high variability of edge load, it is difficult to quickly formulate an ideal resource scheduling scheme, which can seriously affect user experience. Furthermore, an unreasonable resource scheduling scheme can lead to excessive operational costs or violations of SLAs. Unlike traditional centralized load forecasting, the proposed MC-2PF employs a federated learning-based collaborative training method in multi-edge environments. Specifically, the site ID of the edge server is used as the basis for dividing the collaborative training region. Edge servers within the same site are considered clients in the federated learning process, and they will collaborate with cloud servers (considered parameter servers in the federated learning process) for training. The specific workflow of MC-2PF is as follows.

[0071] Step 1: The parameter server selects some (or all) clients to participate in this round of federated aggregation and sends work requests to the clients.

[0072] Step 2: The selected client decides whether to participate in this round of federated aggregation based on the current status of the edge server, and returns an accept / reject response to the parameter server.

[0073] Step 3: The parameter server distributes the global model w and global control parameters c to the clients participating in the federated aggregation.

[0074] Step 4: The client preprocesses the historical load data on its edge server, including data cleaning, resampling, normalization and noise reduction, in order to extract the important features from the original data.

[0075] Step 5: Based on the global model w, global control parameter c, and local control parameter c k The client updates its local load prediction model w after several rounds of training, validation, and testing. k and local control parameter c k .

[0076] Step 6: The trained model can be used to predict future edge load, which will help edge service providers develop appropriate resource scheduling schemes to achieve load balancing among edge servers.

[0077] Step 7: The parameter server waits for all clients participating in the federated aggregation to complete their local training and upload their local models and local control parameters.

[0078] Step 8: The parameter server aggregates all received local models and updates the global model, then begins the next round of federated learning training.

[0079] Classical federated learning (such as FedAvg) typically aims to obtain a high-performing global model. Therefore, a dedicated test set is usually created to validate the effectiveness of the global model. However, this approach is not suitable for all federated learning applications. Unlike existing research, this study aims to learn a high-performing and personalized load prediction model for each client (i.e., edge server). Therefore, the focus is on whether MC-2PF can effectively improve the prediction performance of client models. Specifically, a test set is extracted from local historical load data for each client to more specifically test the performance of each client's load prediction model on the local test set. Accordingly, the objective function of MC-2PF is defined as:

[0080]

[0081] Among them, w kFor the local model of client k (such as an RNN-based load prediction model).

[0082] To address the complex and ever-changing patterns of edge load, the proposed MC-2PF minimizes the error between the predicted and actual loads on each edge server, thereby improving the operational efficiency and service quality of edge systems. Specifically, the MC-2PF model uses an RNN-based approach as the client model for federation in time-series forecasting. Through appropriate data preprocessing and model training, the RNN-based approach effectively captures temporal dependencies in sequences and leverages past observations to predict future changes. Based on historical memory and current input, the future predicted output is defined as:

[0083]

[0084] Among them, s t x t and y t Let represent the hidden state, input, and output at time t, respectively.

[0085] In addition, the following accuracy metrics commonly used in time series prediction problems were employed to comprehensively evaluate the performance of MC-2PF.

[0086] (1) MSE (Mean Squared Error) is defined as:

[0087]

[0088] Among them, y t Let be the actual load value at time t. Let L be the predicted load value at time t, and L be the prediction length.

[0089] (2) MAE (Mean Absolute Error) is the mean absolute error, which is defined as:

[0090]

[0091] (3)R 2 R-squared is the coefficient of determination, reflecting the model's fit, and is defined as:

[0092]

[0093] in, R 2 The value of is in the interval [-∞, 1], when R 2 The closer the value is to 1, the better the model fits.

[0094] Overview of the proposed MC-2PF technical solution

[0095]

[0096]

[0097] Algorithm 1 demonstrates the key steps of MC-2PF. First, the parameter server initializes the global model w, global control parameters c, and the number of clients K (line 2). Each client maintains its local control parameters c. k c and c k All are initialized to 0 to ensure For each communication round, the parameter server randomly selects max(C·K,1) clients from all K clients according to the selection ratio C to participate in federated aggregation and form a client set S (lines 4-5). The function max(,) is used to avoid training interruption due to no clients being selected. Then, the parameter server distributes the global model and global control parameters (w,c) to the clients in S (line 6).

[0098] Next, the client k (k∈S) will receive (w,c) and update its local model to w. k ←w (lines 8-10). Specifically, each client retrieves historical load data locally for training its local model (line 15). This load data includes runtime information of the edge server (such as CPU utilization, memory utilization, disk utilization, and upload / download bandwidth). Using all this runtime information as training data would undoubtedly increase the redundancy and complexity of the model training process. Therefore, CPU utilization, which best reflects the performance of the edge system, is used as the primary predictive metric. For an edge server, x t The CPU utilization at time t is represented by X = {x1, x2, ..., xt}. The historical load sequence of this edge server can be denoted as X = {x1, x2, ..., xt}. n Since the original edge load data is sampled at a high frequency, it inevitably contains a lot of noise. For invalid data, it was replaced by taking the interval average during data preprocessing and compressed through resampling, thus effectively extracting the important features of the load data. Furthermore, the original load data exhibits significant differences in value range across different time periods, so it was normalized (line 16) to accelerate the convergence speed of the local model. After normalization, the original data was mapped to the interval [0,1]. This process is defined as:

[0099]

[0100] Where X represents the original load data. max and X min These represent the maximum and minimum CPU utilization rates in the original load data, respectively.

[0101] To address the issue of high noise levels that severely impact prediction accuracy and lead to high computational complexity, a Savitzky-Golay (SG)-based filter was designed to preprocess the raw load data (line 16) to reduce noise interference and improve data smoothness. Specifically, a filter for... subsequence of:

[0102]

[0103] in, q is S q The center point, S q The length is (2w+1), where w is half the length of the window.

[0104] Next. For S q Fit each element in:

[0105]

[0106] in, x and x′ are the elements to be fitted and the fitted elements, respectively. a is the fitting parameter, and γ is the polynomial degree.

[0107] Therefore, by solving a system of γ-variable linear equations, fitting the elements within the window can be achieved. Specifically, through S... q The elements in the equations form (2w+1) equations, constituting a system of γ-variable linear equations. The fitting parameter 'a' can then be determined using the least squares method. The proposed SG-based filter operates in a sliding window manner until all load data has been processed.

[0108] Next, the preprocessed load X′ is divided into multiple batches to serve as local training data for the client (line 17). Each client performs E rounds of local model training (lines 18-22), and the local model w k The update definition is as follows:

[0109]

[0110] Among them, the difference term (c) is introduced. w -c) is used to estimate the client's offset and correct the direction of local updates in a timely manner.

[0111] After round E of local updates, the local control parameters are updated (line 23), a process defined as follows:

[0112]

[0113] Where η is the learning rate. Local control parameter ck The update comprehensively considers the global control parameter c and the difference between the global model and the updated local model, denoted as (ww k ).

[0114] After completing local training, client k uploads the updated w. k and c k To the parameter server (line 24). After receiving all model updates uploaded by clients, the parameter server updates the global model w and the global control parameters c and begins the next round of FL training. This process is defined as:

[0115]

[0116] Improve convergence by controlling parameters

[0117] To better illustrate the control parameters c and c k The positive impact on improving edge load prediction performance is first explained by briefly addressing the client offset phenomenon in classical federated learning, followed by a theoretical analysis of c and c. k How the design effectively solved this problem.

[0118] Figure 2 This demonstrates the changes in the convergence of the load prediction model across two clients, including the convergence directions of the local and global models, as well as the convergence points of the client models. Due to significant differences in data distribution between different clients, the final convergence points of each client model also differ. In this situation, the convergence direction of some clients is deviated by the influence of other clients, a phenomenon known as client bias. This phenomenon greatly complicates FL training and severely degrades model performance and generalization ability. In the classic FedAvg, each client receives the global model w and performs local updates:

[0119]

[0120] When all clients' data are independent and identically distributed, the update process is unbiased, meaning each client's update moves towards its respective convergence point. However, when there are significant differences in data distribution among clients, the convergence directions of the clients can vary considerably. In this case, the classic FedAvg model cannot guarantee that all client updates will move towards their convergence points, severely impacting the model's accuracy and convergence speed.

[0121] To solve this problem, control parameters c and c' were designed. kThe convergence directions of the global and local models are estimated. By using c, each client can obtain relevant information about the update directions of the global model and other local models during its local model update process. As shown in formula (9), a correction term (c) is added during the local model update process. k -c). Therefore, the local update movement will be corrected back to the vicinity of the ideal convergence direction, thus achieving better convergence results. After the client completes the local model update, according to formulas (10) and (11), the local control parameter c of each client will be adjusted. k Update the global control parameter c. k The update of c estimates the convergence direction of the local model, while the update of c estimates the convergence direction of the global model, which also includes information on the update directions of all clients.

[0122] Application of RNN-based load forecasting model in MC-2PF

[0123] To verify the feasibility and generality of the proposed MC-2PF, an example is provided demonstrating its application to the training process of an RNN-based load prediction model (i.e., LSTM). Specifically, LSTM is used in this example to verify the feasibility of MC-2PF. It is worth noting that MC-2PF can also be applied to other RNN-based models, such as GRU, BiLSTM, and Seq2Seq.

[0124] LSTM is an improved variant of RNN, enhancing its ability to judge information compared to the classic RNN. LSTM can selectively forget or retain information, solving the gradient vanishing or exploding problems that exist in RNNs when predicting long sequences. Figure 3 As shown, the LSTM unit uses three gates to control the incoming information flow, including input gate i t Forgotten Gate t and output gate o t . and y t These represent the current and previous cell states, respectively. t Decision made The information discarded in the middle, i t It has been decided where to store y t New information within, o t Calculate the hidden layer output h at time t. t Specifically, they are defined as:

[0125]

[0126] The trainable parameters are defined as the set W = {W y W f Wi W o} and b = {b y ,b f ,b i ,b o For the parameter server and each client in MC-2PF, according to W and b Accordingly, a set of control parameters with the same structure was designed, denoted as W. c ={W y ′,W f ′,W i ′,W o ′} and b c ={b′ y ,b′ f ,b i ′,b o ′}, and ensure that the global control parameter is the average value of all local control parameters.

[0127] Based on the above settings, the parameter server selects clients to participate in federated aggregation, and the selected clients perform E rounds of training locally. Specifically, each client updates W and b using stochastic gradient descent (SGD). During each local model update, W... c and b c The parameters are used to correct the update direction of the learnable parameters. Once a client completes its local model update, it also updates the control parameters. After all clients upload their local models and control parameters, the parameter server performs a federated average of the learnable and control parameters. Then, MC-2PF enters the next round of training, continuously updating until the model converges. The proposed MC-2PF is also applicable to other RNN-based load prediction models (such as GRU, BiLSTM, and S2S), and the feasibility and generality of MC-2PF are fully validated through extensive experiments in the next section.

[0128] Method Evaluation

[0129] Dataset and Experiment Setup

[0130] A real-world testbed was built using hardware to simulate multi-edge collaborative load prediction scenarios. The testbed consists of a workstation and a group of Jetson TX2 processors. The workstation, acting as a cloud data center (i.e., a parameter server), is equipped with two NVIDIA GeForce GTX 3090 GPUs, one Intel(R) Xeon(R) CPU Silver 4208@2.1GHz, and 32GB of memory. The Jetson TX2s, acting as edge servers (i.e., clients), each have an NVIDIA Pascal GPU with 256 CUDA cores and a CPU cluster consisting of 2-core Denver2 and 4-core ARM Cortex A57 CPUs. The workstation and all Jetson TX2s are located on the same local area network, with end-to-end communication between the workstation and Jetson TX2s established using the FLASK framework. The testbed runs on Ubuntu 18.04, with CUDA v100 and cuDNN v7.5.0.

[0131] A real-world edge load dataset was used, recording the load changes of 6870 edge servers over one month with a sampling frequency of 1 minute. Specifically, CPU utilization was used as a performance indicator, and key information such as the site ID of the edge server, start time, end time, and sampling frequency were extracted. The dataset was divided into a training set (50%), a validation set (25%), and a test set (25%). The training set was used for model training (calculating the weights of the neural network), the validation set was used for selecting model parameters (selecting hyperparameters and preventing overfitting), and the test set was used for evaluating model performance. Multiple load instances were generated based on the load input and prediction length, and three scenarios were set with prediction lengths of minutes, hours, and days. For minute-level prediction scenarios, the input and prediction lengths were set to 10 minutes; for hour-level prediction scenarios, the input and prediction lengths were set to 1 hour; and for day-level prediction scenarios, the input and prediction lengths were set to 1 day.

[0132] To verify the superiority of the proposed MC-2PF, its application on different RNN-based models was compared with the following three benchmark frameworks:

[0133] (1) Local: Each edge server uses its local data only as an independent training model.

[0134] (2) FedAvg: Multiple edge servers use the classic FedAvg collaborative training model.

[0135] (3) FedPer: Improves the performance and generalization ability of personalized models by combining personalized model aggregation and parameter update strategies.

[0136] (4) pFedMe: Personalized federated learning is achieved using a two-layer optimization structure and Moreau envelope regularization method.

[0137] Each edge server employs five RNN-based load prediction models, including:

[0138] (1) LSTM: As an improved RNN, LSTM captures and processes long-term dependencies by introducing gating structures and memory units.

[0139] (2) GRU: Compared to LSTM, GRU simplifies the gating structure and is designed to handle long-term dependencies more efficiently.

[0140] (3) BiLSTM: Compared with LSTM, BiLSTM considers both forward and reverse sequence information to capture the context and dependencies in time series data more comprehensively.

[0141] (4) S2S-LSTM: S2S-LSTM combines the gating structure of LSTM with the sequence generation capability of Seq2Seq.

[0142] (5) S2S-GRU: S2S-GRU combines the gating structure of GRU with the sequence generation capability of Seq2Seq.

[0143] Based on Python 3.9 and PyTorch 1.12.1, MC-2PF and FedAvg, as well as five RNN-based load prediction models, were implemented. For MC-2PF and FedAvg, the maximum number of communications was 500, the number of local training iterations per client was 10, the batch size was 64, and the learning rate was 0.001. For the five RNN-based load prediction models, similar network structures were adopted to ensure they had similar model sizes. Specifically, in LSTM and GRU, four hidden layers were set, each containing 64 neurons. In BiLSTM, due to its twice the parameter size of LSTM, two hidden layers were set, each containing 64 neurons. In S2S, two encoder layers and two decoder layers were set, each containing 64 neurons.

[0144] Experimental Results and Analysis

[0145] First, we analyze the selection of hyperparameters for the SG filter. When using the SG filter to preprocess the load data, it is necessary to set the window length w and the order p of the fitting polynomial. The window length refers to the size of the sliding window used to fit the data, which determines the number of neighboring data points of the fitting polynomial coefficients. Figure 4 The load data before and after using the SG filter were compared. For example... Figure 4 As shown in (a), when using a shorter window length (w=7) and a larger order of the fitted polynomial (p=5), the SG filter can retain more details of the data, but it cannot smooth it very well. In contrast, as... Figure 4 As shown in (c), when using a longer window length (w=30) and a smaller order of the fitted polynomial (p=2), the SG filter can smooth the data better, but may lose some details.

[0146] Furthermore, the prediction performance of the SG filter was tested in three different prediction scenarios when different combinations of window lengths and fitting polynomial coefficients were used. Figure 5 As shown in (a), in minute-level prediction scenarios, prediction accuracy improves with increasing window length. Higher prediction accuracy is achieved when using a smaller order of the fitting polynomial. Therefore, in minute-level prediction scenarios, a combination of a larger window length and a smaller order of the fitting polynomial is more suitable. This is because minute-level load change patterns are easier to capture, and a longer window length and a lower-order polynomial help to clarify these patterns, leading to better prediction results. In minute-level prediction scenarios, there are limitations on the temporal length of the input network; therefore, subsequent experiments in minute-level prediction scenarios will use the largest possible window and the smallest possible order of the fitting polynomial. Figure 5 As shown in (b) and (c), the model prediction performance did not show a significant upward trend with the increase of window length, and even showed a downward trend in some cases. This indicates that in scenarios with large prediction lengths, excessively large window lengths reduce prediction accuracy. Meanwhile, while reducing the order of the fitting polynomial can reduce computational complexity, it reduces the model's prediction accuracy. This is because as the prediction length increases, the load variation model becomes more complex. In this case, using a lower fitting polynomial order and a larger window length can improve the smoothness of the load data, but at the same time, it loses key features of the data, which affects prediction accuracy. Based on the test results, in hourly prediction scenarios, the fitting polynomial order was set to 3 and the window length to 29. In daily prediction scenarios, the fitting polynomial order was set to 6 and the window length to 21.

[0147] Figure 6 The convergence of load prediction models trained on different edge servers using FedAvg and MC-2PF is demonstrated, where MC-2PF converges to a better loss value faster. Specifically, as shown... Figure 6As shown in (a), MC-2PF exhibits the most significant convergence advantage over FedAvg from 100 to 300 rounds. This is because the designed control parameters can correct the model's convergence direction back to near the ideal direction, thus allowing it to find the optimal solution faster than FedAvg. Figure 6 As shown in (b), (c), and (d), the Loss curve exhibits a higher frequency of abnormal fluctuations when FedAvg is used. This is because the data distribution varies significantly among the edge servers, and the performance of the federated model may decrease compared to the previous local model. Therefore, the edge servers require multiple rounds of local training and adjustments to recover to the performance before aggregation. By introducing control parameters, MC-2PF corrects the model's convergence direction and significantly reduces large fluctuations during model training, thus achieving more stable convergence. This makes the resulting global model more adaptable to all edge environments, effectively reducing fluctuations during training.

[0148] For scenarios with different prediction lengths (including min-level, hour-level, and day-level), the performance and convergence speed of five RNN-based load prediction models (including GRU, LSTM, BiLSTM, and S2S) under different training frameworks were evaluated. As shown in Table 1, when using the Local training framework, R... 2 The values ​​are all negative, reflecting the model's poor fitting ability. This is because training a prediction model independently on a single edge is unlikely to yield a good model. Compared to Local, models trained using the FedAvg framework show a significant performance improvement. This is because FedAvg not only uses local load data during model training but also draws on the experience of other edge servers within the same region to some extent, demonstrating good predictive performance when facing highly variable loads. However, the significant differences in data distribution between different edge servers severely limit the generalization ability of the FedAvg training framework, potentially requiring more data and training time for each model to achieve higher prediction accuracy. Furthermore, FedPer and pFedMe perform poorly in edge load prediction because they lack specific design for edge environments; their generalized yet personalized approach confuses the data pattern relationships between edge clients, resulting in poor performance. Compared to FedAvg, MC-2PF further improves the predictive performance of various models when facing highly variable loads. This is because MC-2PF uses control parameters to allow the load predictions of edge servers within the region to be mutually referenced, optimizing the convergence direction of each model and thus improving the prediction performance of each model.

[0149] Figure 7 This paper compares the number of communication rounds required to achieve corresponding performance metrics (MSE = 0.008, 0.009, and 0.012 in min-level, hour-level, and day-level prediction scenarios) using different training frameworks (FedAvg and MC-2PF). Under different local training round settings (5, 10, and 20), MC-2PF achieves the target accuracy with fewer communication rounds compared to the classic FedAvg in most prediction scenarios. In minute-level prediction scenarios, the required number of communication rounds is significantly less than in hourly and day-level prediction scenarios because load variation patterns are easier to capture. As the prediction length increases, load fluctuations become more complex and variable, thus requiring more communication rounds to achieve the target accuracy. In scenarios with more complex load conditions (such as hourly and day-level prediction scenarios), MC-2PF can reduce the number of communication rounds by 15–25% compared to FedAvg. This is because MC-2PF uses control parameters to correct model training, which significantly improves the efficiency of multi-edge collaborative training.

[0150] Table 1. Performance of RNN-based models under different training frameworks at various prediction lengths.

[0151]

[0152] Next, the performance of the S2S-GRU prediction model trained by the proposed MC-2PF was tested when faced with different prediction lengths (i.e., min-level, hour-level, and day-level) and multiple types of loads (i.e., highly periodic and highly random). Figure 8 The performance of the S2S-GRU trained with MC-2PF is demonstrated at different prediction lengths. Specifically, the trained model achieves accurate predictions in various scenarios, with the best performance in minute-level prediction scenarios. As the prediction length gradually increases, the model's prediction performance decreases somewhat, but it can still capture the overall trend of load fluctuations. Figure 9 This paper demonstrates the performance of the S2S-GRU trained using MC-2PF in minute-level prediction scenarios for different types of loads. The trained model achieves good prediction performance when facing highly periodic or highly random load variation patterns. Specifically, when facing highly periodic loads, the prediction model can accurately capture load variation patterns. For example, when encountering sudden load peaks, such as... Figure 9The model can accurately predict changes at the 8000-minute mark in (b), which helps to provide early warnings or pre-allocate resources during runtime. Furthermore, as... Figure 9 As shown in (c) and (d), the trained model can still accurately capture the load variation patterns and maintain high prediction accuracy when faced with different types of highly random loads. This is because the proposed MC-2PF effectively improves the model's generalization ability, enabling it to adaptively achieve efficient and accurate prediction of marginal loads.

[0153] Finally, to verify the effectiveness of the designed SG filter in edge environments, comparative tests were conducted using BiLSTM, LSTM_S2S, and GRU_S2S backbone networks under three federated learning algorithms: MC-2PF, pFedMe, and FedAvg. The results are shown in Table 2. Experimental results show that in all cases, the performance of most models significantly improved after adding the SG filter. For example, in MC-2PF, S2S-GRU improved performance by 50% in environments with prediction units of days. This indicates that the SG filter plays a crucial role in reducing noise and smoothing data, contributing to improved accuracy and reliability of edge load prediction. It effectively suppresses high-frequency noise while retaining key low-frequency signals, enabling the model to better capture the essential features of the data. Smoothing enhances key features in the data, facilitating data pattern learning. It also improves the model's generalization ability, making it more adaptable to the complex and varied data distributions in edge environments.

[0154] Table 2 Ablation experiments based on SG filters with various RNN-based models under different training frameworks.

[0155]

[0156] Advantages and uses of this product

[0157] A novel general framework for multi-edge collaborative load prediction based on deep learning (MC-2PF) is proposed. MC-2PF integrates multi-edge collaboration and personalized federated deep learning, addressing key issues such as high noise, high variability, insufficient data volume, and poor model generalization ability of load data, adaptively achieving efficient and accurate prediction of edge loads. Specifically, an SG-based filter is designed to smooth the original load data, reducing noise interference and mitigating model overfitting. In particular, a novel set of model control parameters is designed to solve the client drift phenomenon caused by large data distribution differences in classical federated learning, and the effectiveness of the design is theoretically analyzed and verified. Extensive experiments using real-world testbeds and edge load datasets verify the effectiveness and practicality of the proposed MC-2PF. The results show that the SG-based filter can improve prediction accuracy and reduce computational complexity. Furthermore, MC-2PF can be applied to the training of various advanced RNN-based load prediction models (including LSTM, GRU, BiLSTM, and S2S) with efficient and accurate prediction performance. Compared to single-edge training frameworks applied to different RNN-based prediction models, MC-2PF exhibits stronger adaptability and achieves higher prediction accuracy. Compared to the advanced FedAvg, MC-2PF can further improve prediction accuracy under different prediction lengths and different types of workloads, while reducing the number of communication rounds by 15-25%.

[0158] Product usage process or method

[0159] (1) MC-2PF randomly selects a certain proportion of clients and distributes the generated global model and global control parameters to the selected clients.

[0160] (2) The MC-2PF uses an SG filter to denoise the load data for each customer.

[0161] (3) MC-2PF uses a pre-designed model parameter update formula on the client side to update the model parameters in order to avoid the convergence direction from shifting.

[0162] (4) MC-2PF sends the client parameters to the parameter server to update the global model and global control parameters.

[0163] The client's load prediction results are sent to the edge server so that the edge computing service provider can adjust its resource management strategy.

Claims

1. A multi-edge collaborative load prediction method based on personalized federated deep learning, characterized in that, Includes the following steps: Step 1: The parameter server selects clients to participate in this round of federated aggregation and sends work requests to the clients; Step 2: The selected client decides whether to participate in this round of federated aggregation based on the current status of the edge server, and returns an accept / reject response to the parameter server; Step 3: The parameter server distributes the global model w and global control parameters c to the clients participating in the federated aggregation; Step 4: The client preprocesses the historical load data on its edge server, including data cleaning, resampling, normalization and noise reduction, in order to extract important features from the original data; Step 5: Based on the global model w, global control parameter c, and local control parameter c k In addition to historical load data, the client updates its local load prediction model after several rounds of training, validation, and testing. k and local control parameter c k ; include The preprocessed load X′ is divided into multiple batches to serve as local training data for the client; each client performs E rounds of local model training; the local model w k The update definition is as follows: Among them, the difference term (c) is introduced. k -c) to estimate the client's offset and correct the direction of local updates in a timely manner; After round E of local updates, the local control parameters are updated; the process is defined as follows: Where η is the learning rate; and the local control parameter c k The update comprehensively considers the global control parameter c and the difference between the global model and the updated local model, denoted as (ww k ); Step 6: The trained model can be used to predict future edge load, which will help edge service providers to develop appropriate resource scheduling schemes to achieve load balancing among edge servers. Step 7: The parameter server waits for all clients participating in the federated aggregation to complete their local training and upload their local models and local control parameters; Step 8: The parameter server aggregates all received local models and updates the global model, then begins the next round of federated learning training.

2. The multi-edge collaborative load prediction method based on personalized federated deep learning according to claim 1, characterized in that, The objective function of MC-2PF, a general framework for multi-edge collaborative load prediction based on personalized federated deep learning, is defined as follows: Among them, w k This is the local model for client k.

3. The multi-edge collaborative load prediction method based on personalized federated deep learning according to claim 1, characterized in that, The MC-2PF model uses an RNN-based approach as the client model for federated time series forecasting. Through data preprocessing and model training, the RNN-based approach captures temporal dependencies in the sequence and leverages past observations to predict future changes. Based on historical memory and current input, the future predicted output is defined as: Among them, s t x t and y t Let represent the hidden state, input, and output at time t, respectively.

4. The multi-edge collaborative load prediction method based on personalized federated deep learning according to claim 3, characterized in that, The metrics for evaluating the performance of the MC-2PF include: (1) MSE is the mean squared error, which is defined as: Among them, y t Let be the actual load value at time t. Let L be the predicted load value at time t, and L be the prediction length. (2) MAE is the mean absolute error, which is defined as: (3)R 2 The coefficient of determination reflects the model's fit and is defined as: in, R 2 The value of is in the interval [-∞, 1], when R 2 The closer the value is to 1, the better the model fits.

5. The multi-edge collaborative load prediction method based on personalized federated deep learning according to claim 1, characterized in that, The workflow of the MC-2PF model includes: First, the parameter server initializes the global model w, global control parameters c, and the number of clients K; each client maintains its local control parameters c. k c and c k All are initialized to 0 to ensure For each communication round, the parameter server randomly selects max(C·K,1) clients from all K clients according to the selection ratio C to participate in federated aggregation and form a client set S. The function max(,) is used to avoid training interruption due to no clients being selected. Then, the parameter server distributes the global model and global control parameters (w,c) to the clients in S.

6. The multi-edge collaborative load prediction method based on personalized federated deep learning according to claim 5, characterized in that, The workflow of the MC-2PF model includes: Next, the client k (k∈S) receives (w,c) and updates its local model to w. k ←w; Specifically, each client retrieves historical load data locally for training its local model; this load data contains runtime information about the edge server; CPU utilization, which best reflects the performance of the edge system, is used as the primary predictive metric; for an edge server, x t The CPU utilization at time t is represented by X = {x1, x2, ..., xt}. The historical load sequence of this edge server can be denoted as X = {x1, x2, ..., xt}. n During data preprocessing, the data is replaced by the interval average and compressed through resampling; it is then normalized to accelerate the convergence of the local model; after normalization, the original data is mapped to the interval [0,1]; and is defined as: Where X represents the original load data; X max and X min These represent the maximum and minimum CPU utilization rates in the original load data, respectively. Design a Savitzky-Golay-based filter to preprocess the raw load data, and define a filter based on... subsequence of: in, q is S q The center point, S q The length is (2w+1), where w is half the length of the window.

7. The multi-edge collaborative load prediction method based on personalized federated deep learning according to claim 5, characterized in that, The workflow of the MC-2PF model includes: Next, for S q Fit each element in: in, x and x′ are the elements to be fitted and the fitted elements, respectively; a is the fitting parameter, and γ is the polynomial degree; By solving a system of γ-variable linear equations, the elements within the window are fitted; specifically, through S... q The elements in the equation form (2w+1) equations, which constitute a system of γ-variable linear equations. The fitting parameter a can then be determined using the least squares method. The proposed SG-based filter operates in a sliding window manner until all the load data has been processed.

8. The multi-edge collaborative load prediction method based on personalized federated deep learning according to claim 5, characterized in that, The workflow of the MC-2PF model includes: after completing local training, the client k uploads the updated w. k and c k The parameter server receives all model updates uploaded by clients; after receiving these updates, the parameter server updates the global model w and the global control parameters c and begins the next round of FL training; the process is defined as follows:

9. The multi-edge collaborative load prediction method based on personalized federated deep learning according to claim 8, characterized in that, Design control parameters c and c k The convergence directions of the global and local models are estimated; by using c, each client obtains relevant information about the update directions of the global model and other local models during its local model update process; as shown in formula (9), a correction term (c) is added during the local model update process. k -c); The locally updated movement will be corrected back to the vicinity of the ideal convergence direction, thereby achieving better convergence results; After the client completes the local model update, according to formulas (10) and (11), the local control parameter c of each client is adjusted. k Update the global control parameter c; k The update estimate of c is the convergence direction of the local model, and the update estimate of c is the convergence direction of the global model, which also contains information about the update directions of all clients.

Images