Personalized federal map learning Transform-based space-time load prediction method in multi-edge system

Abstract

The invention provides a space-time load prediction method based on a personalized federal map learning Transform in a multi-edge system. The space-time load prediction method comprises the following steps: step 1, constructing a multivariable sequence inversion Transform as a load prediction model of an edge node; 2, designing a PFG, and generating personalized model parameters and spatial features for each edge node through a super network and a graph neural network GNN; the parameters are updated and transmitted to an FL server to calculate parameter changes of the prediction model, and the FL server updates other parameters, super network parameters and node feature vectors of the global model according to the parameters returned by the client; and step 3, combining the MIT and the PFG to form a PFG former, and generating a personalized load for each edge node for prediction. According to the technical scheme, the coding block and the multi-layer perception processing prediction task are used, complicated decoding blocks are abandoned, the prediction precision is improved, and the model parameter quantity is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of Internet of Things, and in particular to a spatiotemporal load prediction method based on personalized federated graph learning Transformer in a multi-edge system. Background Art

[0002] With the rapid development of the Internet of Things (IoT), a large number of smart applications have emerged, such as smart homes, VR / AR, and energy management. Due to the computationally intensive and latency-sensitive characteristics of these smart applications, existing resource-first IoT devices are difficult to ensure a good user experience (Quality-of-Experience, QoE). Classic cloud computing has exposed bottleneck problems in terms of transmission delay, energy consumption, and privacy security. As an emerging computing paradigm, Mobile Edge Computing (MEC) reduces the dependence of IoT devices on remote clouds by sinking computing and storage resources to edge nodes closer to data sources, achieving faster response and higher reliability. In MEC, the resource requirements of edge nodes usually depend on their geographical distribution, and the node heterogeneity poses a huge resource management challenge to edge service providers (ESPs). By introducing virtualization technology, the flexibility and efficiency of edge system resource supply can be improved. However, it takes a certain amount of time to start virtual machines (VMs) to provide them to users. For example, it takes several minutes on average to start a VM, and it usually takes a few seconds to a minute to start a container (such as Docker, Podman, etc.). Therefore, it is necessary to effectively capture the load characteristics of the edge system and predict future load changes to support ESP to actively pre-configure resources, thereby achieving efficient and reasonable resource supply.

[0003] In recent years, Transformer has been widely used in time series forecasting problems. By introducing the self-attention mechanism to achieve global context modeling and parallel computing, Transformer avoids gradient vanishing and is suitable for handling long-term dependencies. However, existing Transformer-based frameworks still have limitations when facing multivariate time series forecasting problems. Typically, these frameworks embed multiple variables at the same timestamp into indistinguishable channels and use attention mechanisms to capture temporal dependencies. However, this approach obscures the important associations between variables, misleads the learning direction of the self-attention mechanism, and reduces the prediction performance. In addition, when facing multivariate time series forecasting problems, the linear layer with a simple structure can better ensure variable independence and learn multivariate correlations, and its prediction accuracy and efficiency even exceed the complex Transformer. Consider reversing the time series so that the complete time series of each variable is embedded in the input channel independently. This design will expand the local perception ability of the prediction model, enabling it to make full use of the self-attention mechanism to associate the potential features between multiple variables. In addition, there are the following open issues to be solved in edge load forecasting.

[0004] (1) Inefficient processing of high-dimensional features. In the MEC environment, edge nodes usually have high-dimensional cross-domain features, including hardware configuration (e.g., CPU and memory) and node location, which affect the change of edge load in different aspects. Existing studies usually use static feature encoding methods, which are difficult to fully characterize high-dimensional cross-domain data.

[0005] (2) Limitations of single edge load data. Edge nodes are distributed discretely, and the load data of each node is limited and distributed in a single way. To ensure good prediction performance, deep learning-based models usually need to rely on a large amount of diverse historical data for training. Therefore, it is difficult to ensure high prediction performance by training a prediction model using only the load data of a single edge node.

[0006] (3) Insufficient utilization of spatial features. Most existing studies use centralized training methods, which do not fully explore the potential value of spatial features of different edge nodes in improving load prediction performance. For example, edge nodes at different locations may have similar load change patterns. Therefore, effective utilization of spatial features of different edge nodes can help load prediction. Summary of the invention

[0007] In view of this, the purpose of the present invention is to provide a spatiotemporal load prediction method based on personalized federated graph learning Transformer in a multi-edge system. The encoding block and multi-layer perception process the prediction task, abandoning the complicated decoding block, which improves the prediction accuracy and reduces the number of model parameters.

[0008] To achieve the above object, the present invention adopts the following technical solution: a spatiotemporal load prediction method based on personalized federated graph learning Transformer in a multi-edge system, comprising the following steps:

[0009] Step 1: Build a multivariate sequence reversal Transformer as a load prediction model for edge nodes;

[0010] Step 2: Design PFG to generate personalized model parameters and spatial features for each edge node through the hypernetwork and graph neural network GNN; update the parameters and transmit them to the FL server to calculate the parameter changes of the prediction model. The FL server updates the remaining parameters of the global model, the hypernetwork parameters, and the node feature vector according to the parameters returned by the client;

[0011] Step 3: Combine MIT and PFG to form PFGformer, which generates personalized load prediction for each edge node.

[0012] In a preferred embodiment, the graph neural network GNN is used to embed and reason about graph data, and its input includes: a graph consisting of nodes and edges, represented by an adjacency matrix A∈R K×K ; and the node feature matrix X∈R K×dmodel , where K represents the number of nodes and dmodel represents the number of node features; the graph neural network GNN updates the node embedding according to the information of adjacent nodes, which is defined as:

[0013] S (l+1) =GNN(A,S (l) ,w (l) ), (1)

[0014] Where GNN(·) represents the graph aggregation function, which is average, weighted average, or maximum / minimum pooling; S (l) represents the node embedding of layer l, w (l) represents the trainable weights of layer l.

[0015] In a preferred embodiment, the aggregation process of the FL server is defined as

[0016]

[0017] Among them, θ i is the local model parameter of the i-th client, θ c+1 is the global model parameter for the next iteration, K is the number of participating clients, and D i is the number of samples of the ith client, and D is the total number of samples of all clients; Indicates the contribution of each client to the global model.

[0018] In a preferred embodiment, the edge load includes dynamic and static characteristics of date, node location, and disk capacity; under a fixed prediction window, multiple load sequences on the same edge node are integrated into the historical load input, which is defined as

[0019]

[0020] Where Γ represents the last time index observed in the historical load data, d t is the dynamic feature dimension, d t >1; For a set of load data input, the prediction process is expressed as

[0021]

[0022] in, represents the predicted load sequence, L is the prediction length, and f is the prediction model; represents the static characteristics, where d s Represents the static feature dimension d s >1;

[0023] To evaluate the accuracy of edge load prediction, MSE and MAE are considered as performance indicators, which are defined as

[0024]

[0025] Among them, y i represents the actual load within an interval, and S is the number of load sequences.

[0026] In a preferred embodiment, MIT consists of an embedding layer, an encoder layer, and an output layer.

[0027] In a preferred embodiment, the embedding layer is composed of a sequence reversal layer and a static content perception layer; the sequence reversal specifically divides the time series along the node dimension, which is expressed as

[0028]

[0029] Among them, permute(X t ,[1,3,2]) means swapping the second and third dimensions of the original tensor Xt to achieve the interchange of time and node dimensions;

[0030] The proposed sequence reversal uses the self-attention mechanism to extract the association between different features and combines it with SALayer to integrate multi-dimensional features into the temporal dynamic features; MIT uses a feedforward neural network to efficiently extract the temporal features in the sequence, and then combines them with the spatial features to form spatiotemporal information to support accurate edge load prediction;

[0031] The static content perception layer uses the SA layer and self-attention mechanism to extract the potential associations between different features; the SA layer is used to combine static context with dynamic load sequence data, and use high-dimensional cross-domain data as additional static content; the SA layer uses cross attention to select the most effective static content for integration according to the current sequence, and its input includes the load sequence and static feature data P t ; First, connect the reversed load sequence and timestamp as the timing dynamic feature, which is defined as

[0032]

[0033] Next, the detailed information contained in the SA Layer is expressed as

[0034]

[0035] α t =softmax -1 (Q t )

[0036] V=Linaer2(P t )

[0037]

[0038] Among them, softmax -1 Express Q t The last dimension of is softmaxed, Linear1 and Linear2 represent different linear layers, and Norm represents layer normalization;

[0039] By integrating load sequences and static features, SA Layer learns the static content and inherent properties of edge nodes and filters out redundant information in high-dimensional data; at the same time, the number of channels and parameters of the prediction model will also be reduced.

[0040] In a preferred embodiment, the encoding layer is used to extract spatiotemporal correlation and high-dimensional static features. The encoding layer includes a feedforward neural network and a self-attention layer module. The feedforward neural network is used to capture the relationship between time points and discard the decoding block. The feedforward neural network is defined as:

[0041]

[0042] in, represents the state of the l-1th hidden layer, W (l) represents the weight matrix, b (l) is the bias vector, gelu(·) is the activation function;

[0043] The self-attention layer is used to capture the correlation of multiple variables, which integrates the high-dimensional static features extracted by the SA Layer; specifically, the self-attention layer converts each element in the input sequence into a query Q, key K, and value V vector, and calculates the dot product of the query with all keys; the process is defined as

[0044]

[0045] In addition, the multi-head attention mechanism consists of multiple self-attention layers, focusing on the representation subspace information from different positions; specifically, multiple parallel attention layers are adopted and embedded in the Transformer encoder, which is defined as

[0046] H l =MultiHead(Q,K,V)=Concat(head1,...,head n )W O (11)

[0047] Among them, MultiHead(·) represents the multi-head attention mechanism, l={0,…,L} represents the encoding layer index, head n represents the nth parallel attention layer, is the parameter matrix.

[0048] In a preferred embodiment, the objective function of PFGformer is defined as

[0049]

[0050] in, represents the loss function of the i-th client;

[0051] In PFGformer, the FL server will learn a hypernetwork to generate personalized self-attention matrix parameters and linear layer parameters for each client;

[0052] The model parameters θ of MIT i Split into W i , κ i and i , respectively represent the self-attention matrix parameters, linear layer parameters and other layer parameters; input sequence X t The length of the sequence is recorded as T. After the sequence passes through the embedding layer, the input self-attention layer is converted into an embedding matrix The query, key, and value of the self-attention mechanism are represented as and Where W i =[W i Q ,W i K ,Wi V ];

[0053] The super network is denoted as Hnet(φ,z i ), where φ represents the parameters of the hypernetwork, z i ∈R dmodel represents the feature vector of client i; let the FL server self-learn z i , which will be automatically generated by the Embedding layer; the hypernetwork is implemented with n fully connected layers, where the first n-1 layers are shared by all client Transformer encoding blocks, and the last layer is unique for each encoding block; the hypernetwork is introduced to learn the self-attention matrix parameters and linear layer parameters {W i ,κ i =Hnet(φ,z i )}, to realize a personalized MIT-based prediction model; the prediction model and its loss function are defined as

[0054]

[0055] In each local epoch, the local model parameters are updated using stochastic gradient descent, which is defined as

[0056]

[0057] Among them, B i It is from D i A batch sampled from , α represents the client learning rate;

[0058] According to the chain rule, φ and z i The gradient of is calculated as

[0059]

[0060] Where C represents the total communication rounds, ΔW i =W i J -W i 0 , Δκ i =κ i J -κ i 0 represents the changes in the self-attention matrix parameters and linear layer parameters after J local training rounds;

[0061] After each communication round, use the gradient and To update the hypernetwork parameters φ and the client embedding z i , which is defined as

[0062]

[0063] Where β represents the learning rate of the FL server, and the superscript c represents the cth communication round.

[0064] In a preferred embodiment, EMD is introduced to reconstruct and generate a multi-order adjacency matrix between edge nodes. For any two clients i and i′, the similarity is calculated as

[0065]

[0066] Next, the similarity of the nodes is constructed as a similarity matrix S EMD ∈R K×K , where s ii ′ represents the element of the i-th row and i′th column; in this way, a personalized adjacency matrix is ​​generated for each node; in s EMD Select several most relevant neighbors to aggregate multi-order neighbors; introduce GraphSAGE to process the adjacency matrix and use the client's feature vector z i as node vectors; GraphSAGE is used to perform aggregation operations by sampling neighbor nodes, thereby generating spatial feature vectors of nodes and transmitting them to the client; the client connects the spatial feature vectors with the original feature vectors for subsequent predictions;

[0067] (1) Node k receives the embedding vectors generated by all direct neighbor nodes N(k) in the previous layer It is defined as:

[0068]

[0069] Among them, AGGREGATE(·) represents the aggregation function, represents the neighbor aggregation information received by node k at layer t; The initial input is the feature vector of the node

[0070] (2) Aggregate the information in adjacent nodes through the connection function; the embedding of node k in layer t is and The concatenation of

[0071]

[0072] Among them, σ(·) is a nonlinear activation function; CONCAT(·) represents the connection function, which uses MEAN as an aggregator; and After the connection, an average operation is performed on each dimension of the vector, and the result is nonlinearly transformed to obtain the embedding vector of node k at layer t;

[0073] (3) The embedding output of each node is

[0074] In a preferred embodiment, a λ aggregation mechanism is designed. Specifically, a hyperparameter λ is introduced as the weight when updating the global and local models, which is defined as

[0075]

[0076] in, represents the remaining parameters of the local model of client i in round c, represents the remaining parameters of the global model in round c;

[0077] The update deviation of the local model is corrected back to the ideal convergence direction to improve the prediction performance of local models of different clients; the loss function of PFGformer is rewritten as

[0078]

[0079] Compared with the prior art, the present invention has the following beneficial effects:

[0080] (1) A novel edge load prediction model based on MIT is proposed. Specifically, a novel feature fusion method is first designed to allow the prediction model to select the most appropriate static features for fusion. Therefore, cross-domain static features can be reasonably utilized without being interfered by redundant features, which improves the model training speed under multi-variable dimensions. At the same time, due to the reduction of feature dimensions, the number of channels and the amount of model parameters are also reduced, which greatly reduces the communication overhead. Different from the existing method that divides load data based on timestamps, sequence reversal is introduced and data is divided along the variable dimension. This design makes the variable dimension unobstructed and expands local perception, so the self-attention mechanism can effectively extract the potential relationship between variables. In addition, only the encoding block and the multilayer perceptron (MLP) are used to process the prediction task, and the complicated decoding block is abandoned, which improves the prediction accuracy and reduces the amount of model parameters. Therefore, MIT can better utilize spatial feature information through feature fusion and self-attention mechanism.

[0081] (2) A new graph topology generator is designed. Based on Earth Mover's Distance (EMD), a multi-order adjacency matrix between nodes is generated according to the inconsistently distributed edge nodes, their data volume and hardware configuration, and personalized graph topology data is generated for each node, which contains the spatial feature information of neighboring nodes. Therefore, the proposed graph topology generator can truly reflect the spatial relationship between nodes to better support the extraction of node spatial feature information.

[0082] (3) An innovative PFG framework is developed, which consists of a hypernetwork and Graph Sample and Aggregate (GraphSAGE). First, MLP is used to generate node feature vectors as input for the hypernetwork and GraphSAGE. By leveraging the feature vectors, the MIT model on each edge node can learn personalized self-attention matrix parameters and linear layer parameters. Then, a λ aggregation mechanism is introduced to balance the difference between the global and local models to further optimize the personalized model on the edge nodes. At the same time, GraphSAGE is applied on the personalized graph to generate personalized spatial features for each edge node. In particular, the excellent generalization ability and convergence of PFGformer are proved through rigorous derivation.

[0083] (4) Based on multiple real-world edge workload datasets and testbeds, extensive experiments verify the effectiveness of the proposed PFGformer. Compared with the state-of-the-art framework, PFGformer achieves higher prediction accuracy at different prediction lengths. For different Transformer-based centralized and FL training frameworks, PFGformer achieves higher prediction accuracy and lower communication and training overhead. In addition, ablation experiments verify the effectiveness of each component in PFGformer. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 It is a schematic diagram of an overview of MIT in a preferred embodiment of the present invention;

[0085] Figure 2 It is a schematic diagram of the overview of the PFGformer of the preferred embodiment of the present invention;

[0086] Figure 3 A schematic diagram of an overview of HyperNetworks according to a preferred embodiment of the present invention;

[0087] Figure 4 Schematic diagram of load changes and similarity matrices of different edge nodes in a preferred embodiment of the present invention, wherein (a) is load changes and (b) is a similarity matrix;

[0088] Figure 5 Schematic diagrams of different load modes of different edge nodes in different data sets in a preferred embodiment of the present invention, where (a) is ECW, (b) is Alibaba, and (c) is NEP;

[0089] Figure 6 Schematic diagram of different load modes of different edge nodes in different data sets of a preferred embodiment of the present invention, wherein (a) is a prediction length of 24, (b) is a prediction length of 48, and (c) is a prediction length of 96;

[0090] Figure 7 Schematic diagrams of different load modes of different edge nodes in different data sets of a preferred embodiment of the present invention, wherein (a) is MIT, (b) is Crossformer, (c) is Pyraformer, (d) is Flashformer, (e) is Flowformer, and (f) is Inoformer;

[0091] Figure 8 Schematic diagram for comparing the communication and training costs between different frameworks of the preferred embodiment of the present invention, wherein (a) is the communication and training costs of PFGformer using different prediction models, and (b) is the communication and training costs of different personalized federated learning frameworks. DETAILED DESCRIPTION

[0092] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0093] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0094] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations.

[0095] A spatiotemporal load prediction method based on personalized federated graph learning Transformer in multi-edge systems, reference Figure 1-8 , firstly proposed MIT as the load prediction model for edge nodes, aiming to efficiently utilize the high-dimensional features of edge load and node spatial features. Then, PFG was designed, aiming to generate personalized model parameters and spatial features for each edge node through hypernetwork and GNN. Finally, PFGformer was proposed by combining MIT and PFG to generate a personalized load prediction model for each edge node.

[0096] In PFGformer, hypernetwork, GraphSAGE and λ aggregation mechanism are integrated to generate a personalized MIT-based edge load prediction model for each client. PFGformer can be well applied to the Transformer model based on the self-attention mechanism, and FedGNN is reasonably combined with multi-edge scenarios to improve the performance of the load prediction model. The training process of PFGformer is shown in Algorithm 1. For the FL server, the required parameters and spatial feature vectors are first generated using hypernetwork and GraphSAGE (lines 8-11). Then, the client parameters are updated and transmitted to the FL server to calculate the parameter changes of the prediction model (lines 12-13). Finally, the FL server updates the remaining parameters of the global model, hypernetwork parameters and node feature vectors according to the parameters returned by the client (lines 15-17). For each client, the client model parameters are first updated (lines 19-20). Then, the client updates the client model parameters according to gradient descent using local data (lines 21-24). Finally, the client uploads the model parameters to the FL server (line 25).

[0097]

[0098]

[0099] Next, we will introduce the definitions of graph neural networks, federated learning, and edge load prediction.

[0100] Graph Neural Networks (GNN)

[0101] GNN can be used to embed and reason about graph data. Its input includes: 1) a graph consisting of nodes and edges, represented as an adjacency matrix A∈R K×K ; 2) Node feature matrix X∈R K×dmodel , where K represents the number of nodes and (dmodel) represents the number of node features. GNN updates node embedding based on the information of adjacent nodes. The process is defined as:

[0102] S (l+1) =GNN(A,S (l) ,w (l) ), (1)

[0103] Where GNN(·) represents the graph aggregation function, which can be average, weighted average, or maximum / minimum pooling. (l) represents the node embedding of layer l, w (l) represents the trainable weights of layer l.

[0104] Federated Learning (FL)

[0105] In classic machine learning, all user raw data needs to be transmitted to a central server for model training, which may expose user privacy. By performing local model training on the client, the new FL avoids the aggregation of raw data and thus protects user privacy.

[0106] In FL, user raw data is distributed on the clients. The clients independently train local models using local data and upload the model parameters to the central server. The local model parameters are aggregated to generate a global model, which is distributed to each client. Therefore, the features of different clients can be shared without transmitting the raw data. The aggregation process of FL is defined as

[0107]

[0108] Among them, θ i is the local model parameter of the i-th client, θ c+1 is the global model parameter for the next iteration, K is the number of participating clients, and D i is the number of samples of the ith client, and D is the total number of samples of all clients. It indicates the contribution of each client to the global model. Clients with more samples have a greater impact on the global model update.

[0109] Edge Load Prediction

[0110] By predicting future load changes (such as CPU, memory, and bandwidth), edge systems can be helped to improve resource utilization and service quality. Edge load contains dynamic and static features such as date, node location, and disk capacity, so edge load prediction can be considered as a multivariate and multi-sequence prediction problem. Under a fixed prediction window, multiple load sequences on the same edge node can be integrated into historical load input, which is defined as

[0111]

[0112] Where Γ represents the last time index observable in the historical load data, d t is the dynamic feature dimension (d t >1).

[0113] For a set of load data input, the prediction process can be expressed as

[0114]

[0115] in, represents the predicted load sequence, L is the prediction length, and f is the prediction model. represents the static characteristics, where d srepresents the static feature dimension (d s >1).

[0116] To evaluate the accuracy of edge load prediction, Mean Squared Error (MSE) and Mean Absolute Error (MAE) are considered as performance indicators, which are defined as

[0117]

[0118] Among them, y i represents the actual load within an interval, and S is the number of load sequences.

[0119] MIT: Multivariate Sequence Reversal Transformer

[0120] The proposed MIT is used as a load prediction model for edge nodes. Figure 1 As shown in Figure 1, MIT consists of an embedding layer, an encoder layer, and an output layer. The key steps are shown in Algorithm 1. First, the embedding layer performs sequence reversal and extracts static features of the load data (lines 5 to 7). Next, the encoding layer uses Transformer to capture long-term and short-term and feature dependencies (lines 8 to 11). Finally, the output layer uses a linear layer to predict future loads (lines 12 to 13). The detailed design of the embedding layer and the encoding layer is as follows.

[0121]

[0122]

[0123] MIT Embedding Layer

[0124] In the MIT method, the embedding layer (Input Embedding) consists of two parts, including sequence inversion (Sequence Inversion) and static context aware layer (Static Context Aware Layer, SA Layer).

[0125] A. Sequence reversal

[0126] In the classic Transformer, the self-attention mechanism can learn the long-term and short-term dependencies by calculating the correlation between each vector in the input sequence. Usually, this mechanism divides the time series along the time dimension, so the input vector of each time step may contain multiple variable information with different meanings and importance, which will be embedded in the same input channel, resulting in dimensional confusion. Therefore, the classic attention mechanism may not be able to accurately distinguish the relationship between variables when learning variable dependencies. At the same time, the compression of high-dimensional time vectors will also lead to the loss of feature information. Therefore, the classic Transformer cannot effectively capture the time series representation and describe the correlation of multiple variables, which limits its generalization ability on different time series data. In order to solve these problems, sequence reversal is introduced to ensure the independence between variables and then use the self-attention mechanism to extract the dependencies between variables. Specifically, consider dividing the time series along the node dimension to avoid the high complexity caused by processing high-dimensional vectors. The process can be expressed as

[0127]

[0128] Among them, permute(X t ,[1,3,2]) means to convert the original tensor X t The second and third dimensions of the shard are swapped to achieve the interchange of time and node dimensions.

[0129] The proposed sequence reversal uses the self-attention mechanism to extract the association between different features and combines it with SALayer to integrate multi-dimensional features into the temporal dynamic features. Through this design, MIT can use feedforward neural networks to efficiently extract temporal features in sequences, and then combine them with spatial features to form spatiotemporal information to support accurate edge load prediction.

[0130] B. Static Content Awareness Layer

[0131] Due to the heterogeneity of different edge nodes, their load data usually has high-dimensional features. Existing methods usually use static feature encoding methods, which are difficult to fully characterize high-dimensional cross-domain data. At the same time, excessive static features will lead to a sharp increase in the number of model parameters, which seriously affects the efficiency of model training and inference. In order to make full use of high-dimensional data and reduce model complexity, SA Layer and self-attention mechanism are used to extract potential correlations between different features. SA Layer is used to combine static context with dynamic load sequence data, and use high-dimensional cross-domain data (such as edge node CPU, memory, and location) as additional static content. Specifically, SA Layer adopts cross attention to select the most effective static content for integration according to the current sequence. Its input includes the load sequence and static feature data P tFirst, the reversed load sequence is connected with the timestamp as the timing dynamic feature, which is defined as

[0132]

[0133] Next, the detailed information contained in the SA Layer can be expressed as

[0134]

[0135] Among them, softmax -1 Express Q t The last dimension of is softmaxed (weights are calculated for each static feature of each sequence sample), Linear1 and Linear2 represent different linear layers, and Norm represents layer normalization.

[0136] By integrating the load sequence and static features, SA Layer can learn the static content and inherent properties of edge nodes, which effectively filters out redundant information in high-dimensional data. At the same time, due to the reduction in the number of static features, the number of channels and parameters of the prediction model will also decrease, alleviating the problem of excessive communication overhead in FedGNN.

[0137] MIT coding layer

[0138] At MIT, the Transformer-based encoding layer is used to extract spatiotemporal correlations and high-dimensional static features, which includes modules such as feedforward neural networks and self-attention layers.

[0139] A. Feedforward Neural Network

[0140] In a feedforward neural network, data is propagated forward from the input layer to the output layer. Based on the universal approximation theorem, feedforward neural networks can describe time series by extracting complex representations. Recent work on MLP (Multilayer Perceptron, MLP) has shown that linearizing time series can help improve prediction performance. In view of this, a feedforward neural network is used to capture the relationship between time points and the decoding block is abandoned, which greatly reduces the number of model parameters. Specifically, the feedforward neural network is defined as

[0141]

[0142] in, represents the state of the l-1th hidden layer, W (l) represents the weight matrix, b (l) is the bias vector and gelu(·) is the activation function.

[0143] B. Self-Attention Layer

[0144] The self-attention layer is used to capture the correlation of multiple variables, which integrates the high-dimensional static features extracted by the SA Layer. Specifically, the self-attention layer converts each element in the input sequence into a query (Q), key (K), and value (V) vector, and calculates the dot product of the query with all keys. The process is defined as

[0145]

[0146] In addition, the multi-head attention mechanism consists of multiple self-attention layers, focusing on the representation subspace information from different positions. Specifically, multiple parallel attention layers are adopted and embedded in the Transformer encoder, which is defined as

[0147] H l =MultiHead(Q,K,V)=Concat(head1,...,head n )W O , (11)

[0148] Among them, MultiHead(·) represents the multi-head attention mechanism, l={0,…,L} represents the encoding layer index, head n represents the nth parallel attention layer, is the parameter matrix.

[0149] PFGformer:Personalized Federated Graph Learning with Transformer

[0150] Edge load has the characteristics of high variance, high dimensionality and spatial correlation, which poses great challenges to accurate and efficient load prediction. Specifically, high variance means that the load may fluctuate violently in the short term. High dimensionality means that the load data covers a large number of features from various edge nodes in different regions. Spatial correlation means that there are different degrees of similarity between edge loads in different regions, which increases the difficulty of extracting valuable information. To address these challenges, a novel PFGformer framework is proposed to introduce a collaborative training method based on personalized federated graph learning in a multi-edge environment. Each client (i.e., edge node) runs MIT as its local model, and its input is the multi-dimensional edge load data of the local region. The FL server (i.e., cloud server) uses FedAvg to aggregate the MIT model parameters except the self-attention matrix parameters and linear layers, and uses hypernetwork and GraphSAGE to generate personalized self-attention matrix parameters and spatial features for each client, which are then distributed to each client. When updating the local model, a lambda aggregation mechanism is used to balance the difference between global and local model parameters.

[0151] It is expected that the client can learn a good personalized prediction model for different load patterns and spatial features in its edge environment. Therefore, the proposed PFGformer aims to effectively solve the data heterogeneity problem and extract personalized spatial features to improve the performance of local prediction models. Specifically, the hypernetwork, GraphSAGE and λ aggregation mechanism are used in the FL server to train the load prediction model of each client in a personalized way. The objective function of PFGformer is defined as

[0152]

[0153] in, Represents the loss function of the i-th client.

[0154] PFGformer Super Network

[0155] In PFGformer, the FL server will learn a hypernetwork to generate personalized self-attention matrix parameters and linear layer parameters for each client; specifically, the model parameters θ of MIT are i Split into three parts, including W i , κ i and i , respectively represent the self-attention matrix parameters, linear layer parameters and other layer parameters. Input sequence X t The length of the sequence is recorded as T. After the sequence passes through the embedding layer, the input self-attention layer is converted into an embedding matrix Therefore, the query, key, and value of the self-attention mechanism are represented as and Where W i =[W i Q ,W i K ,W i V ]. Through this design, parameters can be effectively shared between clients and personalization based on the MIT prediction model can be maintained.

[0156] The super network is denoted as Hnet(φ,z i ), where φ represents the parameters of the hypernetwork, z i ∈R dmodel represents the feature vector of client i. i It can be uploaded to the FL server by the local client or self-learned by the FL server. In order to reduce communication overhead and avoid privacy leakage, the FL server is allowed to self-learn z i, which will be automatically generated by the Embedding layer. The hypernetwork is implemented with n fully connected layers, where the first n-1 layers are shared by all client Transformer encoding blocks, and the last layer is unique to each encoding block. By introducing the hypernetwork to learn the self-attention matrix parameters and linear layer parameters {W i ,κ i =Hnet(φ,z i )}, a personalized MIT-based prediction model can be implemented. The model and its loss function are defined as

[0157]

[0158] In each local epoch, the local model parameters are updated using stochastic gradient descent, which is defined as

[0159]

[0160] Among them, B i It is from D i A batch sampled from , α represents the client learning rate.

[0161] According to the chain rule, φ and z i The gradient of can be calculated as

[0162]

[0163] Where C represents the total communication rounds, ΔW i =W i J -W i 0 , Δκ i =κ i J -κ i 0 Represents the changes in the self-attention matrix parameters and linear layer parameters after J local training rounds.

[0164] After each communication round, use the gradient and To update the hypernetwork parameters φ and the client embedding z i , which is defined as

[0165]

[0166] Where β represents the learning rate of the FL server, and the superscript c represents the cth communication round.

[0167] Through the above design, clients can effectively share parameters and make better use of the self-attention mechanism. At the same time, the prediction model can also guarantee good performance when facing scenarios with a large number of clients. The generalization bounds of PFGformer will be analyzed in detail in Section 4.3.

[0168] PFGformer graph topology generator and GraphSAGE

[0169] The spatial correlation of loads at different edge nodes is not only related to the node location, but also closely related to the data pattern. Figure 4 (a) shows the load variation pattern of different nodes, where nodes 1 and 2 are quite different, but nodes 1 and 12 are relatively similar. This shows that the spatial correlation of each node has a tight coupling relationship, which will affect the accuracy of load prediction, but has been ignored in existing studies. To address this problem, Earth Mover's Distance (EMD) is introduced to reconstruct and generate a multi-order adjacency matrix between edge nodes, aiming to generate a personalized graph for each node, thereby more accurately reflecting the spatial relationship between nodes. For any two clients i and i′, their similarity can be calculated as

[0170]

[0171] Next, the similarity of the nodes can be constructed as a similarity matrix S EMD ∈R K×K , where s ii′ Denotes the element in row i and column i′. In this way, a personalized adjacency matrix is ​​generated for each node. Figure 4 (b) visualizes the similarity matrix of 28 nodes, where the correlations between nodes are randomly distributed and not determined by spatial distance. Specifically, in s EMD In order to extract the spatial features of edge nodes, GraphSAGE is introduced to process the adjacency matrix and the client’s feature vector z is used. i As a node vector. Unlike the classic GCN that requires multiple message passing, GraphSAGE can efficiently process large-scale graph data because it only needs to perform one aggregation operation on each node. At the same time, GraphSAGE can perform incremental learning, which can be updated based on new nodes or edges without retraining the entire model. Therefore, when a node joins or leaves, GraphSAGE can predict it based on the local neighborhood information of the new node. In view of this, GraphSAGE is used to perform aggregation operations by sampling neighbor nodes, and then generate spatial feature vectors of nodes and transmit them to the client. The client connects the spatial feature vector with the original feature vector for subsequent predictions.

[0172] (1) Node k receives the embedding vectors generated by all direct neighbor nodes N(k) in the previous layer It is defined as:

[0173]

[0174] Among them, AGGREGATE(·) represents the aggregation function, Represents the neighbor aggregation information received by node k at layer t. The initial input is the feature vector of the node

[0175] (2) Aggregate the information in adjacent nodes through the connection function. The embedding of node k in layer t is and The concatenation of

[0176]

[0177] Among them, σ(·) is a nonlinear activation function. CONCAT(·) represents the connection function, which uses MEAN as an aggregator. and After the connection, an average operation is performed on each dimension of the vector, and the result is nonlinearly transformed to obtain the embedding vector of node k at layer t.

[0178] (3) The embedding output of each node is

[0179] PFGformerλ aggregation mechanism

[0180] Due to the large differences in load data distribution, the convergence of client models is uneven. Affected by other clients, the update direction of some clients may deviate, which is called client drift, which seriously affects the performance and generalization ability of the model. To solve this problem, for other parameters ξ, a λ aggregation mechanism is designed to replace the classic way of using the global model to cover the local model. The proposed λ aggregation mechanism can correct the update direction of the global and local models and overcome the client drift problem caused by inconsistent data distribution in classic FL. Specifically, a hyperparameter λ is introduced as the weight when updating the global and local models, which is defined as

[0181]

[0182] in, represents the remaining parameters of the local model of client i in round c, represents the remaining parameters of the global model at round c.

[0183] Through this design, the update deviation of the local model can be corrected back to the ideal convergence direction, thereby improving the prediction performance of local models on different clients. Therefore, the loss function of PFGformer can be rewritten as

[0184]

[0185] Method evaluation

[0186] First, we introduce the dataset and experimental settings. Then, we conduct a large number of comparative experiments on the proposed PFGformer method and other baseline methods.

[0187] Dataset and Experimental Settings

[0188] A real hardware testbed was built to simulate a multi-edge system. The testbed consists of a workstation and multiple Jetson TX2s. The workstation acts as a cloud server, equipped with two NVIDIA GeForce GTX 3090 GPUs, an Intel(R) Xeon(R) CPU Silver 4208@2.1GHz and 32GB of memory. Jetson TX2s act as edge nodes, each Jetson TX2 is equipped with a 256-core NVIDIA Pascal(R) GPU, a CPU cluster consisting of a dual-core Denver 2 64-bit CPU and a quad-core Arm(R) Cortex(R)-A57 MPCore processor, and 8G memory. The operating systems of the workstation and JetsonTX2s are both Ubuntu 18.04, they are located in the same local area network, and end-to-end communication is established based on the FLASK framework.

[0189] To evaluate the performance of the proposed PFGformer, the following three real edge load datasets are used. In order to better show the load change pattern of different edge nodes, the load data of five edge nodes are randomly selected from each of the three datasets for visualization. Figure 5 As shown in Figure 2, for the same dataset, the load change patterns of different edge nodes are mostly different. For different datasets, the load change patterns of different edge nodes are more significantly different. Therefore, these three datasets can be used to fully test the generalization of the model.

[0190] (1) ECW records the changes in upload bandwidth of 28 edge nodes over 720 hours in August 2022, including 12-dimensional static features such as location and number of node CPUs. For this dataset, upload bandwidth is considered as load.

[0191] (2) Alibaba recorded the operating status of 4,000 servers over 8 days, including machine ID, timestamp, CPU utilization, memory utilization and other operating information. For this dataset, CPU utilization is considered as load.

[0192] (3) NEP recorded the operation information of edge servers on the NEP platform from June 1 to September 1, 2020 with a sampling frequency of 1 minute. The load data of 6,870 edge servers during June was used, where CPU utilization was regarded as load, and characteristics such as server location and user ID were taken into account.

[0193] The dataset is divided into training set, validation set and test set in the ratio of 7:1:2. The default input and prediction sequence lengths are both 96, and load instances are generated according to the input and prediction sequence lengths. MIT extracts content from static features and receives location information features of edge nodes, which are used as inputs of the prediction model together with the load data. The communication rounds are set to 5, the number of local training rounds is 10, the client sampling rate is 0.8, the number of hypernetwork layers is 3, the learning rate of FL server and client is 0.0001, the number of Transformer encoding layers is 2, the batch size is 32, and λ is 0.1. The time granularity of the dataset is resampled to 1 hour. Based on the above experimental settings, the following state-of-the-art FL frameworks are compared to verify the superiority of the proposed PFGformer.

[0194] (1) Centralized training: All client load data are concentrated on the FL server to train the prediction model.

[0195] (2) FedAvg: Each client trains a local prediction model and uploads the model parameters to the FL server for average aggregation.

[0196] (3) FedProx: Based on the FedAvg framework, it allows the client to adopt different local training rounds and adds a regularization term in the loss function to alleviate the device heterogeneity problem.

[0197] (4) pFedMe: Introduces Moreau Envelopes optimization based on the client-side loss function, allowing the client to update the local model in different directions without deviating from the global model, thereby learning a personalized model.

[0198] (5) FedPer: By combining personalized model aggregation and parameter update strategies, the generalization ability of personalized models is improved while protecting user privacy.

[0199] (6) Vanilla Personlized-T: Use the locally trained self-attention matrix parameters to replace the self-attention matrix parameters after federation aggregation.

[0200] (7) FedSage+: A linear predictor is introduced on the client side to repair missing connections in cross-subgraphs.

[0201] (8) FedGCN: The entire network is divided into multiple sub-networks through community detection, and FL and GCN are combined on a sub-network basis.

[0202] On each edge node, a comparison is made with the following state-of-the-art Transformer framework to verify the superiority of the proposed MIT.

[0203] (1) Crossformer: uses a two-stage attention mechanism and a hierarchical encoder-decoder architecture to capture the correlations between multivariate variables.

[0204] (2) Pyraformer: Introduces a pyramid-based attention mechanism to strike a balance between obtaining long-term dependencies and low time and space complexity.

[0205] (3) Informer: It combines ProbSpar self-attention mechanism, multi-head self-attention mechanism and knowledge distillation, and enhances the modeling ability of temporal features through temporal factor decomposition.

[0206] (4) Flashformer: uses Flash attention mechanism to improve the scope and speed of perceiving context.

[0207] (5) Flowformer: Based on the network flow conservation theorem, the Flow attention mechanism is designed to reduce the time complexity.

[0208] (6) Transformer: Uses self-attention mechanism to process input sequences in parallel and exploits the relationship between different positions in the sequence to capture long-distance dependencies.

[0209] Experimental results and analysis

[0210] First, the prediction performance of different prediction models under different training frameworks was evaluated. As shown in Table 1, in most cases, the prediction accuracy of the proposed MIT under different training frameworks is better than other methods, showing its superior load prediction performance in complex and changeable edge environments. This is because the MIT method efficiently extracts static features of the load by combining sequence reversal and SA Layer. This design improves the self-attention mechanism in the classic Transformer, making it suitable for load prediction tasks in complex edge environments. At the same time, MIT uses a feedforward neural network to assist in capturing the long-term and short-term dependencies of time series, and the results also verify the effectiveness of this design in improving the accuracy of multivariate time series prediction. The improved versions of existing Transformers usually embed multiple variables of the same timestamp into the same channel, which causes the model to over-focus on local perception and ignore the potential correlation between time series features. In addition, they usually regard node positions as static features without fully considering the spatial information of different nodes. Therefore, they are difficult to effectively adapt to multivariate time series edge load prediction tasks. Specifically, Crossformer focuses on the correlation between multiple variables in the data and proposes Dimension-Segment-Wise (DSW) embedding and attention across time steps, which performs well. However, it exposes a high computational complexity when processing high-dimensional data. At the same time, when facing dynamic edge loads, Crossformer cannot capture its changing features in time, which affects its accuracy. When dealing with dynamic loads, Pyraformer may not be able to quickly update model parameters, resulting in slow response. At the same time, in a multi-edge environment, high-dimensional data is usually accompanied by a large amount of redundant information, and Pyraformer cannot effectively filter out key features, resulting in model overfitting. Informer's self-attention distillation mechanism can reduce feature dimensions and memory consumption, but the use of convolutional layers and maximum pooling to distill features will lead to feature information loss and reduced accuracy. FlashAttention proposed by Flashformer divides the input into blocks and performs an attention mechanism on each block separately, which limits its ability to capture long-term dependencies and contextual information in the entire time series.

[0211] Table 1 Comparison of the proposed MIT and PFGformer with other state-of-the-art frameworks on different datasets

[0212]

[0213]

[0214]

[0215] Furthermore, the proposed PFGformer is compared with other SOTA training frameworks. As shown in Table 1, PFGformer shows superior performance under different datasets. This is because PFGformer uses the hypernet to learn the self-attention matrix parameters and linear layer parameters of Transformer, and introduces the λ aggregation mechanism. This design retains sufficient personalization for each client's Transformer prediction model to prevent it from being overly affected by other client models with low correlation. At the same time, PFGformer introduces a personalized adjacency matrix and uses GraphSAGE to generate personalized spatial feature vectors for each client. This design makes full use of the data correlation between edge nodes, so that each edge node can make full use of personalized information, thereby generating a personalized prediction model with better generalization. The accuracy of the centralized training framework is lower than that of PFGformer. This is because centralized training cannot capture the differences between edge nodes and cannot effectively utilize the correlation of their load patterns. In addition, the performance of the centralized training framework is better than FedProx, pFedMe, and FedPer. This is because the data of all edge nodes are concentrated in the cloud server for model training. In addition, it is difficult for classic FL to directly adapt to the Transformer-based prediction model, and the difference in data distribution also poses a huge challenge to generating a good personalized Transformer. Although Vanilla Personlized-T has made some personalized designs for Transformer, experimental results show that its improvement in prediction performance is still relatively limited. Compared with other frameworks, the proposed PFGformer can generate a superior personalized Transformer model for each client, further improving the accuracy of edge load prediction.

[0216] Comparison between the PFGformer framework and the FedGNN-based framework

[0217] The proposed PFGformer treats GNN as a spatial feature extractor and uses Transformer for load prediction. To further verify the superiority of the design, MIT is used as the backbone network and PFGformer is compared with other two advanced FedGNN-based frameworks on the ECW and Alibaba datasets. Figure 7 As shown in , PFGformer can maintain excellent data fitting ability as the prediction length increases. In contrast, other FedGNN-based frameworks perform poorly when faced with high-dimensional, high-heterogeneity, and high-variance edge loads. Figure 7(c) When the prediction length is 96, Fedsage+ and FedGNN show poor prediction performance. This is because the node connections in the edge environment are usually sparse, and GCN relies on the connection information between neighboring nodes for feature fusion and transmission, which cannot effectively handle sparse graph structures and model temporal correlation. At the same time, GCN performs feature fusion through the connection information between nodes, and has weak feature modeling capabilities for diverse edge nodes. Although FedSage+ can use GraphSAGE to complete the missing connections between some edge nodes, it still cannot fully utilize the feature information of edge nodes with high correlation at long distances, resulting in a decrease in prediction performance. FedGCN uses traditional GCN as a predictor, which can better utilize the data correlation between nodes. In short-term prediction scenarios, FedGCN has good performance, but its performance cannot be guaranteed in long-term prediction scenarios. In contrast, PFGformer constructs a personalized adjacency matrix through EMD and uses GraphSAGE to extract spatial features, which fully utilizes the spatial correlation between edge nodes. At the same time, PFGformer adopts MIT as a predictor, which can effectively capture timing dependencies and timing changes to better adapt to high-latitude and high-dynamic edge environments.

[0218] Parameter sensitivity

[0219] In order to better balance the differences between the global and local models and overcome the data and device heterogeneity between different edge nodes, a λ aggregation mechanism is introduced and the impact of different λ values ​​on the performance of PFGformer is analyzed. Figure 7 As shown in the figure, the performance of different prediction models is tested in the PFGformer framework. The results show that when λ is 0.1, the best load prediction accuracy can be obtained. Too large a λ value will cause the personalized model to focus too much on local data features and ignore valuable information from other clients. For example, when λ=1, PFGformer performs the worst among the three models except MIT. The original intention of PFGformer's design is to reasonably collaborate the models of each client. Therefore, a smaller λ value is conducive to making better use of other clients and solving problems such as insufficient local data scale and single data mode. It is worth noting that when λ=0, the personalization of the local model depends entirely on the aggregation results, causing the local model to excessively forget its own characteristics.

[0220] Communication and training overhead

[0221] In order to make PFGformer better adapt to edge environments with limited communication and computing resources, SALayer is first used in MIT to compress static features to reduce the number of model channels. Then, linear layers are used to replace traditional Transformer encoding blocks to reduce the number of model parameters. Finally, MLP is used to generate node feature vectors for the hypernetwork and GraphSAGE to generate self-attention matrix parameters and spatial feature vectors, reducing communication overhead. Figure 8 In , the bars and lines represent the communication overhead and training overhead of different frameworks respectively. Figure 8 As shown in (a), the proposed MIT can significantly reduce communication overhead compared to other Transformer-based prediction models. At the same time, since MIT only contains encoding layers and linear mapping layers, the training time is greatly shortened. Relatively speaking, other prediction models cause more communication overhead and training overhead. Specifically, Crossformer emphasizes multi-scale feature extraction and has a complex network structure, which leads to excessive training overhead when facing high-dimensional dynamic data. Flashformer has the highest training overhead because it uses three position encodings: additive absolute, additive relative, and RoPE. In addition, the communication and training overheads of the proposed PFGformer are compared with those of other personalized FL frameworks. As shown in Figure 8 As shown in (b), PFGformer does not increase communication overhead while ensuring high accuracy, and has a faster training speed. This is because although PFGformer needs to transmit spatial feature vectors, it does not need to upload self-attention matrix parameters and linear layer parameters, which keeps the communication overhead from increasing. Each client in pFedMe needs to update the loss function multiple times to personalize the local model, resulting in more training overhead.

[0222] Ablation experiment

[0223] In the proposed PFGformer, firstly, a hypernetwork, SA Layer and λ aggregation mechanism are designed in FL to generate a personalized Transformer. Then, the graph topology generator and GraphSAGE are combined to effectively utilize the positional relationship of edge nodes. For the components designed in PFGformer, their effectiveness is verified by ablation experiments. As shown in Table 2, after the introduction of the three components, the prediction accuracy is improved to a certain extent. Specifically, for the Alibaba dataset, GraphSAGE significantly affects the prediction accuracy. This is because there are many node relationships with the same data mode that can be utilized in the Alibaba dataset. At the same time, Hypernetwork and SA Layer have a great role in promoting the improvement of prediction accuracy in different datasets. This shows that the node position and static features have a great influence on the trend of edge load changes, and also proves the rationality of using hypernetwork to update local MIT parameters. In some cases, the introduction of SALayer in Crossformer will result in a decrease in accuracy. This is because Crossformer uses multi-scale and cross-modal representations to capture complex time series features, so the introduction of SA Layer will affect its capture of feature relationships, resulting in a decrease in accuracy.

[0224] Table 2 Ablation experiment of PFGformer ("+": used, "-": not used).

[0225]

[0226]

[0227] Advantages and uses of this product

[0228] In this paper, a novel MIT is first proposed to efficiently capture the important representation of edge load changes, the spatial information of nodes, and the influence of static features. Then, a graph topology generator is designed to generate multi-order adjacency matrices of edge nodes through EMD, and generate personalized graph topology for each node for extracting spatial feature information. Then, the hypernetwork and λ aggregation mechanism are introduced in PFG to generate personalized prediction model parameters, and GraphSAGE is used to extract node spatial features. Finally, by combining MIT and PFG, an innovative PFGformer is designed to efficiently generate a powerful personalized Transformer model for each client. In particular, the excellent generalization ability and convergence of PFGformer are proved through rigorous derivation. Based on multiple real-world edge load datasets and test beds, a large number of experiments verify the superiority of the proposed MIT and PFGformer. The results show that compared with the state-of-the-art frameworks, MIT and PFGformer have achieved higher prediction accuracy in different prediction scenarios. At the same time, PFGformer shows lower communication and training overhead. In addition, the effectiveness of each component in PFGformer is verified by ablation experiments.

[0229] The process or method of using the product

[0230] (1) The PFGformer server randomly selects a certain proportion of clients, uses the hypernetwork to learn the client’s self-attention matrix parameters and linear layer parameters, and generates a spatial feature vector.

[0231] (2) The PFGformer server distributes the self-attention matrix parameters, linear layer parameters, other parameters, and spatial feature vectors to the client for client parameter updates.

[0232] (3) The PFGformer client uploads the model parameters trained with local data to the server.

[0233] (4) The PFGformer server accepts the client model parameters to update the parameters of the hypernetwork.

[0234] The client's load prediction results are sent to the edge server for the edge computing service provider to adjust the resource management strategy.

Claims

1. A spatiotemporal load prediction method based on personalized federated graph learning Transformer in multi-edge systems, characterized by: The following steps are involved: Step 1: Build a multivariate sequence reversal Transformer as a load prediction model for edge nodes; Step 2: Design PFG to generate personalized model parameters and spatial features for each edge node through the hypernetwork and graph neural network GNN; Update the parameters and transmit them to the FL server to calculate the parameter changes of the prediction model. The FL server updates the remaining parameters of the global model, the hypernetwork parameters, and the node feature vectors according to the parameters returned by the client. Step 3: Combine MIT and PFG to form PFGformer, which generates personalized load prediction for each edge node.

2. The spatiotemporal load prediction method based on personalized federated graph learning Transformer in a multi-edge system according to claim 1, characterized in that: Graph neural network GNN is used to embed and reason about graph data. Its input includes: a graph consisting of nodes and edges, represented as an adjacency matrix A∈R K×K ; and the node feature matrix X∈R K×dmodel , where K represents the number of nodes and d mod el represents the number of node features; the graph neural network GNN updates the node embedding based on the information of adjacent nodes, which is defined as: S (l+1) =GNN(A,S (l) ,w (l) ), (1) Where GNN(·) represents the graph aggregation function, which is average, weighted average, or maximum / minimum pooling; S (l) represents the node embedding of layer l, w (l) represents the trainable weights of layer l.

3. The spatiotemporal load prediction method based on personalized federated graph learning Transformer in a multi-edge system according to claim 1, characterized in that: The aggregation process of the FL server is defined as Among them, θ i is the local model parameter of the i-th client, θ c+1 is the global model parameter for the next iteration, K is the number of participating clients, and D i is the number of samples of the ith client, and D is the total number of samples of all clients; Indicates the contribution of each client to the global model.

4. The spatiotemporal load prediction method based on personalized federated graph learning Transformer in a multi-edge system according to claim 1, characterized in that: Edge load contains dynamic and static characteristics of date, node location, and disk capacity; under a fixed prediction window, multiple load sequences on the same edge node are integrated into historical load input, which is defined as Where Γ represents the last time index observed in the historical load data, d t is the dynamic feature dimension, d t >1; For a set of load data input, the prediction process is expressed as in, represents the predicted load sequence, L is the prediction length, and f is the prediction model; represents the static characteristics, where d s Represents the static feature dimension d s >1; To evaluate the accuracy of edge load prediction, MSE and MAE are considered as performance indicators, which are defined as Among them, y i represents the actual load within an interval, and S is the number of load sequences.

5. The spatiotemporal load prediction method based on personalized federated graph learning Transformer in a multi-edge system according to claim 1, characterized in that: MIT consists of an embedding layer, an encoder layer, and an output layer.

6. The spatiotemporal load prediction method based on personalized federated graph learning Transformer in a multi-edge system according to claim 5, characterized in that: The embedding layer consists of a sequence reversal and a static content perception layer; the sequence reversal specifically divides the time series along the node dimension, expressed as Among them, permute(X t ,[1,3,2]) means to convert the original tensor X t The second and third dimensions exchange positions to achieve the interchange of time and node dimensions; The proposed sequence reversal uses the self-attention mechanism to extract the association between different features and combines it with the SA Layer to integrate multi-dimensional features into the temporal dynamic features; MIT uses a feedforward neural network to efficiently extract the temporal features in the sequence, and then combines them with the spatial features to form spatiotemporal information to support accurate edge load prediction; The static content perception layer uses SA Layer and self-attention mechanism to extract potential associations between different features; SALayer is used to combine static context with dynamic load sequence data, and use high-dimensional cross-domain data as additional static content; SALayer uses cross attention to select the most effective static content for integration according to the current sequence, and its input includes load sequence and static feature data P t ; First, connect the reversed load sequence and timestamp as the timing dynamic feature, which is defined as Next, the detailed information contained in the SA Layer is expressed as α t =softmax -1 (Q t ) V=Linaer2(P t ) Among them, softmax -1 Express Q t The last dimension of is softmaxed, Linear1 and Linear2 represent different linear layers, and Norm represents layer normalization; By integrating load sequences and static features, SA Layer learns the static content and inherent properties of edge nodes and filters out redundant information in high-dimensional data; at the same time, the number of channels and parameters of the prediction model will also be reduced.

7. The spatiotemporal load prediction method based on personalized federated graph learning Transformer in a multi-edge system according to claim 5, characterized in that: The encoding layer is used to extract spatiotemporal correlation and high-dimensional static features. The encoding layer contains a feedforward neural network and a self-attention layer module. The feedforward neural network is used to capture the relationship between time points and discard the decoding block. The feedforward neural network is defined as: in, represents the state of the l-1th hidden layer, W (l) represents the weight matrix, b (l) is the bias vector, gelu(·) is the activation function; The self-attention layer is used to capture the correlation of multiple variables, which integrates the high-dimensional static features extracted by the SA Layer; specifically, the self-attention layer converts each element in the input sequence into a query Q, key K, and value V vector, and calculates the dot product of the query with all keys; the process is defined as In addition, the multi-head attention mechanism consists of multiple self-attention layers, focusing on the representation subspace information from different positions; specifically, multiple parallel attention layers are adopted and embedded in the Transformer encoder, which is defined as H l =MultiHead(Q,K,V)=Concat(head1,...,head n )W O (11) Among them, MultiHead(·) represents the multi-head attention mechanism, l={0,…,L} represents the encoding layer index, head n represents the nth parallel attention layer, is the parameter matrix.

8. The spatiotemporal load prediction method based on personalized federated graph learning Transformer in a multi-edge system according to claim 1, characterized in that: The objective function of PFGformer is defined as in, represents the loss function of the i-th client; In PFGformer, the FL server will learn a hypernetwork to generate personalized self-attention matrix parameters and linear layer parameters for each client; The model parameters of MIT are i Split into W i , κ i and i , respectively represent the self-attention matrix parameters, linear layer parameters and other layer parameters; input sequence X t The length of the sequence is recorded as T. After the sequence passes through the embedding layer, the input self-attention layer is converted into an embedding matrix The query, key, and value of the self-attention mechanism are represented as and Where W i =[W i Q ,W i K ,W i V ]; The super network is denoted as Hnet(φ,z i ), where φ represents the parameters of the hypernetwork, z i ∈R dmodel represents the feature vector of client i; let the FL server self-learn z i , which will be automatically generated by the Embedding layer; the hypernetwork is implemented with n fully connected layers, where the first n-1 layers are shared by all client Transformer encoding blocks, and the last layer is unique for each encoding block; the hypernetwork is introduced to learn the self-attention matrix parameters and linear layer parameters {W i ,κ i =Hnet(φ,z i )}, to realize a personalized MIT-based prediction model; the prediction model and its loss function are defined as In each local epoch, the local model parameters are updated using stochastic gradient descent, which is defined as Among them, B i It is from D i A batch sampled from , α represents the client learning rate; According to the chain rule, φ and z i The gradient of is calculated as Where C represents the total communication rounds, ΔW i =W i J -W i 0 , Δκ i =κ i J -κ i 0 represents the changes in the self-attention matrix parameters and linear layer parameters after J local training rounds; After each communication round, use the gradient and To update the hypernetwork parameters φ and the client embedding z i , which is defined as Where β represents the learning rate of the FL server, and the superscript c represents the cth communication round.

9. The spatiotemporal load prediction method based on personalized federated graph learning Transformer in a multi-edge system according to claim 1, characterized in that: EMD is introduced to reconstruct and generate a multi-order adjacency matrix between edge nodes. For any two clients i and i′, their similarity is calculated as Next, the similarity of the nodes is constructed as a similarity matrix S EMD ∈R K×K , where s ii′ represents the element in row i and column i′; In this way, a personalized adjacency matrix is ​​generated for each node; EMD Select several most relevant neighbors to aggregate multi-order neighbors; introduce GraphSAGE to process the adjacency matrix and use the client's feature vector z i as node vectors; GraphSAGE is used to perform aggregation operations by sampling neighbor nodes, thereby generating spatial feature vectors of nodes and transmitting them to the client; the client connects the spatial feature vectors with the original feature vectors for subsequent predictions; (1) Node k receives the embedding vectors generated by all direct neighbor nodes N(k) in the previous layer It is defined as: Among them, AGGREGATE(·) represents the aggregation function, represents the neighbor aggregation information received by node k at layer t; The initial input is the node’s feature vector z k , (2) Aggregate the information in adjacent nodes through the connection function; the embedding of node k in layer t is and The concatenation of Among them, σ(·) is a nonlinear activation function; CONCAT(·) represents the connection function, which uses MEAN as an aggregator; and After the connection, an average operation is performed on each dimension of the vector, and the result is nonlinearly transformed to obtain the embedding vector of node k at layer t; (3) The embedding output of each node is 10. The spatiotemporal load prediction method based on personalized federated graph learning Transformer in a multi-edge system according to claim 1, characterized in that: Design the λ aggregation mechanism. Specifically, introduce a hyperparameter λ as the weight when updating the global and local models, which is defined as in, represents the remaining parameters of the local model of client i in round c, represents the remaining parameters of the global model in round c; The update deviation of the local model is corrected back to the ideal convergence direction to improve the prediction performance of local models of different clients; the loss function of PFGformer is rewritten as

Images