Spatiotemporal load prediction method based on personalized federated graph learning transformer in multi-edge system

Abstract

This invention provides a spatiotemporal load prediction method based on personalized federated graph learning (PFG) Transformer in multi-edge systems, comprising the following steps: Step 1: Constructing a multivariate sequence inversion Transformer as the load prediction model for edge nodes; Step 2: Designing a PFG, generating personalized model parameters and spatial features for each edge node through a hypernetwork and a graph neural network (GNN); updating the parameters and transmitting them to the FL server to calculate the parameter changes of the prediction model; the FL server updates the remaining parameters of the global model, hypernetwork parameters, and node feature vectors according to the parameters returned by the client; Step 3: Combining MIT and PFG to form a PFGformer, generating personalized load for each edge node for prediction. This technical solution uses coded blocks and multilayer perceptron processing for the prediction task, eliminating complex decoding blocks, which improves prediction accuracy and reduces the number of model parameters.

Description

Technical Field

[0001] This invention relates to the field of Internet of Things (IoT) technology, and in particular to a spatiotemporal load prediction method based on personalized federated graph learning Transformer in multi-edge systems. Background Technology

[0002] With the rapid development of the Internet of Things (IoT), numerous smart applications are emerging, such as smart homes, VR / AR, and energy management. Due to the computationally intensive and latency-sensitive nature of these smart applications, existing resource-intensive IoT devices struggle to guarantee a good Quality of Experience (QoE). Classic cloud computing has revealed bottlenecks in areas such as transmission latency, energy consumption, and privacy security. As an emerging computing paradigm, Mobile Edge Computing (MEC) reduces the dependence of IoT devices on remote clouds by pushing computing and storage resources down to edge nodes closer to the data source, achieving faster response and higher reliability. In MEC, the resource requirements of edge nodes typically depend on their geographical distribution, and node heterogeneity poses significant resource management challenges for Edge Service Providers (ESPs). Introducing virtualization technology can improve the flexibility and efficiency of edge system resource provisioning. However, it takes time to start virtual machines (VMs) to provide them to users. For example, starting a virtual machine typically takes several minutes, while starting a container (such as Docker or Podman) usually takes a few seconds to a minute. Therefore, it is necessary to effectively capture the load characteristics of edge systems and predict future load changes to support ESP in proactively pre-configuring resources, thereby achieving efficient and reasonable resource provisioning.

[0003] In recent years, Transformers have been widely used in time series forecasting problems. By introducing self-attention mechanisms to achieve global context modeling and parallel computation, Transformers avoid gradient vanishing and are suitable for handling long-term dependencies. However, existing Transformer-based frameworks still have limitations when facing multivariate time series forecasting problems. Typically, these frameworks embed multiple variables at the same timestamp into indistinguishable channels and use attention mechanisms to capture temporal dependencies. However, this approach obscures important associations between variables, misleads the learning direction of the self-attention mechanism, and reduces prediction performance. Furthermore, when facing multivariate time series forecasting problems, simple linear layers can better ensure variable independence and learn multivariate correlations, with prediction accuracy and efficiency even surpassing the complex Transformer. Consider reversing the time series, so that the complete time series of each variable is independently embedded into the input channel. This design will expand the local awareness of the prediction model, enabling it to fully utilize the self-attention mechanism to associate latent features between multiple variables. In addition, the following open issues remain to be addressed in marginal load forecasting.

[0004] (1) Inefficient processing of high-dimensional features. In MEC environments, edge nodes typically possess high-dimensional cross-domain features, including hardware configuration (e.g., CPU and memory) and node location, which influence edge load variations in various ways. Existing research often uses static feature encoding methods, which are insufficient to fully represent high-dimensional cross-domain data.

[0005] (2) Limitations of Single Edge Load Data. Edge nodes are discretely distributed, and the load data of each node is limited and uniformly distributed. To ensure good prediction performance, deep learning-based models typically need to rely on a large amount of diverse historical data for training. Therefore, it is difficult to guarantee high prediction performance by training a prediction model using only the load data of a single edge node.

[0006] (3) Insufficient utilization of spatial features. Most existing studies adopt a centralized training method, which does not fully explore the potential value of the spatial features of different edge nodes in improving load prediction performance. For example, edge nodes in different locations may have similar load change patterns. Therefore, effectively utilizing the spatial features of different edge nodes can help with load prediction. Summary of the Invention

[0007] In view of this, the purpose of this invention is to provide a spatiotemporal load prediction method based on personalized federated graph learning Transformer in multi-edge systems. The method uses coded blocks and multilayer perceptron processing for prediction tasks, eliminating the need for complex decoding blocks, which improves prediction accuracy and reduces the number of model parameters.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: a spatiotemporal load prediction method based on personalized federated graph learning Transformer in multi-edge systems, comprising the following steps:

[0009] Step 1: Construct a load prediction model using a multivariate sequence inversion Transformer as an edge node;

[0010] Step 2: Design PFG, generate personalized model parameters and spatial features for each edge node through hypernetwork and graph neural network GNN; update the parameters and send them to FL server to calculate the parameter changes of the prediction model. FL server updates the remaining parameters of the global model, hypernetwork parameters and node feature vectors according to the parameters returned by the client.

[0011] Step 3: Combine MIT and PFG to form PFGformer, and generate personalized load predictions for each edge node.

[0012] In a preferred embodiment, a graph neural network (GNN) is used for feature embedding and inference on graph data. Its input includes a graph consisting of nodes and edges, represented as an adjacency matrix A∈R. K×K ; and the node feature matrix X∈R K×dmodel Where K represents the number of nodes and dmodel represents the number of node features; the graph neural network (GNN) updates the node embedding based on neighboring node information, which is defined as:

[0013] S (l+1) =GNN(A,S) (l) ,w (l) ), (1)

[0014] Where GNN(·) represents the graph pooling function, which is an average, weighted average, or max / min pooling function; S (l) w represents the node embedding at layer l. (l) This represents the trainable weights of the l-th layer.

[0015] In a preferred embodiment, the aggregation process of the FL server is defined as follows:

[0016]

[0017] Where, θ i θ is the local model parameter of the i-th client. c+1 These are the global model parameters for the next iteration, where K is the number of participating clients, and D... i Let be the number of samples from the i-th client, and D be the total number of samples from all clients. This indicates the degree of contribution each client makes to the global model.

[0018] In a preferred embodiment, the edge load includes dynamic and static features such as date, node location, and disk capacity; under a fixed prediction window, multiple load sequences on the same edge node are integrated into a historical load input, which is defined as...

[0019]

[0020] Where Γ represents the last time index observed in the historical load data, and d t It is a dynamic feature dimension, d t >1; For a set of load data inputs, the prediction process is represented as

[0021]

[0022] in, This represents the predicted load sequence, where L is the prediction length and f is the prediction model. Denotes static features, where d s d represents the static feature dimension s >1;

[0023] To evaluate the accuracy of edge load prediction, MSE and MAE are considered performance metrics, defined as follows:

[0024]

[0025] Among them, y i This represents the actual load within a certain interval, where S is the number of load sequences.

[0026] In a preferred embodiment, the MIT consists of an embedding layer, an encoder layer, and an output layer.

[0027] In a preferred embodiment, the embedding layer consists of a sequence inversion layer and a static content awareness layer; the sequence inversion specifically divides the time series along the node dimension, represented as follows:

[0028]

[0029] Among them, permute(X t [1,3,2]) represents swapping the second and third dimensions of the original tensor Xt to achieve the interchange of time and node dimensions;

[0030] The proposed sequence inversion combines the self-attention mechanism to extract the correlation between different features with SALayer, thereby integrating multi-dimensional features into temporal dynamic features; MIT uses a feedforward neural network to efficiently extract temporal features from the sequence, and then combines them with spatial features to form spatiotemporal information to support accurate edge load prediction.

[0031] The static content-aware layer uses a static context layer (SA layer) and a self-attention mechanism to extract potential correlations between different features. The SA layer combines static context with dynamic load sequence data, and incorporates high-dimensional cross-domain data as additional static content. The SA layer employs cross-attention to select the most effective static content for integration based on the current sequence, and its input includes the load sequence. and static feature data P t First, the inverted load sequence is concatenated with the timestamp as a time-series dynamic feature, defined as follows:

[0032]

[0033] Next, the detailed information contained in the SA Layer is represented as follows:

[0034]

[0035] α t =softmax -1 (Q t )

[0036] V = Linaer2(P) t )

[0037]

[0038] Among them, softmax -1 Indicates Q t The last dimension is softmaxed, Linear1 and Linear2 represent different linear layers, and Norm represents layer normalization;

[0039] By integrating load sequences and static features, the SA Layer learns the static content and inherent attributes of edge nodes, filtering out redundant information in high-dimensional data; at the same time, the number of channels and parameters of the prediction model will also be reduced.

[0040] In a preferred embodiment, the encoding layer is used to extract spatiotemporal correlations and high-dimensional static features. The encoding layer includes a feedforward neural network and a self-attention layer module. The feedforward neural network is used to capture the relationships between time points and discard the decoding blocks. The feedforward neural network is defined as follows:

[0041]

[0042] in, W represents the (l-1)th hidden layer state. (l) Let b represent the weight matrix. (l) It is the bias vector, and gelu(·) is the activation function;

[0043] The self-attention layer is used to capture the correlation of multiple variables, fusing high-dimensional static features extracted by the SA layer; specifically, the self-attention layer transforms each element in the input sequence into a vector of query Q, key K, and value V, and computes the dot product of the queries with all keys; the process is defined as follows:

[0044]

[0045] Furthermore, the multi-head attention mechanism consists of multiple self-attention layers, focusing on representation subspace information from different locations; specifically, it employs multiple parallel attention layers and embeds a Transformer encoder, which is defined as...

[0046] H l =MultiHead(Q,K,V)=Concat(head1,...,head n W O (11)

[0047] Where MultiHead(·) represents the multi-head attention mechanism, l={0,…,L} represents the coding layer index, and head n This represents the nth parallel attention layer. It is a parameter matrix.

[0048] In a preferred embodiment, the objective function of PFGformer is defined as

[0049]

[0050] in, Let represent the loss function for the i-th client;

[0051] In PFGformer, the FL server will learn a supernetwork to generate personalized self-attention matrix parameters and linear layer parameters for each client;

[0052] The model parameters θ of MIT i Split into W i κ i and ξ i , represent the parameters of the self-attention matrix, the parameters of the linear layer, and the parameters of other layers, respectively; input sequence X t The length of the sequence is denoted as T. After the sequence passes through the embedding layer, the input to the self-attention layer is transformed into an embedding matrix. The query, key, and value in a self-attention mechanism are represented as follows: and Among them W i =[W i Q W i K Wi V ];

[0053] Hypernet is denoted as Hnet(φ,z) i ), where φ represents the parameters of the hypernet, z i ∈R dmodel Let z represent the feature vector of client i; let the FL server learn z itself. i It will be automatically generated through the Embedding layer; the hypernetwork is implemented using n fully connected layers, where the first n-1 layers are shared by all client Transformer encoding blocks, and the last layer is unique for each encoding block; the self-attention matrix parameters and linear layer parameters {W} are learned by introducing the hypernetwork. i ,κ i =Hnet(φ,z) i This enables personalized MIT-based prediction models; the prediction model and its loss function are defined as follows:

[0054]

[0055] In each local epoch, the local model parameters are updated using stochastic gradient descent, which is defined as follows:

[0056]

[0057] Among them, B i From D i A batch is sampled in the middle, and α represents the client learning rate;

[0058] According to the chain rule, φ and z i The gradient is calculated as

[0059]

[0060] Where C represents the total number of communication rounds, ΔW i =W i J -W i 0 , Δκ i =κ i J -κ i 0 This represents the changes in the parameters of the self-attention matrix and the linear layer after J local training rounds;

[0061] After each round of communication, use gradients. and To update the hypernetwork parameters φ and client embedding z i It is defined as

[0062]

[0063] Where β represents the learning rate of the FL server, and the superscript c represents the c-th communication round.

[0064] In a preferred embodiment, EMD is introduced to reconstruct and generate a multi-order adjacency matrix between edge nodes, where the similarity between any two clients i and i′ is calculated as follows:

[0065]

[0066] Next, the similarity of the nodes is constructed into a similarity matrix S. EMD ∈R K×K , where s ii ' represents the element in the i-th row and i'-th column; this method generates a personalized adjacency matrix for each node; in s EMD We select several of the most relevant neighbors to aggregate multi-order neighbors; we introduce GraphSAGE to process the adjacency matrix and use the client's feature vector z. i As a node vector, GraphSAGE is used to perform aggregation operations by sampling neighboring nodes, thereby generating spatial feature vectors for the nodes and transmitting them to the client; the client then concatenates the spatial feature vectors with the original feature vectors for subsequent predictions.

[0067] (1) Node k receives the embedding vectors generated by all its direct neighbor nodes N(k) in the previous layer. It is defined as:

[0068]

[0069] Where AGGREGATE(·) represents an aggregate function, This represents the neighbor aggregation information received by node k at layer t; The initial input is the feature vector of the node.

[0070] (2) Aggregate information from adjacent nodes through a join function; the embedding of node k in layer t is... and The series connection is defined as

[0071]

[0072] Where σ(·) is a nonlinear activation function; CONCAT(·) represents the connection function, which uses MEAN as the aggregator; and After concatenation, an averaging operation is performed on each dimension of the vector, and the result is subjected to a nonlinear transformation to obtain the embedding vector of node k in layer t.

[0073] (3) The embedding output of each node is

[0074] In a preferred embodiment, a λ aggregation mechanism is designed, specifically, a hyperparameter λ is introduced as the weights when updating the global and local models, defined as follows:

[0075]

[0076] in, This represents the remaining parameters of the local model of client i in round c. This represents the remaining parameters of the global model in round c;

[0077] The update offset of the local model is corrected back to the ideal convergence direction, improving the prediction performance of the local model on different clients; the loss function of PFGformer is rewritten as...

[0078]

[0079] Compared with the prior art, the present invention has the following beneficial effects:

[0080] (1) A novel edge load prediction model based on MIT is proposed. Specifically, a novel feature fusion method is designed first, allowing the prediction model to select the most suitable static features for fusion. Therefore, cross-domain static features can be reasonably utilized without being disturbed by redundant features, improving the model training speed under multivariable dimensions. At the same time, due to the reduction in feature dimensions, the number of channels and model parameters are also reduced, greatly reducing communication overhead. Unlike existing methods that divide load data based on timestamps, sequence reversal is introduced to divide data along the variable dimension. This design prevents the variable dimension from being confused and expands local perception, so the self-attention mechanism can effectively extract the potential relationships between variables. In addition, only coded blocks and multilayer perceptrons (MLPs) are used to process the prediction task, eliminating the complex decoding blocks, which improves prediction accuracy and reduces the number of model parameters. Therefore, MIT can better utilize spatial feature information through feature fusion and self-attention mechanisms.

[0081] (2) A novel graph topology generator is designed. Based on Earth Mover's Distance (EMD), a multi-order adjacency matrix between nodes is generated according to the inconsistent distribution of edge nodes, their data volume, and hardware configuration. Personalized graph topology data containing spatial feature information of neighboring nodes is generated for each node. Therefore, the proposed graph topology generator can realistically reflect the spatial relationships between nodes, thus better supporting the extraction of node spatial feature information.

[0082] (3) An innovative PFG framework was developed, consisting of a supernetwork and Graph Sample and Aggregate (GraphSAGE). First, an MLP is used to generate node feature vectors as input for the supernetwork and GraphSAGE. By utilizing these feature vectors, the MIT model at each edge node can learn personalized self-attention matrix parameters and linear layer parameters. Next, a λ-aggregation mechanism is introduced to balance the differences between the global and local models, further optimizing the personalized models at the edge nodes. Simultaneously, GraphSAGE is applied to the personalized graph to generate personalized spatial features for each edge node. In particular, the excellent generalization ability and convergence of PFGformer are rigorously proven through derivation.

[0083] (4) Extensive experiments validated the effectiveness of the proposed PFGformer based on multiple real-world edge load datasets and testbeds. Compared to state-of-the-art frameworks, PFGformer achieved higher prediction accuracy across different prediction lengths. For different Transformer-based centralized and FL training frameworks, PFGformer achieved higher prediction accuracy and lower communication and training overhead. Furthermore, ablation experiments verified the effectiveness of each component in PFGformer. Attached Figure Description

[0084] Figure 1 This is a schematic diagram showing an overview of the MIT (Made in Taiwan) according to a preferred embodiment of the present invention.

[0085] Figure 2 This is a schematic diagram of an overview of the PFGformer according to a preferred embodiment of the present invention;

[0086] Figure 3 This is a schematic diagram of an overview of HyperNetworks according to a preferred embodiment of the present invention;

[0087] Figure 4 This is a schematic diagram of load changes and similarity matrices for different edge nodes in a preferred embodiment of the present invention, wherein (a) represents load changes and (b) represents the similarity matrix;

[0088] Figure 5 This is a schematic diagram of different load modes of different edge nodes in different datasets according to a preferred embodiment of the present invention, wherein (a) is ECW, (b) is Alibaba, and (c) is NEP;

[0089] Figure 6 The diagram illustrates different load patterns of different edge nodes in different datasets according to a preferred embodiment of the present invention, wherein (a) is a prediction length of 24, (b) is a prediction length of 48, and (c) is a prediction length of 96.

[0090] Figure 7 This is a schematic diagram of different load modes of different edge nodes in different datasets in a preferred embodiment of the present invention, wherein (a) is MIT, (b) is Crossformer, (c) is Pyraformer, (d) is Flashformer, (e) is Flowformer, and (f) is Inoformer;

[0091] Figure 8 This is a schematic diagram comparing the communication and training costs between different frameworks in a preferred embodiment of the present invention, wherein (a) represents the communication and training costs of PFGformer using different prediction models, and (b) represents the communication and training costs of different personalized federated learning frameworks. Detailed Implementation

[0092] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0093] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0094] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations according to this application; as used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise; furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0095] A spatiotemporal load prediction method based on personalized federated graph learning Transformer in multi-edge systems, reference Figure 1-8 First, we propose MIT as the load prediction model for edge nodes, aiming to efficiently utilize the high-dimensional features of edge load and the spatial features of nodes. Next, we design PFG, which aims to generate personalized model parameters and spatial features for each edge node through a hypernetwork and GNN. Finally, by combining MIT and PFG, we propose PFGformer to generate a personalized load prediction model for each edge node.

[0096] In PFGformer, a hypernetwork, GraphSAGE, and λ aggregation mechanism are integrated to generate a personalized MIT-based edge load prediction model for each client. PFGformer is well-suited for Transformer models based on self-attention mechanisms and appropriately combines FedGNN with multi-edge scenarios to improve the performance of the load prediction model. The training process of PFGformer is shown in Algorithm 1. For the FL server, the required parameters and spatial feature vectors are first generated using the hypernetwork and GraphSAGE (lines 8-11). Next, the client parameters are updated and sent to the FL server to compute the parameter changes of the prediction model (lines 12-13). Finally, the FL server updates the remaining parameters of the global model, hypernetwork parameters, and node feature vectors according to the parameters returned by the client (lines 15-17). For each client, the client model parameters are first updated (lines 19-20). Next, the client updates the client model parameters using local data according to gradient descent (lines 21-24). Finally, the client uploads the model parameters to the FL server (line 25).

[0097]

[0098]

[0099] The following sections will introduce the definitions of graph neural networks, federated learning, and edge load prediction.

[0100] Graph Neural Networks (GNNs)

[0101] GNNs can be used for feature embedding and inference on graph data. Their input includes: 1) a graph consisting of nodes and edges, represented as an adjacency matrix A∈R. K×K ;2) Node feature matrix X∈R K×dmodel Where K represents the number of nodes, and (dmodel) represents the number of node features. GNN updates node embeddings based on neighboring node information; this process is defined as follows:

[0102] S (l+1) =GNN(A,S) (l) ,w (l) ), (1)

[0103] Where GNN(·) represents the graph pooling function, which can be average, weighted average, or max / min pooling. (l) w represents the node embedding at layer l. (l) This represents the trainable weights of the l-th layer.

[0104] Federated Learning (FL)

[0105] In classic machine learning, all raw user data needs to be transmitted to a central server for model training, which may expose user privacy. By training the model locally on the client side, the new FL avoids the aggregation of raw data, thus protecting user privacy.

[0106] In FL (Flexible Downloader), user raw data is distributed across clients. Each client independently trains a local model using its local data and uploads the model parameters to a central server. These local model parameters are then used to aggregate the data to generate a global model, which is distributed to all clients. Therefore, features from different clients can be shared without transmitting the raw data. FL's aggregation process is defined as follows:

[0107]

[0108] Where, θ i θ is the local model parameter of the i-th client. c+1 These are the global model parameters for the next iteration, where K is the number of participating clients, and D... i Let be the number of samples from the i-th client, and D be the total number of samples from all clients. This indicates the degree of contribution of each client to the global model; clients with more samples have a greater impact on the global model update.

[0109] Edge load prediction

[0110] By predicting future load changes (such as CPU, memory, and bandwidth), edge systems can improve resource utilization and service quality. Edge load includes dynamic and static characteristics such as date, node location, and disk capacity; therefore, edge load prediction can be viewed as a multivariate, multi-sequence prediction problem. Under a fixed prediction window, multiple load sequences on the same edge node can be integrated into a historical load input, defined as...

[0111]

[0112] Where Γ represents the last observable time index in the historical load data, and d t It is a dynamic feature dimension (d) t >1).

[0113] Given a set of load data inputs, the prediction process can be represented as follows:

[0114]

[0115] in, This represents the predicted load sequence, where L is the prediction length and f is the prediction model. Denotes static features, where d sRepresents the static feature dimension (d) s >1).

[0116] To evaluate the accuracy of edge load prediction, Mean Squared Error (MSE) and Mean Absolute Error (MAE) are considered performance metrics, defined as follows:

[0117]

[0118] Among them, y i This represents the actual load within a certain interval, where S is the number of load sequences.

[0119] MIT: Multivariate Sequence Inversion Transformer

[0120] The proposed load prediction model for MIT as an edge node. For example... Figure 1 As shown, the MIT (Match 1) consists of an embedding layer, an encoder layer, and an output layer. Its key steps are illustrated in Algorithm 1. First, the embedding layer performs sequence inversion and extracts static features of the payload data (lines 5-7). Next, the encoder layer utilizes a Transformer to capture long-short-term and feature dependencies (lines 8-11). Finally, the output layer utilizes a linear layer to predict future payloads (lines 12-13). The detailed design of the embedding and encoder layers is as follows.

[0121]

[0122]

[0123] MIT Embedding Layer

[0124] In the MIT approach, the input embedding consists of two parts: sequence inversion and static context-aware layer (SA layer).

[0125] A. Sequence Reversal

[0126] In the classic Transformer, the self-attention mechanism learns long-short-term dependencies by calculating the correlation between each vector in the input sequence. Typically, this mechanism partitions the time series along the time dimension, so the input vector at each time step may contain multiple variables with different meanings and importance, which are embedded in the same input channel, leading to dimensionality confusion. Therefore, the classic attention mechanism may not accurately distinguish the relationships between variables when learning variable dependencies. Furthermore, compressing high-dimensional time vectors also results in the loss of feature information. Thus, the classic Transformer cannot effectively capture time series representations and describe multivariate correlations, limiting its generalization ability across different time series data. To address these issues, sequence inversion is introduced to ensure the independence of variables and then utilize the self-attention mechanism to extract dependencies. Specifically, the time series is partitioned along the node dimension to avoid the high complexity caused by processing high-dimensional vectors. This process can be represented as...

[0127]

[0128] Among them, permute(X t [1,3,2]) represents the original tensor X t The second and third dimensions are swapped to achieve the interchange of time and node dimensions.

[0129] The proposed sequence inversion combines self-attention mechanism to extract correlations between different features with SALayer, thereby integrating multidimensional features into temporal dynamic features. Through this design, MIT can efficiently extract temporal features from sequences using feedforward neural networks, and then combine them with spatial features to form spatiotemporal information to support accurate edge load prediction.

[0130] B. Static Content Awareness Layer

[0131] Due to the heterogeneity of different edge nodes, their load data typically possess high-dimensional features. Existing methods often employ static feature encoding, which struggles to adequately represent high-dimensional cross-domain data. Furthermore, excessive static features lead to a dramatic increase in model parameters, severely impacting training and inference efficiency. To fully utilize high-dimensional data and reduce model complexity, a Self-Attention (SA) Layer and a self-attention mechanism are used to extract potential correlations between different features. The SA Layer combines static context with dynamic load sequence data, incorporating high-dimensional cross-domain data (e.g., edge node CPU, memory, and location) as additional static content. Specifically, the SA Layer employs cross-attention, selecting the most effective static content for integration based on the current sequence. Its input includes the load sequence. and static feature data P tFirst, the inverted load sequence is concatenated with a timestamp, serving as a time-series dynamic feature, defined as follows:

[0132]

[0133] Next, the detailed information contained in the SA Layer can be represented as follows:

[0134]

[0135] Among them, softmax -1 Indicates Q t The last dimension is softmaxed (weights are calculated for each static feature of each sequence sample), Linear1 and Linear2 represent different linear layers, and Norm represents layer normalization.

[0136] By integrating load sequences with static features, the SA Layer can learn the static content and inherent properties of edge nodes, effectively filtering redundant information in high-dimensional data. Simultaneously, due to the reduction in the number of static features, the number of channels and parameters in the prediction model also decreases, alleviating the problem of excessive communication overhead in FedGNN.

[0137] MIT coding layer

[0138] At MIT, Transformer-based encoding layers are used to extract spatiotemporal correlations and high-dimensional static features, and include modules such as feedforward neural networks and self-attention layers.

[0139] A. Feedforward Neural Network

[0140] In feedforward neural networks, data propagates forward from the input layer to the output layer. Based on the general approximation theorem, feedforward neural networks can describe time series by extracting complex representations. Recent work on Multilayer Perceptron (MLP) indicates that linearizing the time series helps improve prediction performance. Therefore, feedforward neural networks are used to capture the relationships between time points and discard decoding blocks, which significantly reduces the number of model parameters. Specifically, a feedforward neural network is defined as...

[0141]

[0142] in, W represents the (l-1)th hidden layer state. (l) Let b represent the weight matrix. (l) is the bias vector, and gelu(·) is the activation function.

[0143] B. Self-Attention Layer

[0144] Self-attention layers are used to capture the correlations of multiple variables, fusing high-dimensional static features extracted by the SA layer. Specifically, the self-attention layer transforms each element in the input sequence into a query (Q), key (K), and value (V) vector, and computes the dot product of the queries with all keys. This process is defined as...

[0145]

[0146] Furthermore, the multi-head attention mechanism consists of multiple self-attention layers, focusing on representation subspace information from different locations. Specifically, it employs multiple parallel attention layers and embeds a Transformer encoder, which is defined as...

[0147] H l =MultiHead(Q,K,V)=Concat(head1,...,head n W O (11)

[0148] Where MultiHead(·) represents the multi-head attention mechanism, l={0,…,L} represents the coding layer index, and head n This represents the nth parallel attention layer. It is a parameter matrix.

[0149] PFGformer:Personalized Federated Graph Learning with Transformer

[0150] Edge loads are characterized by high variance, high dimensionality, and spatial correlation, posing significant challenges to accurate and efficient load prediction. Specifically, high variance means that loads can fluctuate dramatically in the short term. High dimensionality means that load data encompasses a large number of features from various edge nodes across different regions. Spatial correlation means that edge loads in different regions exhibit varying degrees of similarity, increasing the difficulty of extracting valuable information. To address these challenges, a novel PFGformer framework is proposed, introducing a collaborative training approach based on personalized federated graph learning in multi-edge environments. Each client (i.e., edge node) runs MIT as its local model, with its input being multidimensional edge load data from its local region. The FL server (i.e., cloud server) aggregates the MIT model parameters (excluding the self-attention matrix parameters and linear layers) using FedAvg, and generates personalized self-attention matrix parameters and spatial features for each client using a hypernetwork and GraphSAGE, which are then distributed to each client. When updating the local model, a λ aggregation mechanism is used to balance the differences between global and local model parameters.

[0151] The goal is for clients to learn well-developed, personalized prediction models tailored to the diverse load patterns and spatial characteristics of their edge environments. Therefore, the proposed PFGformer aims to effectively address data heterogeneity and extract personalized spatial features to improve the performance of local prediction models. Specifically, it utilizes hypernetworks, GraphSAGE, and lambda aggregation mechanisms on the FL server to personalize the training of load prediction models for each client. The objective function of PFGformer is defined as follows:

[0152]

[0153] in, Let represent the loss function for the i-th client.

[0154] PFGformer Hypernetwork

[0155] In PFGformer, the FL server learns a hypernetwork to generate personalized self-attention matrix parameters and linear layer parameters for each client; specifically, it uses the MIT model parameters θ i It is divided into three parts, including W i κ i and ξ i , represent the parameters of the self-attention matrix, the linear layer parameters, and other layer parameters, respectively. Input sequence X t The length of the sequence is denoted as T. After the sequence passes through the embedding layer, the input to the self-attention layer is transformed into an embedding matrix. Therefore, the query, key, and value in the self-attention mechanism are represented as follows: and Among them W i =[W i Q W i K W i V This design allows clients to effectively share parameters while maintaining the personalization of the MIT-based prediction model.

[0156] Hypernet is denoted as Hnet(φ,z) i ), where φ represents the parameters of the hypernet, z i ∈R dmodel z represents the feature vector of client i. i Data can be uploaded from a local client to the FL server or it can learn itself from the FL server. To reduce communication overhead and prevent privacy leaks, the FL server is configured to learn itself. iThis will be automatically generated through the embedding layer. The hypernetwork is implemented using n fully connected layers, where the first n-1 layers are shared by all clients' Transformer encoding blocks, and the last layer is unique for each encoding block. The self-attention matrix parameters and linear layer parameters {W} are learned by introducing the hypernetwork. i ,κ i =Hnet(φ,z) i This allows for the creation of personalized MIT-based prediction models. The model and its loss function are defined as follows:

[0157]

[0158] In each local epoch, the local model parameters are updated using stochastic gradient descent, which is defined as follows:

[0159]

[0160] Among them, B i From D i In the sampled batch, α represents the client learning rate.

[0161] According to the chain rule, φ and z i The gradient can be calculated as

[0162]

[0163] Where C represents the total number of communication rounds, ΔW i =W i J -W i 0 , Δκ i =κ i J -κ i 0 This represents the changes in the self-attention matrix parameters and linear layer parameters after J local training rounds.

[0164] After each round of communication, use gradients. and To update the hypernetwork parameters φ and client embedding z i It is defined as

[0165]

[0166] Where β represents the learning rate of the FL server, and the superscript c represents the c-th communication round.

[0167] Through the above design, clients can effectively share parameters and better utilize the self-attention mechanism. Furthermore, the prediction model maintains good performance even in scenarios with a large number of clients. The generalization bound of PFGformer will be analyzed in detail in Section 4.3.

[0168] PFGformer Graph Topology Generator and GraphSAGE

[0169] The spatial correlation of loads on different edge nodes is not only related to node location, but also closely related to data patterns. Figure 4 (a) shows the load variation patterns of different nodes, where nodes 1 and 2 differ significantly, but nodes 1 and 12 are quite similar. This illustrates the tight coupling relationship between the spatial correlations of the nodes, which affects the accuracy of load prediction, but has been neglected in existing research. To address this issue, Earth Mover's Distance (EMD) is introduced to reconstruct and generate a multi-order adjacency matrix between edge nodes, aiming to generate a personalized graph for each node, thereby more accurately reflecting the spatial relationships between nodes. For any two clients i and i′, their similarity can be calculated as follows:

[0170]

[0171] Next, the similarity of the nodes can be constructed into a similarity matrix S. EMD ∈R K×K , where s ii′ This represents the element in the i-th row and i′ column. A personalized adjacency matrix is ​​generated for each node in this way. Figure 4 (b) visualizes the similarity matrix of 28 nodes, where the correlation between nodes is randomly distributed and not determined by spatial distance. Specifically, in s EMD Several of the most relevant neighbors are selected to aggregate multi-order neighbors. To extract the spatial features of edge nodes, GraphSAGE is introduced to process the adjacency matrix, and the client's feature vector z is used. i As node vectors, GraphSAGE, unlike classic GCNs which require multiple message passes, efficiently handles large-scale graph data because it only needs to perform an aggregation operation on each node once. Furthermore, GraphSAGE supports incremental learning, updating based on new nodes or edges without retraining the entire model. Therefore, when a node joins or leaves, GraphSAGE can predict its location based on its local neighborhood information. Given this, GraphSAGE is used to generate spatial feature vectors for nodes by sampling neighboring nodes and performing aggregation operations, which are then transmitted to the client. The client concatenates the spatial feature vectors with the original feature vectors for subsequent predictions.

[0172] (1) Node k receives the embedding vectors generated by all its direct neighbor nodes N(k) in the previous layer. It is defined as:

[0173]

[0174] Where AGGREGATE(·) represents an aggregate function, This represents the neighbor aggregation information received by node k at layer t. The initial input is the feature vector of the node.

[0175] (2) Aggregate information from adjacent nodes using a join function. The embedding of node k in layer t is... and The series connection is defined as

[0176]

[0177] Here, σ(·) is a nonlinear activation function. CONCAT(·) represents the connection function, which uses MEAN as the aggregator. and After concatenation, an averaging operation is performed on each dimension of the vector, and the result is subjected to a nonlinear transformation to obtain the embedding vector of node k in layer t.

[0178] (3) The embedding output of each node is

[0179] PFGformerλ aggregation mechanism

[0180] Due to significant differences in the distribution of load data, the convergence of client models varies considerably. Influenced by other clients, the update direction of some clients may deviate, a phenomenon known as client drift, which severely impacts model performance and generalization ability. To address this issue, a λ aggregation mechanism is designed for other parameters ξ, replacing the classic approach of using a global model to cover the local model. The proposed λ aggregation mechanism can correct the update directions of the global and local models, overcoming the client drift problem caused by inconsistent data distribution in classic Fourier transforms (FL). Specifically, a hyperparameter λ is introduced as the weight when updating the global and local models, defined as...

[0181]

[0182] in, This represents the remaining parameters of the local model of client i in round c. This represents the remaining parameters of the global model in round c.

[0183] This design allows the update offset of the local model to be corrected back to the ideal convergence direction, thereby improving the prediction performance of the local model across different clients. Therefore, the loss function of PFGformer can be rewritten as...

[0184]

[0185] Method Evaluation

[0186] First, the dataset and experimental setup are introduced. Then, extensive comparative experiments are conducted between the proposed PFGformer method and other benchmark methods.

[0187] Dataset and Experiment Setup

[0188] A real hardware testbed was built to simulate a multi-edge system. The testbed consisted of a workstation and multiple Jetson TX2s. The workstation acted as a cloud server, equipped with two NVIDIA GeForce GTX 3090 GPUs, one Intel(R) Xeon(R) CPU Silver 4208@2.1GHz, and 32GB of memory. The Jetson TX2s acted as edge nodes, each equipped with a 256-core NVIDIA Pascal(R) GPU, a CPU cluster consisting of a dual-core Denver 2 64-bit CPU and a quad-core Arm(R) Cortex(R)-A57 MPCore processor, and 8GB of memory. Both the workstation and the Jetson TX2s ran Ubuntu 18.04, were on the same local area network, and established end-to-end communication based on the FLASK framework.

[0189] To evaluate the performance of the proposed PFGformer, three real-world edge load datasets were used. To better illustrate the load variation patterns of different edge nodes, load data from five randomly selected edge nodes in each of these three datasets were visualized. Figure 5 As shown, for the same dataset, the load change patterns of different edge nodes are mostly different. For different datasets, the load change patterns of different edge nodes are even more significantly different. Therefore, these three datasets can be used to fully test the generalization ability of the model.

[0190] (1) ECW recorded the upload bandwidth changes of 28 edge nodes over 720 hours in August 2022, including 12 static features such as location and number of node CPUs. For this dataset, upload bandwidth is considered as load.

[0191] (2) Alibaba recorded the operating status of 4,000 servers over 8 days, including machine ID, timestamp, CPU utilization, and memory utilization. For this dataset, CPU utilization is considered as load.

[0192] (3) NEP recorded the operating information of edge servers on the NEP platform from June 1 to September 1, 2020, with a sampling frequency of 1 minute. Load data of 6,870 edge servers during June were used, where CPU utilization was regarded as load, and characteristics such as server location and user ID were taken into account.

[0193] The dataset was divided into training, validation, and test sets in a 7:1:2 ratio. The default input and prediction sequence lengths were both 96, and load instances were generated based on these lengths. MIT extracted content from static features and received location information features from edge nodes, which, along with the load data, served as input to the prediction model. The setup included 5 communication rounds, 10 local training rounds, a client sampling rate of 0.8, 3 supernetwork layers, a learning rate of 0.0001 for both the FL server and client, 2 Transformer encoding layers, a batch size of 32, and λ of 0.1. The dataset was resampled to a 1-hour temporal granularity. Based on these experimental settings, the proposed PFGformer was compared with the following state-of-the-art FL frameworks to verify its superiority.

[0194] (1) Centralized training: All client load data are centralized on the FL server to train the prediction model.

[0195] (2) FedAvg: Each client trains its own local prediction model and uploads the model parameters to the FL server for average aggregation.

[0196] (3) FedProx: Based on the FedAvg framework, it allows clients to use different local training rounds and adds a regularization term to the loss function to mitigate the problem of device heterogeneity.

[0197] (4) pFedMe: Introduces Moreau Envelopes optimization based on client loss function, allowing the client to update the local model in different directions without deviating from the global model, thereby learning a personalized model.

[0198] (5) FedPer: By combining personalized model aggregation and parameter update strategies, it improves the generalization ability of personalized models while protecting user privacy.

[0199] (6) Vanilla Personlized-T: Replaces the parameters of the self-attention matrix after federated aggregation with the parameters of the self-attention matrix trained locally.

[0200] (7) FedSage+: Introduces a linear predictor on the client side to fix missing connections in the cross subgraph.

[0201] (8) FedGCN: Divides the entire network into multiple sub-networks through community detection and combines FL and GCN on a sub-network basis.

[0202] At each edge node, the superiority of the proposed MIT is demonstrated by comparison with the following state-of-the-art Transformer frameworks.

[0203] (1) Crossformer: Uses a two-stage attention mechanism and a hierarchical encoder-decoder architecture to capture the correlation between multivariate variables.

[0204] (2) Pyraformer: Introduces a pyramid-based attention mechanism to balance the acquisition of long-term dependencies and low time and space complexity.

[0205] (3) Informer: Combines the ProbSpar self-attention mechanism, multi-head self-attention mechanism and knowledge distillation, and enhances the ability to model time features through time factor decomposition.

[0206] (4) Flashformer: Employs the Flash attention mechanism to enhance the scope and speed of perceiving context.

[0207] (5) Flowformer: Based on the network flow conservation theorem, a Flow attention mechanism was designed to reduce time complexity.

[0208] (6) Transformer: Uses a self-attention mechanism to process the input sequence in parallel and uses the relationship between different positions in the sequence to capture long-distance dependencies.

[0209] Experimental Results and Analysis

[0210] First, the prediction performance of different prediction models under different training frameworks was evaluated. As shown in Table 1, in most cases, the proposed MIT method outperforms other methods in prediction accuracy under different training frameworks, demonstrating its superior load prediction performance in complex and variable edge environments. This is because the MIT method efficiently extracts static features of the load by combining sequence inversion and SA layer. This design improves the self-attention mechanism in the classic Transformer, making it applicable to load prediction tasks in complex edge environments. At the same time, MIT uses a feedforward neural network to help capture the long-short-term dependencies of the temporal sequence, and the results also verify the effectiveness of this design in improving the accuracy of multivariate temporal prediction. Existing improved versions of Transformer usually embed multiple variables at the same timestamp into the same channel, which leads to the model over-focusing on local perception and ignoring the potential correlation between temporal features. In addition, they usually treat node positions as static features and do not fully consider the spatial information of different nodes. Therefore, they are difficult to effectively adapt to multivariate temporal edge load prediction tasks. Specifically, Crossformer focuses on the correlation between multiple variables in the data and proposes Dimension-Segment-Wise (DSW) embedding and cross-time step attention, which has good performance. However, it exhibits high computational complexity when processing high-dimensional data. Furthermore, Crossformer's inability to capture changing features in a timely manner when facing dynamic edge loads affects its accuracy. Pyraformer may fail to update model parameters quickly when handling dynamic loads, resulting in slow response. Additionally, in multi-edge environments, high-dimensional data often contains a large amount of redundant information, and Pyraformer cannot effectively filter out key features, leading to overfitting. Informer's self-attention distillation mechanism can reduce feature dimensionality and memory consumption, but using convolutional layers and max pooling for feature distillation leads to feature information loss and accuracy degradation. Flashformer's FlashAttention divides the input into blocks and performs attention mechanisms separately on each block, limiting its ability to capture long dependencies and contextual information throughout the time series.

[0211] Table 1 compares the proposed MIT and PFGformer with other state-of-the-art frameworks on different datasets.

[0212]

[0213]

[0214]

[0215] Furthermore, the proposed PFGformer was compared with other state-of-the-art (SOTA) training frameworks. As shown in Table 1, PFGformer exhibits superior performance across different datasets. This is because PFGformer utilizes the self-attention matrix parameters and linear layer parameters that are important in supernet learning of Transformers, and introduces a λ aggregation mechanism. This design preserves sufficient individuality for the Transformer prediction model for each client, avoiding excessive influence from other client models with low correlation. Simultaneously, PFGformer introduces a personalized adjacency matrix and uses GraphSAGE to generate personalized spatial feature vectors for each client. This design fully leverages the data correlations between edge nodes, enabling each edge node to fully utilize personalized information, thereby generating a more generalized personalized prediction model. The accuracy of centralized training frameworks is lower than that of PFGformer. This is because centralized training cannot capture the differences between edge nodes and cannot effectively utilize the correlation of their load patterns. Moreover, the performance of centralized training frameworks outperforms FedProx, pFedMe, and FedPer. This is because the data from all edge nodes is centralized on a cloud server for model training. Furthermore, classic FL (Flexible Transformer) frameworks struggle to directly adapt to Transformer-based prediction models, and variations in data distribution pose a significant challenge to generating high-quality, personalized Transformers. While Vanilla Personlized-T incorporates some personalized designs for the Transformer, experimental results show that its improvement in prediction performance remains limited. Compared to other frameworks, the proposed PFGformer can generate superior personalized Transformer models for each client, further enhancing the accuracy of edge load prediction.

[0216] Comparison between the PFGformer framework and the FedGNN-based framework

[0217] The proposed PFGformer treats the GNN as a spatial feature extractor and uses a Transformer for load prediction. To further validate the superiority of this design, MIT was used as the backbone network, and PFGformer was compared with two other state-of-the-art FedGNN-based frameworks on the ECW and Alibaba datasets. Figure 7 As shown, PFGformer consistently maintains excellent data fitting capabilities as the prediction length increases. In contrast, other FedGNN-based frameworks perform poorly when facing high-dimensional, highly heterogeneous, and high-variance marginal loads. Figure 7(c) When the prediction length is 96, FedSage+ and FedGNN exhibit poor prediction performance. This is because node connections in edge environments are typically sparse, and GCN relies on connection information between neighboring nodes for feature fusion and propagation, failing to effectively handle sparse graph structures and model temporal correlations. Furthermore, GCN's feature fusion based on node connections is weak for modeling diverse edge node features. While FedSage+ can utilize GraphSAGE to fill in some missing connections between edge nodes, it still cannot fully utilize the highly correlated features of distant edge nodes, leading to a decline in prediction performance. FedGCN uses a traditional GCN as the predictor, making better use of data correlations between nodes. FedGCN performs well in short-term prediction scenarios, but its performance cannot be guaranteed in long-term prediction scenarios. In contrast, PFGformer constructs a personalized adjacency matrix through EMD and extracts spatial features using GraphSAGE, fully utilizing the spatial correlations between edge nodes. Meanwhile, PFGformer uses MIT as the predictor, which can effectively capture temporal dependencies and temporal changes to better adapt to high-latitude, highly dynamic edge environments.

[0218] Parameter sensitivity

[0219] To better balance the differences between the global and local models and overcome the heterogeneity of data and devices among different edge nodes, a λ aggregation mechanism was introduced, and the impact of different λ values ​​on the performance of PFGformer was analyzed. Figure 7 As shown, the performance of different prediction models was tested within the PFGformer framework. The results indicate that the best load prediction accuracy is achieved when λ is 0.1. Excessively large λ values ​​can cause personalized models to overemphasize local data features while ignoring valuable information from other clients. For example, when λ = 1, PFGformer performs the worst among the three models except for MIT. PFGformer is designed to facilitate reasonable collaboration among client models; therefore, smaller λ values ​​are beneficial for fully utilizing other clients and addressing issues such as insufficient local data scale and monotonous data patterns. It is worth noting that when λ = 0, the personalization of the local model relies entirely on the aggregation results, causing the local model to excessively forget its own features.

[0220] Communication and training overhead

[0221] To better adapt PFGformer to edge environments with limited communication and computational resources, SALayer was first used at MIT to compress static features and reduce the number of model channels. Next, linear layers were used to replace traditional Transformer encoding blocks to reduce the number of model parameters. Finally, an MLP was used to generate node feature vectors, which were then used by the hypernetwork and GraphSAGE to generate self-attention matrix parameters and spatial feature vectors, reducing communication overhead. Figure 8 In the diagram, the bars and lines represent the communication and training overhead of different frameworks, respectively. For example... Figure 8 As shown in (a), the proposed MIT significantly reduces communication overhead compared to other Transformer-based prediction models. Furthermore, since MIT only includes encoding and linear mapping layers, it greatly shortens training time. In contrast, other prediction models incur higher communication and training overhead. Specifically, Crossformer emphasizes multi-scale feature extraction and has a complex network structure, leading to excessive training overhead when dealing with high-dimensional dynamic data. Flashformer has the highest training overhead because it uses additive absolute, additive relative, and RoPE positional encodings. In addition, the communication and training overhead of the proposed PFGformer is compared with other customized FL frameworks. Figure 8 As shown in (b), PFGformer maintains high accuracy without increasing communication overhead and offers faster training speed. This is because while PFGformer requires the transmission of spatial feature vectors, it does not need to upload attention matrix parameters and linear layer parameters, thus preventing an increase in communication overhead. In contrast, each client in pFedMe needs to update the loss function multiple times to personalize the local model, resulting in higher training overhead.

[0222] ablation experiment

[0223] In the proposed PFGformer, a hypernetwork, SA layer, and λ aggregation mechanism are first designed in FL to generate a personalized Transformer. Then, a graph topology generator and GraphSAGE are combined to effectively utilize the positional relationships of edge nodes. The effectiveness of the components designed in PFGformer is verified through ablation experiments. As shown in Table 2, the prediction accuracy is improved to some extent after introducing the three components. Specifically, for the Alibaba dataset, GraphSAGE significantly affects the prediction accuracy. This is because the Alibaba dataset has many nodes with the same data patterns that can be utilized. Meanwhile, Hypernetwork and SA layer both significantly contribute to improving prediction accuracy in different datasets. This indicates that node location and static features have a significant impact on the edge load change trend, and also proves the rationale for using a hypernetwork to update the local MIT parameters. In some cases, introducing SA layer into Crossformer leads to a decrease in accuracy. This is because Crossformer uses multi-scale and cross-modal representations to capture complex time-series features; therefore, introducing SA layer affects its capture of feature relationships, thus leading to a decrease in accuracy.

[0224] Table 2 Ablation experiments of PFGformer ("+": used, "-": not used).

[0225]

[0226]

[0227] Advantages and uses of this product

[0228] This invention first proposes a novel MIT (Match Targeting) mechanism to efficiently capture important representations of edge load changes, node spatial information, and the influence of static features. Next, a graph topology generator is designed to generate multi-order adjacency matrices for edge nodes via EMD (Earth Targeting Method), creating a personalized graph topology for each node to extract spatial feature information. Then, a hypernetwork and λ-aggregation mechanism are introduced into PFG (Physical Transformer Group) to generate personalized prediction model parameters, and GraphSAGE is used to extract node spatial features. Finally, by combining MIT and PFG, an innovative PFGformer is designed to efficiently generate powerful personalized Transformer models for each client. Specifically, the excellent generalization ability and convergence of PFGformer are rigorously proven through derivation. Extensive experiments based on multiple real-world edge load datasets and testbeds validate the superiority of the proposed MIT and PFGformer. Results show that compared to state-of-the-art frameworks, MIT and PFGformer achieve higher prediction accuracy in different prediction scenarios. Simultaneously, PFGformer exhibits lower communication and training overhead. Furthermore, ablation experiments verify the effectiveness of each component in PFGformer.

[0229] Product usage process or method

[0230] (1) The PFGformer server randomly selects a certain proportion of clients, uses the hypernetwork to learn the self-attention matrix parameters and linear layer parameters of the clients, and generates spatial feature vectors.

[0231] (2) The PFGformer server distributes the self-attention matrix parameters, linear layer parameters and other parameters, as well as the spatial feature vector, to the client for parameter updates.

[0232] (3) The PFGformer client uploads the model parameters trained on local data to the server.

[0233] (4) The PFGformer server accepts the model parameters from the client to update the parameters of the hypernet.

[0234] The client's load prediction results are sent to the edge server so that the edge computing service provider can adjust its resource management strategy.

Claims

1. A spatiotemporal load prediction method based on personalized federated graph learning Transformer in multi-edge systems, characterized in that, Includes the following steps: Step 1: Construct a multivariate sequence inversion Transformer, MIT, as a local load prediction model for edge nodes; Step 2: Design PFG, and generate personalized model parameters and spatial features for each edge node through hypernetwork and graph neural network (GNN); The parameters are updated and sent to the FL server to calculate the parameter changes of the prediction model. The FL server updates the remaining parameters of the global model, the hypernetwork parameters, and the node feature vectors according to the parameters returned by the client. Step 3: Combine MIT and PFG to form PFGformer, and generate personalized load predictions for each edge node; The MIT consists of an embedding layer, an encoder layer, and an output layer; The embedding layer consists of a sequence inversion layer and a static content awareness layer; the sequence inversion specifically divides the time series along the node dimension, represented as follows: Among them, permute(X t [1,3,2]) represents the original tensor X t The second and third dimensions are swapped to achieve the interchange of time and node dimensions; The proposed sequence inversion combines the self-attention mechanism to extract the correlation between different features with the SA layer, thereby integrating multi-dimensional features into temporal dynamic features; MIT uses a feedforward neural network to efficiently extract the temporal features in the sequence, and then combines them with spatial features to form spatiotemporal information to support accurate edge load prediction. The static content-aware layer uses a static context layer (SA Layer) and a self-attention mechanism to extract potential correlations between different features. The SA Layer combines static context with dynamic load sequence data, and incorporates high-dimensional cross-domain data as additional static content. The SA Layer employs cross-attention to select the most effective static content for integration based on the current sequence; its input includes the load sequence. and static feature data P t First, the inverted load sequence is concatenated with the timestamp as a time-series dynamic feature, defined as follows: Next, the detailed information contained in the SA Layer is represented as follows: α t =softmax -1 (Q t ) V=Linaer2(P t ) Among them, softmax -1 Indicates Q t The last dimension is softmaxed, Linear1 and Linear2 represent different linear layers, and Norm represents layer normalization; By integrating load sequences and static features, the SA Layer learns the static content and inherent attributes of edge nodes, filtering out redundant information in high-dimensional data; at the same time, the number of channels and parameters of the prediction model will also be reduced. The encoding layer is used to extract spatiotemporal correlations and high-dimensional static features. The encoding layer includes a feedforward neural network and a self-attention layer module. The feedforward neural network is used to capture the relationships between time points and discard decoded blocks. The feedforward neural network is defined as follows: in, W represents the (l-1)th hidden layer state. (l) Let b represent the weight matrix. (l) It is the bias vector, and gelu(·) is the activation function; The self-attention layer is used to capture the correlation of multiple variables, fusing high-dimensional static features extracted by the SA layer; specifically, the self-attention layer transforms each element in the input sequence into a vector of query Q, key K, and value V, and computes the dot product of the queries with all keys; the process is defined as follows: Furthermore, the multi-head attention mechanism consists of multiple self-attention layers, focusing on representation subspace information from different locations; specifically, it employs multiple parallel attention layers and embeds a Transformer encoder, which is defined as... H l =MultiHead(Q,K,V)=Concat(head1,...,head n )W O (11) Where MultiHead(·) represents the multi-head attention mechanism, l={0,…,L} represents the coding layer index, and head n This represents the nth parallel attention layer. It is a parameter matrix; The objective function of PFGformer is defined as follows: Among them, l i Let θ represent the loss function for the i-th client; i D is the local model parameter of the i-th client. i Let be the number of samples from the i-th client, and D be the total number of samples from all clients. In PFGformer, the FL server will learn a supernetwork to generate personalized self-attention matrix parameters and linear layer parameters for each client; The model parameters θ of MIT i Split into W i κ i and ξ i , represent the parameters of the self-attention matrix, the parameters of the linear layer, and the parameters of other layers, respectively; input sequence X t The length of the sequence is denoted as T. After the sequence passes through the embedding layer, the input to the self-attention layer is transformed into an embedding matrix. The query, key, and value in a self-attention mechanism are represented as follows: and Among them W i =[W i Q W i K W i V ]; Hypernet is denoted as Hnet(φ,z) i ), where φ represents the parameters of the hypernet, z i ∈R dmodel Let z represent the feature vector of client i; let the FL server learn z itself. i It will be automatically generated through the Embedding layer; the hypernetwork is implemented using n fully connected layers, where the first n-1 layers are shared by all client Transformer encoding blocks, and the last layer is unique for each encoding block; the self-attention matrix parameters and linear layer parameters {W} are learned by introducing the hypernetwork. i ,κ i =Hnet(φ,z) i This enables personalized MIT-based prediction models; the prediction model and its loss function are defined as follows: In each local epoch, the local model parameters are updated using stochastic gradient descent, which is defined as follows: Among them, B i From D i A batch is sampled in the middle, and α represents the client learning rate; According to the chain rule, φ and z i The gradient is calculated as Where C represents the total number of communication rounds, ΔW i =W i J -W i 0 , Δκ i =κ i J -κ i 0 This represents the changes in the parameters of the self-attention matrix and the linear layer after J local training rounds; After each round of communication, use gradients. and To update the hypernetwork parameters φ and client embedding z i It is defined as Where β represents the learning rate of the FL server, and the superscript c represents the c-th communication round.

2. The spatiotemporal load prediction method based on personalized federated graph learning Transformer in a multi-edge system according to claim 1, characterized in that, Graph Neural Networks (GNNs) are used for feature embedding and inference on graph data. Their input consists of a graph of nodes and edges, represented as an adjacency matrix A∈R. K×K ; and the node feature matrix X∈R K×dmodel Where K represents the number of nodes and dmodel represents the number of node features; the graph neural network (GNN) updates the node embedding based on neighboring node information, which is defined as: S (l+1) =GNN(A,S (l) ,w (l) ), (1) Where GNN(·) represents the graph pooling function, which is an average, weighted average, or max / min pooling function; S (l) w represents the node embedding at layer l. (l) This represents the trainable weights of the l-th layer.

3. The spatiotemporal load prediction method based on personalized federated graph learning Transformer in a multi-edge system according to claim 1, characterized in that, The aggregation process of the FL server is defined as follows: Where, θ i θ is the local model parameter of the i-th client. c+1 These are the global model parameters for the next iteration, where K is the number of participating clients, and D... i Let be the number of samples from the i-th client, and D be the total number of samples from all clients. This indicates the degree of contribution each client makes to the global model.

4. The spatiotemporal load prediction method based on personalized federated graph learning Transformer in a multi-edge system according to claim 1, characterized in that, Edge load includes dynamic and static characteristics such as date, node location, and disk capacity; under a fixed prediction window, multiple load sequences on the same edge node are integrated into a historical load input, which is defined as follows: Where Γ represents the last time index observed in the historical load data, and d t It is a dynamic feature dimension, d t >1; Given a set of load data inputs, the prediction process is represented as follows: in, This represents the predicted load sequence, where L is the prediction length and f is the prediction model. Denotes static features, where d s d represents the static feature dimension s >1; To evaluate the accuracy of edge load prediction, MSE and MAE are considered performance metrics, defined as follows: Among them, y i This represents the actual load within a certain interval, where S is the number of load sequences.

5. The spatiotemporal load prediction method based on personalized federated graph learning Transformer in a multi-edge system according to claim 1, characterized in that, EMD is introduced to reconstruct and generate multi-order adjacency matrices between edge nodes. For any two clients i and i′, their similarity is calculated as follows: Next, the similarity of the nodes is constructed into a similarity matrix S. EMD ∈R K×K , where s ii′ This represents the element in the i-th row and i'-th column; This method generates a personalized adjacency matrix for each node; in s EMD We select several of the most relevant neighbors to aggregate multi-order neighbors; we introduce GraphSAGE to process the adjacency matrix and use the client's feature vector z. i As a node vector, GraphSAGE is used to perform aggregation operations by sampling neighboring nodes, thereby generating spatial feature vectors for the nodes and transmitting them to the client; the client then concatenates the spatial feature vectors with the original feature vectors for subsequent predictions. (1) Node k receives the embedding vectors generated by all its direct neighbor nodes N(k) in the previous layer. It is defined as: Where AGGREGATE(·) represents an aggregate function, This represents the neighbor aggregation information received by node k at layer t; The initial input is the feature vector of the node. (2) Aggregate information from adjacent nodes through a join function; the embedding of node k in layer t is... and The series connection is defined as Where σ(·) is a nonlinear activation function; CONCAT(·) represents the connection function, which uses MEAN as the aggregator; and After concatenation, an averaging operation is performed on each dimension of the vector, and the result is subjected to a nonlinear transformation to obtain the embedding vector of node k in layer t. (3) The embedding output of each node is 6. The spatiotemporal load prediction method based on personalized federated graph learning Transformer in a multi-edge system according to claim 1, characterized in that, The design incorporates a λ aggregation mechanism, specifically by introducing a hyperparameter λ as the weights for updating the global and local models. λ is defined as follows: in, This represents the remaining parameters of the local model of client i in round c. This represents the remaining parameters of the global model in round c; The update offset of the local model is corrected back to the ideal convergence direction, improving the prediction performance of the local model on different clients; the loss function of PFGformer is rewritten as... l i (W i ,k i ,ξ)=l i (Hnet(φ,z i ),(1-λ)ξ g +lx i ) (21).

Images