Personalized federated learning method based on graph convolutional hypernetwork

Abstract

The present invention discloses a personalized federated learning method based on a graph convolutional hypernetwork: each client initializes local model parameters and divides the model into training sets and test sets; each client uses its training set to update its local model N times, and uploads the local gradient vector to a server; the server calculates the cosine distance of all gradient vectors and constructs a client correlation matrix; the server performs principal component analysis and dimensionality reduction on all gradient vectors to serve as the embedding vector of each client; the server initializes the graph convolutional hypernetwork; the correlation matrix and the embedding vector are input into the initialized graph convolutional hypernetwork to generate a local model for each client, which is sent to the client, and the loss value and gradient are calculated. The gradient is transmitted to the server, and the gradient of the client's local loss to the graph convolutional hypernetwork parameters is calculated, and the graph convolutional hypernetwork parameters are updated; the embedding vector and the correlation matrix are input into the trained graph convolutional hypernetwork to generate a personalized local model for each client.

Description

Technical Field

[0001] The present invention relates to the field of federated learning, and more specifically, to a personalized federated learning method based on graph convolutional hypernetwork. Background Art

[0002] Federated learning is a distributed machine learning framework with privacy protection. Its core concept is to collaboratively train machine learning models between multiple clients with local data. By exchanging model parameters without exchanging local data, a global model based on global data distribution is constructed, thus achieving a balance between privacy protection and shared computing. With the development of the Internet of Things and the widespread use of IoT devices, federated learning has also evolved. However, because devices belong to individual users, enterprises, and scenarios, their data distribution varies greatly, meaning they are not independent and identically distributed (IID). This makes it difficult to collaboratively train a global model that generalizes well across all data sources. Summary of the Invention

[0003] To overcome the shortcomings of existing technologies, this paper proposes a personalized federated learning method based on a graph convolutional hypernetwork. Rather than using a single global model to meet the needs of all clients, this method learns an optimal, personalized local model for each client. Furthermore, this method leverages the topological relationships between client data distributions, enabling clients with similar data distributions to collaborate and further improve the performance of personalized local models.

[0004] The objectives of the present invention are achieved through the following technical solutions.

[0005] The personalized federated learning method based on graph convolutional hypernetwork of the present invention includes the following steps:

[0006] Step S101: Each client initializes local model parameters and divides its local data set into a training set and a test set;

[0007] Step S102: Each client updates its local model N times using its training set through the stochastic gradient descent algorithm, and uploads the local gradient vectors accumulated during the update to the server.

[0008] Step S103: The server flattens the received client model gradient vectors, calculates the cosine distances between all gradient vectors, and constructs a client correlation matrix;

[0009] Step S104: The server performs principal component analysis on the gradient vectors of all received client models and reduces their dimensionality, and uses the reduced dimensionality vectors as the embedding vectors of each client;

[0010] Step S105: The server initializes a graph convolutional hypernetwork, whose input dimension is the dimension of the client embedding vector and whose output dimension is the dimension of the client local model;

[0011] Step S106: Input the correlation matrix obtained in step S103 and the client embedding vector obtained in step S104 into the graph convolutional hypernetwork initialized in step S105 to generate local models for each client. These models are then sent to the client, and the loss value and gradient are calculated locally on the client. The gradient is transmitted to the server, and the gradient of the client's local loss with respect to the graph convolutional hypernetwork parameters is calculated by the chain rule. The graph convolutional hypernetwork parameters are continuously updated through the backpropagation algorithm until convergence;

[0012] Step S107: Input the client's embedding vector and correlation matrix into the graph convolutional hypernetwork trained in step S106 to generate a local model personalized for each client.

[0013] In step S101 , each client adopts a different neural network model according to the different formats of data collected by each client.

[0014] The updating process of the local models of n clients in step S102 is as follows:

[0015]

[0016] Among them, w k is the local model of the kth client, w′ k is the updated local model of the k-th client, is the gradient of the local model of the kth client calculated on its local data, and η is the update step size. This process is performed N times on the client. In addition, the gradient vector obtained by updating the local model of the kth client to be uploaded in step S102 is the difference between the two local models before and after the update:

[0017] h k =w′ k -w k

[0018] The client correlation matrix in step S103 is:

[0019] A=(A ij ) n×n ,

[0020] Among them, A ij =1-cos <h i ,g j >, h i The gradient vector obtained by updating the local model of the i-th client, h jThe gradient vector obtained by updating the local model of the jth client is obtained; in addition, the values of the elements in the client correlation matrix A that are greater than the set threshold are set to 0, and the remaining values are set to 1, thereby converting the client correlation matrix A into a 0-1 adjacency matrix.

[0021] The embedding vector obtained after dimensionality reduction in step S104 is as follows:

[0022]

[0023] in, It is the result of PCA dimensionality reduction of the client gradient vector.

[0024] The graph convolutional super network structure in step S105 includes:

[0025] (1) Graph Convolutional Neural Network: Obtain client-side fusion features through graph convolution operations;

[0026] (2) Feedforward neural network: Each client shares the feedforward neural network model, and the features of each client are integrated to generate its local personalized model;

[0027] In addition, the input of the graph convolutional hypernetwork is the embedding vector and the correlation matrix, and the output is the client local model parameters; its forward propagation process is as follows:

[0028]

[0029] Among them, φ is the parameter of graph convolution, are the parameters of the feedforward neural network, Represents the forward propagation process of graph convolution after T times, V (T) represents the output of the T-th layer graph convolutional network, H represents the feedforward neural network, The local models of n clients are generated by the graph convolutional hypernetwork. GHN stands for graph convolutional hypernetwork.

[0030] The gradient of the graph convolutional hypernetwork parameters in step S106 is calculated by the chain rule as follows:

[0031]

[0032]

[0033] in, The training loss of the k-th client is the graph convolutional hypernetwork parameter. The gradients of φ, which can be obtained by the chain rule, Represents the k-th client training set loss weight w k The gradient, Represents the output of the feedforward neural network, i.e. w kFeedforward neural network parameters The gradient, Indicates w k Output V of the graph convolutional network (T) The gradient, Represents the graph convolutional network output V (T) Gradient of the graph convolutional network parameters φ. After multiple rounds of stochastic gradient descent, the graph convolutional hypernetwork is trained.

[0034] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0035] (1) Compared with the traditional federated averaging algorithm, the present invention learns the optimal personalized model for each client instead of learning a shared global model, which is more applicable in the case of non-independent and identically distributed (IID) distribution.

[0036] (2) The present invention uses a graph convolutional hypernetwork to enable clients with close distribution to jointly learn, significantly improving the effect of personalized local models.

[0037] (3) The present invention uses a graph convolutional hypernetwork to learn the optimal aggregation method for the clients participating in the federation, significantly improving the performance of federated learning in scenarios where user data is not independent and identically distributed.

[0038] This paper leverages graph convolutional neural networks (GCNNs) to fully exploit the topological relationships among clients—that is, the relationships between their local data distributions—to enable collaborative learning among closely distributed clients. Furthermore, this paper abandons the traditional weighted average aggregation method of federated learning and instead uses GCNNs to learn the optimal aggregation method and parameters for federated learning participants, significantly improving the effectiveness of federated learning in scenarios where user data is not independent and identically distributed (IID). BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a flow chart of the personalized federated learning method based on graph convolutional hypernetwork of the present invention;

[0040] Figure 2 Schematic diagram of the graph convolutional hypernetwork training process of the personalized federated learning method based on the graph convolutional hypernetwork of the present invention. DETAILED DESCRIPTION

[0041] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0042] like Figure 1As shown in FIG, the personalized federated learning method based on the graph convolutional hypernetwork of the present invention includes the following processes:

[0043] Step S101: Each client initializes local model parameters and divides its local data set into a training set and a test set.

[0044] Client datasets are often non-IID (independent and identically distributed), meaning that data distributions vary across clients. Data collection typically occurs locally on IoT devices. Depending on the data format collected by each client, different convolutional neural network models can be used. For example, for image data, each client initializes a local convolutional neural network model. The local models and tasks for each client are consistent, e.g., the local data is always image data, and the local model is always a convolutional neural network model.

[0045] Step S102: Each client uses its training set to update its local model through the stochastic gradient descent algorithm for N rounds of training, and uploads the local gradient vectors accumulated during the update to the server.

[0046] The update process of n client local models is as follows:

[0047]

[0048] Among them, w k is the local model of the kth client, w′ k is the updated local model of the k-th client, is the gradient of the local model of the kth client calculated on its local data, and η is the update step size. This process is repeated N times on the client. In addition, the gradient vector obtained by updating the local model of the kth client to be uploaded in step S102 is the difference between the two local models before and after the update:

[0049] h k =w′ k -w k (2)

[0050] Step S103: The server flattens the received client model gradient vectors, calculates the cosine distances between all gradient vectors, and constructs a client correlation matrix.

[0051] The client correlation matrix is:

[0052] A=(A ij ) n×n (3)

[0053] Among them, A ij =1-cos <h i ,h j>, h i The gradient vector obtained by updating the local model of the i-th client, h j The gradient vector obtained by updating the local model of the jth client is obtained; in addition, the values of the elements in the client correlation matrix A that are greater than the set threshold are set to 0, and the remaining values are set to 1, thereby converting the client correlation matrix A into a 0-1 adjacency matrix.

[0054] Step S104: The server performs principal component analysis and dimensionality reduction on the gradient vectors of all received client models, and uses the reduced dimensionality vectors as the embedding vector of each client.

[0055] The embedding vector obtained after dimensionality reduction is as follows:

[0056]

[0057] in, It is the result of PCA dimensionality reduction of the client gradient vector.

[0058] Step S105: The server initializes a graph convolutional hypernetwork, whose input dimension is the dimension of the client embedding vector and whose output dimension is the dimension of the client local model.

[0059] Among them, the graph convolutional hypernetwork structure includes:

[0060] (1) Graph Convolutional Neural Network: Client fusion features are obtained through graph convolution operations; graph convolution fusion methods include averaging, maximum pooling, LSTM and other methods.

[0061] (2) Feedforward neural network: including fully connected layers, nonlinear activation layers, and dropout layers; each client shares the feedforward neural network model, and generates a local personalized model for each client by integrating features.

[0062] In addition, the input of the graph convolutional hypernetwork is the embedding vector and the correlation matrix, and the output is the client local model parameters. Its forward propagation process is as follows:

[0063]

[0064] Among them, φ is the parameter of graph convolution, are the parameters of the feedforward neural network, Represents the forward propagation process of graph convolution after T times, V (T) represents the output of the T-th layer graph convolutional network, H represents the feedforward neural network, The local models of n clients are generated by the graph convolutional hypernetwork. GHN stands for graph convolutional hypernetwork.

[0065] Step S106: Input the correlation matrix obtained in step S103 and the client embedding vector obtained in step S104 into the graph convolutional hypernetwork initialized in step S105 to generate local models for each client. These models are then sent to the client. The loss value and gradient are calculated locally on the client, and the gradient is transmitted to the server. The gradient of the client local loss to the graph convolutional hypernetwork parameters is calculated by the chain rule. The graph convolutional hypernetwork parameters are continuously updated through the backpropagation algorithm until convergence, that is, the model is saved when the preset stop condition is met. The graph convolutional hypernetwork parameter training process is as follows: Figure 2 shown.

[0066] The gradient of the graph convolutional hypernetwork parameters is calculated by the chain rule as follows:

[0067]

[0068]

[0069] in, The training loss of the k-th client is the graph convolutional hypernetwork parameter. The gradients of φ, which can be obtained by the chain rule, Represents the k-th client training set loss weight w k The gradient, Represents the output of the feedforward neural network, i.e. w k Feedforward neural network parameters The gradient, Indicates w k Output V of the graph convolutional network (T) The gradient, Represents the graph convolutional network output V (T) Gradient of the graph convolutional network parameters φ. After multiple rounds of stochastic gradient descent, the graph convolutional hypernetwork is trained.

[0070] Step S107: Input the client's embedding vector and correlation matrix into the graph convolutional hypernetwork trained in step S106, and generate a local model personalized for each client through forward propagation.

[0071] Through the above process, the present invention can conduct collaborative learning among clients with closely distributed private data, and learn the optimal aggregation method through the graph convolutional hypernetwork, significantly improving the performance of federated learning in non-independent and identically distributed scenarios.

[0072] Example 1

[0073] This embodiment of the personalized federated learning method based on the graph convolutional hypernetwork includes the following steps:

[0074] Step S101: Each client initializes local model parameters and divides the dataset into training and test sets in a 7:3 ratio. In this example, a convolutional neural network model is selected as the local model, and the image classification task on the CIFAR-10 dataset is selected as the client's local task. Because data distribution varies across clients in reality, this example sets each client to have data from only two random classes.

[0075] Step S102: Each client uses its training set to update its local model using the stochastic gradient descent algorithm for N rounds, and uploads the local gradient vectors accumulated during the update to the server. The update process of the local models of n clients is shown in formula (1).

[0076] Step S103: The server flattens the received client model gradient vectors, calculates the cosine distances between all gradient vectors, and constructs the client correlation matrix A according to formula (3).

[0077] Step S104: The server performs principal component analysis on the gradient vectors of all received client models and reduces the dimension of the vectors after dimension reduction. as the embedding vector for each client.

[0078] Step S105: The server initializes a graph convolutional hypernetwork, whose input dimension is the dimension of the client embedding vector and whose output dimension is the dimension size of the client local model.

[0079] Step S106: Input the correlation matrix obtained in step S103 and the client embedding vector obtained in step S104 into the graph convolutional hypernetwork initialized in step S105 to generate local models for each client. These models are then sent to the client, and the loss value and gradient are calculated locally on the client. The gradient is transmitted to the server, and the chain rule is used to calculate the relationship between the client local loss and the graph convolutional hypernetwork parameters. The gradient of φ is used to continuously update the graph convolutional hypernetwork parameters through the back-propagation algorithm until convergence.

[0080]

[0081]

[0082] Among them, α and β represent the two parts of the graph convolutional hypernetwork, namely the training steps of the feedforward neural network and the graph convolutional network, respectively. They represent the parameters of the feedforward neural network and graph convolutional network in the tth round respectively.

[0083] Step S107: Input the client's embedding vector and correlation matrix into the graph convolutional hypernetwork trained in step S106, and generate a local model personalized for each client through forward propagation.

[0084]

[0085] Among them, φ* and The network parameters of the graph convolutional hypernetwork after training convergence.

[0086] This example tests performance in scenarios where 10, 50, and 100 clients participate in federation. The experimental results are shown in the following table. The results reported in the table are the average test accuracy across all clients:

[0087] Table 1

[0088]

[0089]

[0090] Although the functions and working processes of the present invention have been described above in conjunction with the accompanying drawings, the present invention is not limited to the above-mentioned specific functions and working processes. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which are all protected by the present invention.

Claims

1. A personalized federated learning method based on graph convolutional hypernetwork, characterized by: The following processes are included: Step S101: Each client initializes local model parameters and divides its local data set into a training set and a test set; the local data is image data; Step S102: Each client updates its local model N times using its training set through the stochastic gradient descent algorithm, and uploads the local gradient vectors accumulated during the update to the server. Step S103: The server flattens the received client model gradient vectors, calculates the cosine distances between all gradient vectors, and constructs a client correlation matrix; Step S104: The server performs principal component analysis on the gradient vectors of all received client models and reduces their dimensionality, and uses the reduced dimensionality vectors as the embedding vectors of each client; Step S105: The server initializes a graph convolutional hypernetwork, whose input dimension is the dimension of the client embedding vector and whose output dimension is the dimension of the client local model; The graph convolutional hypernetwork structure includes: (1) Graph Convolutional Neural Network: Obtain client fusion features through graph convolution operations; (2) Feedforward neural network: Each client shares the feedforward neural network model, and the features of each client are integrated to generate its local personalized model; In addition, the input of the graph convolutional hypernetwork is the embedding vector and the correlation matrix, and the output is the client local model parameters; its forward propagation process is as follows: ， in, is the parameter of graph convolution, are the parameters of the feedforward neural network, Represents the forward propagation process of graph convolution after T times, represents the output of the T-th layer graph convolutional network, H represents the feedforward neural network, The local models of n clients generated by the graph convolutional hypernetwork, GHN stands for graph convolutional hypernetwork; Step S106: Input the correlation matrix obtained in step S103 and the client embedding vector obtained in step S104 into the graph convolutional hypernetwork initialized in step S105 to generate local models for each client. These models are then sent to the client, and the loss value and gradient are calculated locally on the client. The gradient is transmitted to the server, and the gradient of the client's local loss with respect to the graph convolutional hypernetwork parameters is calculated by the chain rule. The graph convolutional hypernetwork parameters are continuously updated through the backpropagation algorithm until convergence; Step S107: Input the client's embedding vector and correlation matrix into the graph convolutional hypernetwork trained in step S106 to generate a local model personalized for each client.

2. The personalized federated learning method based on graph convolutional hypernetwork according to claim 1, characterized in that: In step S101 , each client adopts a different neural network model according to the different formats of data collected by each client.

3. The personalized federated learning method based on graph convolutional hypernetwork according to claim 1, characterized in that The updating process of the local models of n clients in step S102 is as follows: ， in, is the local model of the k-th client, is the updated local model of the k-th client, is the gradient of the local model of the kth client calculated on its local data, is the update step size; this process is performed N times on the client. In addition, the gradient vector obtained by updating the k-th client local model to be uploaded in step S102 is the difference between the two local models before and after the update: 。 4. The personalized federated learning method based on graph convolutional hypernetwork according to claim 1, characterized in that The client correlation matrix in step S103 is: ， in, , The gradient vector obtained by updating the local model of the i-th client, The gradient vector obtained by updating the local model of the jth client; in addition, the client correlation matrix The values of the elements in the matrix that are greater than the set threshold are set to 0, and the remaining values are set to 1, thereby converting the client correlation matrix A into a 0-1 adjacency matrix.

5. The personalized federated learning method based on graph convolutional hypernetwork according to claim 1, characterized in that The embedding vector obtained after dimensionality reduction in step S104 is as follows: ， in, It is the result of PCA dimensionality reduction of the client gradient vector.

6. The personalized federated learning method based on graph convolutional hypernetwork according to claim 1, characterized in that The gradient of the graph convolutional hypernetwork parameters in step S106 is calculated by the chain rule as follows: ， ， in, 、 The training loss of the k-th client is the graph convolutional hypernetwork parameter. The gradients of , which can be obtained by the chain rule, Represents the k-th client training set loss weight The gradient, The output of the feedforward neural network is Feedforward neural network parameters The gradient, express Output of graph convolutional network The gradient, Represents the output of the graph convolutional network Graph Convolutional Network Parameters The gradient of is trained through multiple stochastic gradient descents to train the graph convolutional hypernetwork.

Images