A clustering-based federated graph learning method and system

Abstract

A clustering-based federated graph learning method improves the performance of a global model obtained by federated learning of a graph network with uneven data distribution based on a clustering aggregation algorithm, and the method comprises the following steps: S1: a client obtains graph data through a respective data collection method, constructs a graph network, calculates the structural difference E of each type of label, and uploads to a server; S2: clients with different distributed data perform local training until the model converges to obtain the model parameters a of the output layer out and upload to the server. The server processes the two parameters uploaded by the client as the basis for dividing the client into different cluster clusters; S3: each client uploads its model parameters to the server when the local training reaches a certain batch. The server updates the global model parameters through intra-cluster aggregation and inter-cluster aggregation and sends them to each client. The client updates the model and continues the training of the next batch until the model converges. The present application can better complete the node classification and link prediction tasks.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of federated graph learning, and more particularly, to a clustering-based federated graph learning method and system. BACKGROUND

[0002] With the advent of the information age, graph data similar to social networks has exploded exponentially. Graph neural networks can handle various complex tasks on graphs, such as node classification, graph classification, link prediction, etc. Its excellent performance enables it to be widely used in various real-world scenarios, such as semantic segmentation, recommendation systems, text classification, social influence prediction, and brain neural prediction. Like other deep learning models, the superior performance of graph neural networks also relies on a large amount of data for training. In the case of insufficient data at a single data holder, training the algorithm with multiple parties' data can improve the effectiveness of graph neural networks, but considering data security and privacy protection issues, it is often difficult to achieve. Therefore, how to coordinate multiple parties to complete the task while protecting data privacy becomes particularly important.

[0003] In this context, federated learning emerged, aiming to train a common global model by sharing gradient parameters to achieve data privacy protection without leaving the local. Federated learning has achieved good results in the image field and can be widely used in target detection, image classification, etc. However, in the field of graph networks, federated learning is still in its infancy. Federated learning makes data on edge devices valuable, but the data between edge devices often has large distribution differences, and federated averaging aggregation algorithms perform poorly in this case.

[0004] By clustering different graph networks with different data distributions into different clusters and then aggregating them, the impact of data distribution differences can be effectively reduced while preserving the privacy protection features of federated learning. The performance of the global model is improved, enabling it to optimize the accuracy of recommendation systems and detect financial criminal behavior across agencies. SUMMARY

[0005] The present application aims to provide a clustering-based federated graph learning method and system. The present application combines clustering and federated graph learning, effectively reducing the impact of large data distribution differences on the performance of the global model.

[0006] The present application provides a new idea for federated aggregation for non-identically and independently distributed graph network data. Instead of being limited to traditional federated averaging aggregation algorithms, a clustering-based federated aggregation algorithm is proposed. The weight parameters of the output layer obtained by local training at the client are used to explore the hidden structural information of the graph network. After processing together with the structural difference index of the label, they are used as the basis for client clustering. Then, step-by-step aggregation is performed: intra-cluster aggregation and inter-cluster aggregation.

[0007] The clustering-based federated graph learning method of the application comprises the following steps:

[0008] S1: The client obtains graph data through respective data collection methods, constructs a graph network and calculates the structural difference E of various labels, and uploads to the server;

[0009] S2: The client trains the graph network separately to obtain the model parameters a out of the output layer and uploads to the server. The server processes the two parameters uploaded by the client as the basis for dividing the client into different clusters;

[0010] S3: When each client locally trains to a certain batch, upload the model parameters to the server. The server updates the global model parameters through intra-cluster aggregation and inter-cluster aggregation and sends them to each client. The client updates the model and continues the next batch of training until the model converges.

[0011] Preferably, the step S1 specifically comprises:

[0012] The client calculates the structural difference e l of various labels of the graph data, and the specific formula is as follows:

[0013]

[0014] Wherein, N l represents the number of nodes of a certain label in the graph network. The nodes of this label are divided into m classes according to the number of neighbor nodes, and the number of nodes in the i-th class is k i . Finally, a L-dimensional vector E = [e1, e2, e l , …, e L ] is obtained and uploaded to the server.

[0015] Preferably, the step S2 specifically comprises:

[0016] S2.1: The client adopts a three-layer multi-head graph attention network model (GAT) for training, the first two layers are used to compress node features, and the last layer is the output layer;

[0017] S2.2: Based on the model structure of step S2.1, the output layer attention coefficient a out is obtained by training to convergence and uploaded to the server. After the server normalizes a out of each client, the principal component analysis method (PCA) is used to reduce the dimension of a out to obtain The specific formula is as follows:

[0018]

[0019] wherein μ is the average of a out and σ is the standard deviation. The client The dimension of the client is determined by the number of clients. Generally speaking, the more the number of clients, the more the divided cluster. u represents the dimension of the client p after dimension reduction. The server realizes the division of the client by using a distance-based clustering method, and the specific formula is as follows:

[0020]

[0021]

[0022] wherein, p, v ∈ q and p ≠ v, and q represents a set of clients participating in federated learning. flag p,v represents a clustering signal between the clients p and v, and when flag p,v = 1, it indicates that the two clients can be divided into the same cluster, and vice versa; δ is a threshold value of clustering, which can be adjusted according to specific conditions.

[0023] Preferably, the step S3 specifically comprises:

[0024] The clients participating in federated learning encrypt the model parameters W, a of each layer GAT based on homomorphic encryption and upload them to the server within a set training batch; the server aggregates the clients in each cluster to obtain the model parameters of each cluster, and then aggregates between clusters to finally obtain the global model parameters. The specific formula is as follows:

[0025]

[0026]

[0027] wherein, represents the weight parameter and the attention coefficient of the client i in the cluster c in the t-th batch, and Q c represents the number of clients in the cluster c, and r represents the number of clusters. The server transmits the global model parameters back to each client, and the client receives and decrypts the global model parameters. t = t + 2, and the above steps are repeated using the parameters until the model converges.

[0028] A clustering-based federated graph learning system, characterized by comprising a parameter acquisition and processing module, a clustering module, and a federated aggregation module connected in sequence;

[0029] The parameter acquisition and processing module is used for collecting and processing parameters for clustering, specifically including: calculation of structural difference indicators of each label of the client, and acquisition of attention coefficients of the output layer of the model trained locally by the client to convergence;

[0030] The clustering module is used for calculating and processing the structural difference indicators of the output layer attention coefficients uploaded by each client and the labels, so as to cluster the clients with different data distributions.

[0031] The federal aggregation module is used for uploading the model parameters of each layer of each client when the training is performed to the t-th batch.

[0032] The present application fully and comprehensively captures the similarity of the clients with uneven label distribution on the network structure, improves the performance of the global model by distributing each client into different clusters for step-by-step aggregation, and enables the global model to better complete the tasks of node classification and link prediction.

[0033] The present application has the advantages that, compared with the prior art, the technical scheme of the present application has the beneficial effects that the structural difference indicators of the output layer attention coefficients of the locally trained and converged model and the labels are used to discover the hidden structural similarity between the graph networks of the clients with uneven data distribution, so that each client is distributed into different clusters for step-by-step aggregation, the performance of the global model is improved, and the global model can better complete the tasks of node classification and link prediction. BRIEF DESCRIPTION OF DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of these drawings.

[0035] Figure 1 The framework diagram of the present application is shown in the figure.

[0036] Figure 2 The general flowchart of the present application is shown in the figure.

[0037] Figure 3 The system structure diagram of the present application is shown in the figure. DETAILED DESCRIPTION

[0038] The various exemplary embodiments of the present application will be described in detail below, and the detailed description should not be considered as limiting the present application, but should be understood as a more detailed description of some aspects, characteristics and embodiments of the present application.

[0039] It should be understood that the terminology used in this invention is merely for describing particular embodiments and is not intended to limit the invention. Furthermore, with respect to numerical ranges in this invention, it should be understood that each intermediate value between the upper and lower limits of the range is also specifically disclosed. Every smaller range between any stated value or intermediate value within a stated range, and any other stated value or intermediate value within said range, is also included in this invention. The upper and lower limits of these smaller ranges may be independently included or excluded from the range.

[0040] Unless otherwise stated, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art. While only preferred methods and materials have been described herein, any methods and materials similar or equivalent to those described herein may be used in the implementation or testing of this invention. All references to this specification are incorporated by way of citation to disclose and describe methods and / or materials associated with those references. In the event of any conflict with any incorporated reference, the content of this specification shall prevail.

[0041] Various modifications and variations can be made to the specific embodiments described in this specification without departing from the scope or spirit of the invention, as will be apparent to those skilled in the art. Other embodiments derived from this specification will also be readily apparent to those skilled in the art. This application specification and embodiments are merely exemplary.

[0042] The terms “include,” “including,” “have,” “contain,” etc., used in this article are all open-ended terms, meaning that they include but are not limited to.

[0043] Unless otherwise specified, the term "parts" in this invention refers to parts by weight.

[0044] This invention provides a user music recommendation method based on clustering-based federated graph learning, such as... Figure 1 As shown, the steps include:

[0045] Step S1: Data acquisition and processing, and obtaining label difference E;

[0046] Specifically: in the application scenario of a recommendation system, taking the music streaming platform LastFM as an example, the task of federated learning is to classify the music preferences of target users, so as to recommend corresponding types of music, which belongs to the node classification task. Each client obtains LastFM music streaming platform related data in various ways, extracts users as a node set V, extracts friendship relationships between users as an edge set E, extracts keywords of user preferences as features X, and deletes isolated nodes to construct a graph network G(V, E, X). Due to different ways of obtaining data, the node of each client often has low coincidence degree and large data distribution difference. Ten clients with a certain amount of data are selected to participate in federated learning, and there are 18 types of labels in the ten clients. The number of nodes of each label in each client is shown in the following table:

[0047]

[0048]

[0049] Thereafter, the client calculates the structural difference e l of each label, and the specific formula is as follows:

[0050]

[0051] Where N l represents the number of nodes of a certain type of label in the graph network, the nodes of this label are divided into m classes according to the number of neighbor nodes, and the number of nodes in the i-th class is k i . Finally, an 18-dimensional vector E = [e1, e2, …, e 18 ] is obtained and uploaded to the server.

[0052] Step S2: The client locally trains to convergence, and obtains the output layer attention coefficient a out ;

[0053] The client inputs the features of the graph network nodes collected by itself into the graph attention network (GAT) for training. During the training process, the parameters of the model are constantly updated through backpropagation. Until the model converges, the attention coefficient a out of the output layer is obtained and uploaded. The server first performs z-score normalization on the attention coefficients a out of the output layer of each client, and then performs principal component analysis (PCA) to reduce the dimension of a out to obtain The specific formula is as follows:

[0054]

[0055] Where μ is the a outwhere is the average value, and is the standard deviation, and PCA is reduced to two dimensions. For client p, Then the clustering signal flag between clients is calculated p,v The clients are divided into different clusters, and the specific formula is as follows:

[0056]

[0057]

[0058] where d p = [d p,1 , d p,2 ], p, v e q and p ≠ v, q represents the number of clients participating in federated learning. flag p,v represents the clustering signal between clients p, v, when flag p,v = 1, it means that the two clients can be divided into the same cluster, according to the calculation, ten clients participating in federated learning will be divided into four clustering clusters.

[0059] Step S3: The clients participating in federated learning encrypt the model parameters W, a of each layer GAT based on homomorphic encryption and upload them to the server in the set training batch; the server aggregates the clients in each clustering cluster, obtains the model parameters of each cluster, and then aggregates between clusters to finally obtain the global model parameters. The specific formula is as follows:

[0060]

[0061]

[0062] where represents the weight parameter and attention coefficient of client i in cluster c in the t-th batch, Q c represents the number of clients in cluster c, and r represents the number of clusters, and at this time r = 4. The server transmits the global model parameters back to each client, and the client receives and decrypts the global model parameters. t = t + 2, and the above steps are repeated using the parameters until the model converges.

[0063] The node classification accuracy on the LastFM music streaming platform under different aggregation algorithms is shown in the following table:

[0064] Client Individual training Federated average aggregation Clustered aggregation 1 0.3194 0.5290 0.5516 2 0.2790 0.5403 0.5597 3 0.3226 0.5613 0.5710 4 0.3113 0.5468 0.5694 5 0.4645 0.5339 0.5435 6 0.3484 0.5419 0.5468 7 0.2694 0.5097 0.5290 8 0.4613 0.5242 0.5516 9 0.3452 0.5500 0.5645 10 0.4065 0.5323 0.5629 Average 0.3528 0.5369 0.5550

[0065] From the results, it can be seen that the federated aggregation method provided by the present disclosure can more accurately classify users in the aggregation scenario between clients with uneven data distribution, thereby realizing more accurate music preference recommendation.

[0066] Step S4: input the new user features to the model to realize music recommendation;

[0067] When the client obtains a new user network or adds a new node to the original network, the newly constructed graph network is input into the trained model to obtain the classification result of the new node, and then the corresponding type of music is recommended.

[0068] The parameter acquisition and processing module is configured to collect and process parameters for clustering, specifically including: calculation of structural difference indicators of each label of the client, and acquisition of output layer attention coefficients of the model trained locally by the client to convergence;

[0069] The clustering module is configured to calculate and process the output layer attention coefficients uploaded by each client and the structural difference indicators of the labels, so as to cluster the clients with different data distributions.

[0070] The federated aggregation module uploads the model parameters of each layer of each client when the training reaches the t-th batch. The server obtains a global model by intra-cluster aggregation and inter-cluster aggregation and re-sends the global model to each client. The client updates the model and continues the training of the next batch until the model converges.

[0071] The above-described embodiments only describe the preferred modes of the present application, and do not limit the scope of the present application. Without departing from the design spirit of the present application, various modifications and improvements to the technical solutions of the present application made by those skilled in the art shall fall within the protection scope determined by the claims of the present application.

Claims

1. A clustering-based federated graph learning method, the method being implemented by a center server and at least one client, the method comprising the following steps: S1: the client obtains graph data through a respective data collection method, constructs a graph network and calculates the structural difference E of each type of label, and uploads to the server; S2: the client trains the model to convergence by using the graph network alone, and obtains the model parameters a of the output layer out Upload to the server; the server processes the two parameters uploaded by the client and uses them as the basis for dividing the client into different clustering clusters; specifically including: S2.1: the client trains using a three-layer multi-head graph attention network model (GAT), the first two layers being used to compress node features, and the last layer being an output layer; S2.2: Train the model structure based on step S2.1 to convergence to obtain the output layer attention coefficient a out Upload to the server; the server normalizes a out of each client, and then performs principal component analysis (PCA) on a out to reduce the dimension to obtain The specific formula is as follows: where μ is the average of a out of each client, and σ is the standard deviation; the number of clients The dimension of the client is determined by the number of clients; the more the number of clients, the more the divided cluster; u represents the dimension of the client p after dimension reduction; the server adopts a distance-based clustering method to divide the clients, and the specific formula is as follows: wherein, p, v e q and p≠ v, q represents a set of clients participating in federated learning; flag p,v represents the clustering signal between clients p, v, when flag p,v = 1 indicates that the two clients can be divided into the same cluster, otherwise not; δ is the threshold of clustering, which can be adjusted according to specific circumstances; S3: each client uploads its model parameters to the server when local training reaches a certain batch; the server updates the global model parameters through intra-cluster aggregation and inter-cluster aggregation and sends them to each client, the client updates the model and continues training in the next batch until the model converges.

2. The cluster-based federated graph learning method of claim 1, wherein, The step S1 specifically comprises calculating the structural difference e of each type of label of the client computing graph data l The specific formula is as follows: Wherein, N l represents the number of nodes in a certain class of labels in the graph network, and the nodes of the label are divided into m classes according to the number of neighbor nodes, and the number of nodes in the i-th class is k i ; finally, an L-dimensional vector E = [e1, e2, e l ,…, e L ] is obtained and uploaded to the server.

3. The cluster-based federated graph learning method of claim 1, wherein, The step S3 specifically comprises that the client participating in federated learning encrypts the model parameters W, a of each layer GAT based on homomorphic encryption and uploads them to the server within a set training batch; the server aggregates the clients within each cluster to obtain the model parameters of each cluster, and then performs inter-cluster aggregation to finally obtain the global model parameters; the specific formula is as follows: wherein, denote the weight parameters and attention coefficients of client i within cluster c at the t-th batch, Q c denote the number of clients within cluster c, r denotes the number of clusters; The server transmits the global model parameters back to each client, the client receives and decrypts the global model parameters; t = t + 2, and repeats the above steps using the parameters until the model converges.

4. A federated graph learning system based on clustering, characterized in that: comprise a parameter acquisition and processing module, a clustering module, and a federated aggregation module connected in sequence; The parameter acquisition and processing module is used to collect and process parameters for clustering, specifically comprising: calculation of the structural difference index of each label of the client, and acquisition of the attention coefficient of the output layer of the model trained by the client to convergence; specifically comprising: S2.1: the client trains using a three-layer multi-head graph attention network model (GAT), the first two layers being used to compress node features, and the last layer being an output layer; S2.2: Train the model structure based on step S2.1 to convergence to obtain the output layer attention coefficient a out upload to the server; the server normalizes a out of each client, and then performs principal component analysis (PCA) on a out to reduce the dimension to obtain The specific formula is as follows: wherein μ is the average of a out of each client, and σ is the standard deviation; the number of clients The dimension of the client is determined by the number of clients; the more the number of clients, the more the divided cluster; u represents the dimension of the client p after dimension reduction; the server adopts a distance-based clustering method to divide the clients, and the specific formula is as follows: wherein, p, v e q and p≠ v, q represents a set of clients participating in federated learning; flag p,v represents the clustering signal between clients p, v, when flag p,v = 1 indicates that the two clients can be divided into the same cluster, otherwise not; δ is the threshold of clustering, which can be adjusted according to specific circumstances; The clustering module is used to calculate and process the output layer attention coefficient uploaded by each client and the structural difference index of the label, so as to cluster the clients with different data distributions; The federated aggregation module, each client uploads its model parameters of each layer when training reaches the t-th batch; the server obtains the global model by intra-cluster aggregation and inter-cluster aggregation and sends it to each client; the client updates the model and continues training in the next batch until the model converges.

Images