A federated heterogeneous graph learning method based on soft clustering

Abstract

The present application provides a federal heterogeneous graph learning method based on soft clustering to solve the problem of performance decline of the federal learning model caused by the non-independent and identically distributed of the graph structure data and the inconsistent participation of the clients in the training process. The present application decouples the node attribute features and the structure features of the graph structure data for training, and performs personalized soft clustering according to the similarity of the structure feature distribution of the graph structure data of different clients. Finally, the personalized model after iteration is obtained by weighted aggregation in the cluster according to the participation of the client in the training and the model performance. The present application combines the leading technologies in the fields of soft clustering and federal graph learning, which helps to solve the negative impact of the clients with different graph structure data distribution and different training degree on the model performance in the federal graph learning, protects the local data privacy and security, and at the same time trains a personalized model that is more consistent with the actual data distribution of the clients, improves the training efficiency and accuracy of the federal learning model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of joint graph learning, and particularly relates to a federated heterogeneous graph learning method based on soft clustering. BACKGROUND

[0002] Federated graph learning (FGL) is a combination of federated learning (FL) and graph neural network (GNN), which integrates powerful data modeling, feature extraction and enhanced data privacy. FGL has been widely applied in intelligent medical care, traffic prediction, Internet of Things and recommendation system and other fields. It is known that one of the main problems that FL needs to solve is non-independent and identically distributed. However, compared with FL, FGL not only needs to solve the problem of non-independent and identically distributed at the data feature level, but also needs to solve the problem of non-independent and identically distributed at the structure feature level. In addition, in actual application scenarios, it is crucial to solve the problem of inconsistent participation of different clients in training for further promoting the development of FGL. SUMMARY

[0003] The present application aims to provide a federated heterogeneous graph learning method based on soft clustering to solve the problem of non-independent and identically distributed of graph structure data and the problem of performance decline of federated learning model caused by inconsistent participation of clients in training.

[0004] To achieve the above-mentioned purpose, the technical scheme of the present application is as follows: a federated graph learning method for non-independent and identically distributed graphs based on soft clustering, comprising the following steps:

[0005] Step S1, for the structure features of the graph structure data, explicit graph embedding is constructed to decouple the node attribute features and the structure features of the graph structure data.

[0006] Step S2, according to the different structure feature distribution of the graph structure data, soft clustering is performed on the clients with similar structure feature data.

[0007] Step S3, according to the training participation degree of different clients in the clustering cluster and the performance of the model obtained by training, the model is weighted aggregated to obtain an individualized iterative model.

[0008] Preferably, in step S1, for the structure features of the graph structure data, explicit graph embedding is constructed to decouple the node attribute features and the structure features of the graph structure data, which specifically comprises the following steps:

[0009] Step S1A1, for the graph structure data of the local client, explicit structure feature embedding of the graph structure data based on node degree and random walk is constructed respectively;

[0010] Step S1A2, the two structure feature embeddings based on node degree and random walk are spliced according to a certain dimension ratio to construct a comprehensive structure feature embedding.

[0011] Step S1A3, for the heterogeneity problem of node attribute features of the graph structure data owned by different clients, the node attribute features and the comprehensive structure feature embedding of the original graph structure data are respectively decoupled trained through the model structure.

[0012] Preferably, in step S1A1, the explicit construction calculation method of the structure feature embedding based on node degree and random walk is as follows:

[0013]

[0014] wherein, represents the structure feature embedding of node v based on node degree and random walk, is an identity mapping, T = AD -1 is a random walk transition matrix calculated from a binary adjacency matrix A and a pair-wise angle matrix D, k1 and k2 are respectively the dimension number required to be constructed for the structure feature embedding based on node degree and random walk.

[0015] Preferably, in step S1A3, the objective function of decoupled training of the node attribute features and the comprehensive structure feature embedding of the original graph structure data through the model structure is as follows:

[0016]

[0017] wherein, φ n,i ,φ s,i respectively represent the local model parameters of the node attribute features and the structure feature part of the graph structure data extracted by the i-th client, represents the optimal solution of its parameters, D i represents the graph structure data set owned by the i-th client, D represents the total set of graph structure data owned by the clients participating in training, Φ represents the global model obtained by global aggregation, N represents the total number of clients participating in training, argmin(·) represents the inverse function of the minimum value function, represents the loss function of training.

[0018] Preferably, in step S2, according to the different structure feature distributions of the graph structure data, the clients with similar structure feature data are divided into soft clusters, which specifically includes the following steps:

[0019] Step S2A1, for the heterogeneity problem of the structure features of the graph structure data owned by different clients, the principal vectors representing the structure feature distribution are calculated by high-dimensional matrix decomposition on the data matrix;

[0020] Step S2A2: Based on the principal vectors of different clients, calculate the high-dimensional similarity matrix of the principals between the graph structure data structure distributions of different clients, and evaluate the degree of similarity of the structural feature distributions among the clients.

[0021] Step S2A3: Based on the high-dimensional protagonist similarity matrix, clients with similar structural feature distributions are divided into clusters to promote the full learning of structural features that are beneficial to improving model accuracy among similar clients.

[0022] Preferably, in step S2A1, the formula for calculating the principal vector representing the distribution of structural features by performing high-dimensional matrix decomposition on the data matrix is ​​as follows:

[0023] U,s,V T =TSVD(e)

[0024] The truncated singular value decomposition (TSVD) algorithm is used on graph-structured data to calculate the principal vectors representing the distribution of structural features; where s represents the obtained principal vectors, and U and V are... T These represent the left and right matrices obtained through the TSVD algorithm, respectively. e represents the set of graph structure data owned by the client, and TSVD(·) represents the TSVD algorithm computation function.

[0025] Preferably, in step S2A2, the method for calculating the high-dimensional similarity matrix of the main characters between the graph structure data structure distributions of different clients and evaluating the similarity of the structural feature distributions among the clients is as follows:

[0026]

[0027] Among them, high-dimensional protagonist similarity i,j s represents the high-dimensional protagonist similarity between the i-th client and the j-th client. i and s j Let represent the principal vectors of the i-th client and the j-th client respectively, arccos(·) denotes the inverse cosine function, and |·| denotes the modulo operation of the vector.

[0028] Preferably, in step S2A3, the method for dividing clients with similar structural feature distributions into clusters based on the high-dimensional protagonist similarity matrix is ​​calculated as follows:

[0029]

[0030] Preferably, in step S3, the model is weighted and aggregated based on the training participation level of different clients within the cluster and the performance of the trained model to obtain a personalized iterative model, specifically including the following steps:

[0031] Step S3A1, statistics of the training participation degree of different clients in the same cluster and the model performance obtained by training are calculated, and two aggregation weight operators based on the training participation degree and the model performance are calculated respectively;

[0032] Step S3A2, the two aggregation weight operators based on the training participation degree and the model performance are comprehensively considered, the aggregation weight of each client in the current round of clustering cluster is calculated, and the personalized model in the current round of clustering cluster is obtained by weighted aggregation.

[0033] Preferably, in step S3A1, the formula for calculating the two aggregation weight operators based on the training participation degree and the model performance of different clients in the same cluster is as follows:

[0034]

[0035] Wherein, W i represents the two aggregation weight operators based on the training participation degree and the model performance of the i th client respectively, D i and P i represent the data amount and the training participation times of the client i respectively, Acc i represents the accuracy of the model performance, τ represents the offset for controlling the weight value range, and N' represents the number of clients in the clustering cluster.

[0036] In step S3A2, the formula for calculating the aggregation weight of each client in the current round of clustering cluster by comprehensively considering the two aggregation weight operators based on the training participation degree and the model performance is as follows:

[0037] W = λ 1 × W p + λ 2 × W q , st. λ 1 + λ 2 = 1

[0038]

[0039] Wherein, W represents the aggregation weight of the client, λ 1 and λ 2 represent the proportional hyperparameters corresponding to the two aggregation weight operators based on the training participation degree and the model performance respectively, Φ and φ s represent the personalized model parameters obtained by aggregation and the local model parameters of the client extracted structural feature part respectively

[0040] Compared with the prior art, the present application has the following beneficial effects:

[0041] 1. Compared with the existing front FGL method, the present application considers the non-independent and identically distributed problem from two aspects of node attribute features and structural features.

[0042] 2. The application adopts a soft clustering technique to more flexibly cluster clients with different data structures. This method can achieve more effective client cluster division, further improving the efficiency of the structure feature extraction layer aggregation. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 The framework diagram of the federated heterogeneous graph learning method based on soft clustering of the application. DETAILED DESCRIPTION

[0044] The technical solutions of the application will be specifically described below. Figure 1 The technical solutions of the application will be specifically described below.

[0045] The application proposes a kind of federated graph learning method based on soft clustering, which is proposed to solve the problem of performance decline of federated learning model caused by non-independent and identically distributed of graph structure data and inconsistent participation of clients in training process, and the overall framework of the method is as shown in the accompanying Figure 1 Figure. It includes structure-data feature decoupling training module, structure clustering module and full aggregation module. The node attribute features and structure features of the graph structure data are decoupled and trained, and personalized soft clustering is carried out according to the similarity of the structure feature distribution of different clients, and finally the weighted aggregation is carried out according to the participation degree of the client in training within the cluster, to obtain the personalized iterative model. The application combines the front technologies of soft clustering and federated graph learning two fields, which helps to solve the negative impact of clients with different graph structure data distribution and different training degree on the performance of the model in federated graph learning, protects the privacy and security of local data, and trains personalized model that is more consistent with the actual data distribution of the client, improves the training efficiency and accuracy of the federated learning model.

[0046] The specific steps of the application are as follows:

[0047] Step S1, for the structure features of graph structure data, explicit graph embedding is constructed, and the node attribute features and structure features of the graph structure data are decoupled and trained.

[0048] Step S2, according to the different structure feature distribution of graph structure data, the clients with similar structure feature data are divided into soft clusters.

[0049] Step S3, according to the training participation degree of different clients in the cluster and the performance of the model obtained by training, the model is weighted aggregated to obtain the personalized iterative model.

[0050] In step S1, for the structure features of graph structure data, explicit graph embedding is constructed, and the node attribute features and structure features of the graph structure data are decoupled and trained, which specifically includes the following steps:

[0051] Step S1A1, for the client local graph structure data, respectively, explicitly construct graph structure data based on node degree, random walk structure feature embedding.

[0052] Step S1A2, according to a certain dimension ratio, splice the two structure feature embeddings based on node degree and random walk, and construct a comprehensive structure feature embedding.

[0053] Step S1A3, for the heterogeneity problem of node attribute features of the graph structure data owned by different clients, the model structure is used to decouple the node attribute features of the original graph structure data and the comprehensive structure feature embedding.

[0054] In step S1A1, the explicit construction method of the structure feature embedding based on node degree and random walk is as follows:

[0055]

[0056] Wherein, represents the structure feature embedding based on node degree and random walk of node v, is an identity mapping, T = AD -1 is a random walk transition matrix calculated by a binary adjacency matrix A and a pair-wise angle matrix D, k1 and k2 are respectively the dimension number required to construct the structure feature embedding based on node degree and random walk.

[0057] In step S1A3, the objective function of decoupling training of node attribute features of the original graph structure data and the comprehensive structure feature embedding by the model structure is as follows:

[0058]

[0059] Wherein, φ n,i ,φ s,i respectively represent the local model parameters of the i-th client to extract the node attribute features and structure feature parts of the graph structure data, represents the optimal solution of its parameters, D i represents the graph structure data set owned by the i-th client, D represents the total set of graph structure data owned by the clients participating in training, Φ represents the global model obtained by global aggregation, N represents the total number of clients participating in training, argmin(·) represents the inverse function of the minimum value function, represents the loss function of training.

[0060] In step S2, according to the different structure feature distribution of the graph structure data, the clients with similar structure feature data are divided into soft clusters, which specifically includes the following steps:

[0061] Step S2A1, in view of the heterogeneity of the structure characteristics of the graph structure data owned by different clients, the principal vectors representing the structure characteristic distribution are calculated by high-dimensional matrix decomposition on the data matrix.

[0062] Step S2A2, according to the principal vectors of different clients, the high-dimensional principal angle similarity matrix between the structure distribution of the graph structure data of different clients is calculated, and the similarity degree of the structure characteristic distribution between each client is evaluated.

[0063] Step S2A3, according to the high-dimensional principal angle similarity matrix, the clients with similar structure characteristic distribution are divided into clustering clusters, and the sufficient learning of the structure characteristics between similar clients which is beneficial to improve the model accuracy is promoted.

[0064] In step S2A1, the formula for calculating the principal vector representing the structure characteristic distribution by high-dimensional matrix decomposition on the data matrix is as follows:

[0065] U,s,V T =TSVD(e)

[0066] The present application uses the truncated singular value decomposition algorithm (TSVD) on the graph structure data to obtain the principal vector representing the structure characteristic distribution. Wherein, s represents the obtained principal vector, U, V T respectively represent the left matrix and the right matrix obtained by the TSVD algorithm, e represents the set of graph structure data owned by the client, and TSVD(·) represents the TSVD algorithm calculation function.

[0067] In step S2A2, the method for calculating the high-dimensional principal angle similarity matrix between the structure distribution of the graph structure data of different clients and evaluating the similarity degree of the structure characteristic distribution between each client is as follows:

[0068]

[0069] Wherein, the high-dimensional principal angle similarity i,j represents the high-dimensional principal angle similarity between the i-th client and the j-th client, s i and s j respectively represent the principal vector of the i-th client and the j-th client, arccos(·) represents the inverse cosine function, and |·| represents the vector module operation.

[0070] In step S2A3, the method for dividing the clients with similar structure characteristic distribution into clustering clusters according to the high-dimensional principal angle similarity matrix is calculated as follows:

[0071]

[0072] In step S3, based on the level of training participation of different clients within the cluster and the performance of the trained model, the model is weighted and aggregated to obtain a personalized iterative model. This specifically includes the following steps:

[0073] Step S3A1: Statistically analyze the training participation level and model performance of different clients within the same cluster, and calculate two aggregation weight operators based on training participation level and model performance, respectively.

[0074] Step S3A2: Taking into account both the training participation level and the model performance-based aggregation weight operators, calculate the aggregation weight of each client in this round of clustering, and perform weighted aggregation to obtain the personalized model in this round of clustering.

[0075] In step S3A1, the training participation level of different clients within the same cluster and the performance of the trained model are statistically analyzed. The formulas for calculating the two aggregation weight operators based on training participation level and model performance are as follows:

[0076]

[0077] in, Let D represent the two aggregation weight operators for the i-th client, one based on training participation and the other based on model performance. i and P i These represent the amount of data the client possesses and the number of training sessions, respectively. Acc i The accuracy of the model performance is represented by τ, the offset used to control the weight range is represented by N′, and the number of clients within the cluster is represented by N′.

[0078] In step S3A2, considering both aggregation weight operators based on training participation and model performance, the aggregation weights of each client within the current cluster are calculated, and the weighted aggregation yields the personalized model within the current cluster using the following formula:

[0079] W = λ1 × W p +λ2×W q ,st.λ1+λ2=1

[0080]

[0081] Where W represents the client's aggregation weight, λ1 and λ2 represent the proportional hyperparameters corresponding to the two aggregation weight operators based on training participation and model performance, respectively, and Φ and φ s These represent the personalized model parameters obtained from aggregation and the local model parameters of the structural feature parts extracted by the client, respectively.

[0082] The above is the preferred embodiment of the present application, any change made according to the technical solution of the present application, as long as the function generated does not exceed the scope of the technical solution of the present application, belongs to the protection scope of the present application.

Claims

1. A federated heterogeneous graph learning method based on soft clustering, characterized in that, Comprising the following steps: Step S1, for the structure characteristics of the graph structure data, construct explicit graph embedding, decouple the node attribute characteristics and structure characteristics of the graph structure data for training; Step S2, according to the different structure characteristic distribution of the graph structure data, soft clustering is divided for the clients with similar structure characteristic data; Step S3, according to the training participation degree of different clients in the clustering cluster and the performance of the model trained, the model is weighted aggregated to obtain the personalized iterative model; In step S3, according to the training participation degree of different clients in the clustering cluster and the performance of the model trained, the model is weighted aggregated to obtain the personalized iterative model, comprising the following steps: Step S3A1, the training participation degree and the performance of the model trained of different clients in the same clustering cluster are counted, and two aggregation weight operators based on the training participation degree and the model performance are calculated respectively; Step S3A2, the aggregation weight of each client in the current clustering cluster is calculated by comprehensively considering the two aggregation weight operators based on the training participation degree and the model performance, and the personalized model in the current clustering cluster is obtained by weighted aggregation; In step S3A1, the training participation degree and the performance of the model trained of different clients in the same clustering cluster are counted, and the formulas of the two aggregation weight operators based on the training participation degree and the model performance are as follows: where, respectively represent the two aggregation weight operators of the ith client based on training participation degree and model performance, D i and P i respectively represent the data volume and the number of training participation of the client i, Acc i denotes the accuracy of the model performance, τ represents the offset for controlling the weight value domain, and N' represents the number of clients in the clustering cluster. In step S3A2, the aggregation weight of each client in the current clustering cluster is calculated by comprehensively considering the two aggregation weight operators based on the training participation degree and the model performance, and the formulas of the personalized model in the current clustering cluster obtained by weighted aggregation are as follows: where W i denotes the aggregation weight of the i-th client, λ1, λ2 denote the proportion hyperparameters corresponding to two aggregation weight operators based on training participation degree and model performance, respectively, Φ, ∅ s,i denote the personalized model parameters aggregated and the local model parameters of the i-th client extracting the structural feature part, respectively. 2.The federated heterogeneous graph learning method based on soft clustering according to claim 1, wherein, In step S1, for the structure characteristics of the graph structure data, construct explicit graph embedding, decouple the node attribute characteristics and structure characteristics of the graph structure data for training, comprising the following steps: Step S1A1, for the graph structure data of the client locally, respectively construct the structure characteristic embedding of the graph structure data based on node degree and random walk; Step S1A2, according to a certain dimension ratio, splice the two structure characteristic embeddings based on node degree and random walk to construct comprehensive structure characteristic embedding; Step S1A3, for the heterogeneity problem of the node attribute characteristics of the graph structure data owned by different clients, respectively decouple the original graph structure data node attribute characteristics and comprehensive structure characteristic embedding for training by model structure. 3.The soft clustering based federated heterogeneous graph learning method of claim 2, wherein, In step S1A1, the explicit construction calculation method of the structure characteristic embedding based on node degree and random walk is as follows: wherein, k1, k2 are the dimensionality of the structure feature embedding based on node degree, random walk respectively to be constructed. is the identity mapping, T = AD -1 is the random walk transition matrix computed from the binary adjacency matrix A and the diagonal degree matrix D, k1, k2 are the dimensionality of the structure feature embedding based on node degree, random walk respectively to be constructed. 4.The soft clustering based federated heterogeneous graph learning method of claim 2, wherein, In step S1A3, the objective function of decoupling training of the node attribute characteristics of the original graph structure data and the comprehensive structure characteristic embedding by model structure is as follows: where, ∅ n,i ,∅ s,i respectively represent the local model parameters of the i-th client extracting the node attribute feature and structure feature part of the graph structure data, represent the optimal solution of its parameters, D i represents the graph structure data set owned by the i-th client, D represents the total set of graph structure data owned by the clients participating in training, represents the global model obtained through global aggregation, N represents the total number of clients participating in training, and argmin(·) represents the inverse function of the minimum value function, represents the loss function of training.

5. The federated heterogeneous graph learning method based on soft clustering according to claim 1, characterized in that, In step S2, according to the different structure characteristic distribution of the graph structure data, soft clustering is divided for the clients with similar structure characteristic data, comprising the following steps: Step S2A1, for the heterogeneity problem of the structure characteristics of the graph structure data owned by different clients, the principal vector representing the structure characteristic distribution is calculated by high-dimensional matrix decomposition of the data matrix; Step S2A2, according to the principal vector of different clients, the high-dimensional principal angle similarity matrix between the structure data structure distribution of different clients is calculated, and the similarity degree of the structure feature distribution between each client is evaluated. Step S2A3, according to the high-dimensional principal angle similarity matrix, the clients with similar structure feature distribution are divided into clustering clusters, and the sufficient learning of the structure features between similar clients which is beneficial to improve the model accuracy is promoted.

6. The federated heterogeneous graph learning method based on soft clustering according to claim 5, characterized in that, In step S2A1, the formula for calculating the principal vector representing the structure feature distribution is as follows by high-dimensional matrix decomposition of the data matrix: U, s, V T = TSVD(e) The truncated singular value decomposition algorithm TSVD is used for the graph structure data to obtain eigenvectors representing the structure feature distribution; wherein, s represents the obtained eigenvectors, U, V T respectively represent the left matrix and the right matrix obtained through the TSVD algorithm, e represents a set of graph structure data owned by the client, and TSVD(·) represents a TSVD algorithm calculation function.

7. The federated heterogeneous graph learning method based on soft clustering according to claim 5, characterized in that, In step S2A2, the method for calculating the high-dimensional principal angle similarity matrix between the structure data structure distribution of different clients to evaluate the similarity degree of the structure feature distribution between each client is as follows: wherein the high-dimensional principal character similarity i,j represents the high-dimensional principal character similarity between the ith client and the jth client, s i and s j respectively represent the principal vector of the ith client and the jth client, arccos(·) represents the inverse cosine function, and |·| represents the vector modulus operation. 8.The federated heterogeneous graph learning method based on soft clustering according to claim 5, wherein, In step S2A3, the method for dividing the clients with similar structure feature distribution into clustering clusters according to the high-dimensional principal angle similarity matrix is calculated as follows:

Images