Federal heterogeneous graph learning method based on soft clustering

Abstract

The invention provides a federal heterogeneous graph learning method based on soft clustering, and aims to solve the problem of performance reduction of a federal learning model caused by non-independent identical distribution of graph structure data and inconsistent participation degrees of clients in a training process. According to the method, decoupling training is carried out on node attribute features and structural features of graph structure data, personalized soft clustering is carried out according to the similarity of graph structure data structural feature distribution of different clients, and finally weighted aggregation is carried out in the cluster according to the participation degree of the clients in training and model performance. And obtaining a model after personalized iteration. According to the method, frontier technologies in two fields of soft clustering and federated graph learning are combined, so that the negative influence of clients with different graph structure data distributions and different training degrees on model performance in federated graph learning can be solved, the privacy security of local data is protected, meanwhile, a personalized model better fitting the actual data distribution of the clients is trained, and the user experience is improved. And the training efficiency and accuracy of the federal learning model are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of joint graph learning, and in particular to a federated heterogeneous graph learning method based on soft clustering. Background Art

[0002] Federated Graph Learning (FGL) is a combination of Federated Learning (FL) and Graph Neural Networks (GNNs), which integrates powerful data modeling, feature extraction, and enhanced data privacy. FGL has been widely used in fields such as smart healthcare, traffic prediction, IoT, and recommendation systems. As we all know, one of the main problems that FL needs to solve is non-IID. However, compared with FL, FGL needs to solve the non-IID problem not only at the data feature level, but also at the structural feature level. In addition, in practical application scenarios, solving the problem of unbalanced participation of different clients in training is crucial to further promote the development of FGL. Summary of the invention

[0003] The purpose of the present invention is to propose a federated heterogeneous graph learning method based on soft clustering to solve the problem of performance degradation of the federated learning model caused by non-independent and identically distributed graph structure data and inconsistent client participation during the training process.

[0004] To achieve the above object, the technical solution of the present invention is: a federated graph learning method of non-independent identically distributed graphs based on soft clustering, comprising the following steps:

[0005] Step S1: construct explicit graph embedding based on the structural features of graph structure data, and decouple the node attribute features and structural features of the graph structure data for training.

[0006] Step S2: According to the differences in the distribution of structural features of the graph structure data, the clients with similar structural feature data are divided into soft clusters.

[0007] Step S3: weighted aggregation is performed on the model according to the training participation degree of different clients in the cluster and the model performance obtained by training to obtain a personalized iterative model.

[0008] Preferably, in step S1, an explicit graph embedding is constructed for the structural features of the graph structure data, and node attribute features and structural features of the graph structure data are decoupled for training, specifically including the following steps:

[0009] Step S1A1: for the local graph structure data of the client, explicitly construct structural feature embedding of the graph structure data based on node degree and random walk respectively;

[0010] Step S1A2, concatenating two structural feature embeddings based on node degree and random walk according to a certain dimensional ratio to construct a comprehensive structural feature embedding;

[0011] Step S1A3: To address the heterogeneity of node attribute features of graph structure data owned by different clients, decoupled training is performed on the node attribute features of the original graph structure data and the comprehensive structural feature embedding through the model structure.

[0012] Preferably, in step S1A1, the explicit construction calculation method of embedding structural features based on node degree and random walk is as follows:

[0013]

[0014] in, Represents the structural feature embedding of node v based on node degree and random walk, is an identity mapping, T = AD -1 is the random walk transfer matrix calculated by the binary adjacency matrix A and the pairwise angle matrix D. k1 and k2 are the dimensions that need to be constructed based on the node degree and the structural feature embedding of the random walk, respectively.

[0015] Preferably, in step S1A3, the objective function of decoupling training the node attribute features and the comprehensive structural features of the original graph structure data through the model structure is as follows:

[0016]

[0017] Among them, φ n,i ,φ s,i They represent the local model parameters of the node attribute features and structural features extracted by the i-th client, It indicates the optimal solution of its parameters, D i represents the graph structure data set owned by the i-th client, D represents the total set of graph structure data owned by the clients participating in the training, Φ represents the global model obtained after global aggregation, N represents the total number of clients participating in the training, argmin(·) represents the inverse function of the minimum function, Represents the loss function of training.

[0018] Preferably, in step S2, according to the different distribution of structural features of the graph structure data, the clients with similar structural feature data are divided into soft clusters, which specifically includes the following steps:

[0019] Step S2A1: To solve the heterogeneity problem of structural features of graph structure data owned by different clients, high-dimensional matrix decomposition is performed on the data matrix to calculate the main vector representing the distribution of the structural features;

[0020] Step S2A2: Calculate the similarity matrix of the high-dimensional main elements between the graph structure data structure distributions of different clients based on the main vectors of different clients, and evaluate the similarity of the structural feature distributions between the clients;

[0021] Step S2A3: Based on the high-dimensional protagonist similarity matrix, clients with similar structural feature distributions are divided into clusters to promote full learning of structural features that are beneficial to improving model accuracy among similar clients.

[0022] Preferably, in step S2A1, by performing high-dimensional matrix decomposition on the data matrix, the formula for calculating the principal vector representing the distribution of the structural features is as follows:

[0023] U,s,V T =TSVD(e)

[0024] The truncated singular value decomposition (TSVD) algorithm is used for graph structure data to calculate the main vector that represents the distribution of structural features; where s represents the main vector obtained, U, V T They represent the left matrix and the right matrix obtained by the TSVD algorithm, respectively. e represents the graph structure data set owned by the client. TSVD(·) represents the TSVD algorithm calculation function.

[0025] Preferably, in step S2A2, the similarity matrix of the high-dimensional main body between the graph structure data structure distributions of different clients is calculated, and the method for evaluating the similarity of the structure feature distribution between each client is as follows:

[0026]

[0027] Among them, high-dimensional protagonist similarity i,j represents the high-dimensional protagonist similarity between the i-th client and the j-th client, s i and j denote the main vectors of the ith client and the jth client respectively, arccos(·) denotes the inverse cosine function, and |·| denotes the modulo operation of the vector.

[0028] Preferably, in step S2A3, based on the high-dimensional principal similarity matrix, the method for dividing the clients with similar structural feature distribution into clusters is calculated as follows:

[0029]

[0030] Preferably, in step S3, according to the training participation degree of different clients in the cluster and the model performance obtained by training, the model is weightedly aggregated to obtain a personalized iterative model, which specifically includes the following steps:

[0031] Step S3A1: Count the training participation degree and model performance obtained by training of different clients in the same cluster, and calculate two aggregation weight operators based on the training participation degree and the model performance respectively;

[0032] Step S3A2: Comprehensively consider the two aggregation weight operators based on the degree of training participation and based on the model performance, calculate the aggregation weight of each client in the current clustering cluster, and perform weighted aggregation to obtain the personalized model in the current clustering cluster.

[0033] Preferably, in step S3A1, the training participation degree and the model performance obtained by training of different clients in the same cluster are counted, and the formulas for calculating two aggregation weight operators based on the training participation degree and the model performance are as follows:

[0034]

[0035] in, Denotes the two aggregation weight operators based on the training participation degree and model performance of the i-th client, respectively. i and P i They represent the amount of data and number of training sessions of client i, Acc i represents the accuracy of the model performance, τ represents the offset used to control the weight range, and N′ represents the number of clients in the cluster;

[0036] In step S3A2, the two aggregation weight operators based on the degree of training participation and the model performance are comprehensively considered to calculate the aggregation weight of each client in the current clustering cluster, and the formula for the personalized model in the current clustering cluster is obtained by weighted aggregation as follows:

[0037] W=λ1×W p +λ2×W q ,st.λ1+λ2=1

[0038]

[0039] Where W represents the client's aggregation weight, λ1 and λ2 represent the proportional hyperparameters corresponding to the two aggregation weight operators based on training participation and model performance, respectively, and Φ and φ s They represent the personalized model parameters obtained by aggregation and the local model parameters of the structural features extracted by the client.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] 1. Compared with the existing cutting-edge FGL method, the present invention considers the non-independent and identically distributed problem from two levels: node attribute characteristics and structural characteristics.

[0042] 2. The present invention uses soft clustering technology to perform more flexible clustering of clients with different data structures. This method can achieve more effective client cluster division and further improve the efficiency of structural feature extraction layer aggregation. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a framework diagram of the federated heterogeneous graph learning method based on soft clustering of the present invention. DETAILED DESCRIPTION

[0044] The following is combined with Figure 1 , the technical solution of the present invention is specifically described.

[0045] The present invention proposes a federated graph learning method based on soft clustering to solve the problem of non-independent and identically distributed graph data and inconsistent client participation during training, which leads to the degradation of federated learning model performance. The overall framework of the method is shown in the attached figure. Figure 1 As shown. It includes a structure-data feature decoupling training module, a structure clustering module, and a full aggregation module. The present invention decouples the node attribute features and structure features of the graph structure data for training, and performs personalized soft clustering according to the similarity of the structure feature distribution of the graph structure data of different clients, and finally performs weighted aggregation within the cluster according to the degree of participation of the client in the training to obtain a personalized iterative model. The present invention combines the cutting-edge technologies in the fields of soft clustering and federated graph learning, which helps to solve the negative impact of clients with different graph structure data distributions and different degrees of training on model performance in federated graph learning, protect local data privacy and security, and at the same time train personalized models that are more in line with the actual data distribution of the client, thereby improving the training efficiency and accuracy of the federated learning model.

[0046] The specific steps of the present invention are as follows:

[0047] Step S1: construct explicit graph embedding based on the structural features of graph structure data, and decouple the node attribute features and structural features of the graph structure data for training.

[0048] Step S2: According to the differences in the distribution of structural features of the graph structure data, the clients with similar structural feature data are divided into soft clusters.

[0049] Step S3: weighted aggregation is performed on the model according to the training participation degree of different clients in the cluster and the model performance obtained by training to obtain a personalized iterative model.

[0050] In step S1, an explicit graph embedding is constructed based on the structural features of the graph structure data, and the node attribute features and structural features of the graph structure data are decoupled for training, which specifically includes the following steps:

[0051] Step S1A1: For the local graph structure data of the client, the structural feature embedding of the graph structure data based on node degree and random walk is explicitly constructed.

[0052] Step S1A2: Concatenate the two structural feature embeddings based on node degree and random walk according to a certain dimensional ratio to construct a comprehensive structural feature embedding.

[0053] Step S1A3: To address the heterogeneity of node attribute features of graph structure data owned by different clients, decoupled training is performed on the node attribute features of the original graph structure data and the comprehensive structural feature embedding through the model structure.

[0054] In step S1A1, the explicit construction calculation method of embedding structural features based on node degree and random walk is as follows:

[0055]

[0056] in, Represents the structural feature embedding of node v based on node degree and random walk, is an identity mapping, T = AD -1 is the random walk transfer matrix calculated by the binary adjacency matrix A and the pairwise angle matrix D. k1 and k2 are the dimensions that need to be constructed based on the node degree and the structural feature embedding of the random walk, respectively.

[0057] In step S1A3, the objective function of decoupling training the node attribute features and the comprehensive structural features of the original graph structure data through the model structure is as follows:

[0058]

[0059] Among them, φ n,i ,φ s,i They represent the local model parameters of the node attribute features and structural features extracted by the i-th client, It indicates the optimal solution of its parameters, D i represents the graph structure data set owned by the i-th client, D represents the total set of graph structure data owned by the clients participating in the training, Φ represents the global model obtained after global aggregation, N represents the total number of clients participating in the training, argmin(·) represents the inverse function of the minimum function, Represents the loss function of training.

[0060] In step S2, according to the different distribution of structural features of the graph structure data, the clients with similar structural feature data are divided into soft clusters, which specifically includes the following steps:

[0061] Step S2A1: To address the heterogeneity of structural features of graph structure data owned by different clients, a principal vector representing the distribution of structural features is calculated by performing high-dimensional matrix decomposition on the data matrix.

[0062] Step S2A2: Calculate the similarity matrix of the high-dimensional main points between the graph data structure distributions of different clients based on the main vectors of different clients, and evaluate the similarity of the structural feature distributions between the clients.

[0063] Step S2A3: Based on the high-dimensional protagonist similarity matrix, clients with similar structural feature distributions are divided into clusters to promote full learning of structural features that are beneficial to improving model accuracy among similar clients.

[0064] In step S2A1, by performing high-dimensional matrix decomposition on the data matrix, the formula for calculating the main vector representing the distribution of structural features is as follows:

[0065] U,s,V T =TSVD(e)

[0066] The present invention uses the truncated singular value decomposition algorithm (TSVD) for graph structure data to calculate the main vector that characterizes the distribution of structural features. Where s represents the main vector obtained, U, V T They represent the left matrix and the right matrix obtained by the TSVD algorithm, respectively. e represents the graph structure data set owned by the client. TSVD(·) represents the TSVD algorithm calculation function.

[0067] In step S2A2, the similarity matrix of the high-dimensional main body between the graph structure data structure distributions of different clients is calculated, and the method for evaluating the similarity of the structural feature distribution between each client is as follows:

[0068]

[0069] Among them, high-dimensional protagonist similarity i,j represents the high-dimensional protagonist similarity between the i-th client and the j-th client, s i and j denote the main vectors of the ith client and the jth client respectively, arccos(·) denotes the inverse cosine function, and |·| denotes the modulo operation of the vector.

[0070] In step S2A3, based on the high-dimensional protagonist similarity matrix, the method for dividing the clients with similar structural feature distribution into clusters is calculated as follows:

[0071]

[0072] In step S3, according to the training participation degree of different clients in the cluster and the model performance obtained by training, the model is weighted and aggregated to obtain a personalized iterative model, which specifically includes the following steps:

[0073] Step S3A1: Count the training participation degrees and model performance obtained by training of different clients in the same cluster, and calculate two aggregation weight operators based on the training participation degree and the model performance respectively.

[0074] Step S3A2: Comprehensively consider the two aggregation weight operators based on the degree of training participation and based on the model performance, calculate the aggregation weight of each client in the current clustering cluster, and perform weighted aggregation to obtain the personalized model in the current clustering cluster.

[0075] In step S3A1, the training participation degree and the model performance obtained by training of different clients in the same cluster are counted, and the formulas for calculating two aggregation weight operators based on the training participation degree and the model performance are as follows:

[0076]

[0077] in, Denotes the two aggregation weight operators based on the training participation degree and model performance of the i-th client, respectively. i and P i Respectively represent the amount of data owned by the client and the number of training participations, Acc i represents the accuracy of the model performance, τ represents the offset used to control the weight value range, and N′ represents the number of clients in the cluster.

[0078] In step S3A2, the two aggregation weight operators based on the degree of training participation and the model performance are comprehensively considered to calculate the aggregation weight of each client in the current clustering cluster, and the formula for the personalized model in the current clustering cluster is obtained by weighted aggregation as follows:

[0079] W=λ1×W p +λ2×W q ,st.λ1+λ2=1

[0080]

[0081] Where W represents the client's aggregation weight, λ1 and λ2 represent the proportional hyperparameters corresponding to the two aggregation weight operators based on training participation and model performance, respectively, and Φ and φ s They respectively represent the personalized model parameters obtained by aggregation and the local model parameters of the structural features extracted by the client.

[0082] The above are preferred embodiments of the present invention. Any changes made according to the technical solution of the present invention, as long as the resulting functions do not exceed the scope of the technical solution of the present invention, belong to the protection scope of the present invention.

Claims

1. A federated heterogeneous graph learning method based on soft clustering, characterized in that: The steps include: Step S1: construct explicit graph embedding based on the structural features of graph structure data, and decouple the node attribute features and structural features of the graph structure data for training; Step S2: According to the different distribution of structural features of the graph structure data, the clients with similar structural feature data are divided into soft clusters; Step S3: weighted aggregation is performed on the model according to the training participation degree of different clients in the cluster and the model performance obtained by training to obtain a personalized iterative model.

2. According to claim 1, a federated heterogeneous graph learning method based on soft clustering is characterized in that: In step S1, an explicit graph embedding is constructed based on the structural features of the graph structure data, and the node attribute features and structural features of the graph structure data are decoupled for training, which specifically includes the following steps: Step S1A1: for the local graph structure data of the client, explicitly construct the structural feature embedding of the graph structure data based on node degree and random walk respectively; Step S1A2, concatenating two structural feature embeddings based on node degree and random walk according to a certain dimensional ratio to construct a comprehensive structural feature embedding; Step S1A3: To address the heterogeneity of node attribute features of graph structure data owned by different clients, decoupled training is performed on the node attribute features of the original graph structure data and the comprehensive structural feature embedding through the model structure.

3. The method for learning a federated heterogeneous graph based on soft clustering according to claim 2, characterized in that: In step S1A1, the explicit construction calculation method of embedding structural features based on node degree and random walk is as follows: in, Represents the structural feature embedding of node v based on node degree and random walk, is an identity mapping, T = AD -1 is the random walk transfer matrix calculated by the binary adjacency matrix A and the pairwise angle matrix D. k1 and k2 are the dimensions that need to be constructed based on the node degree and the structural feature embedding of the random walk, respectively.

4. The method for learning a federated heterogeneous graph based on soft clustering according to claim 2, characterized in that: In step S1A3, the objective function of decoupling training the node attribute features and the comprehensive structural features of the original graph structure data through the model structure is as follows: Among them, φ n,i ,φ s,i They represent the local model parameters of the node attribute features and structural features extracted by the i-th client, It indicates the optimal solution of its parameters, D i represents the graph structure data set owned by the i-th client, D represents the total set of graph structure data owned by the clients participating in the training, Φ represents the global model obtained after global aggregation, N represents the total number of clients participating in the training, argmin(·) represents the inverse function of the minimum function, Represents the loss function of training.

5. The method for learning a federated heterogeneous graph based on soft clustering according to claim 1, characterized in that: In step S2, according to the different distribution of structural features of the graph structure data, the clients with similar structural feature data are divided into soft clusters, which specifically includes the following steps: Step S2A1: To solve the heterogeneity problem of structural features of graph structure data owned by different clients, high-dimensional matrix decomposition is performed on the data matrix to calculate the main vector representing the distribution of the structural features; Step S2A2: Calculate the similarity matrix of the high-dimensional main elements between the graph structure data structure distributions of different clients based on the main vectors of different clients, and evaluate the similarity of the structural feature distributions between the clients; Step S2A3: Based on the high-dimensional protagonist similarity matrix, clients with similar structural feature distributions are divided into clusters to promote full learning of structural features that are beneficial to improving model accuracy among similar clients.

6. The method for learning a federated heterogeneous graph based on soft clustering according to claim 5, characterized in that: In step S2A1, by performing high-dimensional matrix decomposition on the data matrix, the formula for calculating the main vector representing the distribution of structural features is as follows: U,s,V T =TSVD(e) The truncated singular value decomposition algorithm TSVD is used for graph structure data to calculate the main vector that represents the distribution of structural features; where s represents the main vector obtained, U, V T They represent the left matrix and the right matrix obtained by the TSVD algorithm, respectively. e represents the graph structure data set owned by the client. TSVD(·) represents the TSVD algorithm calculation function.

7. The method for learning a federated heterogeneous graph based on soft clustering according to claim 5, characterized in that: In step S2A2, the similarity matrix of the high-dimensional main body between the graph structure data structure distributions of different clients is calculated, and the method for evaluating the similarity of the structural feature distribution between each client is as follows: Among them, high-dimensional protagonist similarity i,j represents the high-dimensional protagonist similarity between the i-th client and the j-th client, s i and j denote the main vectors of the ith client and the jth client respectively, arccos(·) denotes the inverse cosine function, and |·| denotes the modulo operation of the vector.

8. The method for learning a federated heterogeneous graph based on soft clustering according to claim 5, characterized in that: In step S2A3, based on the high-dimensional protagonist similarity matrix, the method for dividing the clients with similar structural feature distribution into clusters is calculated as follows:

9. The method for learning a federated heterogeneous graph based on soft clustering according to claim 1, characterized in that: In step S3, according to the training participation degree of different clients in the cluster and the model performance obtained by training, the model is weighted and aggregated to obtain a personalized iterative model, which specifically includes the following steps: Step S3A1: Count the training participation degree and model performance obtained by training of different clients in the same cluster, and calculate two aggregation weight operators based on the training participation degree and the model performance respectively; Step S3A2: Comprehensively consider the two aggregation weight operators based on the degree of training participation and based on the model performance, calculate the aggregation weight of each client in the current clustering cluster, and perform weighted aggregation to obtain the personalized model in the current clustering cluster.

10. The method for learning a federated heterogeneous graph based on soft clustering according to claim 9, characterized in that: In step S3A1, the training participation degree and the model performance obtained by training of different clients in the same cluster are counted, and the formulas for calculating two aggregation weight operators based on the training participation degree and the model performance are as follows: in, Denotes the two aggregation weight operators based on the training participation degree and model performance of the i-th client, respectively. i and P i They represent the amount of data and number of training sessions of client i, Acc i represents the accuracy of the model performance, τ represents the offset used to control the weight range, and N′ represents the number of clients in the cluster; In step S3A2, the two aggregation weight operators based on the degree of training participation and the model performance are comprehensively considered to calculate the aggregation weight of each client in the current clustering cluster, and the formula for the personalized model in the current clustering cluster is obtained by weighted aggregation as follows: W=λ1×W p +λ2×W q ,st.λ1+λ2=1 Where W represents the client's aggregation weight, λ1 and λ2 represent the proportional hyperparameters corresponding to the two aggregation weight operators based on training participation and model performance, respectively, and Φ and φ s They respectively represent the personalized model parameters obtained by aggregation and the local model parameters of the structural features extracted by the client.

Images