Comparative longitudinal federal map learning method based on adaptive multi-kernel clustering

Abstract

The invention relates to a contrast longitudinal federal map learning method based on adaptive multi-kernel clustering, and the method comprises the following steps: S1, initializing a global model: initializing a global model gG through a server, and distributing the global model gG to all K clients; s2, local training: each client performs local training on the graph learning model by using the graph data held by the client, and model update is generated; and S3, parameter uploading: the K clients send the local graph learning model update theta k to the server. And S4, model aggregation: the server sends the aggregated model update theta global back to the clients, and the server aggregates the model parameter update of the K clients through weighted average to construct a global model. According to the method, multi-source graph knowledge can be integrated in a privacy-protected longitudinal federation environment, and the model is trained under the condition of no label based on a comparative learning mechanism. And self-adaptive multi-core graph clustering is integrated, so that the performance of the model is improved.

Description

Technical Field

[0001] The present invention relates to the field of privacy-preserving representation learning and processing of graph network data, and in particular to a comparative longitudinal federated graph learning method based on adaptive multi-core clustering. Background Art

[0002] As data privacy and security issues are gaining more and more attention, traditional centralized data processing and machine learning methods are exposing more and more drawbacks. Especially in the fields of medical care, finance and other fields involving sensitive information, how to make full use of multi-source data for effective model training while protecting data privacy has become a problem that needs to be solved urgently. Federated Learning (FL), as an emerging distributed machine learning method, realizes joint training of models without sharing original data through the collaboration between clients and servers, providing a feasible solution to this challenge.

[0003] On this basis, Vertical Federated Learning on Graphs (VFGL) further expands the application of federated learning and introduces it into the field of graph data processing, especially when data features cannot be shared between different data holders (clients). VFGL combines the advantages of federated learning and graph neural networks (GNNs), not only protecting the privacy of data held by clients, but also integrating data features from different sources, thereby improving the modeling ability of graph data. The core idea of ​​VFGL is to improve the overall performance of the global model through collaborative learning when each client holds the same node but different features. For example, in the scenario of user credit score prediction, a bank and Alipay each have information features about the user, such as name, work unit, age, and income, as well as the user's transaction relationship network. However, due to the requirements of privacy protection, the two institutions will not share the transaction relationship data they hold. It is difficult to accurately predict the user's credit score if only relying on the data of a single institution. Through vertical graph federated learning, it is possible to securely integrate multi-party knowledge without sharing their respective original transaction relationship data to achieve a comprehensive credit score for the user. In the VFGL framework, each client trains a local graph neural network model (GNN) based on the local graph data it holds. Subsequently, the server aggregates the graph neural network model parameters of all clients to update the server's global model parameters. After that, the server distributes the updated global model parameters to the clients, and the clients continue to train the local graph neural network models using the local graph data. Through iterative training of multiple clients and servers, VFGL can fully utilize the potential knowledge of multiple data sources without sharing the original data held by multiple institutions, and achieve more accurate model predictions. Although VFGL has achieved certain results, the effectiveness of existing methods depends largely on high-quality data.

[0004] In real-world scenarios, data often lacks label information, and obtaining this label information often requires a lot of manpower and material resources, which is costly. The lack of data labels makes it difficult to effectively learn data representations in vertical graph federated learning. Therefore, there is an urgent need to combine self-supervised learning methods with vertical graph federated learning, especially when dealing with unlabeled and scattered multi-source graph data. A simple solution is to adopt the contrastive learning strategy in self-supervised learning to encode the graph information of each data source. In recent years, contrastive learning has become an important self-supervised learning technique and has been widely used in many fields. The core idea of ​​contrastive learning is to distinguish similar and dissimilar data pairs by training models. By narrowing the distance between positive sample pairs (i.e., similar samples) and separating negative sample pairs (i.e., dissimilar samples), the model can effectively identify patterns and differences in the data in the absence of sample labels. This approach enhances the model's understanding and representation capabilities of complex data structures. Although there are some techniques that integrate contrastive learning into federated learning settings, these methods usually target image data in Euclidean space. In graph data, the structure composed of nodes and edges is more complex, and the relationships and patterns between nodes are also more complex. Therefore, existing methods are difficult to directly apply to representation learning of multi-source graph data.

[0005] The obtained graph data may contain noise due to the unreliability of the data source or human errors in the data collection process. The complex graph structure may amplify the impact of these noises on the model performance. The introduction of noise in the data may cause random deviations, which in turn blurs the originally well-defined classifications. This makes the model unable to make good linear distinctions between the data, and existing vertical graph federated learning techniques ignore this problem. Summary of the invention

[0007] In view of the defects of the prior art, the purpose of the present invention is to provide a comparative longitudinal federated graph learning method based on adaptive multi-core clustering. In order to achieve the above object, the technical solution adopted by the present invention is: a comparative longitudinal federated graph learning method based on adaptive multi-core clustering, comprising the following steps: S1: Global model initialization: The server initializes a global model g G , and distribute it to all K clients, using G k =(V k ,E k ), k = {1, 2, ..., K} to represent the graph data held by client k, where V k Represents a node set, with n being the number of nodes k To indicate that E kRepresents the set of edges between connected nodes. Each client’s graph encoder uses f k (·), k∈{1,2,…,k,…,K} represents; S2: Local training: Each client uses its own graph data to locally train the graph learning model and generate model updates; S3: Parameter upload: K clients update the local graph learning model parameters θ k Sent to the server. S4: Model aggregation: The server updates the aggregated model parameters θ global Sent back to the client, the server aggregates the model parameter updates of K clients by weighted average to build a global model, θ global The calculation formula is as follows: Among them, θ global represents the global model parameters, θ k represents the model parameters of the kth client, N is the total size of all client datasets, and N k is the size of the local dataset of the kth client.

[0008] Furthermore, step S2 includes the following sub-steps: S21: Graph data augmentation: augment the graph data G on client k k , G k Convert to and Two augmented graph views; S22: Encoding augmented graph view information based on graph convolutional network: The two augmented graph view information are encoded into the vector space through the graph convolutional network. The graph convolutional network uses a neighbor aggregation strategy to learn node representation. The aggregation rule of the lth graph convolutional layer is as follows: in, An adjacency matrix with self-loops representing graph data, express The degree matrix, W (l) is the lth layer parameter of the graph encoder, σ is the activation function, H (l) is the output of the lth graph convolutional layer; By f k (·) is used to represent the graph convolutional network encoding function and nonlinear projection function in client k. and Input into the shared encoder f k (·), we get the corresponding two node representation matrices, which are expressed as and S23: Normalization of node representations in augmented graph view: The two node representation matrices U are normalized by L2 normalization L2(·) k and V k Normalize and get the corresponding two normalized nodes U k-norm and V k-norm , the normalized calculation formula is as follows: Where g(·) represents the normalization function; S24: Kernel function mapping: These two normalized nodes (U k-norm and V k-norm ) Embedded into Hilbert space, two augmented graph views and The overall kernel mapping at client k can be expressed as: Therefore, the entire kernel function can be expressed as: S25: Graph contrastive learning in Hilbert space: by distinguishing the representation of the same node from the representation of other nodes in these two different augmented graph views.

[0009] Further, step S25 includes the following sub-steps: The nodes with the same index i in the two augmented graph views are v i and u i , and its kernel mapping result is obtained by φ(v i ) and φ(u i ), these two results are positive sample pairs, and the kernel mapping results of the nodes other than index i in these two augmented graph views are negative sample pairs. Therefore, node u i The loss function is expressed as: Among them, I [j≠i] ∈{0,1} represents the indicator function, when j≠i, the value of the indicator function is 1, τ represents the temperature parameter, describes the node v in the kernel space i and u i The similarity between them, the second and third terms in the denominator are negative sample pairs. Since the two augmented graph views are symmetrical, the loss function of the other view is expressed as The loss function can be expressed as: in, The subscript con represents the abbreviation of contrastive, and the superscript k represents the kth client; By minimizing the loss function in formula (9), the learnable parameters θ of the model in client k are optimized k .

[0010] Further, step S4 includes the following sub-steps: Adaptive multi-core clustering is introduced in the federated learning framework. The multi-core clustering is expressed as follows: Among them I n represents the identity matrix of size d, Q is the hard label, or the soft label obtained by optimization, K p represents the pth kernel matrix, m represents the number of kernels, α p represents the weight of the p-th kernel matrix, The kernel is selected by an adaptive optimization method. When the selected M kernels are obtained, the overall optimization goal of the federated learning framework is expressed as follows: Among them, the first term represents the loss function of contrast kernel learning, the second term represents multi-kernel clustering, and the parameter λ is used to balance the relative importance of these two terms for the entire model. p Set to 1 / M.

[0011] Beneficial effects: This application proposes a comparative longitudinal federated graph learning method with adaptive multi-core clustering. Under the federated learning framework, based on the comparative learning strategy and multi-core clustering method, it effectively integrates multi-source graph data information scattered on multiple clients and collaboratively trains a global model.

[0012] This application can not only integrate multi-source graph knowledge in a privacy-preserving vertical federation environment, but also train models without labels based on a contrastive learning mechanism. It also integrates adaptive multi-core graph clustering to improve the performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 A schematic diagram of the structure of the present invention;

[0015] Figure 2 Schematic diagram of the federated learning training adopted by the present invention. DETAILED DESCRIPTION

[0017] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0018] Federated Learning (FL) is a distributed machine learning method that allows multiple clients (such as devices or organizations) to collaboratively train a global model without sharing their local data. Federated learning protects data privacy and reduces communication overhead by updating the model locally on each client and aggregating these updates on a central server, thereby achieving joint training of the model.

[0019] There are two main components in federated learning: client and server. Client: A device or server that has data and is capable of local computation. The client trains a local model using its own data and sends model updates to the server. Server: Responsible for aggregating model updates from various clients and generating a global model. The server does not need to access the client's original data.

[0020] The workflow of federated learning is as follows: Initialization: The server initializes the global model and sends it to all clients. Local training: Each client trains the model locally using its own data, generating model updates. Parameter upload: Clients send their model updates to the server. Model aggregation: The server aggregates these updates to improve the global model. The server sends the aggregated model updates back to the client.

[0021] In federated learning, assume that there are K (k∈{1,2,…,k,…,K}) clients, and each client k has a local dataset D k Client k uses its local dataset D k Train its local model g k , and update the model parameters. After all K clients update locally, they send the updated model parameters back to the central server, which aggregates these model parameters to update the global model g G The process is continuously iterated until the global model converges to a desired accuracy level or the predetermined communication round T is reached.

[0022] The optimization goal of federated learning can be expressed as:

[0023]

[0024] Where L(·) represents the loss function, g G Represents the global model.

[0025] The training process of federated learning:

[0026] (1) Global model initialization:

[0027] The central server initializes a global model g G , and distribute it to all K clients.

[0028] (2) Local model training and parameter update:

[0029] In each round of communication t, each client k uses its local dataset D k Train the model and update the model parameters. Assume that the local loss function is l(g k ;D k ), then the local update process of client k in round t can be expressed as:

[0030]

[0031] Where η is the learning rate, Based on the local dataset D k Compute the gradient of the loss function.

[0032] (3) Model parameter aggregation:

[0033] After all clients update locally, they send the updated model parameters back to the central server, which aggregates these model parameters to update the global model. A common aggregation method is a simple weighted average (for example, the FedAvg algorithm):

[0034]

[0035] (4) Iteration process:

[0036] Repeat steps (2) and (3) until the global model converges or the predetermined communication round T is reached.

[0037] The present invention uses the symbol G k =(V k ,E k ), k = {1, 2, ..., K} to represent the local map data held by client k, where V k Represents a node set, with the number of nodes being n k To indicate that E k Represents the edge set between connected nodes. The attribute information of the node is represented by the matrix , and F represents the attribute dimension of the node. The graph data held by different clients describes different types of interaction feature information between nodes. The local model of each client uses g k (·), k∈{1,2,…,k,…,K} indicates that the graph encoding model on the client will be based on the graph data G it holds k Therefore, the optimization goal of the graph federated learning model can be expressed as:

[0038]

[0039] where g GRepresents the global model of the central server. The schematic diagram of the federated learning training used in the present invention is as follows Figure 1 As shown:

[0040] This application designs a contrastive longitudinal federated graph learning method based on adaptive multi-core clustering, aiming to integrate contrastive learning and adaptive multi-core clustering into the longitudinal federated graph learning framework. The overall structure of the designed model is as follows Figure 1 shown.

[0041] First, the global model is initialized. The central server initializes a global model g G And distribute it to all K clients. Secondly, the client model training. For client k, first augment the graph data G on client k k , G k The graph encoder then encodes the augmented two graph view information into node representation matrices, using U k and V k These two matrices U k and V k The graph structure and node attribute characteristics of the two augmented graph view information are retained. Then, based on the kernel function φ(·), the two matrices U k and V k The information is projected into the Hilbert space. In this Hilbert space, the node representation features are learned based on the contrastive learning strategy, so that the distances between positive samples are close to each other and the distances between negative samples are far away from each other, so as to learn discriminative node representations. In order to further improve the model performance and make the learned node representations more discriminative, an adaptive multi-core clustering strategy is introduced in the federated learning process for model training and parameter update. Finally, a global model update is performed. The central server aggregates the model parameters of all clients to update the global model. Afterwards, the server sends the updated global model parameters to each client for the next round of model training. In this iterative process, the client and the global server are continuously trained, so as to integrate the information of multi-source graph data under the premise of protecting data privacy and obtain a jointly trained global graph model.

[0042] The present application provides a comparative longitudinal federated graph learning method based on adaptive multi-core clustering, comprising the following steps: S1: Global model initialization: The server initializes a global model g G , and distribute it to all K clients, using G k =(V k ,E k ), k = {1, 2, ..., K} to represent the graph data held by client k, where V k Represents a node set, with the number of nodes being nk To indicate that E k Represents the set of edges between connected nodes. Each client’s graph encoder uses f k (·), k∈{1,2,…,k,…,K} represents; S2: Local training: Each client uses its own graph data to locally train the graph learning model and generate model updates; S3: Parameter upload: K clients update the local graph learning model to θ k Sent to the server. S4: Model aggregation: The server updates the aggregated model parameters θ global Sent back to the client, the server aggregates the model parameter updates of K clients by weighted average to build a global model, θ global The calculation formula is as follows: Among them, θ global represents the global model parameters, θ k represents the model parameters of the kth client, N is the total size of all client datasets, and N k is the size of the local dataset of the kth client.

[0043] Wherein, step S2 includes the following sub-steps: S21: Graph data augmentation: augment the graph data G on client k k , G k Convert to and Two augmented graph views; The goal of data augmentation is to create novel and plausible data by transforming existing data while preserving the original semantics. Therefore, augmented information can be used to perform graph contrastive learning without label information.

[0044] S22: After obtaining the two augmented graph views, the present invention eliminates the dimensionality difference by encoding the two augmented graph view information into a unified vector space. Encoding augmented graph view information based on graph convolutional network: The two augmented graph view information is encoded into the vector space through the graph convolutional network. The graph convolutional network uses a neighbor aggregation strategy to learn node representations. The aggregation rule of the lth graph convolutional layer is as follows: in, An adjacency matrix with self-loops representing graph data, express The degree matrix, W (l) is the lth layer parameter of the graph encoder, σ is the activation function, H (l) is the output of the lth graph convolutional layer.

[0045] In practical applications, a shared nonlinear projection function is usually applied to the output of the GCN encoder to enhance the capacity of the contrastive learning model. k (·) is used to represent the graph convolutional network encoding function and nonlinear projection function in client k. and Input into the shared encoder f k (·), we get the corresponding two node representation matrices, which are expressed as and

[0046] S23: Normalization of node representations in augmented graph view: The two node representation matrices U are normalized by L2 normalization L2(·) k and V k Normalize and get the corresponding two normalized nodes U k-norm and V k-norm , the normalized calculation formula is as follows: Here, g(·) represents the normalization function.

[0047] Normalization can prevent numerical stability problems that occur when computing the inner product of the kernel function due to large differences in the range of eigenvalues.

[0048] S24: Kernel function mapping: The commonly used graph contrastive learning method is to minimize the distance between positive sample pairs and maximize the distance between negative sample pairs through contrastive learning functions. However, the original data may be noisy when collected, and the complex graph structure will amplify the noise effect, especially when f is used. k (·) after encoding. If you use U k-norm and V k-norm Contrastive learning may lead to linear inseparability of data. Based on the advantages of multi-core learning strategy, the present invention introduces multi-core learning into the graph contrast learning process. k-norm and V k-norm ) is embedded and projected into Hilbert space In the two augmented graph views and The overall kernel mapping at client k can be expressed as: Therefore, the entire kernel function can be expressed as:

[0049] S25: Graph contrast learning in Hilbert space: In the above steps, the present invention projects the information of the two augmented graph views into a unified Hilbert space Now, we will perform contrastive learning, which is to optimize the mapping function and the graph encoder by maximizing the consistency between the two augmented graph views. This is done by distinguishing the representation of the same node from the representation of other nodes in the two different augmented graph views.

[0050] Specifically, step S25 includes the following sub-steps: The nodes with the same index i in the two augmented graph views are v i and u i , and its kernel mapping result is obtained by φ(v i ) and φ(u i ), these two results are positive sample pairs, and the kernel mapping results of the nodes other than index i in these two augmented graph views are negative sample pairs. Therefore, node u i The loss function is expressed as: Among them, I [j≠i] ∈{0,1} represents the indicator function, when j≠i, the value of the indicator function is 1, τ represents the temperature parameter, describes the node v in the kernel space i and u i The similarity between them, the second and third terms in the denominator are negative sample pairs. Since the two augmented graph views are symmetrical, the loss function of the other view is expressed as The loss function can be expressed as: in, The subscript con represents the abbreviation of contrastive, and the superscript k represents the kth client; By minimizing the loss function in formula (9), the learnable parameters θ of the model in client k are optimized k .

[0051] Wherein, step S4 includes the following sub-steps: The global model g is obtained by minimizing the local loss function (9) of each client k in each iteration and aggregating these updates on the central server. G To further improve the performance, adaptive multi-core clustering is introduced in the federated learning framework. The multi-core clustering is expressed as follows: Among them I n represents the identity matrix of size d, Q is the hard label, or the soft label obtained by optimization, K prepresents the pth kernel matrix, m represents the number of kernels, α p represents the weight of the p-th kernel matrix, Existing multi-core clustering methods usually assume that the kernel matrix is ​​pre-computed and the matrix is ​​fixed during the entire clustering iteration process. In the present invention, the kernel is selected by an adaptive optimization method. When the selected M kernels are obtained, the overall optimization goal of the federated learning framework is expressed as follows: Among them, the first term represents the loss function of contrast kernel learning, the second term represents multi-kernel clustering, and the parameter λ is used to balance the relative importance of these two terms for the entire model. p Set to 1 / M. The local client and the central server jointly optimize the global model g through continuous iterative training by minimizing the loss function in formula (11) G This is achieved by minimizing each client's local loss function at each iteration and aggregating these updates at the central server.

[0052] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technician familiar with this profession can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solution of the present invention still fall within the scope of the technical solution of the present invention.

Claims

1. A comparative longitudinal federated graph learning method based on adaptive multi-core clustering, characterized in that: The following steps are involved: S1: Global model initialization: The server initializes a global model g G , and distribute it to all K clients, using G k =(V k ,E k ), k = {1, 2, ..., K} to represent the graph data held by client k, where V k Represents a node set, with the number of nodes being n k To indicate that E k Represents the set of edges between connected nodes. Each client’s graph encoder uses f k (·), k∈{1,2,…,k,…,K} represents; S2: Local training: Each client uses its own graph data to locally train the graph learning model and generate model updates; S3: Parameter upload: K clients update the local graph learning model to θ k Sent to the server. S4: Model aggregation: The server updates the aggregated model parameters θ global Sent back to the client, the server aggregates the model parameter updates of K clients by weighted average to build a global model, θ global The calculation formula is as follows: Among them, θ global represents the global model parameters, θ k represents the model parameters of the kth client, N is the total size of all client datasets, and N k is the size of the local dataset of the kth client.

2. According to claim 1, a comparative longitudinal federated graph learning method based on adaptive multi-core clustering is characterized in that: Step S2 includes the following sub-steps: S21: Graph data augmentation: augment the graph data G on client k k , G k Convert to and Two augmented graph views; S22: Encoding augmented graph view information based on graph convolutional network: The two augmented graph view information are encoded into the vector space through the graph convolutional network. The graph convolutional network uses a neighbor aggregation strategy to learn node representation. The aggregation rule of the lth graph convolutional layer is as follows: in, An adjacency matrix with self-loops representing graph data, represents the degree matrix of A~, W (l) is the lth layer parameter of the graph encoder, σ is the activation function, H (l) is the output of the lth graph convolutional layer; By f k (·) is used to represent the graph convolutional network encoding function and nonlinear projection function in client k. and Input into the shared encoder f k (·), we get the corresponding two node representation matrices, which are expressed as and S23: Normalization of node representations in augmented graph view: The two node representation matrices U are normalized by L2 normalization L2(·) k and V k Normalize and get the corresponding two normalized nodes U k-norm and V k-norm , the normalized calculation formula is as follows: Where g(·) represents the normalization function; S24: Kernel function mapping: These two normalized nodes (U k-norm and V k-norm ) Embedded into Hilbert space, two augmented graph views and The overall kernel mapping at client k can be expressed as: Therefore, the entire kernel function can be expressed as: S25: Graph contrastive learning in Hilbert space: by distinguishing the representation of the same node from the representation of other nodes in these two different augmented graph views.

3. The method for learning a comparative longitudinal federated graph based on adaptive multi-core clustering according to claim 2, characterized in that: Step S25 includes the following sub-steps: The nodes with the same index i in the two augmented graph views are v i and u i , and its kernel mapping result is obtained by φ(v i ) and φ(u i ), these two results are positive sample pairs, and the kernel mapping results of the nodes other than index i in these two augmented graph views are negative sample pairs. Therefore, node u i The loss function is expressed as: Among them, I [j≠i] ∈{0,1} represents the indicator function, when j≠i, the value of the indicator function is 1, τ represents the temperature parameter, describes the node v in the kernel space i and u i The similarity between them, the second and third terms in the denominator are negative sample pairs, and since the two augmented graph views are symmetrical, the loss function of the other view is expressed as l k (v i ,u i ), the loss function can be expressed as: in, The subscript con represents the abbreviation of contrastive, and the superscript k represents the kth client; By minimizing the loss function in formula (9), the learnable parameters θ of the model in client k are optimized k .

4. The method for learning a comparative longitudinal federated graph based on adaptive multi-core clustering according to claim 3, characterized in that: Step S4 includes the following sub-steps: Adaptive multi-core clustering is introduced in the federated learning framework. The multi-core clustering is expressed as follows: Among them I n represents the identity matrix of size d, Q is the hard label, or the soft label obtained by optimization, K p represents the pth kernel matrix, m represents the number of kernels, α p represents the weight of the p-th kernel matrix, The adaptive multi-core method is used to select the core. When the selected M cores are obtained, the overall optimization goal of the federated learning framework is expressed as follows: Among them, the first term represents the loss function of contrast kernel learning, the second term represents multi-kernel clustering, and the parameter λ is used to balance the relative importance of these two terms for the entire model. p Set to 1 / M.

Images