Transform-based federal map data mining method and system

Abstract

The invention provides a transform-based federal map data mining method and system, and the method specifically comprises the following steps: S1, carrying out the mining of a node relation in a client through a simplified transform module, obtaining the high-dimensional representation of all nodes in the client, and carrying out the mining of the high-dimensional representation of all nodes in the client; the high-dimensional representation and the representation output by the GCN layer in the model of the client are input into a full connection layer together for a classification prediction task, a label classification task is obtained, and the negative correlation problem in the attention aggregation process is solved; and S2, according to a label classification task, dividing data in different clients into parameters of a personalized layer and parameters of a common layer, fixing the parameters of the personalized layer without aggregation, continuously uploading the parameters of the common layer for aggregation, and iterating to obtain a convergent federal training model.

Description

Technical Field

[0001] The present invention relates to the technical field of federated graph data mining methods, and specifically relates to a federated graph data mining method and system based on transformer. Background Art

[0002] In the real world, graph data is usually stored in multiple data holders. Due to privacy policies and relevant laws and regulations, the data of different users cannot be directly shared, resulting in a serious problem of data fragmentation. Federated graph learning can train graph data distributively on different machines, breaking the dilemma of data islands and solving the problem of cross-institutional data sharing. However, in the real world, the data held by different institutions has strong heterogeneity, and there are significant differences in the data volume size, feature distribution, labels, and topological structures between subgraphs.

[0003] FedAvg is a simple but effective federated learning optimization algorithm that aggregates the parameters of each client in a weighted manner and is still applied in many federated graph learning scenarios. FedProx controls the deviation between the client weights and the global weights by adding constraints to the client parameters. Although these methods have achieved certain results, they cannot deeply process the topological relationships of graphs, resulting in unsatisfactory model prediction effects. To further mine the graph data relationships between clients, FedSage+ uses the node information in subgraphs to infer the nodes of other subgraphs to make up for the missing edges between subgraphs; Fedego uses a hybrid method to mix and align the data of each client and then trains it by the server; FED-PUB uses similarity scores to calculate the similarity between clients and adaptively transfers parameters to the server in a learnable way. Although these works can mine the association relationships of graphs to a certain extent, they still have great drawbacks. The optimal number of layers of the GNN model is usually limited to no more than 3 layers, resulting in the model being able to capture at most 3-hop relationships. However, in the real world, the topological relationship length of the input graph usually exceeds 3. Due to the inherent constraints of the graph structure, it is difficult for GNN-based models to capture this long-distance information. These methods cannot fully mine the relationships between non-neighbor nodes within the client. The long-distance node dependencies still implicitly contain rich implicit relationships, which are of great significance for improving the effect of model joint training. Moreover, due to the heterogeneity of data between clients, ordinary parameter weighted averaging is difficult to adapt to the local training tasks of each client. The existing methods do not make full use of the data information of other clients and are likely to have a serious impact on local training tasks, resulting in the loss of training accuracy.

[0004] Therefore, the technical problem that needs to be solved urgently at present is: how to mine the neighbor node relationships within the client and solve the possible data fragmentation problem in the process of cross-institutional data sharing. Summary of the Invention

[0005] The present invention is made to solve the above problems, and aims to provide a federated graph data mining method and system based on transformer.

[0006] The present invention provides a federated graph data mining method based on transformer, which has the following features: including: Step S1, using a simplified transformer module to mine the node relationships within the client, obtaining the high-dimensional representations of all nodes within the client, jointly inputting the high-dimensional representations and the representations obtained from the output of the GCN layer in the client's model into a fully connected layer to perform a classification prediction task, obtaining a label classification task, and solving the negative correlation problem in the attention aggregation process; and Step S2, according to the label classification task, dividing the data in different clients into the parameters of the personalized layer and the parameters of the common layer, fixing the parameters of the personalized layer without aggregation, continuously uploading the parameters of the common layer for aggregation, and iteratively obtaining a converged federated training model.

[0007] In the federated graph data mining method based on transformer provided by the present invention, it may also have the following features: Among them, GCN is a classic model in GNN. The core operation mechanism of GNN is message passing, where a node sends messages to its neighbors and receives aggregated messages from its neighbors. In graph representation learning, a large graph is defined as G=(V, E), where the set of nodes is V={v1, v2,..., v n}, and the input node feature matrix X can be expressed as X={x1, x2,..., x n}, the adjacency matrix between nodes is A, and A contains the self-loops of the nodes. GCN updates its own representation by aggregating the information of adjacent nodes, and its aggregation update formula can be summarized as:

[0008]

[0009] Among them, σ represents the activation function, represents the hidden embedding of node u in the l+1 layer, N u represents the neighbors of node u, e i,u represents the edge between node u and node i, AGG(·) represents the aggregation process of the graph neural network, which can specifically be the maximum value method, the minimum value method or the average method, and f(·) represents the message passing mechanism of the graph neural network.

[0010] In the federated graph data mining method based on transformer provided by the present invention, it may also have the following features: Among them, in Step S2, it is defined that a total of M clients participate in the training, and the set is C={C1,..., C M}, the data of each client is heterogeneous, differing in scale, node characteristics, and topological structure. The global optimization objective of federated learning is to minimize the global loss, and the formula is as follows:

[0011]

[0012] where d i is the size of the data of client i, D is the size of the total data of all clients, L i (θ i ) is the optimization objective of the loss function for each client individually, {θ i} i＝1 M represents the personalized parameters of the M clients, and F i (v; θ i ) represents the local model of each client, and l(F i (v; θ i )y) represents the cross-entropy loss used for node classification.

[0013] In the federated graph data mining method based on transformer provided by the present invention, it can also have the following characteristics: Among them, the federated graph data mining method based on transformer further includes the following validity verification method: Step S3, verify on five datasets, and the comparison methods adopted include: classical aggregation methods of federated learning: FedAvg and FedProx, personalized federated algorithm: FedPer, federated graph learning algorithm FED-PUB, and the public datasets are respectively: three citation datasets: Cora, CiteSeer, PubMed; two Amazon shopping datasets: Amazon-Computer and Amazon-Photo. The classification of the training set, validation set, and test set adopts a splitting method of 20%, 40%, and 40%, and the data scenario is set in a scenario with uneven node scales. In this scenario, the number of nodes of different clients varies greatly, and the number of nodes of some clients is more than ten times that of other clients, and the number of edges is more than 100 times. Different datasets can prove the effectiveness of this method.

[0014] In the federated graph data mining method based on transformer provided by the present invention, it can also have the following characteristics: Among them, step S1 specifically includes the following sub-steps: Step S1-1, use a linear fully connected layer to process the input graph data to improve the representation ability of linear attention, and map the input matrix E to a high-dimensional embedding space to obtain the embedding matrix E 0 , the embedding matrix E 0The output obtained after the first layer of the input transformer can, with the iteration of the number of layers, obtain E 1 ; Step S1-2, simplify the multi-head attention mechanism using the single-head attention mechanism. The standard attention calculation formula is as follows:

[0015]

[0016] Q = f Q (E l ),

[0017] K = f K (E l ),

[0018] V = f V (E l )

[0019] where f Q , f K and f V represent three-layer linear fully connected networks, d represents the dimension of the embedding matrix. The calculation process of attention is divided into three parts: First, it is necessary to calculate the correlation scores between the current node and all other nodes in the data, and obtain the association degree between the previous node and all other nodes in the data through these scores; Subsequently, use the correlation scores to calculate the attention weights, and use the Softmax function to normalize the correlation scores into a probability distribution to ensure that the weights sum to 1 and reflect the relative importance of each node in the entire node sequence; Finally, perform a weighted sum of the representations of each node in the sequence according to the attention weights to obtain the output representation at the current position. The model dynamically generates the node representation of the current node according to the importance of each position in the sequence. The node representation is subsequently weighted with the GCN representation to obtain the total output of the graphtransformer; Step S1-3, use the L2 norm to normalize the Q and K matrices to construct a transformer that can be stably trained:

[0020]

[0021] where |Q| L2 and represent the L2 regularization module; Step S1-4, adopt the idea of removing Softmax, and directly calculate the product of the matrix K and V through the multiplication commutative law to calculate the attention with a time complexity of O(N), ensure the non-negativity of the attention matrix, and the simplified single-layer attention calculation process is as follows:

[0022]

[0023] where 1 is a vector of all 1s in N dimensions, and α is the weight controlling the proportion between different layers of attention, that is, the residual weight. and represents the mapping kernel function, and the specific calculation formula is as follows:

[0024]

[0025] In step S1-5, a simple two-layer GCN module is used to aggregate the local neighbor information of the nodes, and the result output by the GCN layer is concatenated with the result output by the attention calculation to form a new feature matrix, which is then input into the linear classification layer to perform the label classification task. The formula is as follows: Y out = f y ((1 - β)E l + βGCN(E, A)) where GCE(E, A)) represents the GCN network layer, and f y represents the classification function, and β is the weight of the output result of the GCN and the output result of the attention layer.

[0026] In the federated graph data mining method based on transformer provided by the present invention, it may also have the following features: Among them, step S2 specifically includes the following sub-steps: Step S2-1, the parameters of the model are divided into personalized layer parameters and common layer parameters. The parameters of the personalized layer are W P , and the parameters of the common layer are W B , then the formula for the label classification task can be expressed as:

[0027]

[0028] where j represents the j-th client; Step S2-2, the data of the client first passes through the common layer, then through the personalized layer, and finally obtains the result of the label classification task. During the federated aggregation process, the client fixes the parameters of the personalized layer without aggregation and only uploads the parameters of the common layer. Through the parameter aggregation process of the federated model server, the weighted average parameters are distributed to the common layers of each client, while the personalized layer always retains the locally trained parameters without uploading; Step S2-3, continuously iterate to complete step S2-2 until the federated training model finally converges.

[0029] The present invention provides a federated graph data mining system based on Transformer, which has the following features: including a node relationship mining module that uses a simplified Transformer module to mine the node relationships within a client, obtains high-dimensional representations of all nodes within the client, and jointly inputs the high-dimensional representations and the representations obtained from the output of the GCN layer of the client's model into a fully connected layer to perform a classification prediction task, obtaining a label classification task, and solving the negative correlation problem in the attention aggregation process; and an aggregation module that, according to the label classification task, divides the data in different clients into the parameters of the personalized layer and the parameters of the common layer, fixes the parameters of the personalized layer without aggregation, continuously uploads the parameters of the common layer for aggregation, and iteratively obtains a converged federated training model.

[0030] In the federated graph data mining system based on Transformer provided by the present invention, it may also have the following features: Among them, a test module validates on five datasets, and the comparison methods adopted include: classical aggregation methods for federated learning: FedAvg and FedProx, personalized federated algorithm: FedPer, federated graph learning algorithm FED-PUB, and the public datasets are respectively: three citation datasets: Cora, CiteSeer, PubMed; two Amazon shopping datasets: Amazon-Computer and Amazon-Photo. The classification of the training set, validation set, and test set adopts a splitting method of 20%, 40%, and 40%, and the data scenario is set in a scenario with uneven node scales. In this scenario, the number of nodes in different clients varies greatly, and the number of nodes in some clients is more than ten times that of other clients, and the number of edges is more than 100 times. Different datasets can prove the effectiveness of this method.

[0031] Functions and effects of the invention

[0032] According to the federated graph data mining method and system based on Transformer involved in the present invention, in order to improve the training effect of federated learning under the condition of heterogeneous data in different clients and ensure the stability of model training, this paper proposes a federated graph data mining method based on Transformer - FedGformer. By using the attention mechanism of Transformer, it realizes the mining of all node relationships within the client and improves the utilization ability of node information. At the same time, through a personalized parameter aggregation method, it realizes the paradigm of personalized training of each client's model. Compared with the previous algorithms, FedGformer has greatly improved the accuracy of node classification. Brief description of the drawings

[0033] Figure 1It is the schematic diagram of the transformer-based federated graph data mining method in the embodiments of the present invention. Detailed implementation manners

[0034] In order to make the technical means, creative features, achieved purposes and effects realized by the present invention easy to understand, the following embodiments will specifically elaborate on the transformer-based federated graph data mining method of the present invention in conjunction with the accompanying drawings.

[0035] The present invention proposes a transformer-based federated graph data mining method (FedGformer) and its system.

[0036] Among them, the architecture of FedGformer is divided into two parts: the client combines a simplified Transformer mechanism with a graph convolutional network GCN, and the server uses a method of personalized cross-aggregation of parameters for distribution.

[0037] The data is first processed through a linear layer to extract features, and then input into a normalization layer for L2 normalization. The processed features are used to calculate the correlation between nodes and information aggregation in the GCN layer. Subsequently, the output is concatenated to form a new feature matrix and input into the classification layer for node prediction. This way of capturing long-distance node dependencies solves the defect of insufficient mining of non-neighbor node relationships within the client, can make full use of the rich implicit dependencies of distant nodes, and improves the stability and accuracy of joint training. In addition, FedGformer divides the parameter aggregation process into a personalized layer and a common layer, and aggregates and receives parameters from the server layer by layer. It solves the problem that the personalized information of the client cannot be fully utilized during the multi-institution joint training process and improves the quality of the federated training task.

[0038] Background and definition of the problem:

[0039] Graph representation learning:

[0040] Define a large graph as G=(V, E), where the set of nodes is V={v1, v2,..., v n}, and the input node feature matrix X can be expressed as X={x1, x2,..., x n}, and the adjacency matrix between nodes is A, and A includes the self-loop of the nodes. In graph representation learning, the core operation mechanism of GNN is message passing, where a node sends messages to its neighbors and receives aggregated messages from its neighbors. GCN is a classic model in GNN, which updates its own representation by aggregating the information of adjacent nodes, and its aggregation update formula can be summarized as:

[0041]

[0042] Among them, σ represents the activation function, represents the hidden embedding of node u in the (l + 1)-th layer, and N u represents the neighbors of node u, and e i,u represents the edge between node u and node i. AGG(·) represents the aggregation process of the graph neural network, which can specifically be the maximum method, the minimum method, or the average method. f(·) represents the message passing mechanism of the graph neural network.

[0043] Federated graph learning:

[0044] The scope of the present invention is the node classification task of federated graph learning, which is used to predict the labels of different nodes and distinguish the categories of nodes. In the training of federated graph learning, it is defined that a total of M clients participate in the training, and the set is C = {C1,..., C M}, and the data of each client is heterogeneous, different in scale, node features, and topological structure. The global optimization objective of federated learning is to minimize the global loss, and the formula is as follows:

[0045]

[0046] where d i is the data volume size of client i, D is the total data volume size of all clients, L i (θ i ) is the optimization objective of the loss function for each client individually, and {θ i} i＝1 M represents the personalized parameters of the M clients themselves, and F i (v; θ i ) represents the local model of each client, and l(F i (v; θ i )y) represents the cross-entropy loss used for node classification. The task of federated graph learning node classification is to predict the unknown node information based on the existing node features, partially known node labels, and the adjacency matrix composed of the relationships between nodes, and optimize the prediction accuracy through the loss function.

[0047] Figure 1 is the structural schematic diagram of the federated graph data mining method based on transformer in the embodiment of the present invention.

[0048] As Figure 1 shown, the federated graph data mining method (FedGFormer) based on transformer in this embodiment specifically includes the following steps:

[0049] Step S1, use a simplified Transformer module to mine the node relationships within the client, obtain the high-dimensional representations of all nodes within the client, and jointly input the high-dimensional representations and the representations obtained from the output of the GCN layer of the model into a fully connected layer to perform a classification prediction task, obtaining a label classification task, thereby solving the negative correlation problem in the attention aggregation process. Among them, the fully connected layer is a classification prediction layer.

[0050] Step S1-1, use a linear fully connected layer to process the input graph data to improve the representation ability of linear attention. By mapping the input matrix E to a high-dimensional embedding space, the embedding matrix E is obtained. 0 。

[0051] Step S1-2, since the layer attention model can be approximately approximated by a single-layer attention model, therefore, according to the energy-constrained graph diffusion model, use the single-head attention mechanism to simplify the multi-head attention mechanism. The standard attention calculation formula is as follows:

[0052]

[0053] Q = f Q (E l ),

[0054] K = f K (E l ),

[0055] V = f V (E l )

[0056] where f Q , F K and f V represent three-layer linear fully connected networks, d represents the dimension of the embedding matrix, and the attention calculation process is divided into three parts:

[0057] First, it is necessary to calculate the correlation scores between the current node and all other nodes in the data, and obtain the association degree between the previous node and all other nodes in the data through these scores.

[0058] Subsequently, use the correlation scores to calculate the attention weights, and use the Softmax function to normalize the correlation scores into a probability distribution to ensure that the weights sum to 1 and reflect the relative importance of each node in the entire node sequence.

[0059] Finally, the representations of each node in the sequence are weighted and summed according to the attention weights to obtain the output representation of the current position. In this way, the model can dynamically generate a representation of the current node position based on the importance of each position in the sequence. The advantage of the self-attention mechanism is that it can establish global semantic associations between nodes at different positions and can adaptively learn the dependencies between different node positions.

[0060] In step S1-3, the existing transformer will become unstable during training as the attention logits grow. To build a transformer that can be stably trained, we use the L2 norm to normalize the Q and K matrices:

[0061]

[0062] Among them, |Q| L2 and Represents the L2 regularization module.

[0063] Step S1-4, in the attention calculation process, although Softmax attention can have provable expressiveness, it is O(N 2 ) will greatly increase the computational efficiency. Therefore, we adopt the idea of removing Softmax and eliminating the nonlinear calculation process. We directly calculate the product of matrix K and V through the multiplication commutative law, and realize the attention calculation with O(N) time complexity instead of O(N) 2 However, if Softmax is directly removed from the attention calculation process, the inner product of Q and K cannot be guaranteed to be non-negative. Relevant research has shown that the non-negativity of the attention matrix is very important in attention calculation. Therefore, a mapping kernel function is used to ensure the non-negativity of the inner product. The simplified single-layer attention calculation process is as follows:

[0064]

[0065] Among them, 1 is an N-dimensional all-1 vector, α is the ratio between different layers of attention, that is, the residual weight, and Represents the mapping kernel function, and the specific calculation formula is as follows:

[0066]

[0067] In step S1-5, a simple two-layer GCN module is used to aggregate the local neighbor information of the node. The output of the GCN layer and the output of the attention layer are concatenated into a new feature matrix, which is then input into the linear classification layer to perform the label classification task. The formula is as follows:

[0068] Y out = f y ((1 - β)E l + βGCN(E, A))

[0069] where GCN(E, A) represents the GCN network layer, and f y represents the classification function, and β is the weight of the output result of GCN and the output result of the attention layer.

[0070] Step S2: According to the label classification task, divide the data in different clients into the parameters of the personalized layer and the parameters of the common layer. Fix the parameters of the personalized layer without aggregation, and continuously upload the parameters of the common layer for aggregation, and iteratively obtain a converged federated training model.

[0071] In the present invention, a personalized method is adopted to aggregate the parameters of the GCN layer and the transformer layer of the model. Specifically, the high-dimensional representation and the representation obtained by the GCN output are weighted and aggregated, and then the fully connected layer is used as the classification prediction layer, and the personalized parameter aggregation method obtains a converged federated training model.

[0072] Step S2-1: Divide the parameters of the model into the parameters of the personalized layer and the parameters of the common layer. The parameters of the personalized layer are W P , and the parameters of the common layer are W B , then the formula for the classification task can be expressed as:

[0073]

[0074] where j represents the j-th client;

[0075] Step S2-2: The data of the client first passes through the common layer, then through the personalized layer, and finally obtains the classification result. In the process of federated aggregation, the client fixes the parameters of the personalized layer without aggregation and only uploads the parameters of the common layer. Through the parameter aggregation process of the federated model server, the weighted average parameters are distributed to the common layers of each client, while the personalized layer always retains the locally trained parameters without uploading.

[0076] Step S2-3: Continuously iterate to complete Step S2-2 until the federated training model finally converges.

[0077] Step S3: Validate on five datasets. The comparison methods adopted include: the classic aggregation methods of federated learning: FedAvg and FedProx, the personalized federated algorithm: FedPer, the federated graph learning algorithm FED-PUB, and the publicly available datasets are: three citation datasets: Cora, CiteSeer, PubMed; two Amazon shopping datasets: Amazon-Computer and Amazon-Photo. The training set, validation set, and test set are classified in a 20%, 40%, and 40% split manner, and the data scenario is set in a scenario with uneven node scales.

[0078] Table 1 shows the accuracy of node classification for different methods.

[0079] Table 1

[0080] Cora CiteSeer PubMed Computer Photo FedAvg 64.26 66.85 77.90 82.72 87.83 FedProx 63.75 65.33 78.17 82.78 88.48 FedPer 63.08 66.78 78.01 82.48 88.34 FED-PUB 67.04 67.58 78.12 82.49 89.28 FedGformer 70.21 68.94 78.49 83.98 91.54

[0081] As shown in Table 1, in this scenario, the number of nodes of different clients varies greatly. The number of nodes of some clients is more than ten times that of other clients, and the number of edges is more than 100 times. Different datasets can prove the effectiveness of this method.

[0082] Those skilled in the art should understand that the present invention is not limited by the above embodiments. What is described in the above embodiments and the specification only illustrates the principle of the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements all fall within the scope of the present invention claimed. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims

1. A federated graph data mining method based on transformers, used to mine data within clients, characterized in that, Specifically, it includes the following steps: Step S1, use a simplified Transformer module to mine the node relationships in the client, obtain the high-dimensional representations of all nodes in the client, and jointly input the high-dimensional representations and the representations obtained from the output of the GCN layer in the model of the client into a fully connected layer to perform a classification prediction task, obtain a label classification task, and solve the negative correlation problem in the attention aggregation process; And Step S2, according to the label classification task, divide the data in different clients into the parameters of the personalized layer and the parameters of the common layer, fix the parameters of the personalized layer without aggregation, and continuously upload the parameters of the common layer for aggregation, and iteratively obtain a converged federated training model.

2. The federated graph data mining method based on Transformer according to claim 1, wherein: Among them, The GCN is a classic model in GNN. The core of the operating mechanism of GNN is message passing, where a node sends messages to its neighbors and receives aggregated messages from its neighbors. In graph representation learning, a large graph is defined as G = (V, E), where the set of nodes is V = {v1, v2,..., v n}, and the input node feature matrix X can be represented as X = {x1, x2,..., x n}. The adjacency matrix between nodes is A, and A includes the self-loops of the nodes. The GCN updates its own representation by aggregating the information of adjacent nodes, and its aggregation update formula can be summarized as: where, σ represents the activation function, represents the hidden embedding of node u in the (l + 1)-th layer, N u represents the neighbors of node u, e i,u represents the edge between node u and node i, AGG(·) represents the aggregation process of the graph neural network, which can specifically be the maximum method, the minimum method, or the average method, and f(·) represents the message passing mechanism of the graph neural network.

3. The federated graph data mining method based on Transformer according to claim 2, wherein: Among them, In the step S2, it is defined that there are a total of M clients participating in the training, and the set is C = {C1,..., C M}, and the data of each client is heterogeneous, different in scale, node characteristics, and topological structure. The global optimization objective of federated learning is to minimize the global loss, and the formula is as follows: where d i is the data volume size of client i, D is the total data volume size of all clients, L i (θ i ) is the optimization objective of the loss function for each client individually, {θ i} i＝1 M represents the personalized parameters of the M clients themselves, F i (v; θ i ) represents the local model of each client, l(F i (v; θ i )y) represents the cross-entropy loss used for node classification.

4. The federated graph data mining method based on Transformer according to claim 3, wherein: Among them, The federated graph data mining method based on Transformer further includes the following effectiveness verification method: Step S3, verify on five datasets. The comparison methods adopted include: classic aggregation methods for federated learning: FedAvg and FedProx, personalized federated algorithm: FedPer, federated graph learning algorithm FED-PUB. The public datasets are respectively: three citation datasets: Cora, CiteSeer, PubMed; two Amazon shopping datasets: Amazon-Computer and Amazon-Photo. The classification of the training set, validation set and test set adopts a splitting method of 20%, 40% and 40%, and the data scenario is set in a scenario where the node scale is uneven. In this scenario, the number of nodes in different clients varies greatly, and the number of nodes in some clients is more than ten times that of other clients, and the number of edges is more than 100 times. Different datasets can prove the effectiveness of this method.

5. The federated graph data mining method based on transformer according to claim 4, wherein, The specific steps of step S1 are as follows: Step S1-1: Process the input graph data using a linear fully-connected layer to enhance the representational ability of linear attention. By mapping the input matrix E to a high-dimensional embedding space, the embedding matrix E is obtained. 0 , the embedding matrix E 0 is the output obtained after inputting it into the first layer of the transformer. As the number of layers iterates, E can be obtained. 1 ; Step S1-2, simplify the multi-head attention mechanism using a single-head attention mechanism. The standard attention calculation formula is as follows: Q = f Q (E l ), K = f K (E l ), V = f V (E l ) Among them, f Q , f K and f V represent a three-layer linear fully connected network, d represents the dimension of the embedding matrix, and the calculation process of the attention is divided into three parts: First, it is necessary to calculate the correlation scores between the current node and all other nodes in the data, and obtain the association degree between the previous node and all other nodes in the data through these scores; Subsequently, the attention weights are calculated using the correlation scores, and the Softmax function is used to normalize the correlation scores into a probability distribution to ensure that the weights sum to 1 and reflect the relative importance of each node in the entire node sequence; Finally, the representations of each node in the sequence are weighted and summed according to the attention weights to obtain the output representation at the current position. The model dynamically generates the node representation of the current node according to the importance of each position in the sequence, and the node representation is subsequently weighted with the GCN representation to obtain the total output of the graph transformer; Step S1-3, use the L2 norm to normalize the Q and K matrices to construct a Transformer that can be stably trained: Among them, |Q| L2 and |K T | L2 represent the L2 regularization module; Step S1-4, adopt the idea of removing Softmax, directly calculate the product of matrix K and V through the multiplication commutative law, and calculate the attention with a time complexity of O(N) to ensure the non-negativity of the attention matrix. The simplified single-layer attention calculation process is as follows: Among them, 1 is an N-dimensional all-1 vector, and α is the ratio that controls the proportion between different layers of attention, that is, the residual weight. and represents the mapping kernel function, and the specific calculation formula is as follows: Step S1-5, use a simple two-layer GCN module to aggregate the local neighbor information of the nodes, and splice the result output by the GCN layer and the result output by the attention calculation together to form a new feature matrix and then input it into a linear classification layer to perform a label classification task. The formula is as follows: Y out = f y ((1 - β)E l + βGCN(E, A)) Among them, GCN(E, A)) represents the GCN network layer, and f y represents the classification function, and β is the weight of the output result of the GCN and the output result of the attention layer.

6. The federated graph data mining method based on transformer according to claim 5, It is characterized in that: Wherein, The specific steps of step S2 include the following sub-steps: Step S2-1: Divide the parameters of the model into personalized layer parameters and common layer parameters. The personalized layer parameters are W P , and the common layer parameters are W B . Then the formula for the label classification task can be expressed as: Among them, j represents the j-th client; In step S2-2, the data of the client first passes through the common layer, then through the personalized layer, and finally obtains the result of the label classification task. During the federated aggregation process, the client fixes the parameters of the personalized layer without aggregation and only uploads the parameters of the common layer. Through the parameter aggregation process of the federated model server, the weighted average parameters are distributed to the common layers of each client, while the personalized layer always retains the locally trained parameters without uploading; In step S2-3, step S2-2 is continuously iterated until the federated training model finally converges.

7. A federated graph data mining system based on transformer, characterized in that It includes: A node relationship mining module that uses a simplified transformer module to mine the node relationships within the client to obtain the high-dimensional representations of all nodes within the client, and jointly inputs the high-dimensional representations and the representations obtained from the output of the GCN layer of the client's model into a fully connected layer for a classification prediction task to obtain a label classification task, solving the negative correlation problem in the attention aggregation process; And An aggregation module that divides the data in different clients into the parameters of the personalized layer and the parameters of the common layer according to the label classification task, fixes the parameters of the personalized layer without aggregation, and continuously uploads the parameters of the common layer for aggregation, and iteratively obtains a converged federated training model.

8. The federated graph data mining system based on transformer according to claim 7, characterized in that, It also includes: A testing module that is verified on five datasets. The comparison methods adopted include: classical aggregation methods for federated learning: FedAvg and FedProx, personalized federated algorithm: FedPer, federated graph learning algorithm FED-PUB. The public datasets are respectively: three citation datasets: Cora, CiteSeer, PubMed; two Amazon shopping datasets: Amazon-Computer and Amazon-Photo. The classification of the training set, validation set, and test set adopts a splitting method of 20%, 40%, and 40%, and the data scenario is set in a scenario with uneven node scales. In this scenario, the number of nodes in different clients varies greatly, and the number of nodes in some clients is more than ten times that of other clients, and the number of edges is more than 100 times. Different datasets can prove the effectiveness of this method.

Images