Graphsmote-based synthetic minority over-sampling technique graph federated learning method

Abstract

The GraphSMOTE-based synthetic minority node oversampling graph federated learning method of the disclosure comprises the following steps: step one, a client uploads model parameters of the last round of iteration training to a server; step two, the server performs weighted averaging on model parameters of each client, and broadcasts global model parameters to each client; step three, the client extracts intermediate features of local minority class nodes by using the global model parameters, finds a pair of nodes with the closest Euclidean distance, and generates synthetic samples by introducing random noise interpolation; step four, the server collects and distributes average minority class node representations and synthetic samples of each client; and step five, the client generates cross-client synthetic samples, predicts new edges for all synthetic samples, and joins the local data set to participate in training. The method combines the advantages of federated learning and graph data enhancement, realizes global circulation of node and edge information on the premise of ensuring data privacy, and reduces the communication cost by adjusting the super parameter to give the client the decision power on the total amount of synthetic samples.

Description

Technical Field

[0001] This invention belongs to the field of graph federated learning technology, and specifically relates to a graph data augmentation federated learning method based on GraphSMOTE, which is used to solve the problem of class imbalance in graph data in federated learning. Background Technology

[0002] As a distributed machine learning framework, federated learning can jointly train neural network models by multiple parties while protecting data privacy, and has broad application prospects. Graphs are one of the most common data structures in the real world, and the class balance of nodes is often not guaranteed, causing classifiers to be unable to effectively learn decision boundaries. Therefore, a federated learning method that can handle the class imbalance problem in graph data is urgently needed in the industry.

[0003] In federated learning algorithms for non-graph data, there are three main solutions to the class imbalance problem. The first solution is implemented locally by the client, employing common methods for handling sample imbalance in integrated environments, including resampling, setting class weights, and changing model class evaluation metrics. However, the rebalancing process lacks information from the global dataset. The second solution requires clients to upload their local data distribution information. A trusted third party will perform data augmentation based on the global data distribution and reschedule the client's training process according to the differences in the client's data distribution. However, the client-uploaded data distribution may pose a threat to data privacy. The third solution does not require clients to upload local data information. Instead, it automatically monitors class imbalance by measuring the differences in model parameters trained by different clients and designs a new loss function to assign different weights to samples of different classes. The weakness of this method is the need for an auxiliary dataset, the quality of which directly affects the monitoring and class balancing effectiveness.

[0004] To address the class imbalance problem in graph data, existing strategies only consider scenarios with centralized data. On the data side, oversampling strategies that directly copy minority class nodes are prone to overfitting, resulting in poor performance on graph data. The GraphSMOTE algorithm effectively avoids this problem by performing interpolation operations within the neighborhood of minority class samples to synthesize new minority class samples and training an edge prediction model to add new edges to the synthesized samples. However, in scenarios restricted by privacy protocols, the GraphSMOTE algorithm cannot directly obtain the intermediate representations of minority class nodes for each client, thus it cannot be directly applied to federated learning. Summary of the Invention

[0005] The technical problem solved by this invention is: given the prevalence of class imbalance in the field of graph federated learning and the lack of corresponding solutions, this invention modifies the GraphSMOTE algorithm to enable each client to share minority class synthetic samples. On the one hand, this can supplement the client's local minority class samples, and on the other hand, it can realize the indirect flow of graph data between clients, thereby improving the classifier's performance.

[0006] A synthetic minority node oversampling graph federated learning method based on GraphSMOTE, the method comprising:

[0007] Step 1: Each customer uploads the parameters f of their trained Embedding GNN model. (i) and Decoder model parameters g (i) ;

[0008] Step 2: The server performs a weighted average of the Embedding GNN model parameters and the Decoder model parameters based on the amount of data from each client, and obtains the weighted average results of the global Embedding GNN model parameters f and the global Decoder model parameters g, and broadcasts them to each client.

[0009] Step 3: After receiving the weighted average result broadcast in Step 2, any customer i among the customers extracts the intermediate representation set of minority class nodes from the local data. And select the nearest minority class node in the local area to synthesize a new sample set. And and Uploaded to the server, among which, For the intermediate representation set of extracted minority class nodes The average value;

[0010] Step four: The server collects and distributes information from other customers to the client. and

[0011] Step 5, calculate for customer i respectively Average number of minority nodes compared to other customers The distance between them, let it be... The closest distance is to customer i and Perform a second nearest neighbor synthesis, and denote the result as... M represents the number of customers;

[0012] Step Six, the customer will Add the new edge to your own training dataset and use the Decoder model to predict new edges between new nodes and connections to the original graph. Then, superimpose the weights α onto the client's original graph adjacency matrix to produce an augmented graph. Participate in local training;

[0013] When the number of iterations reaches the upper limit, or the model's classification performance no longer improves, the synthetic minority oversampled graph federated learning based on GraphSMOTE is completed.

[0014] In step two, the formula for calculating the parameters f of the global embedding GNN model is as follows:

[0015]

[0016] The formula for calculating the global Decoder model parameter g is:

[0017] Among them, f (i) and g (i) These represent the Embedding GNN model parameters and the Decoder model parameters for client i, respectively. That is, the number of nodes N in the local dataset of customer i. (i) The weight is determined;

[0018] In step three, the nearest minority class node pair is selected to synthesize a new sample. The specific method is as follows:

[0019] (31) Let set Let u be the intermediate representation of any minority class node u. Then, traverse the intermediate representations of other minority class nodes in the local array to find the nearest neighbor node of u. The intermediate representation of a minority class node v is v;

[0020] (32) Interpolate the minority class node u and its nearest neighbor to synthesize a new sample set. New sample set The element in is ρ nn(u) +(1-ρ) u Where ρ is random noise between 0 and 1, and the new sample set The category is the same as node u;

[0021] For each customer, a new sample set is synthesized using local data. The size is controlled by hyperparameters;

[0022] In step five, the second nearest neighbor node synthesis process is specifically described as follows:

[0023] (51) Customer i compares with other customers The distance between them helps identify the nearest customers.

[0024] (52) Customer i utilizes local minority samples The synthetic minority class sample set of customer nc(i) To achieve cross-customer sample synthesis, for The intermediate representation of node u is h u ,exist Searching for h u nearest node

[0025] (53) Add a random noise ρ' between 0 and 1, then the elements in the new sample set synthesized by node u and node nn(u) are represented as ρ′nn(u)+(1-ρ′)u;

[0026] The number of cross-customer sample synthesizations is controlled by hyperparameters.

[0027] In step six, during local training, the difference between the data distribution of real samples and synthetic samples is controlled by setting the loss function and training hyperparameters.

[0028] The loss function for local customer training should include not only the cross-entropy loss for classification, but also a penalty term to minimize the distance between the average of the local true minority class samples and the generated samples from other customers.

[0029] The training hyperparameters mentioned above need to be set with the number of epochs and batch size for local training by the customer to avoid excessive differences between the two data distributions due to too many local model updates.

[0030] For each customer, a new sample set is synthesized using local data. The size is controlled by hyperparameters, and the control method is as follows:

[0031] When 0 < λ < 1, a neighborhood interpolation is performed once from the minority class nodes in the local client to generate the same number of synthetic samples. When λ > 1, multiple rounds of neighborhood interpolation are performed on all local minority class nodes, and the number of synthetic nodes is λ times that of the original nodes. By adjusting the size of the hyperparameter λ, the number of synthetic samples generated by a single client and shared globally can be adjusted.

[0032] The number of cross-customer sample synthesized is controlled by hyperparameters, and the control method is as follows:

[0033] When 0 < μ (i) <1, meaning from customer i The extraction ratio is μ (i) The node in the middle represents the synthesized sample of customer nc(i). Find the nearest neighbor and perform an interpolation operation to generate a synthetic sample with the same amount of data as the extracted real data. (i) >1, meaning that the operation is performed on all minority class samples of customer i. Multi-round interpolation synthesis, number of synthesized samples μ (i) A sample size several times larger than that of client i, obtained by adjusting the hyperparameter μ unique to client i. (i) The size of the sample size allows customer i to flexibly control the number of samples synthesized across customers, thereby determining the total number of samples synthesized.

[0034] Beneficial effects

[0035] (1) This disclosed GraphSMOTE-based synthetic minority node oversampling graph federated learning method solves the node class imbalance problem in graph federated learning. An iteration includes the following steps: Step 1, the client uploads the model parameters trained in the previous iteration to the server; Step 2, the server performs a weighted average of the model parameters for each client and broadcasts the global model parameters to each client; Step 3, the client uses the global model parameters to extract intermediate features of local minority class nodes, finds the node pairs with the closest Euclidean distance, and introduces random noise interpolation to generate synthetic samples; Step 4, the server collects and distributes the average minority class node representations and synthetic samples from each client; Step 5, the client generates cross-client synthetic samples and predicts new edges for all synthetic samples, adding them to the local dataset for training. The iteration terminates when the algorithm's termination condition is met. This method combines the advantages of federated learning and graph data augmentation, achieving global flow of node and edge information while ensuring data privacy. Simultaneously, by adjusting hyperparameters, it empowers clients with the decision-making power over the total number of synthetic samples, reducing communication costs.

[0036] (2) The cross-customer sample synthesis of the present invention is closer to the operation of the GraphSMOTE algorithm in a centralized environment, and also complements the single-customer sample synthesis. In a distributed scenario, due to communication cost limitations, the synthesized samples from other customers may not be sufficient to completely balance the classes. In addition, since the dataset sizes of each customer are different and the degree of class imbalance varies, the second cross-customer sample synthesis can give customers greater flexibility, and customers can decide the total number of samples generated according to their actual needs.

[0037] (3) In the method of this invention, the generated data from other customers is generated by interpolation of the output of the global EmbeddingGNN in the previous iteration. However, the parameters of the EmbeddingGNN have changed during the local training process of the customers, resulting in an increasing difference between the distribution of local real node embeddings and the distribution of generated samples from other customers. In addition to limiting the loss function, it is also necessary to reasonably set the number of rounds and batch size of local training for customers to avoid excessive differences in the distribution of the two data due to too many local model updates, and to improve communication efficiency as much as possible while ensuring the training results.

[0038] (4) This invention solves the problem of imbalanced node classes in graph federated learning, integrating the advantages of federated learning and graph data augmentation. Specifically, regarding data security, clients only need to upload model parameters, the average value of minority class node embeddings output by EmbeddingGNN, and synthetic samples to third parties, without involving the transmission of original real data, thus eliminating the risk of privacy leakage. Regarding information flow, clients achieve direct flow of minority class node information by sharing synthetic node embeddings, and indirect flow of edge information by sharing Decoder model parameters, which is particularly beneficial for clients lacking certain information. Regarding communication efficiency, most of the information exchanged consists of single-client synthetic samples, and the amount of this data is adjustable by the hyperparameter λ. Furthermore, λ can be compressed into a reasonable range by increasing the proportion of cross-client synthetic samples. Regarding data augmentation, through single-client and cross-client minority class sample synthesis, each client can obtain data augmentation information from the global database. Moreover, by setting hyperparameters, clients have the autonomy to determine the amount of samples generated, even if their datasets differ. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0040] Figure 2 This is a block diagram of a synthetic minority oversampled graph federated learning algorithm based on GraphSMOTE according to an embodiment of the present disclosure. Detailed Implementation

[0041] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings and examples, so that the process of how the present invention uses technical means to solve technical problems and achieve corresponding technical effects can be fully understood and implemented accordingly. The embodiments of this application and the various features in the embodiments can be combined with each other without conflict, and the resulting technical solutions are all within the protection scope of the present invention.

[0042] Furthermore, the steps illustrated in the flowcharts of the accompanying drawings can be executed in a computer, such as a set of computer-executable instructions. Also, although a logical order is shown in the flowcharts, in some cases the steps shown or described may be performed in a different order than that presented here.

[0043] A synthetic minority node oversampling graph federated learning method based on GraphSMOTE, the method comprising:

[0044] Step 1: Each customer uploads the parameters f of their trained Embedding GNN model. (i) and Decoder model parameters g (i) ;

[0045] Step 2: The server performs a weighted average of the Embedding GNN model parameters and the Decoder model parameters based on the amount of data from each client, and obtains the weighted average results of the global Embedding GNN model parameters f and the global Decoder model parameters g, and broadcasts them to each client.

[0046] Step 3: After receiving the weighted average result broadcast in Step 2, any customer i among the customers extracts the intermediate representation set of minority class nodes from the local data. And select the nearest minority class node in the local area to synthesize a new sample set. And and Uploaded to the server, among which, For the intermediate representation set of extracted minority class nodes The average value;

[0047] Step four: The server collects and distributes information from other customers to the client. and

[0048] Step 5, calculate for customer i respectively Average number of minority nodes compared to other customers The distance between them, let it be... The closest distance is to customer i and Perform a second nearest neighbor synthesis, and denote the result as... M represents the number of customers;

[0049] Step Six, the customer will Add the new edge to your own training dataset and use the Decoder model to predict new edges between new nodes and connections to the original graph. Then, superimpose the weights α onto the client's original graph adjacency matrix to produce an augmented graph. Participate in local training;

[0050] When the number of iterations reaches the upper limit, or the classification performance of Classifier GNN no longer improves, the synthetic minority oversampled graph federated learning based on GraphSMOTE is completed.

[0051] In step two, the formula for calculating the parameters f of the global embedding GNN model is as follows:

[0052]

[0053] The formula for calculating the global Decoder model parameter g is:

[0054] Among them, f (i) and g (i) These represent the Embedding GNN model parameters and the Decoder model parameters for client i, respectively. That is, the number of nodes N in the local dataset of customer i. (i) The weight is determined;

[0055] In step three, the nearest minority class node pair is selected to synthesize a new sample. The specific method is as follows:

[0056] (31) Let set Let u be the intermediate representation of any minority class node u. Then, traverse the intermediate representations of other minority class nodes in the local array to find the nearest neighbor node of u. The intermediate representation of a minority class node v is v;

[0057] (32) Interpolate the minority class node u and its nearest neighbor to synthesize a new sample set. New sample set The element in is ρ nn(u) +(1-ρ) u Where ρ is random noise between 0 and 1, and the new sample set The category is the same as node u;

[0058] For each customer, a new sample set is synthesized using local data. The size is controlled by hyperparameters;

[0059] In step five, the second nearest neighbor node synthesis process is specifically described as follows:

[0060] (51) Customer i compares with other customers The distance between them helps identify the nearest customers.

[0061] (52) Customer i utilizes local minority samples The synthetic minority class sample set of customer nc(i) To achieve cross-customer sample synthesis, for The intermediate representation of node u is h u ,exist Searching for h u nearest node

[0062] (53) Add a random noise ρ' between 0 and 1, then the elements in the new sample set synthesized by node u and node nn(u) are represented as ρ′nn(u)+(1-ρ′)u;

[0063] The number of cross-customer sample synthesizations is controlled by hyperparameters.

[0064] In step six, during local training, the difference between the data distribution of real samples and synthetic samples is controlled by setting the loss function and training hyperparameters.

[0065] The loss function for local customer training should include not only the cross-entropy loss for classification, but also a penalty term to minimize the distance between the average of the local true minority class samples and the generated samples from other customers.

[0066] The training hyperparameters mentioned above need to be set with the number of epochs and batch size for local training by the customer to avoid excessive differences between the two data distributions due to too many local model updates.

[0067] For each customer, a new sample set is synthesized using local data. The size is controlled by hyperparameters, and the control method is as follows:

[0068] When 0 < λ < 1, a neighborhood interpolation is performed once from the minority class nodes in the local client to generate the same number of synthetic samples. When λ > 1, multiple rounds of neighborhood interpolation are performed on all local minority class nodes, and the number of synthetic nodes is λ times that of the original nodes. By adjusting the size of the hyperparameter λ, the number of synthetic samples generated by a single client and shared globally can be adjusted.

[0069] The number of cross-customer sample synthesized is controlled by hyperparameters, and the control method is as follows:

[0070] When 0 < μ () <1, meaning from customer i The extraction ratio is μ () The node in the middle represents the synthesized sample of customer nc(i). Find the nearest neighbor and perform an interpolation operation to generate a synthetic sample with the same amount of data as the extracted real data. () >1, meaning that the operation is performed on all minority class samples of customer i. Multi-round interpolation synthesis, number of synthesized samples μ () A sample size several times larger than that of client i, obtained by adjusting the hyperparameter μ unique to client i. () The size of the sample size allows customer i to flexibly control the number of samples synthesized across customers, thereby determining the total number of samples synthesized.

[0071] Figure 1 A flowchart of a synthetic minority oversampling graph federated learning algorithm based on GraphSMOTE according to an embodiment of the present disclosure is shown. The main algorithmic steps in one iteration include:

[0072] S1, M clients upload the parameters of the trained Embedding GNN and Decoder.

[0073] S2. The server performs a weighted average of the Embedding GNN parameter f and the Decoder parameter g based on the amount of data from each client, and broadcasts the average result to all clients.

[0074] S3. After receiving the global embedding GNN, client i extracts the intermediate embeddings of minority class nodes from the local data. And select the nearest minority class node in the local area to synthesize new samples. Will as well as average Uploaded to the server.

[0075] S4, The server collects and distributes information from other customers to the client. and

[0076] S5 and customer i are calculated respectively. The average of minority samples from other customers The distance between them. If with The closest distance is to customer i and Perform a second nearest neighbor synthesis, and denote the result as...

[0077] S6, The customer embeds the synthesis node. Add the new edge to your own training dataset and use the Decoder model to predict new edges between new nodes and connections to the original graph. Then, superimpose the weights α onto the client's original graph adjacency matrix to produce an augmented graph. Participate in local training.

[0078] The algorithm terminates when the number of iterations reaches its limit or when the classification performance of Classifier GNN no longer improves.

[0079] Example

[0080] like Figure 2 As shown in the diagram, the structure of the GraphSMOTE-based synthetic minority node oversampling graph federated learning method is as follows: In step ①, at the beginning of each iteration, each client uploads the model parameters of the Embedding GNN and Decoder trained locally in the previous iteration to the server. Then, as in steps ② and ③, the server performs weighted aggregation of the model parameters according to the number of nodes of each client and sends the aggregated results back to each client. In step ④, the client inputs the initial features of the local nodes into the global Embedding GNN to obtain the intermediate representations of the nodes. After obtaining the intermediate representations of all minority class nodes in the local dataset, the client executes the SMOTE synthetic oversampling method to generate synthetic samples for a single client, as shown in step ⑤. Steps ⑥ to ⑧ correspond to the second round of communication between the client and the server. The client uploads the average value of the intermediate representations of the real minority class nodes and the synthetic sample set to the server, and obtains synthetic samples generated by other clients from the server. In step ⑦, the server only plays a role in collection and distribution at this stage. In step 9, the customer first identifies another customer whose data distribution is closest to the average of the real samples by comparing the distances between them. Then, cross-customer samples are generated using local real data and synthetic data from this closest customer. Step 10 demonstrates the process of the Decoder model adding new edges to the synthetic nodes. Then, proceed as in step [step number missing]. As shown, new nodes and edges are added to the real graph data and input into the Classifier GNN model to perform the training task. The resulting loss function value is used to update the model parameters.

[0081] While the embodiments disclosed in this invention are as described above, the content is merely for the purpose of facilitating understanding of the invention and is not intended to limit the invention. Any person skilled in the art to which this invention pertains may make any modifications and variations in form and detail of the implementation without departing from the spirit and scope disclosed herein; however, the scope of patent protection for this invention shall still be determined by the scope defined in the appended claims.

Claims

1. A GraphSMOTE-based synthetic minority oversampling technique graph federated learning method, characterized in that The steps of the method include: Step one, each client uploads the trained Embedding GNN model parameters and Decoder model parameters ; Step two, the server respectively makes weighted average on the model parameters of the Embedding GNN and the Decoder according to the data volume of each client, to obtain the weighted average result of the global Embedding GNN model parameters f and the global Decoder model parameters g , and broadcasts to each client; Step three, any one of the clients i After receiving the weighted average result broadcast in step two, the intermediate representation set of the minority class nodes is extracted from the local data as And the nearest local distance minority class nodes are selected to synthesize a new sample set And And Upload to the server, wherein, The average value of the extracted intermediate representation set of the minority class nodes ​ Step four, the server collects and distributes to the client other clients' information with ; Step five, customer i Calculate the distance between and the average of minority class nodes of other customers , ... Set the distance to as the nearest, and the customer i makes the second nearest neighbor synthesis with and , and the synthesis result is recorded as ; M is the number of customers; Step six, the customer adds ( , ... , ) to his own training dataset, and uses the Decoder model to predict new nodes and edges connecting to the original graph, and superimposes them on the customer's original graph adjacency matrix to generate an enhanced graph for local training; The calculation formula of the global Embedding GNN model parameter in the second step is: f ;​ Global Decoder model parameters g The formula for calculating is: ; wherein, and respectively represent the Embedding GNN model parameters and the Decoder model parameters of the customer i ; , is the number of nodes of the local dataset; In step three, the nearest local distance minority class node pair is selected to synthesize new samples, and the specific method is as follows: (31) a record set any minority class node u in the intermediate representation traverses the intermediate representations of the other nodes of the same class locally to find the nearest neighbor node u of the minority class node ; the intermediate representation of the minority class node v is ; (32) interpolating a new sample set from the nearest neighbor nodes of the minority class node u and u , the elements in the new sample set are , where, ρ is a random noise between 0 and 1, and the class of the new sample set is the same as the node u ;​ In step five, the second nearest neighbor node synthesis process, the specific method is as follows: (51) customer i By comparing the distance between , ... the latest customer is locked where represents the average value of ; (52) customer i with local minority class samples and synthetic minority class sample set i for customer nc synthesizing cross-customer samples intermediate representation of nodes u h u find the closest node to u in h The steps of the method include: distance​​ (53) add a random noise between 0 and 1 ρ' then the node u and the node nn( u ) the element in the new set of samples synthesized by ; In step six, the difference between the data distribution of the real sample and the synthesized sample is controlled by the loss function and the setting of the training hyperparameter during local training. The loss function of the customer local training not only includes the cross-entropy loss of classification, but also includes a penalty term, so that the distance between the average value of the customer local real minority class samples and the generated samples from other customers is as small as possible. The training hyperparameter needs to set the number of rounds and the batch size of the customer local training.

2. The GraphSMOTE-based synthetic minority node oversampling graph federated learning method according to claim 1, wherein: When the number of iterations reaches the upper limit, or the classification effect of the Classifier GNN no longer improves, the GraphSMOTE-based synthetic minority oversampling graph federated learning is completed.

3. The GraphSMOTE-based synthetic minority node oversampling graph federated learning method according to claim 1, wherein: For each client, a new sample set is synthesized using local data The size of the set is controlled by a hyperparameter.

4. The GraphSMOTE-based synthetic minority node oversampling graph federated learning method according to claim 3, wherein: For each client, a new sample set is synthesized using local data The size of the set is controlled by a hyperparameter, controlled by When 0 < <1, meaning the proportion taken from the minority class locally by the customer is Each node performs a neighborhood interpolation once, producing the same number of synthetic samples. >1 means performing multiple rounds of neighborhood interpolation on all minority class nodes locally, and the number of nodes represented by the composite node is equal to the number of nodes in the original node. Times, by adjusting hyperparameters The size of the sample size is adjusted to control the number of synthesized samples generated by a single client and shared globally.

5. The GraphSMOTE-based synthetic minority node oversampling graph federated learning method according to claim 4, wherein: The number of cross-customer sample synthesis is controlled by a hyperparameter.

6. The GraphSMOTE-based synthetic minority node oversampling graph federated learning method according to claim 5, wherein: The number of cross-customer sample synthesis is controlled by a hyperparameter, and the control method is: When 0 <1, i.e. from the client i 's drawn proportion of the nodes intermediate representation, find the nearest neighbor in the client nc( i )'s synthetic sample and perform an interpolation operation once, generating an equal amount of synthetic samples as the drawn real data, when >1, i.e. for all minority class samples of the client i perform multiple rounds of interpolation synthesis with , the number of synthetic samples is times the real samples of the client i , by regulating the size of the client i unique hyperparameter , the client i controls the total amount of synthetic samples by controlling the number of synthetic samples across clients.

Images