Privacy protection heterogeneous federal learning method based on split learning

Abstract

The invention relates to a privacy protection heterogeneous federal learning method based on split learning. According to the method, federated learning and split learning are combined, privacy protection on client data in distributed collaborative learning is enhanced by adopting a proper model decomposition strategy, and negative effects of data isomerism are reduced by combining a private group normalization layer into a convolutional neural network. The method comprises the following stages: an initialization stage, a model broadcasting stage, a model training updating stage, a model aggregation stage and a training stop judging stage. Compared with the prior art, the method has the advantages that high model prediction accuracy can be kept under the scene of high client data isomerism in the Internet of Things, the data leakage risk is reduced, and the calculation and communication cost of the client is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of federated learning performance optimization, and in particular to a privacy-preserving heterogeneous federated learning method based on split learning. Background Art

[0002] With the deepening of the digitalization and informatization process of the financial industry, financial data has been widely used, especially in risk management, anti-fraud and customer service. However, it is accompanied by increasing attention to data privacy and security issues, especially in cross-institutional data sharing and compliance, which face unprecedented challenges. The traditional centralized data processing model is stretched in the face of these challenges and is difficult to meet the growing data security and privacy protection needs of the financial industry.

[0003] As a distributed collaborative machine learning algorithm framework, Federated Learning (FL) can use the data generated by a large number of devices in the edge environment of the financial network to train models, effectively solving the problems of data silos and privacy security, and has therefore attracted widespread attention. The most classic federated learning algorithm is the Federated Averaging (FedAvg) algorithm, the basic idea of ​​which is to train the global model locally on each participant (such as a mobile device or edge device), and then send the updated model parameters to the central server for aggregation.

[0004] In the financial field, the data of a single financial institution is often not enough to train a high-accuracy model. For example, in the field of credit card anti-fraud, banks can work with other acquiring institutions to use information such as cardholders' consumption records in different banks to train efficient anti-fraud models, thereby identifying and promptly blocking abnormal credit card transactions. This shows the potential for widespread application of federated learning in the financial field. In multiple fields such as credit card anti-fraud, credit risk assessment, anti-money laundering, credit assessment, insurance risk control, and financial investment, secure federated learning systems can establish efficient and accurate models across sites, which plays an important role in helping all parties involved make reasonable decisions, prevent and deal with financial risks. At the same time, the heterogeneity and sensitivity of financial data also bring additional challenges to federated learning.

[0005] Heterogeneous data refers to the statistical heterogeneity of the local data distribution of different participants, sometimes also called the Non-Independent and Identically Distributed (Non-IID) problem, that is, the data distribution between clients in federated learning is inconsistent and does not obey the same sampling. This data heterogeneity may make it difficult for the federated learning model to converge quickly, or even fail to converge, thereby reducing the model's predictive reasoning accuracy, which also limits the application of federated learning algorithms in large-scale financial scenarios. At the same time, distributed collaborative learning also faces the requirements of data privacy protection and the limited resources of participating distributed devices. Therefore, how to improve the performance and efficiency of federated learning algorithms for financial heterogeneous data while minimizing training overhead and ensuring data privacy security is a question of exploratory significance.

[0006] The most direct solution to the current research on heterogeneous federated learning is to introduce various proximal terms to limit the update of local models to reduce the distance between local models and the global optimum. The FedProx algorithm introduces additional proximal terms in the local optimization objective of each client based on the FedAvg algorithm. To calculate the current global model w t The L2 norm distance between the local model w and the global model w is used to penalize local updates that deviate from the global model. However, adding the proximal term may lead to slower convergence, and the FedProx algorithm requires each client to carefully tune the hyperparameter μ to achieve good accuracy. If μ is too large, the training of the local model will converge slowly. If μ is too small, the proximal term will have no effect. The SCAFFOLD method works by adding a proximal term for server c and all clients c. i A control variable is introduced to estimate the update direction of the global model and the local model respectively, and then the difference between them (c i -c) to correct local updates. However, the introduction of control variables will increase the amount of communication between the server and the client in each training round. Based on the idea of ​​contrastive learning, the MOON method introduces model contrast loss. The added proximal term reduces the difference between the global model and the local model while increasing the distance between the current local model and the local model of the previous round. This means that the client needs to store and maintain multiple models locally, which increases resource consumption and cost.

[0007] Through literature search, I found the following related literature. The methods used in these literatures can improve the performance of federated learning for heterogeneous data and reduce communication costs to a certain extent, but none of them use methods based on split learning and normalization, such as:

[0008] Chinese Patent CN202410043518.7, an accelerated adaptive federated learning algorithm at the client level in a data heterogeneous scenario, patentee: Southeast University. This patent discloses an accelerated adaptive federated learning algorithm at the client level. This technical solution aims at federated learning in a data heterogeneous scenario and provides an algorithm to address the "client drift" phenomenon and the challenge of reducing the communication cost of federated learning. This method modifies the aggregation rule in the global model aggregation stage of federated learning and adjusts the contribution of local models of different heterogeneous clients to the global model using local step sizes before aggregation. At the same time, it adaptively adjusts the size of the local step size following the change in the heterogeneity degree among clients. However, global step size and local step size parameters are introduced in this method, and each client also needs to additionally store the previous local model, the model change amount, and related auxiliary information, increasing the computational and storage overheads of the client. For some resource-constrained clients, the difficulty and cost of implementing model training and inference locally are too high. Summary of the Invention

[0009] The object of the present invention is to provide a privacy-preserving heterogeneous federated learning method based on split learning, which combines federated learning with split learning (SL), strengthens privacy protection by adopting an appropriate model decomposition strategy, and mitigates the negative impact of data heterogeneity by incorporating private group normalization layers into the network. To achieve the object of the present invention, a privacy-preserving heterogeneous federated learning method based on split learning is provided, including:

[0010] Federated server: Coordinates the entire training process and implements model aggregation, communicates with participating clients and the cloud server;

[0011] Cloud server: Responsible for the training task of the server-side model, communicates with participating clients and the federated server;

[0012] Participating clients: Such as sensors, Internet of Things edge devices, etc. The data volumes, data distributions, computing powers, and geographical locations of each client device are different and vary greatly. Model training is performed locally and communication is carried out with the cloud server and the federated server.

[0013] The algorithm includes the following steps:

[0014] 1. A privacy-preserving heterogeneous federated learning method based on split learning, including the following steps:

[0015] Step 1) Initialization phase: K participating clients, a cloud server, and a federated server are arranged in an edge-based Internet of Things environment; A total of T training communication rounds are set. In the t-th communication round (0 < t ≤ T), the federated server initializes the global client model and the global server-side model

[0016] Step 2) Model broadcast phase: The federated server randomly selects a client subset S consisting of m clients from all K clients. t Participate in this round of collaborative training, and then broadcast the model required for this round of training to the participants in the cloud server and client subset and

[0017] Step 3) Model training and updating phase: Client set S t Each selected client k downloads the client model parameters from the federated server Then perform the client local update step in parallel

[0018] Client local update The specific content is:

[0019] 1) Local dataset D k Divide into batches of size B

[0020] 2) For each batch of data In your own client model Perform forward propagation on it until the cutting layer, and get the output of the cutting layer

[0021] 3) Intermediate Activation A k and the corresponding data label y k Sent to the cloud server;

[0022] 4) The cloud server performs the update steps on the cloud server side after receiving the intermediate representation and label from the client in The gradient of the model cutting layer parameters;

[0023] 5) The client uses the gradient fed back by the cloud server Continue the back propagation process and calculate the gradient

[0024] 6) Update client model parameters:

[0025]

[0026] Where η is the learning rate hyperparameter, The gradient value of stochastic gradient descent when the client model is updated; the cloud server updates ServerUpdate (A k ,y k) are as follows:

[0027] 1) The cloud server receives the server-side model parameters broadcast from the federation server

[0028] 2) Use the received intermediate activation A k Perform a forward pass to output the predicted value The output prediction value of the network is a two-dimensional real matrix with shape (N, C), that is:

[0029]

[0030] Each row of this matrix corresponds to the output prediction of an input image, where z ij is the original output value of the i-th input image in category j, N is the number of samples, and C is the number of categories of the data set samples;

[0031] 3) Use the softmax function to output the predicted value of the network Convert to a probability distribution matrix Right now:

[0032]

[0033] where p ij represents the probability that the i-th input image belongs to category j. The softmax is calculated as follows:

[0034]

[0035] 4) Calculate the loss Where LossFunction(·) is the cross entropy loss function, which is calculated as follows:

[0036]

[0037] Where N is the number of samples, C is the number of categories, is the probability that the i-th sample is predicted to be of category c, p i,c is the true label of the i-th sample in category c, which is composed of the true data label y k The value is 0 or 1 after one-hot encoding.

[0038] 5) After finding the loss, use the back propagation algorithm to calculate the model parameter gradient layer by layer

[0039] 6) Use gradient descent method to update the server-side model:

[0040]

[0041] in It is the gradient value of stochastic gradient descent when updating the server-side model;

[0042] 7) Return the model cutting layer parameter gradient Give client k;

[0043] Step 4) Model aggregation phase: All clients and cloud servers participating in this round of training will aggregate the client models and server model Upload to the federated server for weighted averaging to obtain the initial global client model and global server model for the next round of training; in this step, each client extracts the GroupNormalization layer parameters in the model and stores them locally without uploading them, that is, for each neural network layer l, if layer l is not a GN layer, the weighted average of the client-side model is performed:

[0044]

[0045] Weighted average of server-side models:

[0046]

[0047] In equations (3) and (4), n k is the data sample size owned by client k, and n is the total amount of data;

[0048] Step 5) Training stop determination phase: If the current training communication round t exceeds the preset training cycle threshold T, then stop training and use the client model at the federation server and server model To obtain the final complete model; otherwise, let t = t + 1, and then repeat steps 2) to 5) until the specified training rounds are completed.

[0049] The beneficial effects of the present invention are:

[0050] 1) The present invention reduces the risk of data intermediate representation leakage through a model decomposition strategy based on a convolutional neural network structure, and has lightweight privacy protection characteristics;

[0051] 2) Splitting the model reduces the client’s computing and communication costs to a certain extent. The client does not have to bear the complete model training task, and the communication volume between each round of training and the cloud server is reduced;

[0052] 3) A private group normalization layer is introduced into the network to alleviate the negative impact and training bias brought by Non-IID data, which not only enables the collaborative training model to have good global generalization performance, but also adapt to the personalized local data distribution of each client. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a flow chart of the privacy-preserving split federated learning algorithm PPSFL of the present invention.

[0054] Figure 2 This is a schematic diagram of the comparison of local test losses of a privacy-preserving heterogeneous federated learning method based on split learning in the present invention on three datasets, namely Fashion-MNIST, CIFAR-10 and COVID-19 radiological image dataset, in a scenario with non-independent and identically distributed datasets.

[0055] Figure 3 A schematic diagram of the comparison of confusion matrices of a privacy-preserving heterogeneous federated learning method based on split learning of the present invention on a COVID-19 radiological image dataset in a scenario with a non-independent and identically distributed dataset;

[0056] Figure 4 This is a schematic diagram showing a comparison of the communication rounds required for a privacy-preserving heterogeneous federated learning method based on split learning of the present invention to achieve the same target accuracy on three datasets, namely Fashion-MNIST, CIFAR-10 and COVID-19 radiological image datasets, in a scenario with non-independent and identically distributed datasets.

[0057] Figure 5 This is a schematic diagram showing the comparison of the results of image reconstruction on three datasets, namely, CIFAR-10, UTKFace, and COVID-19 radiological image datasets, by a privacy-preserving heterogeneous federated learning method based on split learning in the present invention in a scenario with non-independent and identically distributed datasets. DETAILED DESCRIPTION

[0058] The present application considers a privacy-preserving heterogeneous federated learning algorithm with split learning, which promotes privacy protection and reduces client training overhead through appropriate model decomposition strategies, and mitigates the negative impact of data heterogeneity by incorporating a private group normalization layer into the model. The following will describe in detail the implementation methods of the present invention in conjunction with the accompanying drawings and examples. Through these descriptions, it can be fully understood how the present invention applies technical means to solve technical problems and achieve technical effects, and thus implement it accordingly. It should be pointed out that as long as there is no conflict, the various embodiments and features of the present invention can be combined with each other, and all formed technical solutions belong to the protection scope of the present invention.

[0059] Example 1: See Figure 1 , Figure 1 This is a flowchart of the privacy-preserving split federated learning algorithm PPSFL according to Example 1 of the present invention. The steps of the method are described in detail below:

[0060] Step 1: Initialization. First, the federated server sets the global communication rounds \(T\) and the number of local loop iteration updates \(E\), and determines that the total number of participating clients is \(K\). It should be noted that the present invention adopts a periodic local update scheme. Under this scheme, the communication delay of the client local update model is分摊到 \(E\) iterations. Compared with the stochastic gradient scheme that synchronizes after each training, the communication delay is reduced by \(E\) times. In addition, the periodic communication scheme equalizes the client computing time and alleviates the synchronization problem to a certain extent. Then, in the \(t\) -th communication round (\(0 \lt t\leq T\)), when the collaborative training starts, the federated server takes the feature extractor and classifier of the convolutional neural network as the client - side model and the server - side model and initializes the global client model and the global server - side model

[0061] Step 2: Model broadcasting. The federated server randomly selects a client subset \(S\) consisting of \(m\) clients from all \(K\) clients t to participate in this round of collaborative training. Then it broadcasts the models required for this round of training to the cloud server and the participants in the client subset and

[0062] Step 3: Model training update. Each selected client in the set \(S\) t downloads the client model parameters from the federated server and then starts local training in parallel, performs forward propagation on its own client model until the cut layer. Next, the output \(A\) k of the cut layer and the corresponding data label \(y\) k are sent to the cloud server. At the same time, the cloud server also downloads the server - side model parameters from the federated server and, after receiving the intermediate representation and label from the client, continues to perform forward propagation on the corresponding server - side model of each client model to calculate the loss \(L\), then performs the error back - propagation algorithm, calculates the gradients of the server - side model layer by layer and updates the model parameters After the cloud server calculates the gradients layer by layer to the cut layer, it sends the gradients of the cut layer back to the corresponding client. Each client uses the received gradients to continue completing the back - propagation process of its own model. Then it updates the client model parameters

[0063] Step 4: Model aggregation. At the end of each training communication round, all participating clients and cloud servers aggregate their client models and server model Upload to the federated server for weighted averaging to obtain the initial global client model and global server model for the next round of training. It is particularly important to note that in this step, the client will extract the private GN layer parameters and store them locally without uploading. The aggregation rules for the client model and the server model are as follows:

[0064]

[0065]

[0066] where n k is the number of data samples owned by client k, and n is the total amount of data.

[0067] Step 5: Training stop decision. If the current training communication round t exceeds the preset training cycle threshold T, the training is stopped and the client model at the federation server is used. and server model To obtain the final complete model; otherwise, let t = t + 1, and then repeat steps 2) to 5) until the specified training rounds are completed.

[0068] Embodiment 2: Embodiment 2 of the present invention considers testing different performance indicators on different image classification data sets to verify the availability and effect of the technical method we proposed. The content of Embodiment 2 is divided into three parts:

[0069] (1) We use a Dirichlet distribution-based method to simulate the client’s non-independent and identically distributed datasets and test the average local test loss of the client’s collaborative learning on three datasets: Fashion-MNIST, CIFAR-10, and the COVID-19 radiology image dataset.

[0070] (2) A method based on Dirichlet distribution is used to simulate the client's non-independent and identically distributed data set, and the confusion matrix of the proposed method is tested and evaluated on the COVID-19 radiological image dataset to observe the classification performance for each category of images.

[0071] (3) A method based on Dirichlet distribution is used to simulate the client's non-independent and identically distributed data sets, and the number of communication rounds required for the model obtained by collaborative learning to achieve the target accuracy is tested on three data sets: Fashion-MNIST, CIFAR-10, and COVID-19 radiological image data sets.

[0072] Figure 2This is a schematic diagram of the comparison of local test losses of a privacy-preserving heterogeneous federated learning method based on split learning in the present invention on three datasets, namely Fashion-MNIST, CIFAR-10 and COVID-19 radiological image dataset, in a scenario with non-independent and identically distributed datasets.

[0073] Figure 3 The figure is a schematic diagram of the comparison of confusion matrices of a privacy-preserving heterogeneous federated learning method based on split learning on a COVID-19 radiological image dataset in a scenario with a non-independent and identically distributed dataset. The confusion matrix is ​​a table used to evaluate the performance of a classification model, which summarizes the model's prediction results and actual labels. In the confusion matrix, each row represents the model's prediction results, each column represents the actual category, the elements on its diagonal represent the number of samples correctly classified by the model, and the other elements represent the number of samples incorrectly classified by the model; considering the imbalanced number of samples in each category in the actual classification problem, the Kappa coefficient of each method is further calculated based on the results of the confusion matrix and listed in Table 1. In classification tasks, the Kappa coefficient helps to evaluate the reliability and consistency of the classification process. A high Kappa coefficient means strong consistency between classifiers, indicating that the classification results are consistent and reliable. A low Kappa coefficient indicates poor consistency, indicating that there are inconsistencies or different opinions in the classification process. The calculation formula of the Kappa coefficient is as follows:

[0074]

[0075] Where P o and P e can be calculated from the confusion matrix, where P o is equal to the sum of the diagonal elements of the confusion matrix divided by the total number of samples, and P e is the weighted average of the prediction probability of each category. The calculation results show that PPSFL predicts each category of samples more accurately than other baseline methods and has better classification consistency and reliability.

[0076] Table 1 Kappa coefficient of each method

[0077]

[0078] Table 2 lists the global test accuracy of all methods, which can reflect the generalization ability of the collaborative learning model.

[0079] Table 2 Test accuracy of various distributed collaborative learning methods in the Non-IID case (%)

[0080]

[0081]

[0082] Figure 4 This is a schematic diagram showing a comparison of the communication rounds required for a privacy-preserving heterogeneous federated learning method based on split learning of the present invention to achieve the same target accuracy on three datasets, namely Fashion-MNIST, CIFAR-10 and COVID-19 radiological image datasets, in a scenario with non-independent and identically distributed datasets.

[0083] Figure 5 This is a schematic diagram showing the comparison of the results of image reconstruction on three datasets, namely, CIFAR-10, UTKFace, and COVID-19 radiological image datasets, using a privacy-preserving heterogeneous federated learning method based on split learning in the present invention in a scenario with non-IID datasets. The first row is the original image; the second row is the reconstruction result based on the intermediate activation of the existing splitting methods (SL, SFL, SplitAVG, etc.); the third row is the image reconstruction result based on the intermediate representation of the PPSFL model partition. In each group of reconstruction experiments, it can be observed that the quality of the images reconstructed using the PPSFL method proposed in this paper is lower than that of other methods, indicating that the degree of information leakage of the original data is significantly reduced. Even after reconstruction, most images are still unrecognizable, which makes it difficult for an honest but curious server to extract any useful information from the intermediate representation.

[0084] The above experimental results show that the technical method we proposed shows good convergence effect on different data sets, with higher test accuracy and better classification performance, and greatly reduces the communication cost of the client in distributed collaborative training, while enhancing the privacy protection properties of client data. All the above embodiments are carried out on a Dell Precision 7920Tower workstation, which is equipped with a 10-core Intel Xeon Silver 4210R processor, 64GB memory and NVIDIARTX A5000 GPU. The training operating system is Ubuntu 22.04LTS, and the deep learning framework uses PyTorch 1.9.0 for training.

[0085] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention is described in detail with reference to the embodiments, it should be understood by those skilled in the art that any modification or equivalent replacement of the technical solutions of the present invention does not depart from the spirit and scope of the technical solutions of the present invention and should be covered by the claims of the present invention.

Claims

1. A privacy-preserving heterogeneous federated learning method based on split learning, comprising the following steps: Step 1) Initialization phase: K participating clients, cloud servers, and federated servers are deployed in an edge-based IoT environment; a total of T training communication rounds are set. In the tth communication round (0<t≤T), the federated server initializes the global client model and the global server-side model Step 2) Model broadcast phase: The federated server randomly selects a client subset S consisting of m clients from all K clients. t Participate in this round of collaborative training, and then broadcast the model required for this round of training to the participants in the cloud server and client subset and Step 3) Model training and updating phase: Client set S t Each selected client k downloads the client model parameters from the federated server Then perform the client local update step in parallel Client local update The specific content is: 1) Local dataset D k Divide into batches of size B 2) For each batch of data In your own client model Perform forward propagation on it until the cutting layer, and get the output of the cutting layer 3) Intermediate Activation A k and the corresponding data label y k Sent to the cloud server; 4) The cloud server performs the update steps on the cloud server side after receiving the intermediate representation and label from the client in is the gradient of the model cutting layer parameters; 5) The client uses the gradient fed back by the cloud server Continue the back propagation process and calculate the gradient 6) Update client model parameters: Where η is the learning rate hyperparameter, The gradient value of stochastic gradient descent when the client model is updated; Cloud server update ServerUpdate(A k ,y k ) are as follows: 1) The cloud server receives the server-side model parameters broadcast from the federation server 2) Use the received intermediate activation A k Perform a forward pass to output the predicted value The output prediction value of the network is a two-dimensional real matrix with shape (N, C), that is: Each row of this matrix corresponds to the output prediction of an input image, where z ij is the original output value of the i-th input image in category j, N is the number of samples, and C is the number of categories of the data set samples; 3) Use the softmax function to output the predicted value of the network Convert to a probability distribution matrix Right now: where p ij represents the probability that the i-th input image belongs to category j. The softmax is calculated as follows: 4) Calculate the loss Where LossFunction(·) is the cross entropy loss function, which is calculated as follows: Where N is the number of samples, C is the number of categories, is the probability that the i-th sample is predicted to be of category c, p i，c is the true label of the i-th sample in category c, which is composed of the true data label y k The encoding result is 0 or 1; 5) After finding the loss, use the back propagation algorithm to calculate the model parameter gradient layer by layer 6) Use gradient descent method to update the server-side model: in It is the gradient value of stochastic gradient descent when updating the server-side model; 7) Return the model cutting layer parameter gradient Give client k; Step 4) Model aggregation phase: All clients and cloud servers participating in this round of training will aggregate the client models and server model Upload to the federated server for weighted averaging to obtain the initial global client model and global server model for the next round of training; in this step, each client extracts the GroupNormalization layer parameters in the model and stores them locally without uploading them, that is, for each neural network layer l, if layer l is not a GN layer, the weighted average of the client-side model is performed: Weighted average of server-side models: In equations (3) and (4), n k is the data sample size owned by client k, and n is the total amount of data; Step 5) Training stop determination phase: If the current training communication round t exceeds the preset training cycle threshold T, then stop training and use the client model at the federation server and server model To obtain the final complete model; otherwise, let t = t + 1, and then repeat steps 2) to 5) until the specified training rounds are completed.

Images