Personalized federated learning image classification method and system based on self-supervised contrast learning

Abstract

The application discloses a kind of personalized federated learning image classification method and system based on self-supervised contrast learning, specifically: central server randomly initializes global model, and is issued to each client, the local model of client includes shared layer and individualized layer;Randomly select the client participating in this aggregation, receive global model parameters and as the initialization parameter of local model shared layer, using local image dataset based on SimCLR algorithm is trained individualized layer and shared layer respectively using self-supervised contrast learning, then client shared layer parameter is uploaded to central server;Central server is received to model parameter, based on FedAvg algorithm is weighted average according to client data volume, obtains the global model after aggregation, and is issued to each client;Repeat execution until each client obtains a convergent individualized model, for completing image classification.The application is good in security, high in efficiency, good in accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning and computer technology, and in particular to a personalized federated learning image classification method and system based on self-supervised contrast learning. BACKGROUND

[0002] With the explosive growth of edge devices, these devices have generated a large amount of image data. For example, in the medical system, the research and analysis of biomedical picture data is the premise of carrying out health care and diagnosis. Different medical images correspond to different disease types, and through the analysis of medical image data such as X-ray films and CT, doctors can quickly and accurately identify and diagnose diseases. In modern medical research work, it is necessary to rely on diverse and multi-dimensional samples to effectively improve the accuracy and applicability of medical data analysis. However, the internal data in the medical institutions concerns the privacy information of patients, and has high confidentiality and sensitivity in data storage and data use. Therefore, each country and region has formulated and implemented strict laws and regulations on the collection, dissemination and use of personal privacy data.

[0003] For a large amount of image data, traditional centralized machine learning is to centralize data in a central location and then train in alignment, but centralized training has the following problems: 1) data security problem, because data must be transmitted to the centralized server; 2) when the data volume is very large, centralized learning will face the problem of a large amount of data, resulting in increased training time and resource cost. These problems make it difficult to apply centralized learning to some tasks that need to process a large amount of data.

[0004] The federated learning paradigm is committed to realizing collaborative training of models at the network edge without centralizing raw data, thereby greatly improving the data privacy problem. However, the heterogeneity of data between user devices can seriously reduce the performance of traditional federated averaging. At the same time, there are a large amount of unlabeled data in actual scenarios, and manual labeling of data is time-consuming and laborious, and there are problems such as generalization error, false correlation and adversariality. SUMMARY

[0005] The purpose of the present application is to provide a personalized federated learning image classification method and system based on self-supervised contrast learning with good security, high efficiency and accuracy.

[0006] The technical solution for achieving the purpose of the present application is: a personalized federated learning image classification method based on self-supervised contrast learning, comprising the following steps:

[0007] Step 1, the central server randomly initializes a global model and distributes it to each client. The local model of the client includes a shared layer and a personalized layer;

[0008] Step 2, randomly select the client participating in this aggregation;

[0009] Step 3, the selected clients receive the global model parameters issued in step 1 and use them as the initialization parameters of the local model shared layer, use the local image data set to train the personalized layer and the shared layer based on the SimCLR algorithm for self-supervised contrastive learning, and upload the shared layer parameters and data volume to the central server after completing the local training;

[0010] Step 4, the central server performs weighted averaging based on the FedAvg algorithm according to the client data volume for the received model parameters to obtain the aggregated global model;

[0011] Step 5, the central server issues the aggregated global model to each client;

[0012] Step 6, repeat steps 2 to 5 until each client obtains a converged personalized model for completing image classification.

[0013] Further, in step 1, the global model is an encoder with a network structure of Resnet18.

[0014] Further, in step 1, the shared layer is an encoder composed of a Resnet18 for extracting common features, and the personalized layer is a two-layer MLP network for adapting to local data, and the client initializes the personalized layer before the first round of training.

[0015] Further, in step 3, self-supervised contrastive learning is performed based on the SimCLR algorithm, specifically:

[0016] Without data labels, SimCLR maximizes the consistency of the same data under different augmentations through the contrastive loss of the hidden space to learn the representation, and the loss function is:

[0017]

[0018] where z i ,z j are the outputs of images i, j through the model, sim(z i ,z j ) represents the cosine similarity between z i ,z j , τ is the temperature coefficient of contrastive learning, 2N is the number of data points obtained after pairing augmentation of a small batch sample N, and z k is the output of image k through the model, and l i,j is the loss of the positive sample pair (i, j);

[0019] A small batch of N samples is randomly selected, and paired augmented samples are obtained in the small batch, a total of 2N data points; image i and its augmented sample j constitute a positive sample pair (i, j), the loss function is as shown in the above formula, and finally the loss function on the batch sample The loss average of all positive sample pairs in the 2N data points is calculated.

[0020] Further, in step 3, the personalized layer and the shared layer are trained in turn, specifically:

[0021] In the tth communication round, after the client i loads the global model, the shared layer parameters are first fixed Based on the local data, the personalized layer parameters θ t are trained:

[0022]

[0023] Wherein, is the loss function, is the gradient operator, η is the learning step of the optimizer, is the personalized layer parameter of the client i after the tth round of training;

[0024] Then, based on the local data, the training of the shared layer parameters is carried out by fixing the personalized layer parameters:

[0025]

[0026] Wherein, is the shared layer parameter of the client i after the tth round of training.

[0027] Further, in step 4, the central server carries out weighted average based on the FedAvg algorithm according to the received model parameters and the data quantity of the client, to obtain the aggregated global model, and the formula used is as follows:

[0028]

[0029] Wherein, S t is the set of clients participating in the federated training in the tth round, n i is the data quantity of the client i, is the shared layer parameter of the client i after the tth round of local training, is the initialization parameter of the t+1th round of federated learning.

[0030] The application discloses a personalized federated learning image classification system based on self-supervised contrast learning, and relates to the technical field of federated learning.

[0031] The initialization module randomly initializes a global model through a central server and distributes the global model to each client.

[0032] The client selection module is used for randomly selecting clients participating in this aggregation.

[0033] The self-supervised contrast learning module receives the global model parameters distributed by the initialization module through the selected clients and uses the global model parameters as the initialization parameters of the shared layer of the local model.

[0034] The global model aggregation module performs weighted averaging on the received model parameters based on the FedAvg algorithm according to the data volume of the clients to obtain an aggregated global model.

[0035] The model distribution module distributes the aggregated global model to each client through the central server.

[0036] The client selection module, the self-supervised contrast learning module, the global model aggregation module and the model distribution module are repeatedly executed until each client obtains a converged personalized model for completing image classification.

[0037] The application further discloses a mobile terminal comprising a memory, a processor and a computer program stored in the memory and capable of running on the processor.

[0038] Compared with the prior art, the application has the following advantages: (1) the centralized model training part in the Internet of Things is transferred to an edge device, local data of each organization is combined, and the computing load of a cloud or a server is reduced; (2) privacy data is always kept in the edge device, and the security of the data can be improved; (3) for a large amount of privacy data lacking labels in an actual scene, the cost is reduced through self-supervised contrast learning; (4) for the data imbalance problem among organizations in an actual application scene, the aggregation mode is improved, and the quality of a final model is improved; and (5) each organization can obtain a personalized model suitable for local data after training. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 is a flow chart of the personalized federated learning image classification method based on self-supervised contrastive learning of the present application.

[0040] Figure 2 is a system schematic diagram of the model training process in the implementation of the present application.

[0041] Figure 3 is a performance comparison curve diagram of the method of the present application and the traditional method in the embodiment of the present application. DETAILED DESCRIPTION

[0042] In combination Figure 1 The present application provides a personalized federated learning image classification method based on self-supervised contrastive learning, comprising the following steps:

[0043] Step 1, the central server randomly initializes a global model and distributes it to each client, and the local model of the client includes a shared layer and a personalized layer;

[0044] Step 2, randomly select the client participating in this aggregation;

[0045] Step 3, each selected client receives the global model parameters distributed by step 1 and uses them as the initialization parameters of the shared layer of the local model, uses the local image dataset to train the personalized layer and the shared layer based on the SimCLR algorithm for self-supervised contrastive learning, and after completing the local training, the client uploads the shared layer parameters and data volume to the central server;

[0046] Step 4, the central server performs weighted averaging based on the FedAvg algorithm according to the received model parameters and the data volume of the client, to obtain an aggregated global model;

[0047] Step 5, the central server distributes the aggregated global model to each client;

[0048] Step 6, repeat steps 2 to 5 until each client obtains a converged personalized model for completing image classification.

[0049] As a specific example, in step 1, the global model is an encoder with a network structure of Resnet18.

[0050] As a specific example, in step 1, the shared layer is an encoder composed of a Resnet18, used to extract common features; the personalized layer is a double-layer MLP network, used to adapt to local data, and the client initializes the personalized layer before the first round of training.

[0051] As a specific example, in step 3, self-supervised contrastive learning is performed based on the SimCLR algorithm, specifically:

[0052] Without data labels, SimCLR learns representations by maximizing the consistency of the same data under different augmentations through the contrastive loss of the hidden space, and the loss function is:

[0053]

[0054] where z i ,z j are the outputs of images i, j through the model, sim(z i ,z j ) represents the cosine similarity between z i ,z j , τ is the temperature coefficient of contrastive learning, 2N is the number of data points obtained after a small batch sample N is paired with augmentations, and z k is the output of image k through the model, and l i,j is the loss of the positive sample pair (i, j);

[0055] A small batch sample containing N samples is randomly extracted, and paired augmented samples are obtained in the small batch sample, a total of 2N data points; image i and its enhanced sample j constitute a positive sample pair (i, j), and the loss function is as shown in the above formula, and the loss function of the batch sample is the average loss of all positive sample pairs in the 2N data points.

[0056] As a specific example, in step 3, the personalized layer and the shared layer are trained in turn, specifically:

[0057] At the tth communication round, after the client i loads the global model, the shared layer parameters are first fixed t Based on the local data, the personalized layer parameter θ is trained:

[0058]

[0059] where, is the loss function, is the gradient operator, η is the learning step of the optimizer, is the personalized layer parameter of the client i after the tth round of training;

[0060] Then fix the personalized layer parameters, and train the shared layer parameters based on the local data:

[0061]

[0062] where, is the shared layer parameter of the client i after the tth round of training.

[0063] As a specific example, in step 4, the central server performs weighted averaging based on the FedAvg algorithm according to the amount of client data for the received model parameters to obtain an aggregated global model, and the formula used is as follows:

[0064]

[0065] where S t is the set of clients participating in federated training in the tth round, n i is the data volume of client i, is the shared layer parameter of client i after the tth round of local training, is the initialization parameter of the t+1th round of federated learning.

[0066] The application also provides a personalized federated learning image classification system based on self-supervised contrast learning, which is used to implement the personalized federated learning image classification method based on self-supervised contrast learning, and comprises an initialization module, a client selection module, a self-supervised contrast learning module, a global model aggregation module, and a model distribution module.

[0067] The initialization module randomly initializes a global model through a central server and distributes it to each client, and the local model of the client comprises a shared layer and a personalized layer.

[0068] The client selection module is used to randomly select clients participating in this aggregation.

[0069] The self-supervised contrast learning module receives the global model parameters distributed by the initialization module through the selected clients and uses them as the initialization parameters of the shared layer of the local model, uses the local image dataset to perform self-supervised contrast learning based on the SimCLR algorithm to train the personalized layer and the shared layer respectively, and uploads the shared layer parameters to the central server after completing the local training.

[0070] The global model aggregation module performs weighted averaging based on the FedAvg algorithm according to the amount of client data for the received model parameters through the central server to obtain an aggregated global model.

[0071] The model distribution module distributes the aggregated global model to each client through the central server.

[0072] The client selection module, the self-supervised contrast learning module, the global model aggregation module, and the model distribution module are repeatedly executed until each client obtains a converged personalized model for completing image classification.

[0073] The application further provides a mobile terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the self-supervised contrast learning-based personalized federated learning image classification method when executing the program.

[0074] To sum up, each client respectively trains the model personalization layer and the shared layer based on self-supervised learning using local data, uploads the shared layer model to the server for aggregation after local training ends, and performs local adaptation after receiving the aggregated shared layer model, thereby solving the problem of low model accuracy caused by data heterogeneity of clients in federated learning.

[0075] The technical solutions in the embodiments of the application will be described clearly and in detail below with reference to the drawings in the embodiments of the application. The described embodiments are only some of the embodiments of the application.

[0076] Embodiments

[0077] The present embodiment provides a self-supervised contrast learning-based personalized federated learning image classification method, wherein federated learning, as a distributed machine learning technology, allows multiple users to collaboratively train and effectively helps multiple institutions to perform data use and machine learning modeling while meeting the requirements of user privacy protection, data security, and government regulations, which can effectively solve the data island problem, enable participants to jointly model without sharing data, technically break through the data island, and realize artificial intelligence (AI) collaboration. Self-supervised learning is to automatically generate labels or categories of training samples using the structure or other information of the data itself, thereby avoiding the problem of manually labeling a large amount of data in traditional supervised learning.

[0078] The present embodiment provides a self-supervised contrast learning-based personalized federated learning image classification method, wherein federated learning, as a distributed machine learning technology, allows multiple users to collaboratively train and effectively helps multiple institutions to perform data use and machine learning modeling while meeting the requirements of user privacy protection, data security, and government regulations, which can effectively solve the data island problem, enable participants to jointly model without sharing data, technically break through the data island, and realize artificial intelligence (AI) collaboration. Self-supervised learning is to automatically generate labels or categories of training samples using the structure or other information of the data itself, thereby avoiding the problem of manually labeling a large amount of data in traditional supervised learning.

[0079] Federated learning can aggregate medical model parameters of each medical institution without accessing the original private data, which effectively protects personal privacy and also fuses and analyzes biological medical data of different medical institutions, and the specific process is as follows:

[0080] (1) Each medical institution respectively uses the same initial model and network architecture to train in the centralized federated learning framework, the central server randomly initializes the global model, and distributes it to each medical institution;

[0081] (2) After each medical institution receives the initial global model issued by the server, the local medical picture is used to train the local model by self-supervised contrast learning without labels, and the personalized layer training is performed before the shared layer training, and after each medical institution completes the training, the shared layer parameters and data volume of the local model are uploaded to the server;

[0082] (3) After the central server receives the local model parameters, the weighted average is performed according to the data volume to obtain the aggregated global model parameters;

[0083] (4) The medical institution receives the aggregated global model parameters for the next round of training. Repeat the above process until each medical institution obtains a converged local model. The training process is as shown in Figure 2 .

[0084] The application is applicable to a federated learning framework including a central server with a computing function and multiple participants. By aggregating only the shared layer model parameters of the clients, each medical institution client can obtain a personalized local model.

[0085] The application realizes learning the general features of the data set by learning the similarity or difference of the data points through the trained model in the case that there are a large amount of unlabeled data in the real medical scene, and solves the privacy problem existing in the distributed machine learning by using the federated learning framework. The problems of too few labeled data to perform supervised learning and insufficient local data volume of the participants to train a model with high accuracy and strong applicability are solved. Figure 3 As can be seen from

[0086] In summary, the personalized federated learning image classification method based on self-supervised contrast learning of the embodiment improves the performance of the federated model of the medical institution in the scene of too few labels of medical privacy data and heterogeneous medical privacy data, and realizes the safe and effective sharing of the privacy data between the medical institutions.

Claims

1. A personalized federated learning image classification method based on self-supervised contrastive learning, characterized in that, Comprising the following steps: Step 1, the central server randomly initializes a global model and distributes it to each client, and the local model of the client includes a shared layer and a personalized layer; Step 2, randomly select the client participating in this aggregation; Step 3, the selected client receives the global model parameters distributed by step 1 and uses them as the initialization parameters of the local model shared layer, trains the personalized layer and shared layer based on the SimCLR algorithm using the local image dataset for self-supervised contrastive learning, and after completing the local training, the client uploads the shared layer parameters and data volume to the central server; In step 3, based on the SimCLR algorithm for self-supervised contrastive learning, specifically: Without data labels, SimCLR maximizes the consistency of the same data under different augmentations through the contrastive loss of the hidden space to learn the expression, and the loss function is: ； wherein, is the output of the model is the output of the model represents the cosine similarity between is the temperature coefficient of contrastive learning, 2N is the number of data points obtained after the small batch sample N is augmented in pairs, is the output of the model is the output of the model is the loss of the positive sample pair . A small batch of N samples is randomly selected and paired augmented samples are obtained in the small batch, a total of 2N data points; image and its augmented sample constitute a positive sample pair ( ), the loss function is shown in the formula above, and finally the loss function on the small batch is the average loss of all positive sample pairs in the 2N data points. In step 3, the personalized layer and the shared layer are trained in turn, specifically: In the t-th communication round, the client After loading the global model, first fix the parameters of the shared layer. Personalized layer parameters based on local data Training: ； wherein, is a loss function, is a gradient operator, is a learning step of the optimizer, is the personalized layer parameter of the client i after the t-th round of training. Then fix the personalized layer parameters and train the shared layer parameters based on the local data: ； wherein, is the shared layer parameter of client i after t rounds of training; Step 4, the central server performs weighted averaging on the received model parameters based on the FedAvg algorithm according to the client data volume to obtain the aggregated global model; Step 5, the central server distributes the aggregated global model to each client; Step 6, repeat steps 2 to 5 until each client gets a converged personalized model for image classification.

2. The self-supervised contrastive learning based personalized federated learning image classification method according to claim 1, characterized in that, In step 1, the global model is an encoder with a network structure of Resnet18.

3. The self-supervised contrastive learning based personalized federated learning image classification method according to claim 1, wherein, In step 1, the shared layer is an encoder composed of a Resnet18, used to extract common features; the personalized layer is a two-layer MLP network, used to adapt to local data, and the client initializes the personalized layer before the first round of training.

4. The self-supervised contrastive learning based personalized federated learning image classification method according to claim 1, characterized in that, In step 4, the central server performs weighted averaging on the received model parameters based on the FedAvg algorithm according to the client data volume to obtain the aggregated global model, and the formula used is as follows: ； wherein, is the set of clients participating in federated training in round t, is the data volume of client i, is the shared layer parameters of client i after the tth round of local training, is the initialization parameters of the t+1th round of federated learning.

5. A personalized federated learning image classification system based on self-supervised contrastive learning, characterized in that, The system is used to implement the personalized federated learning image classification method based on self-supervised contrastive learning of any one of claims 1-4, and the system comprises an initialization module, a client selection module, a self-supervised contrastive learning module, a global model aggregation module, and a model distribution module, wherein: The initialization module randomly initializes a global model through the central server and distributes it to each client, and the local model of the client includes a shared layer and a personalized layer; The client selection module is used to randomly select the client participating in this aggregation; The self-supervised contrastive learning module receives the global model parameters distributed by the initialization module through the selected clients and uses them as the initialization parameters of the local model shared layer, trains the personalized layer and shared layer based on the SimCLR algorithm using the local image dataset for self-supervised contrastive learning, and after completing the local training, the client uploads the shared layer parameters to the central server; The global model aggregation module performs weighted averaging on the received model parameters based on the FedAvg algorithm according to the client data volume through the central server to obtain the aggregated global model; The model issuing module issues the aggregated global model to each client through the central server. The client selection module, the self-supervised contrast learning module, the global model aggregation module, and the model issuing module are repeatedly executed until each client obtains a converged personalized model for completing image classification.

6. A mobile terminal comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the self-supervised contrast learning-based personalized federated learning image classification method according to any one of claims 1-4 when executing the program.

Images