Personalized asynchronous federal learning method for medical image segmentation

Abstract

The invention provides a personalized asynchronous federated learning method for medical image segmentation, and the method comprises the following steps: initializing each client, and carrying out the local training of a medical image segmentation model under the guidance of a federated decoder through local data; the client which completes local training sends the encoder part and the federal decoder part to the server; the client asynchronously performs local updating on the local image segmentation model; local personalized updating is carried out; after all the clients finish uploading, the server calculates the correlation between the clients by using the received model parameters and the held evaluation data set, generates a personalized federated decoder for each client in a personalized aggregation manner, and sends the personalized federated decoder back to the corresponding client; when the client receives the personalized federated decoder sent by the server, asynchronous updating of a local model is stopped, and next round of federated learning is started; and repeating the steps until the model converges, and completing the training.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence, and in particular, to a personalized asynchronous federated learning method for medical image segmentation. Background Art

[0002] Federated learning was proposed by Google in 2016 and is a distributed learning machine learning algorithm with privacy protection features. Federated learning protects the local data in each client by sharing the information learned from the local data by each local client, that is, the model of the local parameters, while keeping the local data always local, avoiding the privacy leakage problem caused by data sharing of private data.

[0003] McMahan et al. proposed the Federated Averaging method (FedAvg) [1], which is widely used as the cornerstone of federated learning algorithms. On the basis of this algorithm, a variety of federated learning algorithms with better performance have been successively proposed. In order to further solve the problem that when applying federated learning in actual scenarios, due to the non-independent and identically distributed data between each client, the client drift occurs, resulting in poor performance of the client local model on the local data. The Personalized Federated Learning (PFL) method was proposed. In this kind of federated learning method, each client generates a personalized local model with the characteristics of its local data to alleviate the client drift problem in federated learning.

[0004] In addition to the above problems, the client heterogeneity caused by different network conditions, devices, etc. reduces the training efficiency of federated learning. In synchronous federated learning, the client needs to wait for the slower device before aggregation in each communication. In order to make full use of the waiting time in each client, Asynchronous Federated Learning (AFL) was proposed. [2] first proposed a basic asynchronous federated optimization framework FedAsync, and on this basis, more asynchronous federated learning algorithms have been proposed to improve the training efficiency of federated learning. Although the existing AFL methods have considered the training efficiency, they ignore the impact of heterogeneous data on FL. In order to make reasonable use of the waiting time and solve the problem of data heterogeneity between clients, [3] proposed a personalized asynchronous federated learning method PAFedFV for finger vein recognition. Based on this model, we further explored the personalized asynchronous federated learning algorithm for medical image segmentation, and combined with the characteristics of the image segmentation model, we proposed a new personalized asynchronous federated learning mechanism PAFedMIS for image segmentation.

[0005] Reference Documents:

[0006] [1] McMahan B, Moore E, Ramage D, et al. Communication-efficient learning of deep networks from decentralized data[C] / / Artificial intelligence and statistics. PMLR, 2017:1273-1282.

[0007] [2] Xie C, Koyejo S, Gupta I. Asynchronous federated optimization[J]. arXiv preprint arXiv:1903.03934, 2019.

[0008] Mu H, Guo J, Han C, et al. PAFedFV: Personalized and Asynchronous Federated Learning for Finger Vein Recognition[J]. arXiv preprint arXiv:2404.13237, 2024. Summary of the Invention

[0009] In view of the technical problems mentioned in the above background art, a personalized asynchronous federated learning method for medical image segmentation is provided. This invention is mainly used in the medical image segmentation task carried out under the federated learning framework in the artificial intelligence industry. By designing personalized local training and model aggregation methods, and retaining the personalization of each local client model, the model performance is improved. At the same time, combined with the idea of asynchronous federated learning, during the period when the server waits for all clients to send their local models, the clients in the waiting state are asynchronously locally trained to improve the training efficiency of federated learning.

[0010] The technical means adopted by this invention are as follows:

[0011] A personalized asynchronous federated learning method for medical image segmentation, comprising the following steps:

[0012] Step 1: Each client initializes the encoder, decoder, and federated decoder parts in the local model, and locally trains the medical image segmentation model using local data under the guidance of the federated decoder;

[0013] Step 2: The client that has completed local training sends the encoder part and the federated decoder part to the server; after successfully sending to the server, the client asynchronously locally updates the local image segmentation model;

[0014] Step 3: Local personalized update; after all clients have completed uploading, the server calculates the correlation between clients using the received model parameters and the held evaluation dataset, and aggregates and generates a personalized federated decoder for each client through personalized aggregation, and sends it back to the corresponding client;

[0015] Step 4: When the client receives the personalized federated decoder sent by the server, it stops the asynchronous update of the local model and starts the next round of federated learning;

[0016] Repeat the above steps until the model converges and the training is completed.

[0017] Furthermore, each client model includes: an encoder, a decoder, and a federated decoder; the encoder and decoder are used to form the local image segmentation model; the federated decoder is used to realize information exchange between all clients globally and to guide the training of the local image segmentation model.

[0018] Furthermore, in Step 2, during the process of local update on the client side, the local image segmentation model is iteratively updated using local data;

[0019] Introduce the Dice loss function L Dice and the MSE loss function L MSE :

[0020]

[0021] where N represents the total number of clients participating in federated learning; f i (·) represents the encoder part in the local model of client i; h i (·) represents the decoder part in the local model of client i; x represents the input image; j represents the j-th pixel in the image; y represents the true value of the segmentation image mask; ε represents the Laplace smoothing term.

[0022] The federated decoder in the client uses the output of the local encoder as input to calculate the similarity loss L Sim to guide the update of the local segmentation model; then the total loss function L Total for client i to perform local update is:

[0023] L Total = L Dice + L MSE + L Sim (3);

[0024] After the local update is completed, the client sends the encoder and the federated decoder to the server, and the decoder remains local.

[0025] Further, the calculated similarity loss L Sim is as follows:

[0026]

[0027] where D i represents the local dataset in client i.

[0028] Further, in step 3, after the server receives the model parameters sent by all clients, it starts to perform model aggregation; in order to aggregate and generate a personalized federated decoder, for the encoders of all clients received by the server, weighted average aggregation is performed with the amount of data held by each client as the weight to generate a global encoder; in order to calculate the correlation degree between each client, we input the evaluation image dataset in the server into the global encoder, and sequentially input the output of the global encoder into each federated decoder sent to the server, and use the output of the decoder to calculate the cosine similarity between every two clients, which measures the similarity degree between each client; calculate the similarity degree between client k and client i

[0029]

[0030] where i ≠ k;

[0031] After obtaining the correlation degree between every two clients, personalized model aggregation weights are calculated for each client based on the correlation degree between clients; the federated decoder of client i obtained through personalized model aggregation

[0032] The personalized federated decoder generated by aggregation is sent back to each client respectively, guiding local model training in the next round of federated learning, enabling the local model to balance between global general features and local features; and the parameters of each local segmentation model will not be replaced.

[0033] Further, the federated decoder of client i obtained through personalized model aggregation

[0034]

[0035] where λ represents the proportion of the federated decoder of client i in the previous round in generating the federated decoder in this round, and K represents the set of all clients except client i;

[0036] Further, in step 4, after each client successfully uploads the local model, until it receives the aggregated model sent back by the server, it locally and asynchronously trains the local image segmentation model using local data.

[0037] Compared with the prior art, the present invention has the following advantages:

[0038] While improving the training efficiency of the federated learning model for medical image segmentation, the performance of the local model on local data is enhanced. Based on the existing personalized asynchronous federated learning method for fingerprint recognition, this method combines the characteristics of the image segmentation network and specifically designs a personalized mechanism and an asynchronous learning mechanism. This enables each client model to share information while not losing the characteristics of local data, alleviating to a certain extent the client drift problem caused by data heterogeneity. At the same time, it allows the client to make full use of the waiting time, effectively improving the training efficiency of federated learning for medical image processing. The application effect of federated learning in real medical scenarios has been further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0040] Figure 1 This is the asynchronous training process of PAFedMIS of the present invention.

[0041] Figure 2 This is the overall network structure of the present invention.

[0042] Figure 3 This is the basic framework of federated learning of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0043] In order to enable those skilled in the art to better understand the solutions of the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0044] It should be noted that the terms "first", "second", etc. in the description, claims and the above drawings of the present invention are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily limit to those clearly listed steps or units, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0045] As Figures 1-3 shown, the present invention provides a personalized asynchronous federated learning method for medical image segmentation, comprising the following steps:

[0046] Step 1: Each client initializes the encoder, decoder, and federated decoder parts in the local model, and locally trains the medical image segmentation model using local data under the guidance of the federated decoder; each client model includes: an encoder, a decoder, and a federated decoder; the encoder and decoder are used to construct the local image segmentation model; the federated decoder is used to realize information exchange among all clients globally and to guide the training of the local image segmentation model.

[0047] Step 2: The client that has completed local training sends the encoder part and the federated decoder part to the server; after successfully sending to the server, the client asynchronously updates the local image segmentation model; in Step 2, during the process of local update by the client, the local image segmentation model is iteratively updated using local data;

[0048] Introduce the Dice loss function L Dice and the MSE loss function L MSE :

[0049]

[0050] where N represents the total number of clients participating in federated learning; f i (·) represents the encoder part in the local model of client i; h i (·) represents the decoder part in the local model of client i; x represents the input image; j represents the j-th pixel in the image; y represents the true value of the segmentation image mask; ε represents the Laplace smoothing term;

[0051] The federated decoder in the client calculates the similarity loss L using the output of the local encoder as the input SimGuide the update of the local segmentation model; calculate the similarity loss L Sim is:

[0052]

[0053] where D i represents the local dataset in client i.

[0054] Then the total loss function L for local update of client i Total is:

[0055] L Total = L Dice + L MSE + L Sim (3);

[0056] After the local update is completed, the client sends the encoder and the federated decoder to the server, and the decoder remains local.

[0057] Step 3: Local personalized update; when all clients have completed uploading, the server calculates the correlation between clients using the received model parameters and the held evaluation dataset, and aggregates and generates a personalized federated decoder for each client through personalized aggregation and sends it back to the corresponding client; in Step 3, when the server receives the model parameters sent by all clients, it starts model aggregation; in order to aggregate and generate a personalized federated decoder, the encoders of all clients received by the server are weighted and averaged with the data volume held by each client as the weight to generate a global encoder; in order to calculate the correlation degree between each client, we input the evaluation image dataset in the server into the global encoder, and sequentially input the output of the global encoder into each federated decoder sent to the server, and use the output of the decoder to calculate the cosine similarity between every two clients, which measures the similarity degree between each client; calculate the similarity degree between client k and client i

[0058]

[0059] where i ≠ k;

[0060] After obtaining the correlation degree between every two clients, calculate the personalized model aggregation weight for each client based on the correlation degree between clients; the federated decoder of client i obtained through personalized model aggregation The federated decoder of client i obtained through personalized model aggregation

[0061]

[0062] Among them, λ represents the proportion of the federated decoders of client i in the previous round in the generated federated decoders of this round, and K represents the set of all clients except client i.

[0063] The aggregated personalized federated decoder is sent back to each client to guide the local model training in the next round of federated learning, so that the local model can strike a balance between global common features and local features; and the parameters of each local segmentation model will not be replaced.

[0064] Step 4: When the client receives the personalized federated decoder sent by the server, it stops the asynchronous update of the local model and starts the next round of federated learning. In the traditional federated learning model, due to different network conditions and different local data volumes, the client is heterogeneous, which makes each client take different time to complete local training and upload the local model to the server. Clients with fast upload speeds often face the situation of long waiting time being wasted. Therefore, in order to make better use of the idle waiting time of the model and avoid client idleness as much as possible, we designed an asynchronous training mechanism so that each client can asynchronously use local data to train the local image segmentation model locally after successfully uploading the local model until receiving the aggregated model sent back by the server. It should be noted that asynchronous training only trains the image segmentation model composed of the encoder and decoder in the client, and does not use the federated decoder to guide the training of the segmentation model.

[0065] Repeat the above steps until the model converges and the training is completed.

[0066] Example 1

[0067] Federated machine learning is a distributed machine learning framework with privacy protection and secure encryption technology. It aims to share information among medical institutions by uploading local model parameters without leaking the local data of each client. Using the federated learning framework to process medical data can not only avoid the data storage and huge communication overhead problems caused by uploading a large amount of medical data to a server, but also make full use of more data to generate better models while ensuring data security and patient privacy. The modeling effect is not much different from that of traditional deep learning. The introduction of federated learning effectively solves the privacy and security issues in the current medical field in the process of data sharing and analysis and the problem of insufficient local data for certain diseases.

[0068] The present invention will be further described below in conjunction with cases. Due to the differences in medical devices held by various medical institutions, the parameter settings of the devices, the operation methods of medical staff, the characteristics of patients participating in the diagnosis and treatment, etc., various medical institutions are heterogeneous, that is, traditional federated learning will lead to client drift in medical institutions and a decrease in the training efficiency of the federated learning model. The asynchronous personalized federated learning algorithm for medical image segmentation proposed by this method, in each round of federated learning, when a medical institution successfully uploads its local model, the local model is trained asynchronously locally until all clients have uploaded their local models to the server. Then, the server starts to aggregate the personalized models and sends the generated personalized federated decoder back to the corresponding client to start the next round of federated learning training. To a certain extent, it alleviates the degradation of the local model performance caused by the heterogeneity of medical institutions in the medical image segmentation task and improves the training efficiency of the model. In practical applications, it shortens the training time required to retrain the federated learning model for medical image segmentation when new medical data is added, and at the same time improves the performance of the local segmentation models of each medical institution in segmenting local disease image data.

[0069] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0070] In the above embodiments of the present invention, the descriptions of the various embodiments have their own focuses. For the parts not detailed in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0071] In several embodiments provided by the present application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only illustrative. For example, the division of the units can be a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the units or modules can be in electrical or other forms.

[0072] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0073] In addition, in each embodiment of the present invention, each functional unit may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional unit.

[0074] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, may be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, read-only memories (ROMs), random access memories (RAMs), mobile hard disks, magnetic disks, or optical discs.

[0075] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of each embodiment of the present invention.

Claims

1. A personalized asynchronous federated learning method for medical image segmentation, characterized in that, It includes the following steps: Step 1: Each client initializes the encoder, decoder, and federated decoder parts in the local model, and uses local data to locally train the medical image segmentation model under the guidance of the federated decoder; Step 2: The client that has completed local training sends the encoder part and the federated decoder part to the server; After successfully sending to the server, the client asynchronously performs local updates on the local image segmentation model; Step 3: Local personalized update; When all clients have completed uploading, the server calculates the correlation between clients using the received model parameters and the held evaluation dataset, and aggregates and generates a personalized federated decoder for each client through personalized aggregation, and sends it back to the corresponding client; Step 4: When the client receives the personalized federated decoder sent by the server, it stops the asynchronous update of the local model and starts the next round of federated learning; Repeat the above steps until the model converges and the training is completed.

2. The personalized asynchronous federated learning method for medical image segmentation according to claim 1, wherein Each client model includes: an encoder, a decoder, and a federated decoder; the encoder and decoder are used to form the local image segmentation model; the federated decoder is used to achieve information exchange between all clients globally and to guide the training of the local image segmentation model.

3. A personalized asynchronous federated learning method for medical image segmentation according to claim 1, characterized in that, In Step 2, during the local update of the client, the local image segmentation model is iteratively updated using local data; Introduce the Dice loss function L Dice and the MSE loss function L MSE : Among them, N represents the total number of clients participating in federated learning; f i (·) represents the encoder part in the local model of client i; h i (·) represents the decoder part in the local model of client i; x represents the input image; j represents the j-th pixel in the image; y represents the ground truth of the segmentation image mask; ε represents the Laplacian smoothing term. The federated decoder in the client computes the similarity loss L taking the output of the local encoder as input Sim to guide the update of the local segmentation model; then the total loss function L for local update by client i is Total : L Total = L Dice + L MSE + L Sim (3); After the local update ends, the client sends the encoder and the federated decoder to the server, and the decoder remains local.

4. A personalized asynchronous federated learning method for medical image segmentation according to claim 3, wherein The calculated similarity loss L Sim is as follows: Among them, D i represents the local dataset in client i.

5. A personalized asynchronous federated learning method for medical image segmentation according to claim 1, characterized in that, In Step 3, when the server receives the model parameters sent by all clients, it starts model aggregation; To aggregate and generate a personalized federated decoder, the encoders of all clients received by the server are weighted and averaged with the data volume held by each client as the weight to generate a global encoder; To calculate the correlation degree between each client, we input the evaluation image dataset in the server into the global encoder, and sequentially input the output of the global encoder into each federated decoder sent to the server, and use the output of the decoder to calculate the cosine similarity between every two clients, which measures the similarity between each client; Calculate the similarity between client k and client i where i≠k; After obtaining the correlation degree between each pair of clients, a personalized model aggregation weight is calculated for each client based on the correlation degree between clients; the federated decoder of client i obtained by personalized model aggregation The aggregated and generated personalized federated decoders are respectively sent back to each client to guide the local model training in the next round of federated learning, so that the local model can balance between global general features and local features; and the parameters of each local segmentation model will not be replaced.

6. The personalized asynchronous federated learning method for medical image segmentation according to claim 5, wherein, The federated decoder of client i obtained by aggregating through the personalized model where λ represents the proportion of the federated decoder of client i in the previous round in generating the federated decoder in this round, and K represents the set of all clients except client i.

7. An individualized asynchronous federated learning method for medical image segmentation according to claim 1, characterized in that, In Step 4, after each client successfully uploads the local model, until it receives the aggregated model sent back by the server, it locally and asynchronously trains the local image segmentation model using local data.

Images