Efficient communication federated self-distillation method and system for medical segmentation

Abstract

The present invention provides an efficient communication federated self-distillation method and system for medical segmentation, comprising: a server node S and N institutional clients storing private medical data, wherein the server coordinates parameters and communicates with each client, and the client records the model parameters trained at the beginning of the tth round of communication; the client uses its private data set to start multiple rounds of model training locally; initializes the dynamic parameter difference decomposition threshold T i As well as the client model and weight parameters; the client starts multiple rounds of local training, and the client saves the model prediction results p during each round of training i And the deepest feature map f i The server reconstructs all parameter difference matrices and updates the server model. The client reconstructs the parameter difference information and updates the local model, starts the next communication round, and retrains multiple rounds locally using the private dataset until all client local models converge. This method can improve model training accuracy.

Description

Technical Field

[0001] The present invention relates to the field of federated learning technology, and in particular to an efficient communication federated self-distillation method and system for medical segmentation. Background Art

[0002] In the era of big data, big data within and across industries can be combined to generate enormous value. In the field of artificial intelligence, traditional data processing models are often centralized, with data collected by various parties being brought together for preprocessing, cleaning, and modeling. However, with the development and application of big data-related technologies and artificial intelligence, data security and privacy are receiving increasing attention from relevant departments and enterprises. Increasingly stringent big data privacy requirements have created the problem of "data silos." To address this issue, Google first proposed the concept of federated learning. Federated learning is a machine learning framework that enables data owners to conduct collaborative training while ensuring data security and privacy. This allows different data owners to collaborate on training without sharing data, sharing the value of data without sharing the data.

[0003] In existing related technical literature, [1] McMahan B, Moore E, Ramage D, et al. Communication-efficient learning of deep networks from decentralized data [C] / / Artificial intelligence and statistics. PMLR, 2017: 1273-1282. proposed the FL distributed machine learning paradigm, in which the FedAvg algorithm can effectively solve the data silo problem in distributed scenarios. However, the method only relies on federated averaging of client data, which cannot effectively guarantee high computational efficiency and low communication costs.

[0004] [2] Wu, Chuhan, et al. "Communication-efficient federated learning via knowledge distillation." Nature Communications 13.1(2022):2032. They proposed an adaptive mutual knowledge distillation and dynamic gradient compression technology to improve the efficiency of federated communication. This requires additional training of a complex teacher model to guide local model training, which increases the computational overhead.

[0005] [3] Kim K, Ji BM, Yoon D, et al. Self-knowledge distillation with progressive refinement of targets[C] / / Proceedings of the IEEE / CVFinternational conference on computer vision.2021:6567-6576. A simple and effective regularization method PS-KD is proposed. It combines the hard target and the soft target predicted in the past as the training target, and distills its own knowledge during the training process. This method does not fully learn from its own past knowledge and its application scenarios are relatively limited. Summary of the Invention

[0006] In view of the defects in the prior art, the present invention provides an efficient communication federated self-distillation method and system for medical segmentation.

[0007] According to the present invention, an efficient communication federated self-distillation method and system for medical segmentation is provided, and the scheme is as follows:

[0008] In a first aspect, an efficient communication federated self-distillation method for medical segmentation is provided, the method comprising:

[0009] Step S1: There is a server node S and N institutional clients C1,…,C N , the server coordinates and communicates parameters with each client, and the client C i It is necessary to record the model parameters W at the beginning of the tth round of communication training i,t ; Client C i Using its private dataset D i Start multiple rounds of model training locally; initialize the dynamic parameter difference decomposition threshold T i and client model and weight parameters;

[0010] Step S2: The client starts multiple rounds of local training using its private dataset. The client needs to save the model prediction results p for each round of training. i And the deepest feature map f i ;

[0011] Step S3: After receiving the compressed parameter differences uploaded by all clients, the server reconstructs all parameter difference matrices, aggregates the client parameter differences according to the client weights, and updates the server model;

[0012] Step S4: After receiving the information broadcast from the server, the client reconstructs the parameter difference information and updates the local model, starts the next communication round, and re-trains multiple rounds locally using the private dataset;

[0013] Step S5: Repeat steps S2 to S4 until all client local models converge.

[0014] Preferably, step S2 includes:

[0015] Step S2.1: During each round of training, the client uses the p saved in the previous training round i and f i Use the knowledge of the teacher model as the self-distillation operation, and calculate the prediction distillation loss PDLoss and feature distillation loss FDLoss with the prediction results of the current model training and the deepest feature map;

[0016] Step S2.2: The local model calculates the cross entropy loss MaskLoss between the prediction result and the true label, and performs a weighted sum of the three losses;

[0017] Step S2.3: After the local multiple rounds of training are completed, the client calculates the parameter difference between the current model and the pre-training model at each layer, and dynamically adjusts T according to the test loss of the model. i , dynamic parameter difference compression technology is used to compress the information that the model needs to upload, and then upload it to the server.

[0018] Preferably, the step S3 includes: the server calculates the parameter difference that needs to be updated for each layer of the client model respectively, decomposes it using dynamic parameter difference compression, and broadcasts it to the corresponding client.

[0019] In a second aspect, an efficient communication federated self-distillation system for medical segmentation is provided, the system comprising:

[0020] Module M1: There is a server node S and N institutional clients C1,…,C storing private medical data. N , the server coordinates and communicates parameters with each client, and the client C i It is necessary to record the model parameters W at the beginning of the tth round of communication training i,t ; Client C i Using its private dataset D i Start multiple rounds of model training locally; initialize the dynamic parameter difference decomposition threshold T i and client model and weight parameters;

[0021] Module M2: The client starts multiple rounds of local training using its private dataset. The client needs to save the model prediction results p for each round of training. iAnd the deepest feature map f i ;

[0022] Module M3: After receiving the compressed parameter differences uploaded by all clients, the server reconstructs all parameter difference matrices, aggregates the client parameter differences according to the client weights, and updates the server model;

[0023] Module M4: After receiving the information broadcast from the server, the client reconstructs the parameter difference information and updates the local model, starts the next communication round, and re-trains multiple rounds locally using the private dataset;

[0024] Module M5: Repeatedly trigger modules M2 to M4 until all client local models converge.

[0025] Preferably, the module M2 includes:

[0026] Module M2.1: During each round of training, the client uses the p saved in the previous training round i and f i Use the knowledge of the teacher model as the self-distillation operation, and calculate the prediction distillation loss PDLoss and feature distillation loss FDLoss with the prediction results of the current model training and the deepest feature map;

[0027] Module M2.2: The local model calculates the cross entropy loss MaskLoss between the prediction result and the true label, and performs a weighted sum of the three losses;

[0028] Module M2.3: After multiple rounds of local training, the client calculates the parameter difference between the current model and the pre-training model at each layer, and dynamically adjusts T according to the test loss of the model. i , dynamic parameter difference compression technology is used to compress the information that the model needs to upload, and then upload it to the server.

[0029] Preferably, the module M3 includes: the server calculates the parameter difference that needs to be updated for each layer of the client model, decomposes it using dynamic parameter difference compression, and broadcasts it to the corresponding client.

[0030] In a third aspect, a computer-readable storage medium storing a computer program is provided, which, when executed by a processor, implements the steps in the efficient communication federated self-distillation method for medical segmentation.

[0031] In a fourth aspect, an electronic device is provided, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the efficient communication federated self-distillation method for medical segmentation.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] 1. By adopting a federated learning architecture, this invention enables multiple medical institutions to collaborate and conduct decentralized training of multiple deep neural network models without sharing patient data, effectively avoiding the risk of data privacy leakage;

[0034] 2. The federated learning architecture FedSKD of the present invention decomposes the parameters of the federated communication upload and broadcast phases through dynamic parameter difference compression, while ensuring client privacy and security, which can significantly reduce the communication cost of federated learning.

[0035] 3. The present invention can improve the computational accuracy of the local model without increasing the computational cost by distilling the prediction knowledge and feature knowledge of its own past states. Self-knowledge distillation can also ensure the personalization of the local model to a certain extent.

[0036] Other beneficial effects of the present invention will be explained through the introduction of specific technical features and technical solutions in the specific implementation methods. Those skilled in the art should be able to understand the beneficial technical effects brought about by the introduction of these technical features and technical solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0038] Figure 1 It is the overall flow chart of the present invention;

[0039] Figure 2 This is a system block diagram of the present invention. DETAILED DESCRIPTION

[0040] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0041] Embodiments of the present invention provide an efficient, communication-based federated self-distillation method for medical segmentation. When the client uploads and the server broadcasts parameter differences, the decomposition threshold is dynamically adjusted based on training results. This significantly reduces the communication parameter size between the client and server while minimizing the impact on model accuracy, achieving a good balance between communication cost and model accuracy. Unlike traditional knowledge distillation, which requires the additional training of a cumbersome teacher model, the present invention's self-knowledge distillation framework for local models does not require additional computational overhead. The client model can extract prediction and feature knowledge from its past training rounds to self-distill, thereby improving model training accuracy.

[0042] The present invention comprises two types of node components:

[0043] 1. Client node: A participant in the federated collaboration. Each client uses its own private dataset for local training. After completing local training, the client uploads its model to the server for aggregation. There is no direct communication between clients.

[0044] 2. Server node: The server does not participate in the system's training process. It needs to aggregate the information uploaded by the client and then broadcast it to each client to collaborate with all clients to jointly train a more powerful model.

[0045] Reference Figure 1 and Figure 2 As shown, the method specifically includes:

[0046] Step S1: There is a server node S and N institutional clients C1,…,C N , the server coordinates and communicates parameters with each client, and the client C i It is necessary to record the model parameters W at the beginning of the tth round of communication training i,t ; Client C i Using its private dataset D i Start multiple rounds of model training locally; initialize the dynamic parameter difference decomposition threshold T i and client model and weight parameters;

[0047] Step S2: The client starts multiple rounds of local training using its private dataset. The client needs to save the model prediction results p for each round of training. i And the deepest feature map f i ;

[0048] The step S2 specifically includes:

[0049] Step S2.1: During each round of training, the client uses the p saved in the previous training round i and f iUse the knowledge of the teacher model as the self-distillation operation, and calculate the prediction distillation loss PDLoss and feature distillation loss FDLoss with the prediction results of the current model training and the deepest feature map;

[0050] Step S2.2: The local model calculates the cross entropy loss MaskLoss between the prediction result and the true label, and performs a weighted sum of the three losses. Since the model predictions in later rounds will be more accurate and the feature information will be clearer, the weights of these two losses should gradually increase with the number of rounds.

[0051] Step S2.3: After the local multiple rounds of training are completed, the client calculates the parameter difference of each layer between the current model and the pre-training model, and dynamically adjusts the dynamic parameter difference decomposition threshold T according to the test loss of the model. i , dynamic parameter difference compression technology based on singular value decomposition is used to compress the gradient difference that needs to be uploaded by the model, and then upload it to the server.

[0052] Step S3: After receiving the compressed parameter differences uploaded by all clients, the server reconstructs all parameter difference matrices, aggregates and accumulates the products of all client weights and client parameter differences, and updates the server model.

[0053] The server calculates the parameter difference that needs to be updated for each layer of the client model, decomposes it using dynamic parameter difference compression, and broadcasts it to the corresponding client.

[0054] Step S4: After receiving the information broadcast from the server, the client reconstructs the parameter difference information and updates the local model, starts the next communication round, and re-trains multiple rounds locally using the private dataset;

[0055] Step S5: Repeat steps S2 to S4 until all client local models converge.

[0056] The present invention also provides an efficient communication federated self-distillation system for medical segmentation. The efficient communication federated self-distillation system for medical segmentation can be implemented by executing the process steps of the efficient communication federated self-distillation method for medical segmentation. That is, those skilled in the art can understand the efficient communication federated self-distillation method for medical segmentation as a preferred embodiment of the efficient communication federated self-distillation system for medical segmentation. The system specifically includes the following contents:

[0057] Module M1: Assume there is a server node S and N institutional clients C1,…,C N , the server coordinates and communicates parameters with each client, and the client C i It is necessary to record the model parameters W at the beginning of the tth round of communication training i,t; Client C i Using its private dataset D i Start multiple rounds of model training locally; initialize the dynamic parameter difference decomposition threshold T i and client model and weight parameters;

[0058] Module M2: The client starts multiple rounds of local training using its private dataset. The client needs to save the model prediction results p for each round of training. i And the deepest feature map f i .

[0059] The module M2 specifically includes:

[0060] Module M2.1: During each round of training, the client uses the model prediction result p saved in the previous training round i And the deepest feature map f i The self-distillation operation is performed as the knowledge of the teacher model, and the prediction distillation loss PDLoss and feature distillation loss FDLoss are calculated with the prediction results of the current model training and the deepest feature map.

[0061] Module M2.2: The local model calculates the cross entropy loss MaskLoss between the prediction result and the true label, and performs a weighted sum of the three losses. Since the model predictions in later rounds will be more accurate and the feature information will be clearer, the weights of these two losses should gradually increase with the number of rounds.

[0062] Module M2.3: After multiple rounds of local training, the client calculates the parameter difference between the current model and the pre-training model at each layer, and dynamically adjusts the dynamic parameter difference decomposition threshold T according to the model's test loss. i , dynamic parameter difference compression technology based on singular value decomposition is used to compress the gradient difference that needs to be uploaded by the model, and then upload it to the server.

[0063] Module M3: After receiving the compressed parameter differences uploaded by all clients, the server reconstructs all parameter difference matrices, aggregates and accumulates the products of all client weights and client parameter differences, and updates the server model.

[0064] The server calculates the parameter difference that needs to be updated for each layer of the client model, decomposes it using dynamic parameter difference compression, and broadcasts it to the corresponding client.

[0065] Module M4: After receiving the information broadcast from the server, the client reconstructs the parameter difference information and updates the local model, starts the next communication round, and re-trains multiple rounds locally using the private dataset;

[0066] Module M5: Repeatedly trigger modules M2 to M4 until all client local models converge.

[0067] The embodiment of the present invention provides an efficient communication federated self-distillation method and system for medical segmentation. By adopting a federated learning architecture, multiple medical institutions can collaborate and conduct decentralized training of multiple deep neural network models without sharing patient data, effectively avoiding the risk of data privacy leakage. Under the premise of client privacy and security, the federated learning architecture FedSKD decomposes the parameters of the federated communication upload and broadcast stages through dynamic parameter difference compression, which can significantly reduce the communication cost of federated learning. By distilling the predictive knowledge and feature knowledge of its own past states, the computational accuracy of the local model can be improved without increasing the computational cost. The personalization of the local model can also be guaranteed to a certain extent through self-knowledge distillation.

[0068] Those skilled in the art will appreciate that, in addition to implementing the system and its various devices, modules, and units provided by the present invention in purely computer-readable program code, it is entirely possible to implement the same functions of the system and its various devices, modules, and units provided by the present invention in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; the devices, modules, and units for implementing various functions can also be considered as both software modules implementing the method and structures within the hardware component.

[0069] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. The embodiments of this application and the features in the embodiments may be combined with each other in any manner unless there is a conflict.

Claims

1. An efficient communication federated self-distillation method for medical segmentation, characterized by: include: Step S1: Set up a server node S and N institutional clients C1, ..., C storing private medical data. N , the server coordinates and communicates parameters with each client, and the client C i It is necessary to record the model parameters W at the beginning of the tth round of communication training i，t ; Client C i Using its private dataset D i Start multiple rounds of model training locally; Initialize the dynamic parameter difference decomposition threshold T i and client model and weight parameters; Step S2: The client starts multiple rounds of local training using its private dataset. The client needs to save the model prediction results p for each round of training. i And the deepest feature map f i ; Step S3: After receiving the compressed parameter differences uploaded by all clients, the server reconstructs all parameter difference matrices, aggregates and accumulates the products of all client weights and client parameter differences, and updates the server model; Step S4: After receiving the information broadcast from the server, the client reconstructs the parameter difference information and updates the local model, starts the next communication round, and re-trains multiple rounds locally using the private dataset; Step S5: Repeat steps S2 to S4 until all client local models converge; The step S2 comprises: Step S2.1: During each round of training, the client uses the model prediction result p saved in the previous training round i And the deepest feature map f i Use the knowledge of the teacher model as the self-distillation operation, and calculate the prediction distillation loss PDLoss and feature distillation loss FDLoss with the prediction results of the current model training and the deepest feature map; Step S2.2: The local model calculates the cross entropy loss MaskLoss between the prediction result and the true label, and performs a weighted sum of the three losses; Step S2.3: After the local multiple rounds of training are completed, the client calculates the parameter difference of each layer between the current model and the pre-training model, and dynamically adjusts the dynamic parameter difference decomposition threshold T according to the test loss of the model. i , dynamic parameter difference compression technology based on singular value decomposition is used to compress the gradient difference that needs to be uploaded by the model, and then upload it to the server.

2. The efficient communication federated self-distillation method for medical segmentation according to claim 1, characterized in that The step S3 includes: the server calculates the parameter difference that needs to be updated for each layer of the client model, decomposes it using dynamic parameter difference compression, and broadcasts it to the corresponding client.

3. An efficient communication federated self-distillation system for medical segmentation, characterized by: include: Module M1: There is a server node S and N institutional clients C1, ..., C storing private medical data. N , the server coordinates and communicates parameters with each client, and the client C i It is necessary to record the model parameters W at the beginning of the tth round of communication training i，t ; Client C i Using its private dataset D i Start multiple rounds of model training locally; initialize the dynamic parameter difference decomposition threshold T i and client model and weight parameters; Module M2: The client starts multiple rounds of local training using its private dataset. The client needs to save the model prediction results p for each round of training. i And the deepest feature map f i ; Module M3: After receiving the compressed parameter differences uploaded by all clients, the server reconstructs the parameter difference matrix, aggregates and accumulates the product of all client weights and client parameter differences, and updates the server model; Module M4: After receiving the information broadcast from the server, the client reconstructs the parameter difference information and updates the local model, starts the next communication round, and re-trains multiple rounds locally using the private dataset; Module M5: Repeatedly trigger modules M2 to M4 until all client local models converge; The module M2 includes: Module M2.1: During each round of training, the client uses the model prediction result p saved in the previous training round i And the deepest feature map f i Use the knowledge of the teacher model as the self-distillation operation, and calculate the prediction distillation loss PDLoss and feature distillation loss FDLoss with the prediction results of the current model training and the deepest feature map; Module M2.2: The local model calculates the cross entropy loss MaskLoss between the prediction result and the true label, and performs a weighted sum of the three losses; Module M2.3: After multiple rounds of local training, the client calculates the parameter difference between the current model and the pre-training model at each layer, and dynamically adjusts T according to the test loss of the model. i , dynamic parameter difference compression technology based on singular value decomposition is used to compress the gradient difference that needs to be uploaded by the model, and then upload it to the server.

4. The efficient communication federated self-distillation system for medical segmentation according to claim 3, characterized in that The module M3 includes: the server calculates the parameter difference that needs to be updated for each layer of the client model, decomposes it using dynamic parameter difference compression, and broadcasts it to the corresponding client.

5. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the efficient communication federated self-distillation method for medical segmentation according to any one of claims 1 to 2 are implemented.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the computer program is executed by a processor, the steps of the efficient communication federated self-distillation method for medical segmentation according to any one of claims 1 to 2 are implemented.

Images