Federated learning optimization method and apparatus, system, storage medium
By using a method that generates high-quality samples through cross-cluster distillation and generator training, the inconsistency between models among medical institutions in federated learning is solved, achieving efficient fusion and optimization of the global model and improving the adaptability and generalization ability of cross-border medical data.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GUANGDONG UNIV OF TECH
- Filing Date
- 2025-06-16
- Publication Date
- 2026-05-15
AI Technical Summary
Existing federated learning methods struggle to effectively address the inconsistencies in model knowledge and data heterogeneity among medical institutions in cross-border medical collaborations, leading to a degradation in global model performance, particularly in cross-modal and cross-disease scenarios.
A method using cross-cluster distillation strategy and generator training to generate effective hard samples is adopted. The medical institutions are divided by server-side clustering, and the similarity of model parameters and data statistics is evaluated. High-quality synthetic samples are generated by the generator for global model fine-tuning. Combined with a buffer pool mechanism, key samples are selected and stored to achieve cross-institutional knowledge fusion and optimization.
It significantly alleviates the aggregation bias caused by the inconsistency of model knowledge among medical institutions, improves the adaptability of the global model to niche medical institutions and abnormal data, and enhances the model's convergence speed and generalization ability.
Smart Images
Figure CN120705612B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of data processing technology, and in particular relates to a federated learning optimization method, apparatus, system, and storage medium. Background Technology
[0002] In the field of medical artificial intelligence, individual medical centers struggle to acquire large-scale, diverse datasets for training machine learning (ML) models. Traditional centralized AI requires transferring privacy-sensitive medical data from medical institutions to data centers for fusion training. This model not only faces the risk of patient privacy breaches (such as HIPAA / GDPR compliance challenges) but also incurs high communication resource consumption and energy costs (especially in cross-border data transmission scenarios). Existing research indicates that these issues have become a major bottleneck for conducting cutting-edge scientific research in cross-border medical collaborations. Federated Learning (FL), as an emerging distributed AI framework, achieves the goal of privacy protection by avoiding the sharing of raw data through a collaborative mechanism of localized data training and server-side parameter aggregation. In cross-border medical scenarios, FL allows each medical institution to independently train models based on local data, sharing only encrypted model update parameters (such as weight gradients and data statistics) with the collaborative network, thereby jointly optimizing the global model while avoiding the exposure of raw data. This paradigm not only breaks through the privacy and communication bottlenecks of traditional centralized training, but also significantly improves the model's generalization ability to multi-source heterogeneous medical data (such as image modalities from different devices and regionalized disease distribution) through cross-center knowledge aggregation, providing an innovative technical path for in-depth cooperation in transnational medical research.
[0003] While this method offers good privacy protection and scalability, it often faces severe performance degradation due to non-individualized (Non-IID) heterogeneity in institutional medical data in real-world applications. Traditional methods such as FedAvg, FedProx, SCAFFOLD, and FedGen primarily mitigate inconsistencies between local and global models through model constraints or knowledge distillation, but they struggle to adapt to the characteristic differences across modalities (e.g., MRI / CT) and disease types (e.g., glioma / metastasis). Clustering Federated Learning (CFL) reduces local model conflicts by clustering by case type, but it relies solely on data distribution clustering, ignoring differences in model parameters, which leads to suboptimal convergence of the global model. Summary of the Invention
[0004] The technical problem to be solved by the present invention is to provide a federated learning optimization method, apparatus, system, and storage medium.
[0005] To achieve the above objectives, the present invention adopts the following technical solution:
[0006] A federated learning optimization method includes:
[0007] Step S1: Each client receives the global model and initializes it as a local model for training;
[0008] Step S2: The client uploads model parameter updates to the server. Simultaneously upload local medical data information;
[0009] Step S3: After receiving the model parameters and data statistics from the client, the server performs clustering partitioning;
[0010] Step S4: Use a cross-cluster distillation strategy to fuse knowledge between clusters;
[0011] Step S5: The training generator is trained to generate effective hard samples for fine-tuning the global model.
[0012] Step S6: Select effective hard samples from the buffer pool based on the sample indicators for global model fine-tuning;
[0013] Step S7: Broadcast the fine-tuned global model and generator to each client.
[0014] Preferably, in step S3, each client It is represented by two key elements: vectorized local model parameters and data distribution statistics The similarity between clients is defined as follows:
[0015]
[0016] in, This represents the distance metric algorithm. The weights are represented; the cluster distance is defined as follows:
[0017]
[0018] in, and This represents two clusters; the clustering termination condition is defined as follows:
[0019]
[0020] For clustering The aggregation model parameters are defined as follows:
[0021]
[0022] Preferably, in step S4, the distillation process is defined as follows:
[0023]
[0024] Among them, and This is for estimating the prior label distribution and the corresponding posterior distribution. and These represent the feature extractor and classifier head of the model, respectively. Let be the loss function; the local empirical loss is defined as:
[0025]
[0026] Distillation loss between clusters is defined as:
[0027]
[0028]
[0029] in, This represents the number of clusters.
[0030] Preferably, in step S5, the generator The training objective is defined as follows:
[0031]
[0032] The objective optimization for generating an effective hard sample generator is as follows:
[0033]
[0034] in, Represents entropy;
[0035] The metrics for measuring sample quality are defined as follows:
[0036]
[0037] The objective of fine-tuning the global model is defined as follows:
[0038]
[0039] in, It serves as a buffer pool.
[0040] The present invention also provides a federated learning optimization apparatus, comprising:
[0041] The first processing module is used to receive the global model from each client and initialize it as a local model for training.
[0042] The second processing module is used by the client to upload model parameter updates to the server. Simultaneously upload local medical data information;
[0043] The third processing module is used to perform clustering partitioning after receiving the model parameters and data statistics from the client.
[0044] The fourth processing module is used to fuse knowledge between clusters using a cross-cluster distillation strategy;
[0045] The fifth processing module is used to train the generator to generate effective hard samples for fine-tuning the global model.
[0046] The sixth processing module is used to select effective hard samples from the buffer pool based on the sample's metrics for global model fine-tuning.
[0047] The seventh processing module is used to broadcast the fine-tuned global model and generator to each client.
[0048] The present invention also provides a federated learning optimization system, comprising: a memory and a processor, wherein the memory stores a computer program executed by the processor, and the computer program performs a federated learning optimization method when executed by the processor.
[0049] The present invention also provides a storage medium storing a computer program that executes a federated learning optimization method during runtime.
[0050] This invention comprises three key components: clustering, cross-cluster distillation, and global model fine-tuning. Each medical institution acts as a client, completing model training locally and uploading model parameters and medical data statistics. A data center acts as a server, calculating the similarity between medical institutions to form clusters, achieving a divide-and-conquer approach. Within each cluster, local models are aggregated into aggregate models, each adept at identifying specific pathological features. These aggregate models are then broadcast to clients for cross-cluster knowledge distillation, fusing specific pathological feature knowledge from different clusters and summarizing it into a global model, enabling the global model to adapt to various medical data. Finally, to further enhance the global model, a server-side generator is introduced to generate high-quality hard samples (rare medical data features), which are evaluated using customized quality metrics. A buffer pool stores the best samples, and these samples are selectively replayed through a memory-based mechanism to fine-tune the global model. This fine-tuning stage effectively mitigates model drift and preserves crucial local medical knowledge. This invention offers the following advantages:
[0051] 1. It can alleviate the aggregation bias caused by the inconsistency of model knowledge among medical institutions in different countries; 2. It can improve the adaptability of the global model to niche medical institutions or abnormal data; 3. It can significantly improve the convergence speed and generalization ability of the model by using hard samples and clustering strategies. Attached Figure Description
[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0053] Figure 1 This is a schematic diagram illustrating an application scenario of the federated learning optimization method according to an embodiment of the present invention;
[0054] Figure 2 This is a flowchart of the federated learning optimization method according to an embodiment of the present invention. Detailed Implementation
[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0056] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0057] Example 1:
[0058] like Figure 1 As shown, this embodiment of the invention provides a federated learning optimization method, including:
[0059] Step S1: Each client receives the global model and initializes it as a local model for training;
[0060] Step S2: The client uploads model parameter updates to the server. Simultaneously upload local medical data information;
[0061] Step S3: After receiving the model parameters and data statistics from the client, the server performs clustering partitioning;
[0062] Step S4: Use a cross-cluster distillation strategy to fuse knowledge between clusters;
[0063] Step S5: The training generator is trained to generate effective hard samples for fine-tuning the global model.
[0064] Step S6: Select effective hard samples from the buffer pool based on the sample indicators for global model fine-tuning;
[0065] Step S7: Broadcast the fine-tuned global model and generator to each client.
[0066] As one embodiment of the present invention, in step S3, each client It is represented by two key elements: vectorized local model parameters and data distribution statistics The similarity between clients is defined as follows:
[0067]
[0068] in, This represents the distance metric algorithm. The weights are represented; the cluster distance is defined as follows:
[0069]
[0070] in, and This represents two clusters; the clustering termination condition is defined as follows:
[0071]
[0072] For clustering The aggregation model parameters are defined as follows:
[0073]
[0074] In one embodiment of the present invention, the distillation process in step S4 is defined as follows:
[0075]
[0076] Among them, and This is for estimating the prior label distribution and the corresponding posterior distribution. and These represent the feature extractor and classifier head of the model, respectively. Let be the loss function; the local empirical loss is defined as:
[0077]
[0078] Distillation loss between clusters is defined as:
[0079]
[0080]
[0081] in, This represents the number of clusters.
[0082] As one embodiment of the present invention, in step S5, the generator The training objective is defined as follows:
[0083]
[0084] The objective optimization for generating an effective hard sample generator is as follows:
[0085]
[0086] in, Represents entropy;
[0087] The metrics for measuring sample quality are defined as follows:
[0088]
[0089] The objective of fine-tuning the global model is defined as follows:
[0090]
[0091] in, It serves as a buffer pool.
[0092] Example 2:
[0093] like Figure 2 As shown, this embodiment of the invention provides a federated learning optimization method in a data Non-IID environment, including:
[0094] 1) Local Model Updates and Data Statistics: In cross-border medical collaborations, the clients are various medical institutions worldwide (such as the Mayo Clinic in the United States, Peking Union Medical College Hospital in China, and Heidelberg University Hospital in Germany), and the servers are cross-border medical data centers (such as neutral third-party medical alliance platforms). After each medical institution completes model training locally, it performs the following operations:
[0095] The server sends global model parameters to all clients. and generator Each client Initialization parameters: Train the model on local medical data to obtain local model update parameters. .
[0096] To address the highly non-independent identically distributed (Non-IID) nature of multi-center medical data and cross-border privacy compliance requirements (such as HIPAA, GDPR, and China's Personal Information Protection Law), medical institutions need to achieve clustering without sharing original data within a federated learning framework. Therefore, clients use concatenated vectors to statistically analyze local data distribution. During local training, the number of local medical cases, case categories, and specific pathological characteristics are statistically analyzed. Specifically, for each... , Indicates the number of cases, Indicates the number of samples for each case category, This represents specific pathological features. These components are then connected into a unified representation. This is used to describe the data distribution. Then... Its model parameters and data distribution Send to the server.
[0097] 2) Clustering: After receiving model parameters and data statistics from all clients, the server divides the data into clusters using a hierarchical clustering method. Each client... It is represented by two key elements: vectorized local model parameters and data distribution statistics .
[0098] The first step in clustering is to calculate similarity by evaluating model updates and data distribution. Specifically, for the client... and The similarity between them is calculated as follows:
[0099] (1)
[0100] in A Indicates that the distance metric algorithm belongs to .For example (Manhattan) (Euclidean) and cosine distance metrics. These are the weighting coefficients for the balance parameter and the importance of distribution similarity. Based on this, our method constructs a similarity matrix based on equation (1). .
[0101] The second step is to define the cluster distance. Since hierarchical clustering minimizes the distance (i.e., maximizes similarity), this method clusters two groups... and The distance between them is defined as:
[0102] (2)
[0103] In other words, the cluster distance is determined using the furthest (least similar) pairs. For any cluster... The difference between the maximum and minimum intra-cluster dissimilarity is less than a threshold. Stop clustering when:
[0104] (3)
[0105] In the After the global iteration completes the clustering process, the resulting cluster set is represented as: ,in This represents the number of clusters in this iteration. For each cluster... Perform a weighted average to obtain the aggregate model parameters:
[0106] (4)
[0107] in, The size of the dataset used for training, This represents the corresponding model parameters. The aggregation model for each cluster is adept at specific pathological features. This invention will... The model parameters for all clusters in the next iteration are expressed as follows: Each of them Clustering The average model parameters. Finally, the parameter set... The broadcast was sent to every medical facility.
[0108] 3) Knowledge Distillation: In cross-border medical collaboration scenarios, knowledge distillation mechanisms achieve efficient transfer of diagnostic experience across institutions by integrating medical knowledge from different regions, modalities, and diseases. The client receives data from the server... It belongs to clustering. Each client It will use its corresponding clustering model As its student model, that is, Models from other clusters will serve as teacher models in the distillation process, enabling diagnostic experience learning among medical institutions through cross-cluster distillation. The learning objective for each client consists of three parts: i) empirical risk on the local dataset; ii) cross-cluster distillation; and iii) a loss term for correctly predicting on the generated features. The complete objective definition is as follows:
[0109] (5)
[0110] Here, the local experience loss is defined as:
[0111] (6)
[0112] Distillation loss between clusters is defined as:
[0113] (7)
[0114] in and These represent the feature extractor and classifier head of the model, respectively. The loss function is defined as follows: After local training is completed using cross-cluster distillation, each client updates its model parameters. Uploaded to the server.
[0115] 4) Generator Training: This invention uses a data-free knowledge distillation method. During the distillation process, simulated medical data is generated by a learning generator. To generate synthetic samples that are consistent with the global distribution of medical data and conform to regional characteristics, a conditional distribution needs to be learned. : To capture "disease labels" Y →Image Feature Space X The mapping relationship between " and " should be established. This distribution should approximate the distribution of real medical data while maintaining consistency with the observed statistical characteristics:
[0116] (8)
[0117] in and Given the true prior label distribution and its corresponding posterior distribution, since the true distribution is unavailable, we use an empirical approximation of them. The prior distribution is estimated as follows:
[0118] (9)
[0119] in, It is an indicator function. This can be obtained through the label statistics provided by the client during the model upload process. Similarly, the posterior distribution... Approximation is achieved by aggregating knowledge from the model:
[0120] (10)
[0121] By substituting these approximations into formula (8), the present invention makes the objective function computationally feasible. However, directly in the image feature space... X The above operations may lead to computational inefficiency and pose a potential risk of privacy breaches. To address this issue, this invention instead focuses on the latent feature space. Learning a compact induced distribution This method improves both efficiency and privacy protection:
[0122] (11)
[0123] Based on the above formula, this invention trains a conditional generator by minimizing the following objective function. :
[0124] (12)
[0125] Wherein, the model predictor is represented. The logit output, then through The function generates probability predictions. The generator is based on the labels. Output Samples And only the predictor (i.e. the model head) of the client model needs to be accessed.
[0126] 5) Global model fine-tuning: After each client learns the clustered medical knowledge, the server adjusts the local model parameters. Aggregation yields a global model :
[0127] (13)
[0128] However, directly aggregating local models from medical institutions is prone to loss of local diagnostic knowledge due to differences in data modalities and skewed disease distribution. To address this issue, this invention employs a server-side generator to simulate a dual-loss fine-tuning mechanism involving medical feature samples and the global model, achieving both local knowledge preservation and global performance optimization. The specific solution is as follows:
[0129] Specifically, the server uses a generator to simulate samples. z This was used to reflect the distribution of global healthcare data. These samples were then used to optimize the global model parameters. This is achieved by minimizing the following objective function:
[0130] (14)
[0131] Each local model With global model Model differences Given by the Kullback-Leibler (KL) divergence:
[0132] (15)
[0133] However, in heterogeneous data scenarios, the synthetic samples generated by formula (12) often overfit the feature distribution of a single region, resulting in excessively strong sample discriminative power. This makes it difficult to cover the complex modal differences and disease variations in cross-regional medical care, thus limiting the optimization effect of the global model on marginal cases. To solve this problem, this invention addresses the multimodal heterogeneity and disease distribution of cross-regional medical data by proposing to force the generator to learn the high uncertainty features across regions through entropy regularization of the generator output, thereby obtaining a new training loss function:
[0134] (16)
[0135] here, Indicates the global model in the samples The entropy of the prediction result: where Entropy is represented by minimizing negative entropy, which is then maximized.
[0136] (17)
[0137] By maximizing entropy (i.e. minimizing) The generator is forced to output hard samples whose categories the model struggles to determine; these samples are prevalent at the "data boundary" in cross-border healthcare. However, these samples can also introduce noise, thus degrading model performance. Therefore, a multi-index measurement method is proposed. This method is used to evaluate the quality of generated difficult samples. It comprehensively considers entropy, prediction variance, and gradient-based uncertainty metrics.
[0138] (18)
[0139] The classification loss for each client is defined as follows:
[0140] (19)
[0141] A high value indicates It could be noise or inconsistency with the data distribution.
[0142] To ensure the quality of hard samples during training iterations, a buffer pool was introduced. Used to store and reuse scores based on their ratings High-quality samples selected This buffer pool stores a certain number of tuples. ,in The buffer pool is dynamically updated: it retains samples with high scores and discards samples with low quality, thereby achieving stable and effective fine-tuning.
[0143] The ultimate goal of fine-tuning the global model is to minimize the model variance loss and the supervision loss on the high-quality, hard samples selected from the buffer pool.
[0144] (20)
[0145] The monitoring loss is defined as follows:
[0146] (twenty one)
[0147] By minimizing the loss function This can effectively improve the generalization ability of the global model to data from different regions, modalities, and diseases.
[0148] 6) Global Broadcast: After completing the fine-tuning of the global model, the server will broadcast the updated global model parameters. and generator Broadcast to all clients (medical institutions). Enable multi-center collaborative modeling, ensuring that each medical institution receives the latest shared knowledge adapted to regional data characteristics.
[0149] The algorithm is shown in the table below:
[0150]
[0151] Example 2:
[0152] This invention also provides a federated learning optimization apparatus, comprising:
[0153] The first processing module is used to receive the global model from each client and initialize it as a local model for training.
[0154] The second processing module is used by the client to upload model parameter updates to the server. Simultaneously upload local medical data information;
[0155] The third processing module is used to perform clustering partitioning after receiving the model parameters and data statistics from the client.
[0156] The fourth processing module is used to fuse knowledge between clusters using a cross-cluster distillation strategy;
[0157] The fifth processing module is used to train the generator to generate effective hard samples for fine-tuning the global model.
[0158] The sixth processing module is used to select effective hard samples from the buffer pool based on the sample's metrics for global model fine-tuning.
[0159] The seventh processing module is used to broadcast the fine-tuned global model and generator to each client.
[0160] Example 3:
[0161] This invention also provides a federated learning optimization system, comprising: a memory and a processor, wherein the memory stores a computer program executed by the processor, and the computer program executes a federated learning optimization method when run by the processor.
[0162] Example 4:
[0163] This invention also provides a storage medium storing a computer program that executes a federated learning optimization method during runtime.
[0164] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.
Claims
1. A federated learning optimization method, characterized in that, include: Step S1: Each client receives the global model and initializes it as a local model for training; Step S2: The client uploads model parameter updates to the server. Simultaneously upload local medical data information; Step S3: After receiving the model parameters and data statistics from the client, the server performs clustering partitioning; Step S4: Use a cross-cluster distillation strategy to fuse knowledge between clusters; Step S5: The training generator is trained to generate effective hard samples for fine-tuning the global model. Step S6: Select effective hard samples from the buffer pool based on the sample indicators for global model fine-tuning; Step S7: Broadcast the fine-tuned global model and generator to each client; In this system, each medical institution acts as a client, completing model training locally and then uploading model parameters and medical data statistics. The data center acts as a server, calculating the similarity between medical institutions to form clusters, achieving a divide-and-conquer approach. Within each cluster, local models are aggregated into aggregate models, which are used for specific pathological feature identification. The aggregate models are broadcast to clients for cross-cluster knowledge distillation, fusing specific pathological feature knowledge from different clusters and summarizing it into a global model, making the global model adaptable to various medical data. A server-side generator is introduced to generate high-quality hard samples, which are evaluated using customized quality metrics. Optimal samples are stored in a buffer pool, and samples are selectively replayed using a memory-based mechanism to fine-tune the global model. For each... , Indicates the number of cases, Indicates the number of samples for each case category, It represents specific pathological features; these components are connected into a unified representation. , used to describe data distribution; Its model parameters and data distribution Send to the server; In step S3, each client It is represented by two key elements: vectorized local model parameters and data distribution statistics The similarity between clients is defined as follows: in, This represents the distance metric algorithm. The weights are represented; the cluster distance is defined as follows: in, and This represents two clusters; the clustering termination condition is defined as follows: For clustering The aggregation model parameters are defined as follows: ， in, The size of the dataset used for training; In step S4, the distillation process is defined as follows: in, and This is for estimating the prior label distribution and the corresponding posterior distribution. and These represent the feature extractor and classifier head of the model, respectively. Let be the loss function; the local empirical loss is defined as: Distillation loss between clusters is defined as: in, The number of clusters; In step S5, the generator The training objective is defined as follows: The objective optimization for generating an effective hard sample generator is as follows: in, Represents entropy; The metrics for measuring sample quality are defined as follows: in, Classification loss for each client; The objective of fine-tuning the global model is defined as follows: in, For each local model With global model Differences between models, To monitor losses, It serves as a buffer pool.
2. A federated learning optimization apparatus for implementing the federated learning method of claim 1, characterized in that, include: The first processing module is used to receive the global model from each client and initialize it as a local model for training. The second processing module is used by the client to upload model parameter updates to the server. Simultaneously upload local medical data information; The third processing module is used to perform clustering partitioning after receiving the model parameters and data statistics from the client. The fourth processing module is used to fuse knowledge between clusters using a cross-cluster distillation strategy; The fifth processing module is used to train the generator to generate effective hard samples for fine-tuning the global model. The sixth processing module is used to select effective hard samples from the buffer pool based on the sample's metrics for global model fine-tuning. The seventh processing module is used to broadcast the fine-tuned global model and generator to each client.
3. A federated learning optimization system, characterized in that, include: A memory and a processor, wherein the memory stores a computer program executed by the processor, the computer program performing the federated learning optimization method as described in claim 1 when executed by the processor.
4. A storage medium, characterized in that, The storage medium stores a computer program, which executes the federated learning optimization method as described in claim 1 when it runs.