Asynchronous federated learning method and system based on mutual information and medium
By calculating the mutual information value between the client model update and the global model, screening and weighted aggregation for effective updates, the problem of suboptimal model performance in asynchronous federated learning is solved, and the adaptability and performance of the model in data heterogeneous environments are improved.
Patent Information
- Application Number
- CN202510661393.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-22
- Publication Date
- 2025-09-12
AI Technical Summary
Existing asynchronous federated learning methods cannot effectively evaluate the actual value of model updates, especially when client data distribution is uneven, it is easy to lose unique data distribution information, resulting in suboptimal model performance.
By calculating the mutual information value between the model update submitted by the client and the current global model, valid model updates are screened out and weighted aggregation is performed to generate a new global model.
It improves the adaptability and generalization ability of the model in complex data environments, enhances the flexibility and scalability of the system, protects client data privacy, and improves the performance and communication efficiency of the global model.
Smart Images
Figure CN120633772A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of asynchronous federated learning, and in particular to a mutual information-based asynchronous federated learning method, system, and medium. Background Art
[0002] With the rapid development of artificial intelligence (AI), machine learning models are increasingly being used in diverse fields. Traditional machine learning methods typically require all data to be centralized on a central server for training. This approach not only poses privacy risks but also suffers from low communication efficiency when dealing with large amounts of data and dispersed devices.
[0003] As an emerging distributed machine learning paradigm, federated learning allows multiple clients to train models locally. They only need to share model parameters without uploading original data, which effectively protects data privacy. However, traditional synchronous federated learning requires all clients to train and upload model updates according to a unified time node, which faces many challenges in practical applications. For example, most asynchronous federated learning methods mainly adjust weights based on the lag of model updates (that is, the higher the lag, the lower the weight). Although this method can alleviate the negative impact of stale updates, it cannot effectively address the challenge of data heterogeneity. Specifically, when some clients have unique data distributions, even if the updates they submit lag behind the current global model, these updates may still contain information that is critical to the global model. The simple time-based weight adjustment mechanism ignores the value of the update content itself, which may lead to suboptimal model performance, especially in scenarios where client data distribution is highly uneven. Summary of the Invention The present application provides an asynchronous federated learning method, system, and medium based on mutual information to address the problem that the existing asynchronous federated learning method based on lag degree adjustment weights cannot effectively evaluate the actual value of model updates, and is prone to losing unique data distribution information in a heterogeneous client data environment.
[0004] In a first aspect, the present application provides an asynchronous federated learning method based on mutual information, the method comprising: The server sends the global model to all clients and sets a time window after sending. The client obtains the global model, trains the global model based on local data, and sends model updates and requests a new global model to the server after training is complete. The server continuously receives model updates uploaded by different clients within a time window. After the time window ends, the server calculates the mutual information value between the model update submitted by each client and the current global model. Based on the mutual information value, the server obtains the valid model update. The valid model updates are weightedly aggregated to obtain a new global model. The new global model is distributed to the clients that requested the update.
[0005] In one implementation of the present application, the server sends the global model to all clients and sets a time window after the sending is completed, specifically including: When the global model is first issued, The server randomly initializes the global model , the model Distribute to all clients. After sending, set a time window and wait for the client to upload the model update.
[0006] In one implementation of the present application, the client obtains a global model, trains the global model based on local data, and sends a model update and a request for a new global model to the server after training is completed, specifically including: Client Using local data after receiving the global model Start training; Send model updates after training is complete ) to the server and request new global model parameters, where For local models, Based on the The locally trained model version of the global model for the round.
[0007] In one implementation of the present application, the server calculates the mutual information value between the model update submitted by each client and the current global model, specifically including: By formula: , calculate the mutual information value between the model update submitted by the i-th client and the current global model ; in, is the local model weight parameter, Represents the current global model weight parameter, represents the entropy function.
[0008] In one implementation of the present application, obtaining an effective model update based on the mutual information value specifically includes: The server calculates the mean of all mutual information values and standard deviation , filter out mutual information values less than or greater than Model update.
[0009] In one implementation of the present application, weighted aggregation is performed on valid model updates, specifically including: By formula: , obtain the weight of the i-th valid model update; , get the weight value of the i-th valid model update ; in, represents the i-th valid model update; n represents the number of valid model updates, represents the balance factor, t represents the current time, Indicates the last update time.
[0010] In one implementation of the present application, obtaining a new global model specifically includes: By formula: , the weight parameters in the updated global model , obtain a new global model; Where i belongs to [1,n], n represents the number of valid model updates, represents the weight value of the updated i-th valid model, is the local model weight parameter.
[0011] In a second aspect, the present application provides an asynchronous federated learning system based on mutual information, the system comprising: The server is used to send the global model to all clients and set a time window after sending; The client is used to obtain the global model, train the global model based on local data, and send model updates and request a new global model to the server after training is completed; The server is used to continuously receive model updates uploaded by different clients within the time window; After the time window ends, the server calculates the mutual information value between the model update submitted by each client and the current global model; obtains the valid model update based on the mutual information value; performs weighted aggregation on the valid model updates to obtain a new global model; and distributes the new global model to the clients that requested the update.
[0012] In one implementation of the present application, the server includes a computing module. Used by the formula: , calculate the mutual information value between the model update submitted by the i-th client and the current global model ; in, is the local model weight parameter, Represents the current global model weight parameter, represents the entropy function.
[0013] In a third aspect, the present application provides a non-volatile computer storage medium having computer instructions stored thereon, which, when executed, implement an asynchronous federated learning method based on mutual information as described above.
[0014] It can be seen from the above technical solutions that this application has the following advantages: This application evaluates the actual value of each model update by calculating the mutual information between the client-submitted model update and the current global model. Compared to traditional methods that adjust weights based on lag, this application no longer relies solely on time to judge the importance of model updates. Instead, it deeply considers the amount of information contained in the update content and its potential contribution to the global model, thus avoiding the suboptimal model performance caused by simply adjusting weights based on time.
[0015] In scenarios where client data distribution is highly uneven, this application can fully mine and utilize each client's unique data distribution information. Even if updates submitted by some clients lag behind the current global model, as long as these updates contain information that is critical to the global model, this application can still identify and incorporate them into the effective model update through the calculation of mutual information values. This effectively solves the problem that existing asynchronous federated learning methods are prone to losing lagged updates containing unique data distribution information in heterogeneous data environments, and improves the model's adaptability and generalization capabilities in complex data environments.
[0016] By performing weighted aggregation of effective model updates based on mutual information values, this application can ensure that the global model fully absorbs valuable information from each client during the update process, thereby improving the performance of the global model.
[0017] This application sets a time window to receive client model updates and performs unified processing and aggregation after the time window ends. This asynchronous processing method enhances the flexibility and scalability of the system. The server can flexibly adjust the length of the time window according to actual conditions to adapt to communication delays and differences in client computing power in different scenarios. At the same time, this application is also easy to expand to large-scale distributed systems, providing technical support for building an efficient and stable federated learning platform. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the technical solution of the present invention, the following is a brief introduction to the drawings required for the description. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0019] Figure 1 This is a flowchart of an asynchronous federated learning method based on mutual information provided in an embodiment of the present application.
[0020] Figure 2 This is a schematic diagram of the internal structure of an asynchronous federated learning system based on mutual information provided in an embodiment of the present application. DETAILED DESCRIPTION
[0021] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.
[0022] It should be understood by those skilled in the art that the embodiments described below are merely preferred embodiments of the present disclosure and do not imply that the present disclosure can only be implemented through these preferred embodiments. These preferred embodiments are merely intended to explain the technical principles of the present disclosure and are not intended to limit the scope of protection of the present disclosure. Based on the preferred embodiments provided by the present disclosure, all other embodiments obtained by those skilled in the art without creative effort should still fall within the scope of protection of the present disclosure.
[0023] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.
[0024] The technical solutions proposed in the embodiments of the present application are described in detail below with reference to the accompanying drawings.
[0025] The embodiment provides an asynchronous federated learning method based on mutual information, such as Figure 1 As shown, the method provided in the embodiment of the present application mainly includes the following steps: Step 110: The server sends the global model to all clients and sets a time window after sending.
[0026] In some embodiments, the server sends the global model to all clients and sets a time window after sending the global model, specifically including: When the global model is first issued, The server randomly initializes the global model , the model Distribute to all clients. After sending, set a time window and wait for the client to upload the model update.
[0027] It's important to note that when the server first sends the global model, it randomly initializes the model and distributes it to all clients. This ensures that all clients have the same initial model state at the start of training, avoiding unfair training due to initial model differences. Each client performs local training from the same starting point, making subsequent model updates more comparable and convergent.
[0028] After distributing the global model, the server sets a time window during which it waits for clients to upload model updates. This mechanism provides flexibility, allowing the server to adjust the length of the time window based on factors such as actual communication latency, differences in client computing power, and network conditions. This helps ensure efficient collection of model updates while avoiding the loss of some client updates due to a too-short time window, or system response delays due to an overly long time window.
[0029] By setting a time window, the server can centrally receive and process client model updates within a fixed period of time, thereby improving the overall efficiency of the system. At the same time, this asynchronous collaboration method allows clients to conduct local training in their own free time and upload updates within the time window, eliminating the need for strict synchronization with the server and other clients, reducing communication overhead and coordination costs.
[0030] Specific examples: Consider a scenario where federated learning of medical data involves multiple hospitals in different regions as clients. The server wants to train a global model for disease prediction.
[0031] The server randomly initializes a disease prediction model, which includes some basic neural network structure and parameters. The server then distributes this initial model to all participating hospital clients. This ensures that each hospital begins local training based on the same model, ensuring fair training.
[0032] Considering that hospitals in different regions may have varying network conditions and computing capabilities, the server sets a relatively long time window, such as 24 hours. Within this time window, each hospital can conduct local training based on its own specific circumstances and upload model updates after training is complete. Hospitals with poor network conditions or limited computing capabilities have more time to complete training and upload updates, avoiding update losses due to time constraints.
[0033] After the time window ends, the server collects model updates from all hospitals and performs weighted aggregation to generate a new global model. Due to the time window setting, the server can centrally process all updates within a fixed period of time, improving system efficiency. At the same time, individual hospitals can conduct training in their own free time without the need for strict synchronization with other hospitals, reducing communication overhead and coordination costs. For example, a hospital may be busy with medical treatment during the day but can use its free time at night to conduct local training and upload updates the next day. This does not affect the normal work of the hospital while contributing to the training of the global model.
[0034] Step 120: The client obtains the global model, trains the global model based on local data, and sends a model update and a request for a new global model to the server after training is completed.
[0035] This step can be specifically as follows: Client Using local data after receiving the global model Start training; Send model updates after training is complete ) to the server and request new global model parameters, where For local models, Based on the The locally trained model version of the global model for the round.
[0036] It's important to note that after obtaining the global model, the client performs personalized training using local data. This allows each client to adjust the model based on its unique data distribution and characteristics. After training is complete, the client sends the model update to the server, which helps the server gather diverse information from different data sources and optimize the model globally. This combination of personalized training and global optimization improves the model's adaptability and generalization capabilities in complex, heterogeneous data environments.
[0037] The client only sends model updates to the server without uploading the original data, which effectively protects the client's data privacy. Since model updates are changes to model parameters, they typically do not contain content that can directly identify individuals or sensitive information, thereby reducing the risk of data leakage.
[0038] When the client sends a model update, it simultaneously requests new global model parameters. This mechanism reduces the number of communications and improves communication efficiency. After receiving the model update, the server can immediately send the new global model parameters to the client, allowing the client to quickly start the next round of training, thereby accelerating the convergence of the entire federated learning process.
[0039] Clients continuously send model updates and request new global models, which promotes continuous iteration and evolution of the model. Each round of training is based on the latest global model, allowing the model to continuously absorb new information from different clients, thereby gradually improving performance.
[0040] Specific examples: Consider an intelligent medical diagnosis system where multiple hospitals participate in federated learning as clients. The server wants to train a global model for disease diagnosis.
[0041] After receiving the global model from the server, each hospital trains it using its local medical data. For example, Hospital A might have a large amount of data on heart disease, while Hospital B might have more data on diabetes. Each hospital will customize the global model based on its local data distribution and characteristics to better suit its local diagnostic needs.
[0042] After training is complete, both Hospital A and Hospital B send their model updates to the server. These updates incorporate the new knowledge about disease diagnosis that each hospital has learned during training. At the same time, they also request new global model parameters from the server to begin the next round of training.
[0043] Throughout the entire process, neither Hospital A nor Hospital B uploaded any raw medical data to the server. They only sent model updates, which were changes to the model parameters and did not contain any information that could directly identify the patient or sensitive medical information. This effectively protected patient data privacy.
[0044] After receiving model updates from Hospitals A and B, the server immediately performs weighted aggregation to generate new global model parameters and sends them to both hospitals. This allows Hospitals A and B to quickly begin the next round of training and further optimize based on the latest global model. This mechanism improves communication efficiency and promotes continuous iteration and evolution of the model.
[0045] Step 130: The server continuously receives model updates uploaded by different clients within the time window. After the time window ends, the server calculates the mutual information value between the model update submitted by each client and the current global model.
[0046] The server calculates the mutual information between the model update submitted by each client and the current global model, which can be specifically: By formula: , calculate the mutual information value between the model update submitted by the i-th client and the current global model ; in, is the local model weight parameter, Represents the current global model weight parameter, represents the entropy function.
[0047] It's important to note that by calculating the mutual information between each client's submitted model update and the current global model, the server can better understand the uniqueness of each client's model and its contribution to the global model. This helps maintain global model optimization while also accommodating the personalized needs of each client, making the global model more adaptable to the data distribution and characteristics of different clients.
[0048] Calculating mutual information helps the server identify client updates that have a significant impact on the global model, prioritizing these updates. This not only reduces unnecessary communication overhead but also improves the quality and efficiency of model updates, allowing the global model to converge to a more optimal solution more quickly.
[0049] By comprehensively considering the model updates of each client, the global model can better adapt to diverse data environments, enhancing its robustness and generalization capabilities. The introduction of mutual information allows the model to take into account the information differences and complementarities between different clients during the update process, thereby generating a more robust and universal model.
[0050] Specific examples: Consider an intelligent medical diagnosis system where multiple hospitals participate in federated learning as clients. The server wants to train a global model for disease diagnosis.
[0051] Hospital A might have a large amount of data on heart disease, while Hospital B has more data on diabetes. By calculating mutual information, the server can determine that Hospital A's model updates contribute significantly to the global model's performance in heart disease diagnosis, while Hospital B's updates have a significant impact on diabetes diagnosis. This allows the server to more effectively balance updates from various clients when aggregating the model, ensuring that the global model achieves optimal performance for both heart disease and diabetes diagnosis.
[0052] When communication resources are limited, the server can prioritize client updates with higher mutual information values. For example, if the mutual information value between Hospital C's model update and the global model is high, indicating that its update has a significant impact on the global model, the server can prioritize receiving and processing Hospital C's update, thereby improving communication efficiency and the quality of the model update.
[0053] By comprehensively considering model updates from each hospital, the global model can better adapt to the data environments and patient characteristics of different hospitals. For example, hospitals in a certain region may have more data on a certain rare disease. By calculating mutual information, the server can understand the importance of these updates to the global model and incorporate them into global model training. This ensures that the global model demonstrates good diagnostic performance and generalization capabilities across hospitals and patients in different regions.
[0054] Step 140: Based on the mutual information value, obtain valid model updates; perform weighted aggregation on the valid model updates to obtain a new global model; and distribute the new global model to the client requesting the update.
[0055] In some embodiments, obtaining an effective model update based on the mutual information value specifically includes: The server calculates the mean of all mutual information values and standard deviation , filter out mutual information values less than or greater than Model update.
[0056] Perform weighted aggregation on valid model updates, including: By formula: , obtain the weight of the i-th valid model update; , get the weight value of the i-th valid model update ; in, represents the i-th valid model update; n represents the number of valid model updates, represents the balance factor, t represents the current time, Indicates the last update time.
[0057] Obtain a new global model, including: By formula: , the weight parameters in the updated global model , obtain a new global model; Where i belongs to [1,n], n represents the number of valid model updates, represents the weight value of the updated i-th valid model, is the local model weight parameter.
[0058] It should be noted that this step screens out valid model updates (updates with mutual information values within a reasonable range) by calculating the mean and standard deviation of the mutual information values, avoiding interference with the global model from invalid or low-quality updates, thereby improving the efficiency of model updates.
[0059] Weighted aggregation effectively updates the model, allowing the global model to comprehensively consider the contribution of each client while reducing the impact of outliers, thereby enhancing the robustness and stability of the model.
[0060] Through screening and weighted aggregation, the server can allocate computing resources more reasonably, focus on processing effective model updates, and avoid resource waste.
[0061] The weighted aggregation of valid model updates helps generate a more accurate and generalizable global model, thus improving overall performance.
[0062] As described above, this embodiment can evaluate the actual value of each model update by calculating the mutual information between the model update submitted by the client and the current global model. Compared to the traditional method of adjusting weights based on the degree of lag, this application no longer relies solely on time factors to judge the importance of model updates. Instead, it deeply considers the amount of information contained in the update content itself and its potential contribution to the global model, thus avoiding the suboptimal model performance problem caused by simply adjusting weights based on time.
[0063] In scenarios where client data distribution is highly uneven, this application can fully mine and utilize each client's unique data distribution information. Even if updates submitted by some clients lag behind the current global model, as long as these updates contain information that is critical to the global model, this application can still identify and incorporate them into the effective model update through the calculation of mutual information values. This effectively solves the problem that existing asynchronous federated learning methods are prone to losing lagged updates containing unique data distribution information in heterogeneous data environments, and improves the model's adaptability and generalization capabilities in complex data environments.
[0064] By performing weighted aggregation of effective model updates based on mutual information values, this application can ensure that the global model fully absorbs valuable information from each client during the update process, thereby improving the performance of the global model.
[0065] This application sets a time window to receive client model updates and performs unified processing and aggregation after the time window ends. This asynchronous processing method enhances the flexibility and scalability of the system. The server can flexibly adjust the length of the time window according to actual conditions to adapt to communication delays and differences in client computing power in different scenarios. At the same time, this application is also easy to expand to large-scale distributed systems, providing technical support for building an efficient and stable federated learning platform.
[0066] In addition, this application Figure 2 An asynchronous federated learning system based on mutual information is provided in the embodiment of this application. Figure 2 As shown, the system provided in the embodiment of the present application mainly includes: The server 210 is used to send the global model to all clients 220 and set a time window after sending; Client 220 is used to obtain the global model, train the global model based on local data, and send model updates and request a new global model to server 210 after training is completed; The server 210 is configured to continuously receive model updates uploaded by different clients 220 within a time window; After the time window ends, the server 210 is used to calculate the mutual information value between the model update submitted by each client 220 and the current global model; based on the mutual information value, obtain the valid model update; perform weighted aggregation on the valid model updates to obtain a new global model; and distribute the new global model to the client 220 that requested the update.
[0067] The server 210 includes a computing module, Used by the formula: , calculate the mutual information value between the model update submitted by the i-th client 220 and the current global model ; in, is the local model weight parameter, Represents the current global model weight parameter, represents the entropy function.
[0068] In addition, an embodiment of the present application further provides a non-volatile computer storage medium on which executable instructions are stored. When the executable instructions are executed, an asynchronous federated learning method based on mutual information as described above is implemented.
[0069] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. An asynchronous federated learning method based on mutual information, characterized in that: The method comprises: The server sends the global model to all clients and sets a time window after sending. The client obtains the global model, trains the global model based on local data, and sends model updates and requests a new global model to the server after training is complete. The server continuously receives model updates uploaded by different clients within a time window. After the time window ends, the server calculates the mutual information value between the model update submitted by each client and the current global model. Based on the mutual information value, the server obtains the valid model update. The valid model updates are weightedly aggregated to obtain a new global model. The new global model is distributed to the clients that requested the update.
2. The asynchronous federated learning method based on mutual information according to claim 1, characterized in that: The server sends the global model to all clients and sets a time window after sending it. Specifically, it includes: When the global model is first released, The server randomly initializes the global model , the model Distribute to all clients. After sending, set a time window and wait for the client to upload the model update.
3. The asynchronous federated learning method based on mutual information according to claim 1, characterized in that: The client obtains the global model, trains the global model based on local data, and sends model updates and requests a new global model to the server after training. This includes: Client Using local data after receiving the global model Start training; Send model updates after training is complete ) to the server and request new global model parameters, where For local models, Based on the The locally trained model version of the global model for the round.
4. The asynchronous federated learning method based on mutual information according to claim 1, characterized in that: The server calculates the mutual information between each client's submitted model update and the current global model, specifically: By formula: , calculate the mutual information value between the model update submitted by the i-th client and the current global model ; in, is the local model weight parameter, Represents the current global model weight parameter, represents the entropy function.
5. The asynchronous federated learning method based on mutual information according to claim 1, characterized in that: Based on the mutual information value, effective model updates are obtained, including: The server calculates the mean of all mutual information values and standard deviation , filter out mutual information values less than or greater than Model update.
6. The asynchronous federated learning method based on mutual information according to claim 1, characterized in that: Perform weighted aggregation on valid model updates, including: By formula: , obtain the weight of the i-th valid model update; , get the weight value of the i-th valid model update ; in, represents the i-th valid model update; n represents the number of valid model updates, represents the balance factor, t represents the current time, Indicates the last update time.
7. The asynchronous federated learning method based on mutual information according to claim 1, characterized in that: Obtain a new global model, including: By formula: , the weight parameters in the updated global model , obtain a new global model; Where i belongs to [1,n], n represents the number of valid model updates, represents the weight value of the updated i-th valid model, is the local model weight parameter.
8. An asynchronous federated learning system based on mutual information, characterized in that: The system comprises: The server is used to send the global model to all clients and set a time window after sending; The client is used to obtain the global model, train the global model based on local data, and send model updates and request a new global model to the server after training is completed; The server is used to continuously receive model updates uploaded by different clients within the time window; After the time window ends, the server calculates the mutual information value between the model update submitted by each client and the current global model; obtains the valid model update based on the mutual information value; performs weighted aggregation on the valid model updates to obtain a new global model; and distributes the new global model to the clients that requested the update.
9. The asynchronous federated learning system based on mutual information according to claim 8, characterized in that: The server includes a computing module, Used by the formula: , calculate the mutual information value between the model update submitted by the i-th client and the current global model ; in, is the local model weight parameter, Represents the current global model weight parameter, represents the entropy function.
10. A non-volatile computer storage medium, characterized in that Computer instructions are stored thereon, and when the computer instructions are executed, they implement an asynchronous federated learning method based on mutual information as described in any one of claims 1 to 7.