Federal learning communication overhead dynamic optimization method, system and device and storage medium

Abstract

The invention provides a federated learning communication overhead dynamic optimization method. The method comprises the following steps: monitoring a multi-dimensional state index of a participant in a federated learning process; when the participant meets a preset communication triggering condition, dynamically adjusting the communication frequency between the participant and the cooperative party according to the multi-dimensional state index, aggregating model parameters of the participant, and updating the aggregated model parameters to a global model of the cooperative party; and outputting a final global model until the global model meets a preset convergence condition. Compared with the prior art, the multi-dimensional state indexes of the participants are monitored to comprehensively perceive the training and resource states, the communication frequency is dynamically adjusted based on the multi-dimensional indexes to adapt to the demands of the participants and balance the communication overhead and the model performance, and the global model convergence is performed through repeated iteration until the preset conditions are met. Therefore, the problems that the communication overhead is too high, the resource utilization rate is low and the state of each participant cannot be dynamically adapted in federated learning are effectively solved.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, and in particular to a method, system, device, and storage medium for dynamically optimizing communication overhead in federated learning. Background Technology

[0002] Federated learning is a distributed machine learning framework that allows multiple participants to collaboratively train a global model without sharing their local raw data. In a typical federated learning process, participants first download the global model from a central server, train the model using their local data to obtain updated parameters, and then upload the updated parameters back to the server for aggregation, generating a new global model. This process requires multiple iterations until the model converges. However, federated learning faces a significant challenge: substantial communication overhead. During each training round, participants and the server need to transfer model parameters. When the model is complex (such as a deep neural network), there are many participants, or network bandwidth is limited, communication overhead becomes a bottleneck restricting the efficiency of federated learning.

[0003] Currently, optimization schemes for federated learning communication overhead can be mainly categorized as follows: Fixed-strategy communication frequency control: This method uses a fixed communication round interval, where participants communicate with the server after a fixed number of training rounds locally. While simple to implement, it cannot adapt to the heterogeneity of different participants in terms of training progress, data distribution, and resource availability. Premature communication by participants with faster training speeds may waste resources, while delayed communication by participants with slower training speeds may slow down the overall convergence speed. Model compression and quantization: This method directly reduces the amount of data transmitted in each communication by compressing or quantizing the uploaded model update parameters. For example, gradient sparsity only transmits some important gradient values. The disadvantage of this type of method is that the compression or quantization process itself may introduce additional overhead. Computational overhead: Excessive compression leads to information loss, affecting model accuracy and convergence stability. Furthermore, compression levels often need to be preset based on experience, making it difficult to dynamically adapt to the training process. Asynchronous update mechanism: Allowing participants to upload updates at different times, with the server asynchronously aggregating them to reduce waiting time, but this asynchronous mechanism may result in aggregated model updates coming from parameters from different training stages, easily introducing stale gradients and affecting the accuracy and stability of model convergence. Client selection: In each round of communication, only some participants are selected to upload updates. While this reduces the number of participants in a single round, the computational resources of unselected participants are wasted, and it may affect the representativeness of the model, especially when the data is not independent and identically distributed.

[0004] In summary, existing technologies for optimizing communication overhead in federated learning generally suffer from insufficient flexibility, poor adaptability, and a single control dimension. Existing technologies employ fixed, unadaptive strategies or focus solely on reducing the amount of data transmitted per transaction, failing to comprehensively consider multi-dimensional information such as the real-time training status of participants, dynamic resource load, and model performance changes for collaborative dynamic optimization. This results in a difficulty in balancing communication efficiency and model performance in real-world scenarios with heterogeneous resources, variable network environments, and complex data distributions.

[0005] Therefore, there is an urgent need for an optimization method that can adaptively and multidimensionally regulate the federated learning communication process in order to significantly reduce the overall communication overhead while ensuring model convergence performance. Summary of the Invention

[0006] This invention aims to address the technical problems of insufficient flexibility, poor adaptability, and limited control dimensions in existing federated learning communication optimization methods. Specifically, this invention provides a scheme for dynamically optimizing communication overhead based on multi-dimensional indicators. By dynamically adjusting the communication frequency between the participants and collaborators through multi-dimensional state indicators, it overcomes the inflexibility of fixed-frequency communication, the risk of accuracy loss in compression quantization methods, and the weak scenario adaptability caused by a single optimization dimension.

[0007] The purpose of this invention is to provide a method for dynamically optimizing communication overhead in federated learning; The technical solution provided by this invention is as follows: A method for dynamically optimizing communication overhead in federated learning includes the following steps: Monitor the multi-dimensional status indicators of participants in the federated learning process; When the participating party meets the preset communication triggering conditions, the communication frequency between the participating party and the collaborating party is dynamically adjusted according to the multi-dimensional status indicators, the model parameters of the participating party are aggregated, and the aggregated model parameters are updated to the global model of the collaborating party. The final global model is output when the global model meets the preset convergence condition.

[0008] Preferably, the multi-dimensional status indicators of the monitoring participants during the federated learning process specifically include: Monitor the rate of change of model parameters of the participants during the federated learning process; Monitor the rate of change in model accuracy of the participants during the federated learning process; Monitor the overall resource utilization rate of the participating parties during the federated learning process.

[0009] Preferably, monitoring the rate of change of model parameters of the participating parties during the federated learning process includes: Monitor the model parameter weight vector of the participant in the current training round and the model parameter weight vector in the previous training round; Calculate the Euclidean distance between the model parameter weight vector of the current training round and the model parameter weight vector of the previous training round to obtain the rate of change of the model parameters.

[0010] Preferably, monitoring the overall resource utilization rate of the participants in the federated learning process includes: Monitor the computing resource utilization and network bandwidth utilization of the participating parties; The overall resource utilization rate is obtained by weighted summation of the computing resource utilization rate and the network bandwidth utilization rate.

[0011] Preferably, the preset communication triggering conditions include: The current training round of the participating party has reached the round threshold corresponding to the communication frequency; The preset communication triggering conditions also include at least one of the following: The rate of change of the model parameters of the participating party exceeds the threshold of the first parameter change rate; The rate of change in the model accuracy of the participating party exceeds the first rate of change threshold. The overall resource utilization rate of the participating parties exceeds the first overall resource utilization rate threshold.

[0012] Preferably, dynamically adjusting the communication frequency between the participants and collaborators based on the multi-dimensional status indicators includes: If the rate of change of the model parameters of the participating party exceeds the first parameter change rate threshold, and the rate of change of the model accuracy is greater than zero, then the communication frequency is reduced based on the first adjustment step size. If the rate of change of the model parameters of the participating party is less than the second parameter rate of change threshold, and the absolute value of the rate of change of the model accuracy of the participating party is not greater than the first accuracy rate of change threshold, then the communication frequency is increased based on the first adjustment step size.

[0013] Preferably, the step of dynamically adjusting the communication frequency between the participants and collaborators based on the multi-dimensional status indicators further includes: If the overall resource utilization rate of the participating party is greater than the first overall resource utilization rate threshold, the communication frequency is reduced based on the second adjustment step size. If the overall resource utilization rate of the participating parties is less than the second overall resource utilization rate threshold, the communication frequency is reduced based on the second adjustment step size.

[0014] The second objective of this invention is to provide a system for dynamically optimizing communication overhead in federated learning. The technical solution provided by this invention is as follows: A federated learning communication overhead dynamic optimization system includes: The monitoring module is used to monitor the multi-dimensional status indicators of participants during the federated learning process; The frequency adjustment and aggregation update module is used to dynamically adjust the communication frequency between the participant and the collaborator according to the multi-dimensional status indicators when the participant meets the preset communication triggering conditions, aggregate the model parameters of the participant, and update the aggregated model parameters to the global model of the collaborator. The iterative control module is used to control the monitoring module and the frequency adjustment and aggregation update module to execute repeatedly until the global model meets the preset convergence condition, and then output the final global model.

[0015] The third objective of this invention is to provide a computer device; The technical solution provided by this invention is as follows: A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the federated learning communication overhead dynamic optimization method as described in any one of the claims.

[0016] A fourth objective of this invention is to provide a computer-readable storage medium; The technical solution provided by this invention is as follows: A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the federated learning communication overhead dynamic optimization method as described in any one of the claims.

[0017] This invention provides a method for dynamically optimizing communication overhead in federated learning, comprising the following steps: monitoring multi-dimensional state indicators of participants during the federated learning process; when a participant meets a preset communication triggering condition, dynamically adjusting the communication frequency between the participant and collaborators based on the multi-dimensional state indicators, aggregating the model parameters of the participant, and updating the aggregated model parameters to the global model of the collaborators; and outputting the final global model when the global model meets a preset convergence condition. Compared to existing technologies where fixed communication frequencies lead to resource waste or convergence delays, traditional optimization methods suffer from accuracy loss or poor universality, and single-dimensional control is rigid, this application comprehensively perceives training and resource status by monitoring multi-dimensional state indicators of participants. It dynamically adjusts communication frequencies based on multi-dimensional indicators to adapt to the needs of heterogeneous participants. When preset conditions are met, it aggregates the model parameters of the participants and updates the aggregated model parameters to the global model of the collaborating parties to balance communication overhead and model performance. Through repeated iterative control until preset conditions are met, it ensures global model convergence. This effectively solves the technical problems of excessive communication overhead, low resource utilization, slow model convergence, and inability to dynamically adapt to the heterogeneous states of participants in federated learning.

[0018] This invention also provides a system for dynamically optimizing communication overhead in federated learning. Since this system and the method for dynamically optimizing communication overhead in federated learning solve the same technical problem and belong to the same technical concept, they should have the same beneficial effects, and will not be described in detail here. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of a method for dynamically optimizing communication overhead in federated learning, as described in an embodiment of the present invention. Figure 2 This is a schematic diagram of the structure of a federated learning communication overhead dynamic optimization system according to an embodiment of the present invention; Figure 3 This is an internal structural diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0021] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] like Figure 1 As shown, this embodiment of the invention provides a method for dynamically optimizing the communication overhead of federated learning, including the following steps: S1. Monitor the multi-dimensional status indicators of participants in the federated learning process; In practical applications, before step S1, the following is also included: S0. Dynamically set the initial communication frequency based on the comprehensive resource score of the participants.

[0023] It should be noted that step S0 calculates the comprehensive resource score of the participants based on their computing resources, network bandwidth, and storage capacity, thereby determining the initial communication frequency.

[0024] In practical application, step S0 specifically includes the following steps: S01. Define Application Requirements: First, clarify the specific requirements of the federated learning application. In this embodiment, a CS architecture is adopted (where S is the collaborator and C is the participant). For example, there is one central server (collaborator) and three participants. Each participant trains its neural network using its own local dataset and performs federated learning by sharing neural network gradients with the assistance of the collaborator. Specifically, this involves aggregating the model parameters of each participant into the global model of the collaborator, realizing collaboration among participants, and constructing a global model.

[0025] S02. Analyze the resource characteristics of the participants: Collect the computing resources (CPU / GPU computing power, denoted as...) of each participant. Network bandwidth (denoted as) Storage capacity (denoted as) ), calculate the overall resource score of the participants The formula is as follows:

[0026] in , , These are the weighting coefficients for computing resources, network bandwidth, and storage capacity, respectively. , Indicates the number of participants. This can be adjusted based on task requirements (e.g., in communication-sensitive scenarios). (larger) , The higher the value, the more abundant the resources of the participants.

[0027] S03. Setting the initial communication frequency: Under normal circumstances, after each participant completes training based on its local dataset, it will upload its model parameters to the collaborating party. Therefore, the initial communication frequency... Defined as the threshold for the initial local training rounds of a participant, i.e., the number of rounds a participant completes. After local training, communication with collaborators is conducted. A comprehensive resource-based scoring system is then implemented. Assign initial allocations to different participants The formula is as follows:

[0028] in The base initial frequency (default value is 1). This is the resource coefficient (default value is 0.5). This indicates rounding down; participants with more resources initially... The larger the value, the fewer the initial communication attempts.

[0029] S04, Preset multi-dimensional indicator threshold parameters: including the preset first parameter change rate threshold. First accuracy rate change threshold First comprehensive resource utilization threshold It is used for subsequent communication frequency adjustment judgment.

[0030] It should be noted that the multi-dimensional state indicators in step S1 include the rate of change of model parameters, the rate of change of model accuracy, and the overall resource utilization rate.

[0031] Preferably, step S1 includes the following steps: S11. Monitor the rate of change of model parameters among participants during the federated learning process; S12. Monitor the rate of change in model accuracy among participants during the federated learning process; S13. Monitor the overall resource utilization rate of participants in the federated learning process.

[0032] Preferably, step S11 includes the following steps: Monitor the model parameter weight vector of the participant in the current training round and the model parameter weight vector of the previous training round; Calculate the Euclidean distance between the model parameter weight vector in the current training round and the model parameter weight vector in the previous training round to obtain the rate of change of the model parameters.

[0033] In practical application, step S11, monitoring the rate of change of model parameters, specifically involves: after each round of local training, the participant calculates the current model parameter weight vector. Compared with the previous round of parameter weight vector The Euclidean distance D is calculated using the following formula:

[0034] Where n represents the dimension of the parameters, and i represents the i-th parameter in the model. This represents the i-th parameter in the updated weight vector. represents the i-th parameter of the weight vector before the update. D represents the Euclidean distance, which is a quantitative representation of the difference between the weight vectors before and after the model update. In other words, the Euclidean distance D is a quantitative representation of the rate of change of the model parameters. The larger the Euclidean distance D is, the greater the difference between the two vectors, that is, the more significant the change in the model parameters.

[0035] Preferably, step S12 specifically involves: the rate of change of model accuracy of the global model after aggregating the model parameters of the participating parties.

[0036] In practical application, the monitoring model accuracy change rate in step S12 is specifically as follows: the collaborating party uses the test set to calculate the global model accuracy after each round of aggregation. Compared with the accuracy rate of the previous round The rate of change in model accuracy is calculated using the following formula:

[0037] Preferably, step S13 includes the following steps: Monitor the computing resource utilization and network bandwidth utilization of the participants; The overall resource utilization rate is obtained by weighted summation of computing resource utilization and network bandwidth utilization.

[0038] In practical applications, step S13 involves monitoring the local comprehensive resource utilization rate of participating parties: real-time collection of the utilization rate of participating parties' computing resources (CPU / GPU). Network bandwidth utilization Calculate the comprehensive value of comprehensive resource utilization rate The formula is as follows:

[0039] in , The weighting coefficients are consistent with those in step S0 for the comprehensive resource score; , The higher the value, the heavier the resource load on the participants.

[0040] S2. When the participants meet the preset communication trigger conditions, the communication frequency between the participants and collaborators is dynamically adjusted according to multi-dimensional status indicators, the model parameters of the participants are aggregated, and the aggregated model parameters are updated to the global model of the collaborators. Preferably, the preset communication triggering conditions include: The participant's current training round has reached the round threshold corresponding to the communication frequency; The preset communication trigger conditions also include at least one of the following: The rate of change of the model parameters of the participating party exceeds the threshold of the rate of change of the first parameter; The rate of change in the model accuracy of the participating party exceeds the first accuracy rate of change threshold; The overall resource utilization rate of the participating parties exceeds the first overall resource utilization rate threshold.

[0041] In practical application, the current communication frequency of each participant is the local training round threshold. The initial communication frequency of each participant is determined in step S0. When a participant meets the preset communication triggering conditions, it is dynamically adjusted through multi-dimensional status indicators. Therefore, the round threshold corresponding to the communication frequency is also constantly changing. The preset communication triggering conditions in step S2 are specifically: communication with the assisting party is triggered when a participant meets any of the following conditions: The number of local training rounds for each participant reaches the current participant's communication frequency. (This indicates that the maximum communication frequency has been reached); Model parameter changes (This indicates that the parameters have changed significantly, and the communication frequency needs to be adjusted in a timely manner.) resource utilization rate (This indicates that resources are overloaded, and local training should be reduced while communication should be increased.) Accuracy Change Rate (This indicates that the model's performance fluctuates greatly and needs to be adjusted in a timely manner.)

[0042] As one implementation method, the first parameter change rate threshold The value is 0.1, the first accuracy rate of change threshold. The value is 0.1, the first comprehensive resource utilization threshold. The value is 0.7.

[0043] Preferably, the communication frequency between participants and collaborators is dynamically adjusted based on multi-dimensional status indicators, including: If the rate of change of the model parameters of the participating party exceeds the first parameter change rate threshold, and the rate of change of the model accuracy is greater than zero, then the communication frequency is reduced based on the first adjustment step size. If the rate of change of the model parameters of the participating party is less than the second parameter rate of change threshold, and the absolute value of the rate of change of the model accuracy of the participating party is not greater than the first accuracy rate of change threshold, then the communication frequency is increased based on the first adjustment step size.

[0044] Preferably, dynamically adjusting the communication frequency between participants and collaborators based on multi-dimensional status indicators further includes: If the overall resource utilization rate of the participants is greater than the first overall resource utilization rate threshold, the communication frequency will be reduced based on the second adjustment step size. If the overall resource utilization rate of the participants is less than the second overall resource utilization rate threshold, the communication frequency will be reduced based on the second adjustment step size.

[0045] It should be noted that the threshold for the rate of change of the second parameter and the threshold for the comprehensive resource utilization rate are preset thresholds and can be adjusted according to the actual situation. The first adjustment step size and the second adjustment step size are preset values, and the range of the first adjustment step size and the second adjustment step size is 1-5. Preferably, the first adjustment step size and the second adjustment step size can be 3.

[0046] As one implementation method, the second parameter change rate threshold can be half of the first parameter change rate threshold, and the second comprehensive resource utilization rate threshold can be half of the first comprehensive resource utilization rate threshold.

[0047] Preferably, the dynamic adjustment of the communication frequency adopts a hierarchical adjustment strategy, including: a first adjustment layer based on the rate of change of model parameters and model accuracy, a second adjustment layer based on resource utilization, and a third adjustment layer based on smooth adjustment constraints.

[0048] In practical application, step S2 dynamically adjusts the communication frequency between participants and collaborators based on multi-dimensional status indicators. Specifically, the assisting party updates the participants' communication frequency using a hierarchical adjustment strategy based on the multi-dimensional indicators triggered by communication. The details are as follows: First adjustment layer: Adjustment based on changes in model parameters and the rate of change in model accuracy. Adjust strategy: if and (Indicating significant parameter changes and improved model performance):

[0049] if and (This indicates that the parameters change smoothly and the model performance is stable):

[0050] in, For the first adjustment step, max(1,.) ensures Not less than 1.

[0051] Second adjustment layer: Adjustment based on comprehensive resource utilization rate Adjust strategy: if (Resource overload):

[0052] if (Sufficient resources):

[0053] in, For the second adjustment step size, max(1,.) ensures A value of at least 1 is required to prioritize timely communication for participants with excessive resource loads.

[0054] Third adjustment layer: Smoothing adjustment constraints To prevent excessive fluctuations in the communication frequency after adjustments by the first and second adjustment layers, an upper limit for the adjustment range needs to be set: |

[0055] in, To adjust the amplitude coefficient and ensure smooth frequency adjustment, reducing oscillations, preferably, the amplitude coefficient is adjusted... It is 0.3. To adjust the communication frequency after the hierarchical adjustment strategy is updated, This is the communication frequency for the current round.

[0056] It should be noted that in step S2, the model parameters of the participants are aggregated. Specifically, the collaborating party aggregates the local models of each participant based on the overall resource score and changes in model parameters using an aggregation algorithm. The aggregation algorithm can be FedAvg, FedProx, SCAFFOLD, etc.

[0057] As one implementation method, each participant uploads its local model gradients to the collaborator each time communication is triggered. The collaborator then aggregates these gradients using the FedAvg algorithm, which incorporates a comprehensive resource score for each participant. The weighting coefficients for the model parameter variation D are given by the following formula:

[0058]

[0059] in, These are the aggregated global model parameters. For the local model parameters of the kth participant, Let be the weight coefficient of the k-th participant. This is the balancing coefficient (default value is 0.5); participants with sufficient resources and significant parameter changes have higher weight coefficients, thus improving the representativeness of the aggregation model.

[0060] It should be noted that step S2, updating the aggregated model parameters to the collaborator's global model, specifically involves: updating the aggregated model parameters... The data is updated in the global model and sent to all participants for use in the next round of local training.

[0061] Compared with existing technologies, this application dynamically adjusts the communication frequency through multi-dimensional indicators, effectively solving the problems of resource waste and convergence delay in existing technologies, and achieving multiple goals such as reducing communication overhead, improving resource utilization, and optimizing model performance.

[0062] S3. Output the final global model when the global model meets the preset convergence condition.

[0063] It should be noted that step S3 specifically includes: repeating steps S1-S2, automatically adjusting the communication frequency based on the rate of change of global model parameters and communication triggering conditions, until the global model meets the preset convergence condition, thus obtaining the final global model of the entire federated training. The collaborating party then broadcasts the final global model to all participating parties for actual application inference. The global model meeting the preset convergence condition can mean that the global model reaches a specific convergence condition or that the number of iterations reaches a preset iteration threshold.

[0064] This invention applies to a typical client-server (CS) architecture federated learning system. The system includes a collaborator (i.e., a central server) and multiple participants (i.e., clients). Each participant holds a local private dataset and, under the coordination of the collaborator, jointly trains a global model. The core of this invention lies in dynamically determining, during the training process, how many rounds of local training each participant should undergo before communicating with the server once, i.e., dynamically managing its communication frequency.

[0065] Compared to existing technologies, this application dynamically adjusts the communication frequency based on the participants' status and network conditions, ensuring communication only when necessary. This reduces the number of communications and the amount of data transmitted, thus lowering communication overhead. Dynamically adjusting the communication frequency also allows for better utilization of participants' computing resources and bandwidth, avoiding unnecessary waiting time and resource waste, and improving training efficiency. Furthermore, the dynamic adjustment of the communication frequency can be flexibly adjusted based on the participants' convergence speed indicators. When a participant's model is close to convergence or its convergence speed is slow, the communication frequency can be increased to promote faster global model convergence. Finally, the dynamic adjustment of the communication frequency can be adaptively adjusted according to actual conditions, taking into account the characteristics of the participants, task requirements, and network environment, making the communication strategy more adaptable and flexible.

[0066] This paper also demonstrates the effectiveness of the proposed method using experimental data. First, the experimental verification environment is introduced, followed by the corresponding experimental results.

[0067] (1) Experimental environment System architecture: 1 collaborator (server: CPU Intel Xeon Gold 6330, GPU NVIDIA A100), 5 participants (client: CPU Intel i7-12700H, GPU NVIDIA RTX 3060, network bandwidth varies from 100Mbps to 1Gbps). Dataset: The CIFAR-10 image dataset is used. The local data of the participants are divided into training set and validation set in an 8:2 ratio, and the test set of the collaborators consists of 10,000 images. Model: ResNet-18 deep learning model, initial learning rate 0.01, batch size 32.

[0068] Table 1. Experimental Results

[0069] As shown in Table 1, the more abundant the resources of the participants, the higher the final communication frequency, the fewer the number of communication rounds, and the shorter the training time. Moreover, the global model accuracy steadily improves with the improvement of resource adaptability, which verifies the effectiveness and superiority of the proposed solution.

[0070] In one embodiment, such as Figure 2 As shown, a federated learning communication overhead dynamic optimization system includes: The monitoring module is used to monitor the multi-dimensional status indicators of participants during the federated learning process; The frequency adjustment and aggregation update module is used to dynamically adjust the communication frequency between the participants and collaborators based on multi-dimensional status indicators when the participants meet the preset communication trigger conditions, aggregate the model parameters of the participants, and update the aggregated model parameters to the global model of the collaborators. The iterative control module is used to control the repeated execution of the monitoring module, frequency adjustment and aggregation update module until the global model meets the preset convergence conditions, and then outputs the final global model.

[0071] Specific limitations regarding a dynamic optimization system for federated learning communication overhead can be found in the limitations of a dynamic optimization method for federated learning communication overhead described above, and will not be repeated here. Each module in the aforementioned dynamic optimization system for federated learning communication overhead can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0072] like Figure 3 As shown, in one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the following steps: Monitor the multi-dimensional status indicators of participants in the federated learning process; When the participants meet the preset communication trigger conditions, the communication frequency between the participants and collaborators is dynamically adjusted according to multi-dimensional status indicators, the model parameters of the participants are aggregated, and the aggregated model parameters are updated to the global model of the collaborators. The final global model is output when the global model meets the preset convergence condition.

[0073] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0074] In one embodiment, a computer-readable storage medium is provided, storing a computer program that, when executed by a processor, performs the following steps: Monitor the multi-dimensional status indicators of participants in the federated learning process; When the participants meet the preset communication trigger conditions, the communication frequency between the participants and collaborators is dynamically adjusted according to multi-dimensional status indicators, the model parameters of the participants are aggregated, and the aggregated model parameters are updated to the global model of the collaborators. The final global model is output when the global model meets the preset convergence condition.

[0075] In the embodiments provided in this application, it should be understood that the disclosed methods and systems can be implemented in other ways. The system embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple modules or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, and can be electrical, mechanical, or other forms.

[0076] Furthermore, in the various embodiments of the present invention, each functional module can be fully integrated into a processor, or each module can be a separate device, or two or more modules can be integrated into a device; each functional module in the various embodiments of the present invention can be implemented in hardware or in the form of hardware plus software functional units.

[0077] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by program instructions and related hardware. The aforementioned program instructions can be stored in a computer-readable storage medium. When the program instructions are executed, they perform the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0078] It should be understood that the use of terms such as "system," "apparatus," "unit," and / or "module" in this application is only applicable to distinguishing different components, elements, parts, sections, or assemblies at different levels. However, if other terms can achieve the same purpose, they may be replaced by other expressions.

[0079] As indicated in this application and claims, unless the context clearly indicates otherwise, the words "a," "an," "a," and / or "the" are not specifically singular and may include the plural. Generally, the terms "comprising" and "including" only indicate the inclusion of expressly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements. An element defined by the phrase "comprising an..." does not exclude the presence of other identical elements in the process, method, product, or apparatus that includes the element.

[0080] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "a plurality of" or "several" means two or more, unless otherwise explicitly specified.

[0081] If a flowchart is used in this application, it is used to illustrate the operations performed by the system according to embodiments of this application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, the steps can be processed in reverse order or simultaneously. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0082] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for dynamically optimizing communication overhead of federated learning, characterized in that, The method comprises the following steps: monitoring multi-dimensional state indicators of the participant in the federated learning process; when the participant meets a preset communication triggering condition, dynamically adjusting a communication frequency between the participant and a collaborator according to the multi-dimensional state indicators, aggregating model parameters of the participant, and updating the aggregated model parameters to a global model of the collaborator; until the global model meets a preset convergence condition, outputting a final global model. 2.The method of claim 1, wherein, The monitoring of the multi-dimensional state indicators of the participant in the federated learning process comprises the following steps: monitoring a model parameter change rate of the participant in the federated learning process; monitoring a model accuracy change rate of the participant in the federated learning process; monitoring a comprehensive resource utilization rate of the participant in the federated learning process. 3.The method of claim 2, wherein, The monitoring of the model parameter change rate of the participant in the federated learning process comprises: monitoring a model parameter weight vector of a current training round of the participant and a model parameter weight vector of a previous training round; calculating a Euclidean distance between the model parameter weight vector of the current training round and the model parameter weight vector of the previous training round to obtain the model parameter change rate. 4.The method of claim 2, wherein, The monitoring of the comprehensive resource utilization rate of the participant in the federated learning process comprises: monitoring a computing resource utilization rate and a network bandwidth utilization rate of the participant; performing weighted summation on the computing resource utilization rate and the network bandwidth utilization rate to obtain the comprehensive resource utilization rate. 5.The method of claim 2, wherein, The preset communication triggering condition comprises: the current training round of the participant reaches a round threshold corresponding to the communication frequency; The preset communication triggering condition further comprises at least one of the following: the model parameter change rate of the participant exceeds a first parameter change rate threshold; the model accuracy change rate of the participant exceeds a first accuracy change rate threshold; the comprehensive resource utilization rate of the participant exceeds a first comprehensive resource utilization rate threshold.

6. The method of claim 5, wherein, The dynamically adjusting of the communication frequency between the participant and the collaborator according to the multi-dimensional state indicators comprises: if the model parameter change rate of the participant exceeds the first parameter change rate threshold and the model accuracy change rate is greater than zero, reducing the communication frequency based on a first adjustment step; if the model parameter change rate of the participant is less than a second parameter change rate threshold and an absolute value of the model accuracy change rate of the participant is not greater than the first accuracy change rate threshold, increasing the communication frequency based on the first adjustment step.

7. The method of claim 6, wherein, The dynamically adjusting of the communication frequency between the participant and the collaborator according to the multi-dimensional state indicators further comprises: if the comprehensive resource utilization rate of the participant is greater than the first comprehensive resource utilization rate threshold, reducing the communication frequency based on a second adjustment step; if the comprehensive resource utilization rate of the participant is less than a second comprehensive resource utilization rate threshold, reducing the communication frequency based on the second adjustment step.

8. A federated learning communication overhead dynamic optimization system, characterized in that, The method comprises the following steps: a monitoring module configured to monitor multi-dimensional state indicators of a participant in a federated learning process; a frequency adjustment and aggregation updating module, configured to dynamically adjust a communication frequency between the participant and a collaborator according to the multi-dimensional state indicator when the participant meets a preset communication trigger condition, aggregate model parameters of the participant, and update the aggregated model parameters to a global model of the collaborator; an iteration control module, configured to control the monitoring module and the frequency adjustment and aggregation updating module to repeatedly execute until the global model meets a preset convergence condition, and output a final global model.

9. A computer device, comprising: A computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the federated learning communication overhead dynamic optimization method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the federated learning communication overhead dynamic optimization method according to any one of claims 1-7.

Images