Dynamic clustering federal learning method, device and system and storage medium

Abstract

The invention discloses a dynamic clustering federal learning method, device and system, and a storage medium, and the method comprises the steps: in each round of training, a client side uploads a gradient to a gradient cache region in a server side; the server calculates a gradient quantity required by an adaptive-smooth gradient correction module; the adaptive-smooth gradient correction module is used for correcting the correction amount of the calculation pair based on the adaptive-smooth gradient; after the final gradient correction is obtained, a final gradient is synthesized based on correction, and the updating process of gradient descent is applied; and iteratively carrying out local model training and a global aggregation process until the model converges to obtain a model which finally meets the accuracy requirement.

Description

Technical Field

[0001] This invention belongs to the field of machine learning processing technology, specifically relating to a dynamic clustering federated learning method, apparatus, system, and storage medium. Background Technology

[0002] Machine learning, especially deep learning, has greatly propelled the development and deployment of artificial intelligence across a wide range of applications. However, with the growing concern for data privacy, traditional centralized machine learning (which relies on aggregating user data to a central server for training) faces increasing limitations. To address this issue, federated learning (FL) has emerged as a promising paradigm, enabling collaborative model training while raw data is stored on local clients. Extensive research has expanded upon this paradigm, enhancing its effectiveness in practical applications.

[0003] Despite the immense potential of learning algorithms, significant challenges remain in practical deployment, especially in highly heterogeneous environments such as the Internet of Things (IoT). Due to variations in deployment environments and intended uses, IoT devices typically exhibit significant differences in data volume, structure, and tag distribution. This statistical heterogeneity violates the independent and identically distributed (IID) assumption, causing each client to converge to its own local optimum. Consequently, global model aggregation becomes suboptimal, thus degrading overall training performance.

[0004] Besides the heterogeneity of data distribution among clients, the negative impact of heterogeneity in federated learning also stems from the dynamic and limited participation of clients. For the same data distribution, if all clients participate in each training round, the bias of the final aggregated gradient will be limited to the bias caused by the client's local data. As the proportion of participating clients decreases in each round, the bias between the gradient received by the server and the unbiased gradient in each round increases. This invention observes that while the bias of the non-independent and identically distributed gradients and the error relative to the unbiased gradient can explain the decrease in accuracy of FedAvg, and many existing methods (e.g., FedProx) attempt to improve this by limiting such error, this invention finds that this non-independent and identically distributed gradient caused by limited client participation can be more fundamentally analyzed from the convergence properties of the objective function. Although existing work mitigates the heterogeneity differences between clients and servers through algorithmic or model design, these methods often neglect the impact on the properties of the optimization objective due to heterogeneity visible to some clients. In this case, methods that fix the properties of the objective function and passively apply the same parameters are unlikely to achieve optimal convergence accuracy. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention provides a dynamic clustering federated learning method, apparatus, system, and storage medium.

[0006] To achieve the above objectives, the present invention provides the following solution: A dynamic clustering federated learning method includes: Step S1: In each round of training, the client uploads the gradient to the gradient cache on the server. Step S2, Server-side adaptive computation -smooth The amount of gradient required by the gradient correction module; adaptive. The -smooth gradient correction module uses a gradient correction method based on... -smooth adaptive gradient correction calculation Correction amount ; Step S3: After obtaining the final gradient correction amount, based on... Correction Synthesizing the final gradient and in The update process applies gradient descent; iterative local model training and global aggregation are performed until the model converges, resulting in a model that finally meets the accuracy requirements.

[0007] As a preferred option, based on -Smooth adaptive gradient correction includes: L-smooth adaptive gradient correction and Non-IID dynamic adaptive gradient correction.

[0008] Preferably, the aggregated global gradients in the federated learning aggregation process will be used. , replace with in The update process defined in the -smooth adaptive gradient correction algorithm .

[0009] This invention also provides a dynamic clustering federated learning method, comprising: The first processing module is used to upload gradients from the client to the gradient cache on the server in each round of training. The second processing module is used for server adaptive computation. -smooth The amount of gradient required by the gradient correction module; adaptive. The -smooth gradient correction module uses a gradient correction method based on... -smooth adaptive gradient correction calculation Correction amount ; The third processing module is used to, after obtaining the final gradient correction amount, based on... Correction Synthesizing the final gradient and in The update process applies gradient descent; iterative local model training and global aggregation are performed until the model converges, resulting in a model that finally meets the accuracy requirements.

[0010] As a preferred option, based on -Smooth adaptive gradient correction includes: L-smooth adaptive gradient correction and Non-IID dynamic adaptive gradient correction.

[0011] Preferably, the aggregated global gradients in the federated learning aggregation process will be used. , replace with in The update process defined in the -smooth adaptive gradient correction algorithm .

[0012] The present invention also provides a dynamic clustering federated learning system, comprising: a memory and a processor, wherein the memory stores a computer program executed by the processor, and the computer program executes a dynamic clustering federated learning method when executed by the processor.

[0013] The present invention also provides a storage medium on which a computer program is stored, the computer program executing a dynamic clustering federated learning method during runtime.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. The FedDS framework was proposed to specifically address the objective function issues caused by participation heterogeneity. -smooth Dynamic issues; by according to Actively switching correction schemes to address the posterior error caused by -smooth can improve convergence and convergence speed compared to SOTA.

[0015] 2. Transform the non-IID problem caused by the limited number of participants in federated learning into a method that addresses the convergence properties of the objective function, especially... The dynamic changes of -smooth. Attached Figure Description

[0016] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1This is a flowchart of the dynamic clustering federated learning method according to an embodiment of the present invention; Figure 2 This provides the framework for the dynamic clustering federated learning method in this embodiment of the invention. Figure 3 The results are presented for FedDS (the method of this invention), FedAvg, and SOTAs at t=1000 under different non-IID levels. Figure 4 This diagram illustrates a comparison of communication volume between the method of this invention and existing federated learning methods. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] Example 1 like Figure 1 As shown, this invention provides a dynamic clustering federated learning method, comprising: Step S1: In each round of training, the client uploads gradients to the gradient cache on the server. Each uploaded gradient is marked with a unique client identifier and timestamp, which can be used to find the historical gradients that are paired with the current client gradient. Step S2, Server-side adaptive computation -smooth The amount of gradient required by the gradient correction module (in...) Figure 2 (referred to as a, b, and c respectively); adaptive The -smooth gradient correction module uses a gradient correction method based on... -smooth adaptive gradient correction calculation Correction amount ,according to Compared to and Different positions apply Calculation method. The equivalent Drop and Stale weighted combination weights corresponding to FedDS due to relative position are represented by a grayscale gradient in the module diagram. The darker the color, the more the update method of the current round is equivalent to directly applying Drop, and vice versa. Based on -Smooth adaptive gradient correction includes: L-smooth adaptive gradient correction and Non-IID dynamic adaptive gradient correction.

[0021] Step S3: After obtaining the final gradient correction amount, based on... Correction Synthesizing the final gradient and in The update process applies gradient descent; iterative local model training and global aggregation are performed until the model converges, resulting in a model that finally meets the accuracy requirements.

[0022] In one embodiment of the present invention, the gradient correction of the L-smooth adaptive gradient in step S2 is as follows: Specifically, this refers to some gradients and their statistics in Stale, namely the (historical) global gradients. Historical gradient of the client And the client gradient for the current round. Through simple analysis, it can be mapped to two stages of non-IID FL, namely the approximate IID training stage (corresponding to the smaller stage). -smooth flat region), and significantly non-IID training phase (corresponding to larger and rapidly changing regions). -smooth region).

[0023] (1) Approximate IID training phase Under the approximate IID training phase, it is assumed The present invention defines the approximate IID training phase as satisfying the following conditions. (1) The approximate IID training phase corresponds to the initial stage of model training and convergence. During the approximate IID training phase, because... Therefore, the effect of non-IID on the convergence process can be ignored.

[0024] For non-IID FLs with limited participation, assuming in a certain round A certain client exists in the subset of clients participating in the process. This invention defines the average of all historical gradients of all clients visible to the server in the current round. Client historical gradient and client Current round gradient Assume that in the initial approximate IID stage... satisfy Then it must exist. and .

[0025] Based on the assumptions of L-smooth and the assumptions of -convex, assuming the current time point is From an average perspective, the last time each client appeared was... Assuming any client The last time it appeared was In the approximate IID stage, the following relative relationships need to be satisfied: (2) (2) Significant non-IID training phase In the stage of significant non-independent identical distribution, the present invention makes the following assumptions.

[0026] (3) Under the non-independent identically distributed function (FL), this invention considers This occurs when the model converges to its optimal position. At this point, there must exist... Furthermore, based on the convex function assumption in L-smooth, we can obtain... Therefore, the following relative relationship exists.

[0027] (4) It can be seen that, for the final convergence state, the optimization algorithm should provide... The update amount. For the original state, since it also uses the client's historical gradients, its optimization objective can be expressed as follows: (5) Accordingly, for the FedAvg(Drop) method, the gradient calculation method for each round can be changed. Write it as similar to way (6) It can be seen This does not satisfy the assumption. This is another reason why applying Drop in the non-independent and identically distributed stage can lead to suboptimal convergence.

[0028] As one embodiment of the present invention, in step S2, the Non-IID dynamically adaptive gradient correction is as follows: To accommodate the non-IID heterogeneity caused by limited participation, the optimization objectives of Drop and Stale need to be adapted to a method that can be based on... The system integrates dynamic changes and adaptive switching of the smoothness property.

[0029] based on , , This invention can represent optimization functions for Drop applied in the approximate IID training phase and Stale applied in the significantly non-IID training phase. Furthermore, based on the above... , , The analysis of the relative relationships among the three elements allows this invention to achieve... and The adaptive switching between these parameters aims to achieve the optimal convergence speed and lower bound.

[0030] Specifically, analysis of the approximate IID training phase shows that it must satisfy... Therefore, under this gradient relative relationship, the present invention applies... To achieve a convergence speed approximating the optimal speed during the IID training phase. (Excludes...) After that, the remaining two relative relationships are... and These correspond to two scenarios during the significantly non-IID training phase. Although This invention can be directly applied to both of these situations, but it can prove that... It is possible , , Optimization is performed under the representation of .

[0031] Without prior knowledge of the specific form of the objective function The given gradient needs to satisfy two boundary conditions, which correspond to... Given the partial gradient The maximum value and the minimum value that can be estimated under the condition of unbiased gradient. Therefore, the present invention addresses... The following conditions and restrictions are given. (7) This will be discussed separately. Equal conditions for boundary conditions.

[0032] (1) By simplification, we can obtain This means that this situation only exists in the approximate IID training phase. Furthermore, because this invention employs [a specific method / mechanism] in the approximate IID training phase... Therefore, this situation can be ignored.

[0033] (2) Through simplification, the present invention can be obtained It is worth noting that in non-independent and identically distributed scenarios, due to ,here As the degree of non-independent identical distribution increases, relative to This can be disregarded. Furthermore, as the degree of non-independent identically distributed (IID) training decreases, the importance of the significantly non-IID training phase relative to the approximately IID training phase also decreases. Therefore, from a simplificational analysis perspective, this invention can assume that the condition satisfying this boundary is... .

[0034] Therefore, based on the above analysis, we can conclude that when When applied to the training phase of significant non-IID, the boundary conditions are taken into account. and The analysis leads to the conclusion that, in order to satisfy the convergence assumption without prior conditions... Only when the following conditions are met hour, Only applied to .

[0035] Therefore, the entire optimization process can be expressed as: ,in yes (8) Based on Drop and Stale Based on the -smooth adaptive gradient correction, this invention presents the Federated Improved Drop & Stale (FedDS) algorithm to improve upon the limitations caused by limited participation. -smooth Dynamic heterogeneity effect.

[0036] Compared to the basic FedAvg federated learning framework, the FedDS algorithm of this invention aggregates the global gradient during the federated learning aggregation process. , replace with in The update process defined in the -smooth adaptive gradient correction algorithm .

[0037] Compared to FedAvg, this invention dynamically integrates the unbiasedness of stale through a scene-adaptive correction mechanism, overcoming the limitations of convergent non-IID due to... The convergence difficulty of naive FedAvg is caused by the reduction of -smoothness.

[0038] To demonstrate the effectiveness of the FedDS algorithm, model test accuracy was primarily used as the performance evaluation metric, and comparative experiments were conducted on four datasets: MNIST, FMNIST, CIFAR10, and CINIC10. For method comparisons, FedAws and FedDyn were used as benchmarks in non-IID scenarios. Additionally, a comparison with FedAvg was included to illustrate the performance achievable by naive federated learning in experimental scenarios.

[0039] The methods of this invention, including FedDS, were validated in experimental environments with varying degrees of non-IID, along with FedAvg and SOTAs. The model convergence accuracy at that time. Experimental results are shown in [link to experimental results]. Figure 3 For different datasets, three non-IID environments were set up in the same way for each dataset: IID as a control group, non-IID only for client-local data, and "full non-IID" which has both data non-IID and client-sampled non-IID. Specifically, "local non-IID" means that each client locally contains only data with a random label of one class. "Full non-IID" is based on "local non-IID" but samples only clients with a random label of one class in each round of client sampling. The experimental results show that for different methods, the accuracy of the trained model decreases to varying degrees as the degree of non-IID increases. However, under the same settings, the comparison between methods shows that the FedDS method of this invention adapts to the increasing degree of non-IID. -smooth design can maintain a relatively small decrease in model accuracy.

[0040] Figure 4 A comparison of communication traffic between the method of this invention and existing federated learning methods is presented. The x-axis represents the percentage of accuracy achieved compared to the method with the lowest accuracy among those trained models, and the y-axis represents the average communication traffic required to train the model to a specified accuracy using the corresponding methods. Experimental results show that, across multiple datasets, the method of this invention requires less communication traffic than existing federated learning methods to achieve the same training effect.

[0041] The complete experimental results are shown in Table 1. As can be seen from the accuracy (%) of different algorithms on different datasets presented in Table 1, under moderate non-IID (Local non-IID), the method of this invention maintains a slight advantage over state-of-the-art (SOTA) methods in most scenarios. With increasing non-IID (Full non-IID), FedDS exhibits less accuracy degradation compared to SOTA methods and significantly outperforms SOTA methods in all scenarios.

[0042] Table 1

[0043] This invention proposes the FedDS framework to specifically address the objective function issues arising from participation heterogeneity. - The issue of smooth dynamics. Federated learning has two naive solutions for limited participation: simply discarding gradients from non-participating clients (Drop), and utilizing historical gradients from non-participating clients (Stale). The Drop approach leverages the unbiased nature of client data under IID, directly utilizing the gradients of currently participating clients without processing. While it achieves good convergence speed under IID, it suffers severe accuracy degradation under non-IID. The Stale approach incorporates historical gradients into the gradient averaging process of the current round, mitigating the gradient bias caused by only some clients being visible under non-IID. However, it sacrifices some convergence speed due to errors introduced by stale gradients. By adaptively combining Drop and Stale within the FedDS framework, we can simultaneously balance convergence speed and accuracy, achieving compatibility and leveraging the strengths of both methods in non-IID FL.

[0044] Example 2 The present invention also provides a dynamic clustering federated learning device, comprising: The first processing module is used to upload gradients from the client to the gradient cache on the server in each round of training. The second processing module is used for server adaptive computation. -smooth The amount of gradient required by the gradient correction module; adaptive. The -smooth gradient correction module uses a gradient correction method based on... -smooth adaptive gradient correction calculation Correction amount ; The third processing module is used to, after obtaining the final gradient correction amount, based on... Correction Synthesizing the final gradient and in The update process applies gradient descent; iterative local model training and global aggregation are performed until the model converges, resulting in a model that finally meets the accuracy requirements.

[0045] As one embodiment of the present invention, based on -Smooth adaptive gradient correction includes: L-smooth adaptive gradient correction and Non-IID dynamic adaptive gradient correction.

[0046] As one embodiment of the present invention, the aggregated global gradient in the federated learning aggregation process will be used... , replace with in The update process defined in the -smooth adaptive gradient correction algorithm .

[0047] Example 3 The present invention also provides a dynamic clustering federated learning system, comprising: a memory and a processor, wherein the memory stores a computer program executed by the processor, and the computer program executes a dynamic clustering federated learning method when executed by the processor.

[0048] Example 4 The present invention also provides a storage medium on which a computer program is stored, the computer program executing a dynamic clustering federated learning method during runtime.

[0049] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A dynamic clustering federated learning method, characterized in that, Comprising: Step S1, in each round of training, the client uploads the gradient to the gradient buffer area in the server end; Step S2, server computes adaptation - the amount of gradient required by the smooth gradient correction module; adaptation - the smooth gradient correction module corrects the gradient based on - the smooth adaptation gradient correction calculates the correction amount for ;​ Step S3, after getting the final gradient correction, based on corrected synthesizing final gradient and in application of gradient descent update process; iterative local model training and global aggregation process until the model converges, get the final model to achieve the accuracy requirements.

2. The dynamic clustering federated learning method of claim 1, wherein, Based on Smooth adaptive gradient correction includes: L-smooth adaptive gradient correction and Non-IID dynamic adaptive gradient correction.

3. The dynamic clustering federated learning method of claim 2, wherein, The aggregated global gradient at the federated learning aggregation process will be replaced by the update process defined in the -smooth adaptive gradient correction algorithm . ​ 4. A dynamic clustering federated learning method, characterized in that, Comprising: A first processing module, configured to upload the gradient to the gradient buffer area in the server end in each round of training; a second processing module for server computing adaptive the amount of gradient required by the smooth gradient correction module; adaptive the smooth gradient correction module by basing the adaptive gradient correction computation on the amount of correction ; The third processing module is used to, after obtaining the final gradient correction amount, based on... Correction Synthesizing the final gradient and in The update process applies gradient descent; iterative local model training and global aggregation are performed until the model converges, resulting in a model that finally meets the accuracy requirements.

5. The dynamic clustering federated learning apparatus of claim 4, wherein, based on smooth adaptive gradient correction includes: L-smooth adaptive gradient correction and Non-IID dynamic adaptive gradient correction.

6. The dynamic clustering federated learning apparatus of claim 5, wherein, The aggregated global gradient at the federated learning aggregation process will be replaced by the update process defined in the -smooth adaptive gradient correction algorithm . ​ 7. A dynamic clustering federated learning system, comprising: Comprising: A memory and a processor, wherein the memory has stored thereon a computer program which, when executed by the processor, performs the dynamic clustering federated learning method according to any one of claims 1-3.

8. A storage medium, characterized by The storage medium has stored thereon a computer program which, when executed, performs the dynamic clustering federated learning method according to any one of claims 1-3.

Images