Non-IID data-oriented cross-client global classifier dynamic fine tuning method

Abstract

The invention relates to the technical field of federated learning, in particular to a non-IID data-oriented cross-client global classifier dynamic fine tuning method, which comprises the following steps that: each client carries out local training on a local model based on local data, updates parameters of a feature extractor layer of the local model, and obtains a feature extractor layer of the local model based on a label of the local data; calculating a category average feature of each category; after completing local training, each client uploads data to the server according to the current communication stage; the server updates the global model according to the updated model parameters uploaded by the clients, calculates a weighted average value of the category average features uploaded by the clients, and performs fine adjustment on a classifier layer of the global model based on the weighted average value to obtain an updated global model; and judging whether the updated global model is converged or not, and if yes, issuing the updated global model to each client and ending federated learning. The method effectively improves the accuracy and stability of the global model.

Description

Technical Field

[0001] Embodiments of the present application relate to the technical field of federated learning, and particularly to a method for dynamically fine-tuning a cross-client global classifier for non-IID data. Background Art

[0002] With the rapid development of Internet of Things, intelligent devices and big data technologies, distributed learning methods, especially federated learning methods, have become important technologies for processing large-scale heterogeneous data. Federated learning processes local data on multiple clients and uploads model parameters to a central server for aggregation, avoiding the risk of centralized storage of sensitive data and having significant privacy protection advantages. However, traditional federated learning methods still face core challenges such as data heterogeneity, communication efficiency bottlenecks, device heterogeneity, and cross-platform deployment complexity.

[0003] Traditional federated learning frameworks usually assume that client data satisfies independent and identically distributed (IID), that is, the data distributions among clients are the same. However, in practical applications, most clients face a non-IID data environment where there are significant differences in the data distributions among different clients, which directly affects the training effect of the global model. Problems such as inconsistent feature distributions and label shifts lead to a decline in the generalization ability of the global model. Especially when dealing with minority class samples, the performance of the model is often unstable. In the face of this problem, research teams at home and abroad have proposed several improvement strategies to improve model performance, mainly including two types: optimization techniques based on federated averaging and communication efficiency optimization techniques.

[0004] For the optimization techniques based on federated averaging, foreign research teams have proposed a method combining a global classifier fine-tuning mechanism and graph data modeling to optimize the performance of federated learning in dynamic graphs. This method reduces the impact of data heterogeneity on the global model by uploading class-averaged features to the server. The Chinese patent with publication number CN119514727A proposes a personalized federated learning method based on adaptive local model initialization and dual knowledge distillation, which reduces model drift caused by non-IID data by improving the personalized adjustment of the client model. The Chinese patent with publication number CN119294560A proposes a dynamic knowledge distillation and regularization mechanism to reduce the bias of client data and improve the generalization ability of the global model in a non-IID environment. Other studies such as Fed2 and FedCCFA, etc., reduce the impact brought by data heterogeneity through feature alignment techniques and improve the personalized performance of the model.

[0005] Communication overhead is a key bottleneck in federated learning. Especially in bandwidth-constrained environments, frequent communication operations not only consume a large amount of bandwidth but also increase the computational burden. Based on this, communication efficiency optimization techniques have emerged. In federated learning systems, communication efficiency optimization techniques are mainly achieved through parameter compression and improvement of transmission mechanisms. However, the currently proposed solutions generally face the trade-off dilemma between accuracy loss and efficiency improvement. The Chinese patent with the publication number CN119312947A proposed the idea of splitting the global model into local models adapted to different clients, indirectly reducing communication overhead and improving training efficiency. The Chinese patent with the publication number CN119293861A proposed using differential privacy protection when clients upload model parameters and reducing communication overhead by dynamically allocating privacy budgets. To further reduce communication overhead, some methods have adopted compression and quantization techniques. For example, FedCompress combines periodic compressed communication and local computation, and QFed uses quantization techniques to compress model parameters and reduce the amount of data transmitted per round. Although these methods reduce the communication cost, a compromise needs to be made between accuracy and computational efficiency.

[0006] Although the above methods attempt to alleviate the data heterogeneity problem by graph data modeling and uploading class-averaged features, the effect is still limited when faced with large-scale data distribution differences. Although the currently proposed communication efficiency optimization techniques can reduce communication overhead, frequent communication operations are still required, and there is a trade-off between communication and training efficiency. Finally, it is often difficult to achieve a balance between personalized training and global model optimization in the currently proposed methods. Although some research teams have proposed adaptive local model initialization and dual knowledge distillation methods, in a large-scale client environment, personalized training may still lead to instability in the performance of the global model and it is difficult to meet the personalized needs of each client. Summary of the Invention

[0007] In view of this, embodiments of the present application propose a cross-client global classifier dynamic fine-tuning method for non-IID data, which can significantly improve the adaptability of the global model to non-IID data. When the client data distribution changes, it can quickly adjust the model decision boundary, thereby improving the accuracy and stability of the model, effectively reducing the negative impact of the data distribution difference between clients on the global model training, balancing the model optimization effect and communication overhead by dynamically adjusting the communication frequency, significantly improving communication efficiency, and optimizing the use of system resources.

[0008] In a first aspect, an embodiment of the present application proposes a method for dynamically fine-tuning a cross-client global classifier for non-IID data, which is applied to a federated learning framework composed of a server and multiple clients. The method includes the following steps: Each client locally trains its local model based on local data, freezes the classifier layer of the local model, and only updates the parameters of the feature extractor layer of the local model. During the local training process, based on the labels of the local data, calculate the class-average features for each category; After each client completes local training, upload data to the server according to the current communication stage; Among them, if the current communication stage is the high-frequency communication stage, each client uploads the class-average features and the updated model parameters to the server. If the current communication stage is the transition stage, each client only uploads the updated model parameters to the server. If the current communication stage is the low-frequency communication stage, each client uploads the updated model parameters to the server at the optimal upload frequency; The server updates the global model according to the updated model parameters uploaded by each client, calculates the weighted average of the class-average features uploaded by each client, and fine-tunes the classifier layer of the global model based on the weighted average to obtain the updated global model; Determine whether the updated global model converges. If it converges, send the updated global model to each client and end the federated learning. If it does not converge, send the updated global model to each client, and each client uses the updated global model as the local model for a new round of local training.

[0009] A method for dynamically fine-tuning a cross-client global classifier for non-IID data proposed in this application redefines the optimization process of the global model. Different from the traditional way of optimizing the global model by aggregating local models, this application effectively addresses the challenges brought by data heterogeneity among clients by introducing an adaptive global classifier fine-tuning mechanism. Each client uploads the class-average features to the server, and the server uses the weighted average of these class-average features to fine-tune the decision boundary of the classifier layer of the global model. This mechanism enables the global model to exhibit stronger adaptability in an environment with heterogeneous data distributions, reduces the overfitting problem of majority-class samples, and improves the classification effect of minority-class samples. To further optimize the global model, this application also implements a lightweight feature consistency calibration mechanism on the client side. During the local training process, the parameters of the classifier layer are frozen, and only the parameters of the feature extractor layer are optimized, thereby reducing the distribution difference of features of the same class among different clients. This strategy effectively alleviates the impact of feature noise and label shift on the global model. In addition, this application combines a phased dynamic communication strategy. In the initial stage of training, the feature distributions of clients are aligned through frequent communication to accelerate the model convergence process. As the model gradually converges, the communication frequency is dynamically reduced, thereby reducing communication overhead while maintaining the stability of model performance. Through this innovative technical solution, this application not only significantly improves the generalization ability of the global model but also effectively reduces the communication cost.

[0010] Optionally, in each round of local training, the k-th client optimizes the parameters of the feature extractor layer based on the local dataset D k Optimize the parameters of the feature extractor layer The optimization objective is to freeze the parameters of the classifier layer Minimize the following local loss function:

[0011]

[0012] where represents the local loss function of the k-th client, represents the feature output of the feature extractor layer for the local data, represents the predicted output of the classifier layer for the input features, |D k | represents the size of the local dataset D k l{·, y i} represents the selected base loss function, and t represents the current local training round is the t-th round.

[0013] Optionally, after each round of local training is completed, the k-th client uses the updated local model to calculate the class-average features of each class based on the labels of the local data in the local dataset D k respectively. Denote the class-average feature of the m-th class as The calculation formula is as follows:

[0014]

[0015] yi = m

[0016] Wherein, represents the total number of local data with the m-th category label in the local dataset D k and t represents that the current local training round is the t-th round.

[0017] Optionally, in the high-frequency communication stage, the client uploads the category average features and the updated model parameters to the server, including the category average features of each category calculated after completing the current round of local training and the updated model parameters obtained from each generation of training in the current round of local training

[0018]

[0019] Wherein, M represents the total number of categories, represents the category average feature of the m-th category, τ represents the total number of generations of the current round of local training, and represents the updated model parameters obtained from the θ-th generation of training in the current round of local training.

[0020] Optionally, in the high-frequency communication stage, after each round of global model update is completed, the server uses the test data to evaluate the accuracy of the updated global model. When the change rate of the accuracy of the global model in consecutive T1 rounds of updates is lower than the first preset threshold ε1, the communication stage switches to the transition stage, and each client will only upload the updated model parameters obtained from each generation of training in the current round of local training until the number of training rounds in the transition stage reaches the second preset threshold, and then the communication stage switches to the low-frequency communication stage.

[0021] Optionally, in the low-frequency communication stage, the server determines the optimal upload frequency based on Bayesian optimization, and each client uploads the updated model parameters obtained from each generation of training in the corresponding round of local training according to the optimal upload frequency

[0022] The objective function of Bayesian optimization is:

[0023]

[0024] where, Δt represents the uploading frequency, E represents the number of local training epochs, ΔA(t) represents the change rate of the accuracy of the global model after the current round of update compared to that after the previous round of update, and α, β, and γ are all preset balance parameters, which are used to control the relative importance of model performance, uploading frequency, and the number of local training epochs respectively, Δt * and E * are the optimal uploading frequency and the number of local training epochs for the corresponding round respectively;

[0025] In the low-frequency communication stage, the updated model parameters uploaded by the k-th client are expressed as:

[0026]

[0027] Optionally, the weighted average of the class-average features uploaded by each client is calculated, which is expressed by the formula:

[0028]

[0029] where, S represents the set of clients;

[0030] When fine-tuning the classifier layer of the global model based on the weighted average, the objective function is defined as:

[0031]

[0032] where, represents the prediction of the classifier layer of the global model for the m-th class, ω c,t represents the parameters of the classifier layer of the global model after the t-th round of update, and y m represents the true label of the m-th class.

[0033] Optionally, the update of the parameters of the feature extractor layer of the global model and the update of the parameters of the classifier layer are expressed by the formula:

[0034]

[0035] where, η g represents the learning rate, represents the gradient of ω c,t , |S| represents the total number of clients, and ω c,f+1 and ω c,t+1 represent the parameters of the feature extractor layer and the parameters of the classifier layer of the updated global model respectively.

[0036] In a second aspect, an embodiment of the present application provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor is enabled to execute a method for dynamically fine-tuning a cross-client global classifier for non-IID data as described in the first aspect above.

[0037] In a third aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program, which when executed by a processor, can implement a method for dynamically fine-tuning a cross-client global classifier for non-IID data as described in the first aspect above.

[0038] It can be understood that the beneficial effects of the above second aspect to the third aspect can be referred to the relevant descriptions in the first aspect above, and will not be elaborated here. The beneficial effects generated by the above second aspect to the third aspect can also be applied to the first aspect. Description of the Drawings

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the related art, the drawings required for use in the description of the embodiments of the present application or the related art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0040] Figure 1 is a flowchart of a method for dynamically fine-tuning a cross-client global classifier for non-IID data provided in an embodiment of the present application;

[0041] Figure 2 is a schematic diagram of a federated learning process provided in an embodiment of the present application;

[0042] Figure 3 is a schematic diagram of the high-frequency communication stage, transition stage, and low-frequency communication stage of a progressive communication optimization strategy provided in an embodiment of the present application;

[0043] Figure 4 is a schematic diagram of the comparison between the FedAvg method and a method for dynamically fine-tuning a cross-client global classifier for non-IID data proposed in the present application provided in an embodiment of the present application;

[0044] Figure 5 is a schematic diagram of the structure of an electronic device provided in another embodiment of the present application. Detailed Embodiments

[0045] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the following will elaborate on each embodiment of this application in conjunction with the accompanying drawings. In various embodiments of this application, many technical details are proposed to enable readers to better understand this application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solutions claimed in this application can still be implemented. The division of the following embodiments is only for convenient description and should not constitute any limitation on the specific implementation of this application. The various embodiments can be combined and cross-referenced with each other on the premise of not being contradictory.

[0046] To address the problem that traditional federated learning methods have limited effectiveness in the face of large-scale data distribution differences, an embodiment of this application proposes a cross-client global classifier dynamic fine-tuning method for non-IID data, which is applied to a federated learning framework composed of a server and multiple clients. The following specifically describes the implementation details of the cross-client global classifier dynamic fine-tuning method for non-IID data proposed in this embodiment. The following content is only implementation details provided for easy understanding and is not essential for implementing this solution.

[0047] The specific process of the cross-client global classifier dynamic fine-tuning method for non-IID data proposed in this embodiment can be as Figure 1 shown and includes:

[0048] Step 101, each client locally trains its local model based on local data, freezes the classifier layer of the local model, and only updates the parameters of the feature extractor layer of the local model. During the local training process, based on the labels of the local data, calculate the class average feature of each class.

[0049] Specifically, in the federated learning framework, each client needs to locally train its local model based on local data. To improve its adaptability to non-IID data while maintaining the stability of the classifier layer of the global model in the subsequent process, each client needs to adopt a method based on lightweight feature consistency calibration for local training, that is, it is necessary to freeze the classifier layer of the local model and only update the parameters of the feature extractor layer of the local model, thereby reducing the complexity of local training and reducing the risk of overfitting. In addition, during the local training process, each client also needs to calculate the class average feature of each class based on the labels of the local data.

[0050] In an example, the federated learning framework proposed in this application can be as Figure 2 shown.

[0051] It is understandable that in the traditional federated learning framework, it is difficult to fully capture the impact brought by the heterogeneous feature distributions among different clients only relying on the training of the classifier layer of the global model (hereinafter simply referred to as the global classifier). This is because the optimization effect of the global classifier depends on the class-averaged features uploaded by the clients, and the feature distributions of the clients may have noise or inconsistencies between classes due to data heterogeneity, thus restricting the generalization ability of the global model. Therefore, simply optimizing the global classifier cannot completely solve the problem of heterogeneous feature distributions. For this reason, this embodiment proposes and applies a lightweight cross-client feature consistency mechanism to optimize the adaptability of the global classifier in the federated learning environment and reduce the feature distribution differences between clients.

[0052] In one example, in each round of local training, the k-th client optimizes the parameters of the feature extractor layer based on the local dataset D k The optimization objective is to freeze the parameters of the classifier layer and minimize the following local loss function:

[0053]

[0054] where represents the local loss function of the k-th client, represents the feature output of the feature extractor layer for the local data, represents the predicted output of the classifier layer for the input features, |D k | represents the size of the local dataset D k and l{·,y i} represents the selected base loss function (classification loss), and t represents that the current local training is the t-th round.

[0055] In one example, after each round of local training is completed, the k-th client will use the updated local model in this round and calculate the class-averaged features of each category based on the labels of the local data in the local dataset D k Let the class-averaged feature of the m-th category be denoted as The calculation formula of

[0056]

[0057] yi = m

[0058] where represents the total number of local data with the label of the m-th category in the local dataset D k and t represents that the current local training is the t-th round.

[0059] ​Step 102, after each client completes local training, it uploads data to the server according to the current communication phase. Among them, if the current communication phase is the high-frequency communication phase, each client uploads the class average features and the updated model parameters to the server. If the current communication phase is the transition phase, each client only uploads the updated model parameters to the server. If the current communication phase is the low-frequency communication phase, each client uploads the updated model parameters to the server according to the optimal upload frequency.

[0060] Specifically, after each client completes local training, it will upload data to the server according to the current communication phase. Among them, the communication phase is divided into three phases, namely the high-frequency communication phase (the first phase), the transition phase, and the low-frequency communication phase (the second phase). If the current communication phase is the high-frequency communication phase, each client uploads the class average features and the updated model parameters to the server. If the current communication phase is the transition phase, each client only uploads the updated model parameters to the server. If the current communication phase is the low-frequency communication phase, each client uploads the updated model parameters to the server according to the optimal upload frequency.

[0061] It can be understood that although this embodiment effectively solves the impact of local feature distribution differences on the global model optimization through the lightweight cross-client feature consistency mechanism, during the optimization process of the global classifier, it still needs to face the high communication cost brought by multiple rounds of communication, especially in a bandwidth-constrained environment such as a narrowband network. Traditional communication strategies require communication in each round. Although this method can ensure the stable update of the model, frequent communication may lead to resource waste and is not necessary in some stages. Therefore, this embodiment proposes and applies a progressive communication optimization strategy to optimize the communication efficiency of the global classifier during the distributed training process.

[0062] By analyzing the communication requirements at different time periods during the global model optimization process, we found that the communication requirements for model optimization in different stages are different. Specifically, in the initial stage of the global model, in order to quickly align the feature distributions between clients, a higher communication frequency is required. While in the later stage when the model gradually converges, the benefits of frequent communication will significantly decrease. This observation indicates that dynamically adjusting the communication frequency at different time periods of global optimization can not only balance the communication cost and the model optimization effect, but also make more efficient use of resources. Therefore, this embodiment designs a progressive communication optimization strategy, which significantly reduces the communication cost while ensuring the global model optimization effect by dynamically adjusting the communication frequency and the uploaded content. Figure 3 Shows the overall framework of this communication optimization strategy, specifically including the interaction methods and communication content between the client and the server at different stages.

[0063] Such asFigure 3 As shown, in the high-frequency communication stage, the client uploads the class average features of each category calculated after completing the current round of local training to the server and the updated model parameters obtained from each generation of training in the current round of local training M represents the total number of categories, represents the class average feature of the m-th category, τ represents the total number of generations in the current round of local training, represents the updated model parameters obtained from the θ-th generation of training in the current round of local training. The reason for only uploading the class average features of the last generation is that in the initial stage, the local model of the client has not fully converged, and the feature representation is continuously adjusted with the training rounds. If all generations of feature information are uploaded frequently, it will not only introduce excessive noise and redundant data, but also significantly increase the communication overhead. By only uploading the class average features of the last generation, the data instability can be effectively reduced, ensuring that the uploaded content is more representative and reducing the communication burden at the same time.

[0064] When the local models of each client have achieved a certain degree of convergence during the training process in the high-frequency communication stage, and the feature representations of the clients have gradually stabilized, the server will judge whether to switch to the low-frequency communication stage based on the change rate of accuracy. It should be noted that in order to avoid the adverse impact on the global model optimization caused by prematurely stopping the feature information upload during the switch, we set a transition stage. The high-frequency communication stage will first switch to the transition stage and then to the low-frequency communication stage.

[0065] In one example, in the high-frequency communication stage, after each round of global model update is completed, the server uses the test data to evaluate the accuracy of the updated global model. When the change rate of the accuracy of the global model in T1 consecutive rounds of updates is lower than the first preset threshold ε1, the communication stage switches to the transition stage, and each client will only upload the updated model parameters obtained from each generation of training in the current round of local training Until the number of training rounds in the transition stage reaches the second preset threshold, the communication stage switches to the low-frequency communication stage. It should be noted that both the first preset threshold and the second preset threshold can be set by those skilled in the art according to actual needs.

[0066] In one example, in the low-frequency communication stage, the server determines the optimal upload frequency based on Bayesian optimization, and each client uploads the updated model parameters obtained from each generation of training in the corresponding round of local training according to the optimal upload frequency

[0067] In one example, the objective function of Bayesian optimization is:

[0068]

[0069] where Δt represents the upload frequency, E represents the number of local training epochs, ΔA(t) represents the change rate of the accuracy of the global model after the current round of update compared to that after the previous round of update, and α, β, and γ are all preset balance parameters, which are used to control the relative importance of model performance, upload frequency, and the number of local training epochs respectively. Δt * and E * are the optimal upload frequency and the number of local training epochs for the corresponding round respectively.

[0070] In one example, during the low-frequency communication phase, the updated model parameters uploaded by the k-th client can be expressed as:

[0071] In one example, as Figure 3 shown, in Phase 1, the client uploads the locally trained weights and class-averaged features to the cloud server, and the server uses these features to fine-tune the decision boundary of the global classifier. After entering the transition phase, the client only uploads the locally trained weights and no longer uploads feature information. At this time, the goal is to balance the global model convergence speed and communication efficiency to prepare for the next Phase 2. In Phase 2, the client continues to upload the local weights, but adopts a dynamic upload strategy. According to the convergence of the global model, the upload frequency is dynamically adjusted to further reduce the communication overhead and maintain the optimization effect of the global model. Through the gradual optimization of these three phases, the federated learning framework finally realizes the efficient training of the global classifier while improving the communication efficiency and the use of system resources. Figure 3 The processes of the client and the cloud server in

[0072] Step 103, the server updates the global model according to the updated model parameters uploaded by each client, calculates the weighted average of the class-averaged features uploaded by each client, and fine-tunes the classifier layer of the global model based on the weighted average to obtain the updated global model.

[0073] Specifically, after receiving the updated model parameters uploaded by each client, the server can update the global model according to the updated model parameters uploaded by each client, calculate the weighted average of the class-averaged features uploaded by each client, and fine-tune the classifier layer of the global model based on the weighted average to obtain the updated global model.

[0074] It is understandable that traditional federated learning methods mainly rely on the aggregation of client local model parameters and update the global model through the aggregation results. This approach is difficult to optimize the decision boundary of the global classifier in the face of heterogeneous data distributions among different clients, resulting in overfitting of majority-class samples and a decline in the classification performance of minority-class samples. In this embodiment, the decision boundary of the global classifier is directly fine-tuned on the server side, combined with the class-average feature information uploaded by the clients, enabling the global model to more effectively adapt to the non-independent and identically distributed characteristics of the data and improving the generalization ability of the model.

[0075] In one example, the comparison between the traditional FedAvg method and a cross-client global classifier dynamic fine-tuning method for non-IID data proposed in this application is as Figure 4 shown. The traditional FedAvg method constructs a global model by aggregating the local model parameters uploaded by the clients, and its decision boundary is determined by the combined influence of all local models. The method proposed in this embodiment, on the other hand, uploads the class-average features from the clients and aggregates them on the server side to fine-tune the global classifier. The optimized classifier is then distributed to the clients to guide the adjustment of the local feature extractors, thereby refining the decision boundary of the global model.

[0076] In one example, as Figure 2 shown, on the client side, the feature extractor calculates the class-average features and uploads them to the server together with the local weights. On the server side, the weights are aggregated to update the global model, and the class-average features are used to fine-tune the classifier, optimizing the classifier through the classification loss. The updated global model is then redistributed to each client to guide the optimization process of the local feature extractors. Figure 2 The solid arrows in

[0077] represent the forward process, and the dashed arrows represent the reverse process.

[0078]

[0079] where S represents the set of clients.

[0080] In one example, when the server fine-tunes the classifier layer of the global model based on the weighted average, the objective function can be defined as:

[0081]

[0082] where represents the prediction of the classifier layer of the global model for the m-th class, ω c,t represents the parameters of the classifier layer of the global model after the t-th round of update, ym Represents the true label of the m-th category.

[0083] In one example, the update of the parameters of the feature extractor layer of the global model and the update of the parameters of the classifier layer can be expressed by the formula:

[0084]

[0085] Where η g Represents the learning rate, Represents the gradient of ω c,t , |S| represents the total number of clients, and ω c,f+1 and ω c,t+1 Represent the parameters of the feature extractor layer and the classifier layer of the updated global model, respectively.

[0086] Step 104, determine whether the updated global model converges. If it converges, send the updated global model to each client and end the federated learning. If it does not converge, send the updated global model to each client, and each client uses the updated global model as the local model for a new round of local training.

[0087] A cross-client global classifier dynamic fine-tuning method for non-IID data proposed in this embodiment redefines the optimization process of the global model. Different from the traditional way of optimizing the global model by aggregating local models, this embodiment effectively addresses the challenges brought by data heterogeneity among clients by introducing an adaptive global classifier fine-tuning mechanism. Each client uploads the class average features to the server, and the server uses the weighted average of these class average features to fine-tune the decision boundary of the classifier layer of the global model. This mechanism enables the global model to show stronger adaptability in an environment with heterogeneous data distributions, reduces the overfitting problem of majority-class samples, and improves the classification effect of minority-class samples. To further optimize the global model, this embodiment also designs and implements a lightweight feature consistency calibration mechanism on the client side. During the local training process, the parameters of the classifier layer of the local model are frozen, and only the parameters of the feature extractor layer are optimized, thereby reducing the distribution difference of features of the same category among different clients. This strategy effectively reduces the impact of feature noise and label shift on the global model. In addition, this application combines a phased dynamic communication strategy, aligns the feature distributions of clients through frequent communication in the initial stage of training to accelerate the model convergence process, and as the model gradually converges, the communication frequency is dynamically reduced, thereby reducing the communication overhead while maintaining the stability of the model performance. Through this innovative technical solution, this embodiment not only significantly improves the generalization ability of the global model but also effectively reduces the communication cost.

[0088] Compared with the traditional FedAvg method, the global classifier fine-tuning mechanism in this embodiment significantly improves the generalization ability of the model, especially in the case of uneven client data distribution. The traditional FedAvg method is usually affected by data heterogeneity, resulting in a significant decrease in the accuracy of the global model, especially in a large-scale distributed environment. By dynamically optimizing the decision boundary of the global model and optimizing the consistency of client feature distributions through the feature consistency calibration strategy, this embodiment can effectively adapt to the data distributions of different clients, thereby maintaining a high classification accuracy and the stability of model training. For example, in the experiment on the FMNIST dataset, the accuracy of the global model of the traditional FedAvg method was 87%, while after adopting this method, the accuracy of the global model increased by approximately 4% to reach 90.88%. In the comparative experiment on the CIFAR-10 dataset, compared with the traditional method, this method increased the accuracy by approximately 3%, from 86.1% to 89.93%.

[0089] Compared with the traditional high-frequency communication strategy, the phased dynamic communication strategy in this embodiment can automatically adjust the communication frequency according to the training stage, significantly reducing the communication overhead during training. Traditional federated learning methods require frequent communication rounds, which not only increases the burden on the bandwidth but also leads to unnecessary waste of communication resources. By adopting high-frequency communication in the initial stage of training and gradually reducing the communication frequency in the later stage, this embodiment significantly reduces the communication cost while ensuring the training efficiency of the model, especially suitable for environments with limited bandwidth or limited computing resources. For example, on the FMNIST dataset, compared with the traditional method, this embodiment reduces the communication overhead by approximately 40%, and the convergence speed of the model is not significantly affected. On the CIFAR-10 dataset, after adopting the phased dynamic communication strategy, the communication overhead is reduced by approximately 35%, effectively improving the computing efficiency during training.

[0090] Current federated learning methods, such as FedProx, SCAFFOLD, and FedDyn, although alleviating the non-IID data problem to some extent, still rely on the alignment of local models and global models and cannot effectively handle the dynamic changes in client data distribution. This embodiment introduces an adaptive global classifier optimization mechanism to dynamically adjust the decision boundary of the global model and combines the class-averaged features uploaded by clients for global model optimization. This mechanism enables the global model to quickly respond when the client data distribution changes, thereby improving the generalization ability of the global model, especially when dealing with minority class samples, showing stronger stability and accuracy. Compared with traditional methods, this embodiment not only improves the adaptability of the global model but also significantly improves its performance in complex and dynamic data environments, especially in medical and health scenarios that require high-precision predictions.

[0091] Current federated learning methods usually rely on feature alignment techniques such as Fed2 and FedPAC to mitigate the problems caused by inconsistent data distributions by introducing auxiliary feature alignment variables on the client side. However, feature alignment methods have limitations in extremely non-IID scenarios. Especially when the data distribution differences are large, the feature alignment effect will be greatly reduced. To solve this problem, this embodiment proposes a lightweight feature consistency calibration mechanism, which effectively optimizes the training effect of the global model by reducing the feature distribution differences between clients. This mechanism does not rely on large-scale generated data or complex adversarial training, greatly reducing the computational and communication overheads, and significantly improving the performance of the global model in data heterogeneous environments. Compared with traditional alignment techniques, the mechanism of this embodiment has higher efficiency and lower computational overheads, and is particularly suitable for application scenarios with highly heterogeneous data distributions, such as cross-hospital medical data analysis.

[0092] Communication overhead is a major challenge in federated learning, especially in large-scale distributed systems and bandwidth-constrained network environments. In traditional solutions such as FedProx and SCAFFOLD, although attempts are made to reduce the communication frequency by reducing the local update fluctuations, they still face the problem of frequent communication, resulting in waste of network bandwidth and computing resources. The progressive communication optimization strategy proposed in this embodiment can dynamically adjust the communication frequency, enabling accelerated convergence through frequent communication in the initial stage of model training and reducing unnecessary communication in the later stage, thus reducing the communication burden. By dynamically optimizing the communication frequency according to the model convergence state and training stage, not only the communication cost is reduced, but also the overall efficiency of the system is improved. This strategy is particularly suitable for application scenarios with high bandwidth requirements such as medical image analysis, which can reduce the consumption of network resources and improve the execution efficiency of tasks.

[0093] The step divisions of the above various methods are only for clear description. When implemented, they can be combined into one step or some steps can be split into multiple steps. As long as the same logical relationship is included, they are all within the protection scope of this application; adding insignificant modifications to the algorithm or process or introducing insignificant designs, but without changing the core design of its algorithm and process, are all within the protection scope of this application.

[0094] Another embodiment of this application proposes an electronic device, the specific structure of which is as Figure 5 shown, including: at least one processor 301; and a memory 302 communicatively connected to the at least one processor 301; wherein, the memory 302 stores instructions executable by the at least one processor 301, and the instructions are executed by the at least one processor 301 to enable the at least one processor 301 to execute a method for dynamically fine-tuning a cross-client global classifier for non-IID data as described in the above method embodiment.

[0095] Among them, the memory and the processor can be connected in a bus manner. The bus can include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors and the memory together. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, etc., which are well known in the art and will not be further described herein. The bus interface is responsible for providing an interface between the bus and the transceiver. The transceiver can be a component or multiple components, such as multiple receivers and transmitters, and provides a unit for communicating with various other devices on the transmission medium.

[0096] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interface, voltage regulation, power management, and other control functions. And the memory can be used to store the data used by the processor when executing operations.

[0097] Another embodiment of the present application proposes a computer-readable storage medium storing a computer program, which when executed by a processor, can implement a method for dynamically fine-tuning a cross-client global classifier for non-IID data as described in the above method embodiment.

[0098] That is, those skilled in the art can understand that all or part of the steps in implementing the above embodiment methods can be completed by instructing relevant hardware through a program. This program is stored in a storage medium and includes several instructions to enable a device (such as a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the methods described in various embodiments of the present application. And the foregoing storage medium includes: USB flash drives, mobile hard disks, ROM (Read-Only Memory), RAM (Random Access Memory), magnetic disks, or optical discs and other various media that can store program codes.

[0099] Those of ordinary skill in the art can understand that the above embodiments are specific embodiments for implementing the present application, and in actual applications, various changes can be made in form and details without departing from the spirit and scope of the present application.

Claims

1. A method for dynamically fine-tuning a cross-client global classifier for non-IID data, characterized in that, Applied to a federated learning framework composed of a server and multiple clients, the method includes: Each client respectively performs local training on the local model based on local data, freezes the classifier layer of the local model, only updates the parameters of the feature extractor layer of the local model, and during the local training process, calculates the class average feature of each class based on the labels of the local data; After each client completes local training, it uploads data to the server according to the current communication stage; among them, if the current communication stage is the high-frequency communication stage, each client uploads the class average feature and the updated model parameters to the server, if the current communication stage is the transition stage, each client only uploads the updated model parameters to the server, and if the current communication stage is the low-frequency communication stage, each client uploads the updated model parameters to the server at the optimal upload frequency; The server updates the global model according to the updated model parameters uploaded by each client, calculates the weighted average value of the class average features uploaded by each client, and fine-tunes the classifier layer of the global model based on the weighted average value to obtain the updated global model; Determine whether the updated global model converges. If it converges, the updated global model is sent to each client and the federated learning ends. If it does not converge, the updated global model is sent to each client, and each client uses the updated global model as the local model for a new round of local training.

2. The dynamic fine-tuning method of a cross-client global classifier for non-IID data according to claim 1, wherein In each round of local training, the k-th client optimizes the parameters of the feature extractor layer based on the local dataset D k Optimize the parameters of the feature extractor layer The optimization objective is to minimize the following local loss function by freezing the parameters of the classifier layer Minimize the following local loss function: Among them, represents the local loss function of the k-th client, represents the feature output of the feature extractor layer for the local data, represents the prediction output of the classifier layer for the input features, |D k | represents the size of the local dataset D k The size of, l{·,y i} represents the selected basic loss function, and t represents the current local training round is the t-th round.

3. The dynamic fine-tuning method of a cross-client global classifier for non-IID data according to claim 2, wherein After each round of local training is completed, the k-th client uses the updated local model and calculates the class-average features for each class based on the labels of the local data in the local dataset D k Let the class-average feature of the m-th class be denoted as The calculation formula is as follows: Among them, represents the total number of local data with the m-th category label in the local dataset D k and t represents that the current local training round is the t-th round.

4. The dynamic fine-tuning method of a cross-client global classifier for non-IID data according to claim 3, wherein In the high-frequency communication stage, the client uploads the class average features and the updated model parameters to the server, including the class average features of each class calculated after completing the current round of local training and the updated model parameters obtained from each generation of training in the current round of local training where M represents the total number of categories, represents the average feature of the m-th category, τ represents the total number of generations of local training in the current round, represents the updated model parameters obtained from the θ-th generation of training in the current round of local training.

5. A method for dynamically fine-tuning a cross-client global classifier for non-IID data as described in claim 4, characterized in that, In the high-frequency communication stage, after the server completes one round of global model update each time, it uses test data to evaluate the accuracy of the updated global model. When the change rate of the accuracy of the global model is lower than the first preset threshold ε1 in consecutive T1 rounds of updates, the communication stage switches to the transition stage, and each client will only upload the updated model parameters obtained in each generation of training in the current round of local training. Until the number of training rounds in the transition stage reaches the second preset threshold, the communication stage switches to the low-frequency communication stage.

6. A method for dynamically fine-tuning a cross-client global classifier for non-IID data as described in claim 5, characterized in that In the low-frequency communication phase, the server determines the optimal upload frequency based on Bayesian optimization, and each client uploads the updated model parameters obtained from each generation of training in the corresponding round of local training to the server according to the optimal upload frequency. The objective function of Bayesian optimization is: Among them, Δt represents the upload frequency, E represents the number of local training epochs, ΔA(t) represents the change rate of the accuracy of the global model after the current round of update compared to the accuracy of the global model after the previous round of update, and α, β, and γ are all preset balance parameters, which are used to control the relative importance of model performance, upload frequency, and the number of local training epochs, respectively, Δt * and E * are the optimal upload frequency and the number of local training epochs for the corresponding round, respectively; During the low-frequency communication phase, the updated model parameters uploaded by the k-th client are expressed as:

7. A method for dynamically fine-tuning a cross-client global classifier for non-IID data according to any one of claims 1 to 6, characterized in that Calculate the weighted average value of the class average features uploaded by each client, which is expressed by the formula: where S represents the set of clients; When fine-tuning the classifier layer of the global model based on the weighted average value, the objective function is defined as: Among them, represents the prediction of the classifier layer of the global model for the m-th category, ω c,t represents the parameters of the classifier layer of the global model after the t-th round of update, y m represents the true label of the m-th category.

8. A method for dynamically fine-tuning a cross-client global classifier for non-IID data according to claim 7, characterized in that The update of the parameters of the feature extractor layer of the global model and the update of the parameters of the classifier layer are expressed by the formula: Among them, η g represents the learning rate, represents the gradient of ω c,t , |S| represents the total number of clients, ω c,f+1 and ω c,t+1 represent the parameters of the feature extractor layer and the classifier layer of the updated global model, respectively.

9. An electronic device, characterized in that, Include: At least one processor; And a memory communicatively connected to the at least one processor; Wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a method for dynamically fine-tuning a cross-client global classifier for non-IID data according to any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it can implement a method for dynamically fine-tuning a cross-client global classifier for non-IID data according to any one of claims 1 to 8.

Images