A federated learning method and device based on dynamic expert selection and knowledge fusion

Abstract

This invention provides a federated learning method and apparatus based on dynamic expert selection and knowledge fusion, belonging to the field of federated learning technology. The invention receives the current round's global model broadcast from the server and decomposes it into K virtual expert sets by combining it with the previous round's local old model. Then, a lightweight gating network is used to generate selection weights for each virtual expert. Based on these selection weights, knowledge distillation is used to weightedly fuse the soft outputs of the virtual experts on the local data to generate fused soft labels, which are then used to initialize the local model. The initialized local model is trained, and an adaptive residual compensation mechanism is introduced to upload the adaptive residuals and model parameters to the server. Weighted aggregation and updating of the global model are then performed on the server before broadcasting to the next round. This application enables fine-grained knowledge selection and soft fusion, effectively avoiding knowledge conflicts, significantly reducing communication overhead in highly non-independent and identically distributed environments, and improving the accuracy of personalized models.

Description

Technical Field

[0001] This invention relates to the field of federated learning technology, and more specifically, to a federated learning method and apparatus based on dynamic expert selection and knowledge fusion. Background Technology

[0002] Federated learning, as a distributed machine learning paradigm, allows multiple clients to collaboratively train a shared model while ensuring data privacy, and has been widely applied in fields highly sensitive to data security, such as healthcare, finance, mobile computing, and edge intelligence. However, in practical deployments, federated learning systems generally face severe statistical heterogeneity problems, meaning that the local data of each client differs significantly in distribution, scale, and semantics, making it difficult for a single global model to generalize well across all clients. For example, in cross-hospital medical image analysis tasks, different institutions use different imaging equipment, scanning protocols, and patient group characteristics, causing a uniform model to experience a sharp performance drop on some clients. To address this challenge, personalized federated learning methods have been proposed, aiming to build a dedicated model for each client that adapts to the characteristics of its local data, while making the most of beneficial knowledge from other clients.

[0003] Current mainstream personalized federated learning methods mainly include strategies such as model segmentation, adaptive aggregation, hybrid expert (MoE) architecture, and hypernetwork generation. Among them, FedRep achieves local customization by dividing the model into a shared representation layer and a personalized head layer; Per-FedAvg learns a globally initialized parameter that is friendly to personalized fine-tuning through a meta-learning mechanism; FedALA and other methods dynamically adjust the combination ratio of the global model and the local model by learning fusion weights; while MoE-based methods such as pFedMoE and FedMoE introduce multiple expert sub-models and use gating networks to select activated experts based on input samples to achieve personalized inference; in addition, FedPer and pFedHN use hypernetworks to generate personalized parameters based on client identifiers.

[0004] While the aforementioned methods mitigate the performance degradation caused by statistical heterogeneity to some extent, several key limitations remain: First, most methods treat the global and local models as a whole for coarse-grained fusion, failing to identify and utilize the diverse knowledge contained in different components within the model at a fine-grained level. Second, direct weighted averaging or simple concatenation can easily lead to knowledge conflicts, especially in highly heterogeneous scenarios, where the global model may contain information detrimental to the local task, thus introducing negative transfer. Third, existing MoE-like methods typically employ static expert structures, with the number and topology fixed before training, lacking the ability to dynamically adjust based on data characteristics or the training phase, thus limiting the model's expressive flexibility. Finally, existing solutions generally neglect effective control over communication and computational overhead, making them unsuitable for resource-constrained edge devices or low-bandwidth network environments. More critically, current technologies lack explicit modeling and dynamic discrimination mechanisms for client heterogeneity, making it difficult to achieve accurate, efficient, and robust personalized modeling in complex and ever-changing federated scenarios.

[0005] In view of the above, this application is hereby submitted. Summary of the Invention

[0006] This invention aims to provide a federated learning method and apparatus based on dynamic expert selection and knowledge fusion, in order to solve the technical defects of existing federated learning methods in statistical heterogeneity scenarios, such as coarse knowledge fusion granularity, high risk of knowledge conflict, static and fixed expert structure, large communication and computing overhead, and lack of explicit modeling of heterogeneity.

[0007] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution:

[0008] A federated learning method based on dynamic expert selection and knowledge fusion, applied to the client side, includes:

[0009] S1 receives the global model of the current round broadcast by the server and decomposes it into K sets of virtual experts, with each virtual expert corresponding to a subset of parameters.

[0010] S2, based on local small-batch data, generates selection weights for each virtual expert through a lightweight gating network;

[0011] S3, based on the selection weights, the soft outputs of the selected virtual experts on the local data are weighted and fused through knowledge distillation to generate fused soft labels, and the local model for this round is initialized accordingly;

[0012] S4 trains the local model based on the initialization to obtain the trained local model. At the same time, an adaptive residual compensation mechanism is introduced to upload the adaptive residual and model parameters to the server. The server performs weighted aggregation, updates the global model, and then broadcasts it to the next round.

[0013] Preferably, the previous local old model is decomposed into K sets of virtual experts, each virtual expert corresponding to a subset of parameters, specifically:

[0014] Suppose the neural network of the global model contains Layer, number The layer parameters are set to ;

[0015] The K sets of global virtual experts into which the global model is decomposed are represented as follows: ;

[0016] Each global virtual expert This corresponds to a subset of parameters in the global model, namely: ;

[0017] in, This indicates that the virtual expert is assigned to the global virtual expert. A set of layer indices that satisfies: And when hour, ;

[0018] in, Represents the union; , Representing global virtual experts , The set of layer indexes;

[0019] Similarly, the local old model of client i is decomposed into K sets of local virtual experts.

[0020] Preferably, the expression for the selection weight is:

[0021] ;

[0022] ;

[0023] in, Weights are selected jointly by global and local experts; Indicates client Lightweight gating networks; For the client Gating network parameters; For the client Local data batches; , These are the trainable weight parameters of the gated network; , For the bias of the gated network; For activation functions; This is the normalization function; For batch Average characteristics; Assign a client index number; The encoder for the local model; This is a local data sample; These are the parameters for the local model encoder.

[0024] Preferably, the parameters of the lightweight gating network are obtained by training to minimize the objective loss function; the expression of the objective loss function is:

[0025] ;

[0026] in, Represents the parameters of the gated network Find the minimum value; Let be the objective loss function of the gated network; Indicates the local dataset of client i Data batch Expectations; The task loss function; This is a real label; The output of the model after integrating experts; For sparsity regularization coefficients, This is the smoothness regularization coefficient; For parameters The gradient; Weights are selected jointly by global and local experts; , These are the L1 and L2 norms, respectively.

[0027] Preferably, S3 specifically comprises:

[0028] First, the parameters of the global virtual expert and the local virtual expert are weighted and fused using the joint selection weights output by the gating network to obtain the initial parameters of the local model. The expression is as follows:

[0029] ;

[0030] in, These are the initial parameters for the local model of client i; The number of global virtual experts / local virtual experts; Let i be the normalized selection weight of the kth expert for client i; The parameters are for the k-th global virtual expert; For the first Parameters for a local virtual expert;

[0031] Then, a knowledge distillation mechanism is introduced to calculate a soft label for each expert's output, expressed as:

[0032] ;

[0033] in, The soft label for the k-th expert in the local data batch; The output is the logits of the kth expert; This is a temperature parameter used to control the smoothness of the soft label;

[0034] The expert soft labels are weighted according to the joint selection weights to obtain the fused soft labels, expressed as follows:

[0035] ;

[0036] in, To integrate soft tags;

[0037] The local model is initialized by minimizing the KL divergence between the local model output and the fused soft label, and by introducing an adaptive temperature adjustment mechanism for optimization. The expression is as follows:

[0038] ;

[0039] in, The knowledge distillation loss function; For the local data batch of the i-th client Expectations; Let KL divergence be a metric. To initialize the model's logits output; These are the weighting coefficients; The cross-entropy loss function;

[0040] In addition, an adaptive temperature parameter adjustment mechanism is introduced to dynamically adjust the temperature parameter, the expression of which is:

[0041] ;

[0042] in, The temperature parameters are for the t-th training round; These are the initial temperature parameters; The attenuation coefficient; For the first Total local losses of the wheel; The local total loss is the initial state. It is an exponential function.

[0043] Preferably, when generating the selection weights of each virtual expert through a lightweight gating network, a hierarchical attention mechanism is further introduced. This mechanism divides the lightweight gating network into layers with different semantic abstraction levels, calculates the attention score for each layer, and then applies a hierarchical weighting to the selection weights. The expression is as follows:

[0044] ;

[0045] ;

[0046] in, For the first Attention scores for each layer group; These are the weight parameters for the attention mechanism; , For the attention mechanism, the weight matrix and bias are used. It is the hyperbolic tangent activation function; Let i be the normalized selection weight of the kth expert for client i; The selection weights are the result of hierarchical attention weighting. For expert k's level group The corresponding attention weights;

[0047] Simultaneously, a low-rank decomposition is performed on the weight matrix of the gated network to reduce the number of parameters in the gated network, expressed as:

[0048] ;

[0049] in, Here is the weight matrix of the gated network; This is the left matrix after low-rank decomposition; This is the right matrix after low-rank decomposition; This is the transpose symbol.

[0050] Preferably, it further includes: during local model training, calculating the importance scores of the local model's neural network layers and using Top-k sparsity technology to set differentiated communication update frequencies for different layers to reduce communication overhead, the expression of which is:

[0051] ;

[0052] ;

[0053] ;

[0054] in, For the first Layer importance score; For the local model Layer parameters; for The gradient; For the derivative sign; For the first The dimension of the layer; For the first Layer communication frequency; The highest importance score among all layers; Basic communication frequency; This is a rounding up operation; This is the result of sparsification; For the first The number of parameters retained in the layer; For the first The sparsity of the layer; To select the parameter with the largest absolute value Set one element to 0, and the rest to 0;

[0055] Meanwhile, the sparsity of each layer is dynamically adjusted based on the importance score to balance communication efficiency and model performance, as shown in the formula:

[0056] ;

[0057] in, , These represent the maximum and minimum sparsity values, respectively.

[0058] Preferably, an adaptive residual compensation mechanism is introduced to upload the adaptive residuals and model parameters to the server, so that weighted aggregation can be performed on the server side. Specifically:

[0059] First, calculate the model update residual based on the difference between the parameters trained locally on the client and the initial parameters. The formula is:

[0060] ;

[0061] in, This represents the model update residual for client i in round t; These are the model parameters trained locally by client i in round t. Let be the initial parameters of the local model for client i in round t;

[0062] Then, based on the model-updated residuals combined with an adaptive decay factor, the cumulative value of the historical residuals is updated, as shown in the formula:

[0063] ;

[0064] ;

[0065] in, , They are respectively the tth, The accumulated residual buffer for round client i; It is an adaptive attenuation factor; This is the minimum value of the attenuation factor; The initial decay factor; The decay rate; Let be the local loss gradient of client i in round t; Let be the local loss gradient of client i in round 0; It is an L2 norm; , Let be the local losses of client i in rounds t and 0, respectively;

[0066] A residual selective transmission strategy is adopted for residual transmission, that is, transmission is triggered when the residual norm exceeds a dynamic threshold or reaches a fixed period, as shown in the formula:

[0067] ;

[0068] ;

[0069] in, For transmitting trigger functions, Indicates transmission, Indicates no transmission; For dynamic thresholds; For a set fixed transmission period; This is the initial threshold; This is a threshold decay rate control parameter; For the remainder operation;

[0070] Next, based on the differences in client data volume and distribution, the aggregation weight is calculated using the following formula:

[0071] ;

[0072] in, Let i be the aggregate weight for client i; , Let i and j be the amount of data for clients. This refers to the set of clients participating in this round of training. This is a parameter for adjusting the distribution difference; JavaScript divergence is used to measure the local dataset. With global dataset The distribution differences; For client i's local dataset;

[0073] Finally, update the global model using the following formula:

[0074] ;

[0075] ;

[0076] in, These are the global model parameters for the (t+1)th round; These are the model parameters trained locally by client i in round t. The residual fusion coefficient; For indicator functions; The initial fusion coefficient; This represents the maximum number of training rounds. This is the fusion coefficient attenuation control parameter; , denoted as the global validation loss for rounds t and 0, respectively.

[0077] Preferably, it further includes: training using an end-to-end alternating training method based on the overall optimization objective to collaboratively optimize the gating network, the expert fusion process, and the local model, and accelerating convergence through an adaptive learning rate strategy;

[0078] The expression for the overall optimization objective is:

[0079] ;

[0080] in, These are the model parameters trained locally on client i; For the gating network parameters of client i; Let i be the total local loss for client i in round t. For mission losses; For client i's local dataset; Losses due to gating network issues; For knowledge distillation loss; For regularization terms; , , These are the loss weighting coefficients; These are the initial parameters for the local model of client i;

[0081] When optimizing a gated network, the local model parameters are fixed, and optimization is performed using a small batch of local data. The expression is as follows:

[0082] ;

[0083] in, , They are respectively the tth, Gating network parameters for client i; The learning rate of the gated network; For the loss of the gated network with respect to parameters The gradient;

[0084] Based on the optimized gating network, expert knowledge is fused through knowledge distillation to initialize the initial parameters of the local model after expert fusion, expressed as:

[0085] ;

[0086] in, The initial parameters of the local model after expert fusion in the t-th round of client i; Knowledge distillation is an operation that integrates expert knowledge. The number of global virtual experts / local virtual experts; Let i be the normalized selection weight of the kth expert for client i; For the parameters of the k-th global / local virtual expert; For client i, the local verification set;

[0087] Using the initial parameters of the local model after expert fusion, train it for E epochs on local data, and update the local model parameters using the following expression:

[0088] ;

[0089] ;

[0090] in, Let be the adaptive learning rate of the local model in round t; This represents the number of epochs used for local training. Let w be the gradient of the task loss. For the training batch of the e-th epoch; These are the model parameters trained locally by client i in round t. Let be the global model parameters for round t; Base learning rate; This is the time decay control parameter; The influence coefficients for the distance between the local and global models; This is the scale parameter for the distance.

[0091] This invention also provides a federated learning device based on dynamic expert selection and knowledge fusion, comprising:

[0092] The virtual expert unit is used to receive the global model of the current round broadcast by the server and decompose it into K virtual expert sets in combination with the old local model of the previous round. Each virtual expert corresponds to a parameter subset.

[0093] The gated network unit is used to generate selection weights for each virtual expert based on local small-batch data through a lightweight gated network.

[0094] The expert fusion unit is used to perform weighted fusion of the soft outputs of the selected virtual experts on the local data through knowledge distillation based on the selection weights, generate fused soft labels, and initialize the local model for this round.

[0095] The training update unit is used to train the local model based on the initialization to obtain the trained local model. At the same time, an adaptive residual compensation mechanism is introduced to upload the adaptive residual and model parameters to the server. The server performs weighted aggregation, updates the global model, and then broadcasts it to the next round.

[0096] The present invention also provides a federated learning device based on dynamic expert selection and knowledge fusion, including a processor and a memory, wherein the memory stores a computer program that can be executed by the processor to implement a federated learning method based on dynamic expert selection and knowledge fusion as described above.

[0097] The present invention also provides a computer-readable storage medium storing computer-readable instructions, which, when executed by a processor of the device on which the computer-readable storage medium resides, implement a federated learning method based on dynamic expert selection and knowledge fusion as described above.

[0098] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0099] This invention uses a virtual expert decomposition mechanism to decouple the global and local models into multiple knowledge subspaces (i.e., virtual experts), enabling fine-grained access to knowledge at different levels within the model.

[0100] This invention dynamically generates expert selection weights by integrating hierarchical attention and low-rank decomposition into a lightweight gating network, thereby matching the knowledge selection process with local data characteristics and task semantic requirements.

[0101] This invention performs soft fusion in the output space through knowledge distillation, avoiding knowledge conflicts caused by hard averaging of the parameter space, while an adaptive temperature mechanism balances exploration and convergence.

[0102] This invention significantly reduces uplink bandwidth usage through differentiated communication based on layer importance and dynamic sparsity.

[0103] This invention, through adaptive residual buffering and aggregated weight design, explicitly models historical update bias and client heterogeneity, ensuring training stability and convergence in a highly Non-IID environment.

[0104] This invention can be deployed on resource-constrained devices in typical privacy-sensitive and highly heterogeneous data distribution application scenarios such as medical image analysis, mobile edge intelligence, and financial risk control, and achieves personalized model accuracy superior to existing methods while maintaining low communication overhead. Attached Figure Description

[0105] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0106] Figure 1 This is a schematic diagram of a federated learning method based on dynamic expert selection and knowledge fusion, provided in Example 1.

[0107] Figure 2 The following is a system architecture flowchart of a federated learning method based on dynamic expert selection and knowledge fusion, as provided in Example 1.

[0108] Figure 3 This is a schematic diagram of a federated learning device based on dynamic expert selection and knowledge fusion, provided in Embodiment 2.

[0109] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Detailed Implementation

[0110] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0111] Example 1

[0112] Embodiment 1 of the present invention provides a federated learning method based on dynamic expert selection and knowledge fusion, which can be implemented by a federated learning device based on dynamic expert selection and knowledge fusion (hereinafter referred to as federated learning device), specifically, executed by one or more processors within the federated learning device.

[0113] In this embodiment, the federated learning device may be an electronic device equipped with a processor, which carries a computer program for the federated learning method based on dynamic expert selection and knowledge fusion, and the computer program can be executed, such as a computer, smartphone, smart tablet, workstation, etc., without limitation.

[0114] This embodiment addresses the statistical heterogeneity problem in federated learning, aiming to design a lightweight and efficient personalized federated learning framework. By deconstructing the global and local models into multiple virtual experts and combining adaptive gating mechanisms and knowledge distillation techniques, it achieves fine-grained knowledge selection and fusion, significantly reducing communication and computational overhead while ensuring model performance.

[0115] like Figures 1-2 As shown, a federated learning method based on dynamic expert selection and knowledge fusion includes steps S1 to S4.

[0116] S1 receives the global model for the current round broadcast by the server and decomposes it into K sets of virtual experts, with each virtual expert corresponding to a subset of parameters.

[0117] At the start of each training round, the client downloads the global model and the retained local old model, and breaks them down into K virtual experts.

[0118] Suppose the neural network of the global model contains Layer, number The layer parameters are set to The global / local model is divided into K virtual experts by layer or parameter block, and an expert set of the global model and the local old model is constructed.

[0119] Specifically, there are two methods:

[0120] Method 1: Divide by "layer" (applicable to the number of layers) (Scenarios ≥ K).

[0121] Neural network The layer parameters are grouped into K virtual experts based on whether they are continuous or discrete. Each expert contains parameters from several complete layers.

[0122] Example: If =16, K=4, then the first to fourth layers are Floors 5-8 are , floors 9 - 12 are , floors 13 - 16 are ;

[0123] Core rule: The parameters of each floor only belong to one virtual expert, without overlap and without omission.

[0124] Method 2: Divide by "parameter block" (applicable to the number of floors <K scenarios).

[0125] The parameters of a single floor are sliced into several parameter blocks according to dimensions, and the parameter blocks of different floors are combined into K virtual experts.

[0126] Example: If the parameters of the th floor , and K = 4, then is sliced into , , , , and the k-th parameter block of each floor is combined into the k-th virtual expert.

[0127] The downloaded global model is decomposed into K virtual experts according to the above division method, and the global virtual expert set is defined as: ; Then, each global virtual expert <00​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​First, calculate the average characteristics of the local data batch:

[0133] ;

[0134] Next, the selection weights are calculated:

[0135] ;

[0136] in, Weights are selected jointly by global and local experts; Indicates client Lightweight gating networks; For the client Gating network parameters; For the client Local data batches; , These are the trainable weight parameters of the gated network; , For the bias of the gated network; For activation functions; This is the normalization function; For batch Average characteristics; Assign a client index number; The encoder for the local model; This is a local data sample; These are the parameters for the local model encoder.

[0137] In a preferred embodiment, the parameters of the lightweight gating network are obtained by training by minimizing the target loss function, so that the gating network selects a small number of key experts and remains stable.

[0138] The expression for the target loss function is:

[0139] ;

[0140] in, Represents the parameters of the gated network Find the minimum value; Let be the objective loss function of the gated network; Indicates the local dataset of client i Data batch Expectations; The task loss function; This is a real label; The output of the model after integrating experts; For sparsity regularization coefficients, This is the smoothness regularization coefficient; For parameters The gradient; Weights are selected jointly by global and local experts; , These are the L1 and L2 norms, respectively.

[0141] In another preferred embodiment, to enhance the gating network's ability to perceive the importance of knowledge at different layers, we design a hierarchical attention mechanism. When generating selection weights for each virtual expert using a lightweight gating network, a hierarchical attention mechanism is introduced. This mechanism divides the lightweight gating network into layers with different semantic abstraction levels, calculates the attention score for each layer, and then applies a hierarchical weighting to the selection weights. The expression is as follows:

[0142] ;

[0143] ;

[0144] in, For the first Attention scores for each layer group; These are the weight parameters for the attention mechanism; , For the attention mechanism, the weight matrix and bias are used. It is the hyperbolic tangent activation function; Let i be the normalized selection weight of the kth expert for client i; The selection weights are the result of hierarchical attention weighting. For expert k's level group The corresponding attention weights.

[0145] The hierarchical attention mechanism enables gating networks to dynamically adjust their attention to knowledge at different levels of abstraction according to task requirements, thereby improving personalized performance.

[0146] Meanwhile, to reduce the number of parameters in the gated network, a low-rank decomposition is performed on the weight matrix of the gated network to further reduce the number of parameters. The expression is as follows:

[0147] ;

[0148] in, Here is the weight matrix of the gated network; This is the left matrix after low-rank decomposition; This is the right matrix after low-rank decomposition; This is the transpose symbol.

[0149] This decomposition operation significantly reduces the number of parameters and decreases computational and storage overhead.

[0150] By using virtual expert decomposition in step S1 and gating network selection in step S2, the local data distribution of different clients is adapted to improve the personalized performance of the model.

[0151] S3. Based on the selection weights, the soft outputs of the selected virtual experts on the local data are weighted and fused through knowledge distillation to generate fused soft labels, and the local model for this round is initialized accordingly.

[0152] Based on the expert weights output by the gating network, this step integrates the knowledge of the selected experts into the local model initialization through knowledge distillation.

[0153] Specifically, the parameters of the global virtual expert and the local virtual expert are first weighted and fused using the joint selection weights output by the gating network to obtain the initial parameters of the local model, expressed as:

[0154] ;

[0155] in, These are the initial parameters for the local model of client i; The number of global virtual experts / local virtual experts; Let i be the normalized selection weight of the kth expert for client i; The parameters are for the k-th global virtual expert; For the first Parameters for a local virtual expert;

[0156] To avoid knowledge conflicts caused by simple parameter averaging, a knowledge distillation mechanism is introduced to calculate a soft label for each expert's output, expressed as follows:

[0157] ;

[0158] in, The soft label for the k-th expert in the local data batch; The output is the logits of the kth expert; This is a temperature parameter used to control the smoothness of the soft label;

[0159] The expert soft labels are weighted according to the joint selection weights to obtain the fused soft labels, expressed as follows:

[0160] ;

[0161] in, To integrate soft tags;

[0162] The local model is initialized by minimizing the KL divergence between the local model output and the fused soft label, and by introducing an adaptive temperature adjustment mechanism for optimization. The expression is as follows:

[0163] ;

[0164] in, The knowledge distillation loss function; For the local data batch of the i-th client Expectations; Let KL divergence be a metric. To initialize the model's logits output; These are the weighting coefficients; This is the cross-entropy loss function.

[0165] Furthermore, in a preferred embodiment, to further enhance the fusion effect, an adaptive temperature parameter adjustment mechanism is introduced to dynamically adjust the temperature parameters. The temperature parameters are dynamically adjusted based on the matching degree between expert knowledge and local data, and their expression is:

[0166] ;

[0167] in, The temperature parameters are for the t-th training round; These are the initial temperature parameters; The attenuation coefficient; For the first Total local losses of the wheel; The local total loss is the initial state. It is an exponential function.

[0168] In the early stages of training, higher temperatures encourage the exploration of diverse expert knowledge; as the model converges, the temperature is lowered to refine the fusion.

[0169] The knowledge distillation mechanism in this step avoids knowledge conflicts caused by parameter averaging, preserves expert soft label information, and improves the quality of the initialization model.

[0170] S4 trains the local model based on the initialization to obtain the trained local model. At the same time, an adaptive residual compensation mechanism is introduced to upload the adaptive residual and model parameters to the server. The server performs weighted aggregation, updates the global model, and then broadcasts it to the next round.

[0171] To reduce communication overhead, this embodiment employs a differentiated communication strategy during local model training. Different communication schemes are used for different layers based on their importance and update frequency.

[0172] Specifically, by calculating the importance scores of the local model's neural network layers and employing Top-k sparsity techniques to set differentiated communication update frequencies for different layers, communication overhead is reduced. The expression is as follows:

[0173] ;

[0174] ;

[0175] ;

[0176] in, For the first Layer importance score; For the local model Layer parameters; for The gradient; For the derivative sign; For the first The dimension of the layer; For the first Layer communication frequency; The highest importance score among all layers; Basic communication frequency; This is a rounding up operation; This is the result of sparsification; For the first The number of parameters retained in the layer; For the first The sparsity of the layer; To select the parameter with the largest absolute value Set one element to 0, and the rest to 0.

[0177] Meanwhile, the sparsity of each layer is dynamically adjusted based on the importance score to balance communication efficiency and model performance, as shown in the formula:

[0178] ;

[0179] in, , These represent the maximum and minimum sparsity values, respectively.

[0180] The importance score in this step takes into account both gradient magnitude and parameter size, prioritizing the transmission of layers that have a greater impact on model performance. High-importance layers are updated every round, while low-importance layers are updated once per round. Furthermore, by differentiating sparsity rates, key layers retain more information, while secondary layers are highly sparsified, balancing communication efficiency and model performance.

[0181] To ensure convergence under highly heterogeneous data distribution, this embodiment introduces an adaptive residual compensation mechanism to upload adaptive residuals and model parameters to the server for weighted aggregation.

[0182] Specifically, first, the model update residual is calculated based on the difference between the parameters trained locally on the client and the initial parameters. The formula is as follows:

[0183] ;

[0184] in, This represents the model update residual for client i in round t; These are the model parameters trained locally by client i in round t. Let be the initial parameters of the local model for client i in round t;

[0185] Then, based on the model-updated residuals combined with an adaptive decay factor, the cumulative value of the historical residuals is updated, as shown in the formula:

[0186] ;

[0187] ;

[0188] in, , They are respectively the tth, The accumulated residual buffer for round client i; It is an adaptive attenuation factor; This is the minimum value of the attenuation factor; The initial decay factor; The decay rate; Let be the local loss gradient of client i in round t; Let be the local loss gradient of client i in round 0; It is an L2 norm; , Let be the local losses of client i in rounds t and 0, respectively.

[0189] In the early stages of training, when the gradient is large, the decay factor is small, and old residuals are quickly forgotten; as convergence occurs, the decay factor increases, retaining more historical information.

[0190] A residual selective transmission strategy is also employed for residual transmission. Transmission is triggered when the residual norm exceeds a dynamic threshold or reaches a fixed period. The formula is as follows:

[0191] ;

[0192] ;

[0193] in, For transmitting trigger functions, Indicates transmission, Indicates no transmission; For dynamic thresholds; For a set fixed transmission period; This is the initial threshold; This is a threshold decay rate control parameter; For the remainder operation.

[0194] In the early stages of training, when the loss is large, the threshold is high, so the transmission frequency is reduced; in the later stages, when the loss decreases, the threshold is lowered, and the transmission frequency is increased to ensure convergence accuracy.

[0195] Next, after receiving the model parameters and residuals uploaded by the client, the server calculates the aggregation weight based on the client's data volume and distribution differences. The formula is as follows:

[0196] ;

[0197] in, Let i be the aggregate weight for client i; , Let i and j be the amount of data for clients. This refers to the set of clients participating in this round of training. This is a parameter for adjusting the distribution difference; JavaScript divergence is used to measure the local dataset. With global dataset The distribution differences; This is the local dataset for client i.

[0198] This weighting design gives higher weight to clients with large amounts of data that are distributed almost globally, thus improving the quality of aggregation.

[0199] Finally, update the global model using the following formula:

[0200] ;

[0201] ;

[0202] in, These are the global model parameters for the (t+1)th round; These are the model parameters trained locally by client i in round t. The residual fusion coefficient; For indicator functions; The initial fusion coefficient; This represents the maximum number of training rounds. This is the fusion coefficient attenuation control parameter; , denoted as the global validation loss for rounds t and 0, respectively.

[0203] In the early stages of training, the residual information is abundant, so it is given a high weight; in the later stages, when the model is close to convergence, the residual weights are reduced to stabilize the training.

[0204] This step employs adaptive residual compensation and selective transmission, uploading only key residual information to reduce the communication load between the client and the server; residual buffering and adaptive decay factors stabilize the aggregation and convergence of the global model in heterogeneous data scenarios.

[0205] In another preferred embodiment, training is performed using an end-to-end alternating training method based on the overall optimization objective to collaboratively optimize the gating network, the expert fusion process, and the local model, and to accelerate convergence through an adaptive learning rate strategy.

[0206] The expression for the overall optimization objective is:

[0207] ;

[0208] in, These are the model parameters trained locally on client i; For the gating network parameters of client i; Let i be the total local loss for client i in round t. For mission losses; For client i's local dataset; Losses due to gating network issues; For knowledge distillation loss; For regularization terms; , , These are the loss weighting coefficients; These are the initial parameters for the local model of client i.

[0209] The entire framework employs a three-stage alternating training strategy to achieve end-to-end collaborative optimization:

[0210] Phase 1: When optimizing the gated network, optimization is performed using small batches of local data with fixed local model parameters. The expression is:

[0211] ;

[0212] in, , They are respectively the tth, Gating network parameters for client i; The learning rate of the gated network; For the loss of the gated network with respect to parameters The gradient;

[0213] Phase 2: Based on the optimized gating network, expert knowledge is fused through knowledge distillation to initialize the initial parameters of the local model after expert fusion. The expression is:

[0214] ;

[0215] in, The initial parameters of the local model after expert fusion in the t-th round of client i; Knowledge distillation is an operation that integrates expert knowledge. The number of global virtual experts / local virtual experts; Let i be the normalized selection weight of the kth expert for client i; For the parameters of the k-th global / local virtual expert; This is the local verification set for client i.

[0216] Phase 3: Using the initial parameters of the local model after expert fusion, train the model for E epochs on the local data, and update the local model parameters. The expression is:

[0217] ;

[0218] ;

[0219] in, Let be the adaptive learning rate of the local model in round t; This represents the number of epochs used for local training. Let w be the gradient of the task loss. For the training batch of the e-th epoch; These are the model parameters trained locally by client i in round t. Let be the global model parameters for round t; Base learning rate; This is the time decay control parameter; The influence coefficients for the distance between the local and global models; This is the scale parameter for the distance.

[0220] When the local model deviates significantly from the global model, the learning rate is increased to facilitate rapid exploration; when the deviation is small, the learning rate is decreased to ensure stable convergence.

[0221] This training process can be deployed on resource-constrained devices in typical privacy-sensitive and highly heterogeneous data distribution application scenarios such as medical image analysis, mobile edge intelligence, and financial risk control, while maintaining low communication overhead and achieving personalized model accuracy superior to existing methods.

[0222] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0223] This invention proposes an efficient personalized federated learning method by integrating five techniques: virtual expert decomposition, adaptive gating selection, knowledge distillation fusion, differentiated communication, and residual compensation. This method significantly reduces communication overhead in heterogeneous environments while maintaining or even improving the personalized performance of the model.

[0224] The core advantages of this invention lie in its fine-grained knowledge utilization capability through virtual expert decomposition, and the ability of gated networks to dynamically capture data heterogeneity characteristics compared to traditional fixed aggregation, enabling more accurate knowledge selection. Simultaneously, the knowledge distillation and fusion mechanism effectively avoids knowledge conflicts, residual compensation ensures long-term convergence remains unaffected, and the lightweight gated network design reduces the number of parameters to only 1-5% of the master model, making it suitable for resource-constrained edge devices. Differentiated communication strategies selectively transmit based on layer importance, reducing communication volume by 60-80%. It is particularly suitable for resource-constrained environments with low bandwidth and high latency, and has broad application prospects in scenarios with significant statistical heterogeneity, such as medical image federated learning, mobile device collaborative training, and edge computing.

[0225] Example 2

[0226] like Figure 3 As shown, the second embodiment of the present invention also provides a federated learning device based on dynamic expert selection and knowledge fusion, comprising:

[0227] The virtual expert unit is used to receive the global model of the current round broadcast by the server and decompose it into K virtual expert sets in combination with the old local model of the previous round. Each virtual expert corresponds to a parameter subset.

[0228] The gated network unit is used to generate selection weights for each virtual expert based on local small-batch data through a lightweight gated network.

[0229] The expert fusion unit is used to perform weighted fusion of the soft outputs of the selected virtual experts on the local data through knowledge distillation based on the selection weights, generate fused soft labels, and initialize the local model for this round.

[0230] The training update unit is used to train the local model based on the initialization to obtain the trained local model. At the same time, an adaptive residual compensation mechanism is introduced to upload the adaptive residual and model parameters to the server. The server performs weighted aggregation, updates the global model, and then broadcasts it to the next round.

[0231] Example 3

[0232] The third embodiment of the present invention also provides a federated learning device based on dynamic expert selection and knowledge fusion, which includes a memory and a processor. The memory stores a computer program that can be executed by the processor to implement the federated learning method based on dynamic expert selection and knowledge fusion as described above.

[0233] Example 4

[0234] The fourth embodiment of the present invention also provides a computer-readable storage medium storing computer-readable instructions, which, when executed by the processor of the device where the computer-readable storage medium is located, implement the federated learning method based on dynamic expert selection and knowledge fusion as described above.

[0235] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A federated learning method based on dynamic expert selection and knowledge fusion, applied to a client, characterized in that, include: S1 receives the global model of the current round broadcast by the server and decomposes it into K sets of virtual experts, with each virtual expert corresponding to a subset of parameters. S2, based on local small-batch data, generates selection weights for each virtual expert through a lightweight gating network; S3, based on the selection weights, the soft outputs of the selected virtual experts on the local data are weighted and fused through knowledge distillation to generate fused soft labels, and the local model for this round is initialized accordingly; S4, based on the initialized local model, trains to obtain the trained local model. At the same time, an adaptive residual compensation mechanism is introduced to upload the adaptive residual and model parameters to the server. The server performs weighted aggregation, updates the global model, and then broadcasts it to the next round. Specifically, an adaptive residual compensation mechanism is introduced to upload adaptive residuals and model parameters to the server for weighted aggregation, as follows: First, calculate the model update residual based on the difference between the parameters of the model trained locally on the client and the initial parameters of the local model. The formula is as follows: ； in, This represents the model update residual for client i in round t; These are the model parameters trained locally by client i in round t. Let be the initial parameters of the local model for client i in round t; Then, based on the model-updated residuals combined with an adaptive decay factor, the cumulative value of the historical residuals is updated, as shown in the formula: ； ； in, , They are respectively the tth, The accumulated residual buffer for client i; It is an adaptive attenuation factor; This is the minimum value of the attenuation factor; This is the initial decay factor; The decay rate; Let be the local loss gradient of client i in round t; Let be the local loss gradient of client i in round 0; It is an L2 norm; , Let be the local losses of client i in rounds t and 0, respectively; A residual selective transmission strategy is adopted for residual transmission, that is, transmission is triggered when the residual norm exceeds a dynamic threshold or reaches a fixed period, as shown in the formula: ； ； in, For transmitting trigger functions, Indicates transmission, Indicates no transmission; For dynamic thresholds; For a set fixed transmission period; This is the initial threshold; This is a threshold decay rate control parameter; For the remainder operation; Next, based on the differences in client data volume and distribution, the aggregation weight is calculated using the following formula: ； in, Let i be the aggregate weight for client i; , Let i and j be the amount of data for clients. This refers to the set of clients participating in this round of training. This is an adjustment parameter for the distribution difference; JavaScript divergence is used to measure the local dataset. With global dataset The distribution differences; For client i's local dataset; Finally, update the global model using the following formula: ； ； in, These are the global model parameters for the (t+1)th round; These are the model parameters trained locally by client i in round t. The residual fusion coefficient; For indicator functions; The initial fusion coefficient; This represents the maximum number of training rounds. This is the fusion coefficient attenuation control parameter; , denoted as the global validation loss for rounds t and 0, respectively.

2. The federated learning method based on dynamic expert selection and knowledge fusion according to claim 1, characterized in that... The previous local model was decomposed into K sets of virtual experts, each corresponding to a subset of parameters, as follows: Suppose the neural network of the global model contains Layer, number The layer parameters are set to ; The K sets of global virtual experts into which the global model is decomposed are represented as follows: ； Each global virtual expert This corresponds to a subset of parameters in the global model, namely: ; in, This indicates that the virtual expert is assigned to the global virtual expert. A set of layer indices that satisfies: And when hour, ; in, Represents the union; , Representing global virtual experts , The set of layer indexes; Represents the empty set; Similarly, the local legacy model of client i is decomposed into K sets of local virtual experts. .

3. The federated learning method based on dynamic expert selection and knowledge fusion according to claim 1, characterized in that... The expression for the selection weight is: ； ； in, Weights are selected jointly by global and local experts; Indicates the client Lightweight gating networks; For the client Gating network parameters; For the client Local data batches; , These are the trainable weight parameters of the gated network; , For the bias of the gated network; For activation functions; This is the normalization function; For batch Average characteristics; Assign a client index number; The encoder for the local model; This is a local data sample; These are the parameters for the local model encoder.

4. The federated learning method based on dynamic expert selection and knowledge fusion according to claim 3, characterized in that... The parameters of the lightweight gating network are obtained by training by minimizing the objective loss function; the expression of the objective loss function is: ； in, Represents the parameters of the gated network Find the minimum value; Let be the objective loss function of the gated network; Indicates the local dataset of client i Data batch Expectations; The task loss function; This is a real label; The output of the model after integrating experts; For sparsity regularization coefficients, This is the smoothness regularization coefficient; For parameters The gradient; Weights are selected jointly by global and local experts; , These are the L1 and L2 norms, respectively.

5. A federated learning method based on dynamic expert selection and knowledge fusion according to claim 4, characterized in that... S3 specifically refers to: First, the parameters of the global virtual expert and the local virtual expert are weighted and fused using the joint selection weights output by the gating network to obtain the initial parameters of the local model. The expression is as follows: ； in, These are the initial parameters for the local model of client i; The number of global virtual experts / local virtual experts; Let i be the normalized selection weight of the kth expert for client i; The parameters are for the k-th global virtual expert; For the first Parameters for a local virtual expert; Then, a knowledge distillation mechanism is introduced to calculate a soft label for each expert's output, expressed as: ； in, The soft label for the k-th expert in the local data batch; The output is the logits of the kth expert; This is a temperature parameter used to control the smoothness of the soft label; The expert soft labels are weighted according to the joint selection weights to obtain the fused soft labels, expressed as follows: ； in, To integrate soft tags; The local model is initialized by minimizing the KL divergence between the local model output and the fused soft label, and by introducing an adaptive temperature adjustment mechanism for optimization. The expression is as follows: ； in, The knowledge distillation loss function; For the local data batch of the i-th client Expectations; Let KL divergence be a metric. To initialize the model's logits output; These are the weighting coefficients; The cross-entropy loss function; In addition, an adaptive temperature parameter adjustment mechanism is introduced to dynamically adjust the temperature parameter, the expression of which is: ； in, The temperature parameters are for the t-th training round; These are the initial temperature parameters; The attenuation coefficient; For the first Total local losses of the wheel; This represents the local total loss for the initial state; It is an exponential function.

6. The federated learning method based on dynamic expert selection and knowledge fusion according to claim 3, characterized in that... When generating the selection weights for each virtual expert using a lightweight gating network, a hierarchical attention mechanism is also introduced. This mechanism divides the lightweight gating network into layers with different semantic abstraction levels, calculates the attention score for each layer, and then applies a hierarchical weighting to the selection weights. The expression is as follows: ； ； in, For the first Attention scores for each layer group; These are the weight parameters for the attention mechanism; , For the attention mechanism, the weight matrix and biases are used. It is the hyperbolic tangent activation function; Let i be the normalized selection weight of the kth expert for client i; The selection weights are the result of hierarchical attention weighting. For expert k's level group The corresponding attention weights; Simultaneously, a low-rank decomposition is performed on the weight matrix of the gated network to reduce the number of parameters in the gated network, expressed as: ； in, Here is the weight matrix of the gated network; This is the left matrix after low-rank decomposition; This is the right matrix after low-rank decomposition; This is the transpose symbol.

7. A federated learning method based on dynamic expert selection and knowledge fusion according to claim 5, characterized in that... It also includes: during local model training, calculating the importance scores of the local model's neural network layers and using Top-k sparsity techniques to set differentiated communication update frequencies for different layers to reduce communication overhead. The expression is as follows: ； ； ； in, For the first Layer importance score; For the local model Layer parameters; for The gradient; For the derivative sign; For the first The dimension of the layer; For the first Layer communication frequency; The highest importance score among all layers; Basic communication frequency; This is a rounding up operation; This is the result of sparsification; For the first The number of parameters retained in the layer; For the first The sparsity of the layer; To select the parameter with the largest absolute value Set one element to 0, and the rest to 0; Meanwhile, the sparsity of each layer is dynamically adjusted based on the importance score to balance communication efficiency and model performance, as shown in the formula: ； in, , These represent the maximum and minimum sparsity values, respectively.

8. The federated learning method based on dynamic expert selection and knowledge fusion according to claim 7, characterized in that... It also includes: training using an end-to-end alternating training method based on the overall optimization objective to collaboratively optimize the gating network, expert fusion process and local model, and accelerating convergence through an adaptive learning rate strategy; The expression for the overall optimization objective is: ； in, These are the model parameters trained locally on client i; For the gating network parameters of client i; Let i be the total local loss for client i in round t. For mission losses; For client i's local dataset; Losses due to gating network issues; For knowledge distillation loss; For regularization terms; , , These are the loss weighting coefficients; These are the initial parameters for the local model of client i; When optimizing a gated network, the local model parameters are fixed, and optimization is performed using a small batch of local data. The expression is as follows: ； in, , They are respectively the tth, Gating network parameters for client i; The learning rate of the gated network; For the loss of the gated network with respect to parameters The gradient; Based on the optimized gating network, expert knowledge is fused through knowledge distillation to initialize the initial parameters of the local model after expert fusion, expressed as: ； in, The initial parameters of the local model after expert fusion in the t-th round of client i; Knowledge distillation is an operation that integrates expert knowledge. The number of global virtual experts / local virtual experts; Let i be the normalized selection weight of the kth expert for client i; For the parameters of the k-th global / local virtual expert; For client i, the local verification set; Using the initial parameters of the local model after expert fusion, train it for E epochs on local data, and update the local model parameters using the following expression: ； ； in, Let be the adaptive learning rate of the local model in round t; This represents the number of epochs used for local training. Let w be the gradient of the task loss. For the training batch of the e-th epoch; These are the model parameters trained locally by client i in round t. Let be the global model parameters for round t; Base learning rate; This is the time decay control parameter; The influence coefficients for the distance between the local and global models; This is the scale parameter for the distance.

9. A federated learning device based on dynamic expert selection and knowledge fusion, used to implement a federated learning method based on dynamic expert selection and knowledge fusion as described in any one of claims 1-8, characterized in that, include: The virtual expert unit is used to receive the global model of the current round broadcast by the server and decompose it into K virtual expert sets in combination with the old local model of the previous round. Each virtual expert corresponds to a parameter subset. The gated network unit is used to generate selection weights for each virtual expert based on local small-batch data through a lightweight gated network. The expert fusion unit is used to perform weighted fusion of the soft outputs of the selected virtual experts on the local data through knowledge distillation based on the selection weights, generate fused soft labels, and initialize the local model for this round. The training update unit is used to train the local model based on the initialization to obtain the trained local model. At the same time, an adaptive residual compensation mechanism is introduced to upload the adaptive residual and model parameters to the server. The server performs weighted aggregation, updates the global model, and then broadcasts it to the next round.

Images