Heterogeneous federated learning-based adaptive aggregation system and method

Abstract

The invention discloses an adaptive aggregation system and method based on heterogeneous federated learning, belongs to the technical field of distributed computing, and is applied to heterogeneous transmission of personalized federated learning. The implementation method comprises the following steps: 1, configuring an optimizer and a scheduler for a client split into a base layer and a head layer, and initializing adaptive aggregation parameters of the head layer; 2, training a head layer and a base layer of a client, obtaining update parameters of the head layer and the base layer through a client optimizer, and further obtaining an output prediction value of a local model of the client; 3, calculating probability distribution and information entropy output by the model by using the output predicted value and the global model copy; 4, performing weighted aggregation on the parameters from the clients by the server side, and calculating global model parameters of the server; compared with the prior art, the technical problem that the convergence rate of a local model is reduced due to conflict between a personalized training target of federal learning and a global alignment target is solved.

Description

Technical Field

[0001] The present invention belongs to the field of distributed computing technology and relates to an adaptive aggregation system and method based on heterogeneous federated learning, which is applied to heterogeneous transmission of personalized federated learning. Background Art

[0002] With increasing demands for data privacy and security, federated learning, as a distributed machine learning paradigm, has been widely adopted in scenarios such as finance, healthcare, and smart devices. In practical applications, data distribution varies among participating clients, a phenomenon known as data heterogeneity. This includes diversity in feature space distribution and label distribution. Furthermore, due to differences in terminal device computing power, network conditions, and task scenarios, the deployed model structures may also differ, resulting in model heterogeneity. These two types of heterogeneity significantly impact the training stability and model generalization capabilities of federated learning systems. In particular, during client model updates and server aggregation, conventional, unified global parameter synchronization strategies struggle to fully adapt to diverse local training processes, leading to slow model convergence and unstable performance. In existing practice, most methods still assume consistent model structure, ignoring the complexity of balancing personalization and generalization under heterogeneous conditions. Therefore, building heterogeneous, adaptive federated learning systems that balance individual differences and global coordination has become a research hotspot.

[0003] Currently, existing federated learning has the following defects: on the one hand, the existing model aggregation strategy that relies on static models lacks the ability to dynamically perceive and adapt to differences in tasks or data distribution; on the other hand, the differences in training capabilities, data complexity, and learning rates among existing clients lead to model training distortion; on the other hand, the fine-tuning of local personalized models ignores the impact on global model parameters, resulting in a decline in overall system performance or overfitting of individual models.

[0004] Therefore, in the training of personalized federated learning, how to resolve the conflict between the personalized training goal of existing federated learning and the global alignment goal, which leads to a decrease in the convergence rate of the local model, has become an urgent problem to be solved. Summary of the Invention

[0005] In order to solve the technical problem of how to solve the conflict between the personalized training goal and the global alignment goal of existing federated learning, which leads to a decrease in the convergence rate of the local model, the purpose of the present invention is to propose a heterogeneous data and heterogeneous model optimization method and device based on federated learning. In a resource-constrained distributed computing environment, the present invention preprocesses the heterogeneous training sample set of distributed client nodes to form a data set; inputs the data set into a hierarchical local model for staged training to adapt to the characteristics of heterogeneous data; uses adaptive weight aggregation and information entropy regularization to update the global model, suppresses the adverse effects of heterogeneity, and improves the generalization of the local model.

[0006] The present invention discloses an adaptive aggregation method based on heterogeneous federated learning, which is applied to a federated learning architecture with one server and multiple clients, and includes the following steps:

[0007] Step 1: Configure an independent optimizer and scheduler for the client-side local model split into the base layer and the header layer, and initialize the adaptive aggregation parameters of the client's header layer;

[0008] Step 1.1: Build the local model f corresponding to any k-th client of federated learning k (·;θ k );

[0009] Step 1.2: Set model parameters The representation used to learn the data and task classification head Subsets; among them, Represents the shared base layer parameters of the model; Indicates personalized header layer parameters;

[0010] Step 1.3: The client initializes the local model parameters using the server's global model parameters;

[0011] Step 1.3.1: Get the server's global model parameters

[0012] Step 1.3.2: The client receives the global model parameters from the server and copies them to the client's local model, initializing the client's local model parameters.

[0013] Step 1.4: Configure independent optimizers and schedulers for the base layer and header layer of the client's local model respectively;

[0014] Step 1.4.1: Divide the client's local model into a base layer and a header layer;

[0015] Step 1.4.2: Configure the optimizer and scheduler as shown in formula (1) for the client's base layer;

[0016]

[0017] Among them, the base layer parameters Configuration Optimizer The learning rate is η b , the scheduler is denoted as is the exponential decay rate;

[0018] Step 1.4.3: Configure the optimizer and scheduler as shown in formula (2) for the client's header layer;

[0019]

[0020] Among them, the head layer parameters Configuration Optimizer The learning rate is η h , the scheduler is denoted as

[0021] Step 1.5: Set the initialization parameters of adaptive aggregation for the client's header layer;

[0022] Step 1.5.1: Set the aggregation weight W for the client's head layer to adjust the fusion ratio of the local model and the global model head layer k ={w k,i};

[0023] Step 1.5.2: Set the information entropy adjustment factor λ for the client's header layer to adjust the local model loss calculation MI ;

[0024] Step 1.5.3: Set the learning rate weight update factor η for the client's header layer, which measures the difference between the local model output and the global model output.

[0025] Step 1.5.4: Copy the global model Synchronize to the client's local model;

[0026] Step 2: Train the header layer and base layer of the client's local model, obtain the updated parameters of the client's header layer and base layer through the client optimizer, and then obtain the output prediction value of the client's local model;

[0027] Step 2.1: Set the client local model as the dataset shown in formula (3);

[0028]

[0029] Among them, x i is the input sample, y i For labels;

[0030] Step 2.2: Header layer parameters for the client Training is performed through the client-side header layer optimizer Head layer parameters Make updates;

[0031] Step 2.2.1: Set the number of client head layer training times E p , and the client's base layer parameters Freeze;

[0032] Step 2.2.2: Header layer parameters for the client Use formula (4) to obtain the minimum target loss after training;

[0033]

[0034] in, is the task-related cross entropy loss function;

[0035] Step 2.2.3: Utilize the client-side header optimizer The client's header layer parameters are updated using the method shown in formula (5);

[0036]

[0037] in, Denotes the minimum target loss for training;

[0038] Step 2.2.4: Execute steps 2.2.2 to 2.2.3 in a loop iteration until the number of training times E is reached. p until;

[0039] Step 2.3: Base layer parameters for the client Training is performed through the client-side header layer optimizer Base layer parameters Make updates;

[0040] Step 2.3.1: Set the client's base layer training times E s , and the client's header layer parameters Freeze;

[0041] Step 2.3.2: Base layer parameters for the client Use formula (6) to obtain the minimum target loss after training;

[0042]

[0043] in, is the task-related cross entropy loss function;

[0044] Step 2.3.3: Leverage the client-side base layer optimizer The client's base layer parameters are updated using the method shown in formula (7);

[0045]

[0046] in, Denotes the minimum target loss for training;

[0047] Step 2.3.4: Execute steps 2.3.2 to 2.3.3 in a loop iteration until the number of training times E is reached. s until;

[0048] Step 2.4: Obtain the output prediction value of the client local model as shown in formula (8);

[0049]

[0050] in, represents the base layer feature extraction function, represents the head layer classification function;

[0051] Step 3: Use the local model output prediction value and the global model copy to calculate the probability distribution and information entropy of the model output;

[0052] Step 3.1: Output the predicted value f through the local model k (·;θ k ) and the global model copy f G (·;θ G ) to perform probability calculations and obtain the output probability distributions of the local model and the global model copies as shown in formula (9);

[0053] p k (x i )=softmax(d k (x i θ k )), p G (x i )=softmax(f G (x i θ G )) (9)

[0054] Step 3.2: Use information entropy to obtain the information entropy of the local model and the global model copy as shown in formula (10);

[0055]

[0056] Step 3.3: Evaluate the local model and the global model copy using the information entropy loss shown in Equation (11);

[0057]

[0058] Step 3.4: Using the aggregation weight W k Header layer parameters for the client The weight dimension of is fine-grained and parameter adjustment is performed to form the local model aggregation weight as shown in formula (12);

[0059]

[0060] in, is the information difference gradient in the parameter direction, η is the learning rate of the aggregate weight; clip(·) is used to limit the weight to the interval [0,1];

[0061] Step 3.5: Use the local model aggregation weight to aggregate and update the client's header layer parameters using formula (13);

[0062]

[0063] in, is the header parameter after aggregation;

[0064] Step 4: The server performs weighted aggregation on the local model parameters from each client and calculates the server's global model parameters based on the client data volume and the aggregation weights.

[0065] Step 4.1: The server receives the client set from the current round Local model parameters of The local model parameters include the aggregation update head parameters Base layer parameters Parameters and client data volume n k ;

[0066] Step 4.2: Use formula (14) to obtain the total amount of participating data of the client set in the current round, and calculate the aggregate weight of the client in the current round through the server;

[0067]

[0068] Among them, α k satisfy n k Indicates the amount of data participation of the current client k; N t Indicates the total amount of participation data of the client set;

[0069] Step 4.3: The server calculates the base layer parameters θ b and head layer parameters θ h Make updates;

[0070]

[0071] Among them, the header parameters use the parameters aggregated by the client

[0072] Step 4.4: Use Equation (16) to obtain the global model parameters of the server;

[0073]

[0074] Step 5: Set the global training times to E and use the global model parameters Broadcast to the client and execute steps 1 to 4 in an iterative manner until the global training times reaches E;

[0075] The present invention discloses an adaptive aggregation system based on heterogeneous federated learning, which is used to implement the above method. The present invention discloses an adaptive aggregation system based on heterogeneous federated learning, which includes a client model initialization module, a staged local training module, an information entropy calculation and aggregation adjustment module, a client local model update module, a server aggregation and global model update module, and a model distribution and iteration control module.

[0076] The client model initialization module is used to process the global model parameters sent by the server, including model structure separation, parameter replication and optimizer configuration, and output a structured local model parameter set. and the initial aggregation weight W k ; will serve as input to the staged local training module;

[0077] The staged local training module is used to perform staged training on the initialized local model parameters, including head personalized optimization and base layer shared optimization, and output a set of locally optimized parameters. It will serve as the input of the information entropy calculation and aggregation adjustment module;

[0078] The information entropy calculation and aggregation adjustment module is used to perform information entropy difference analysis on the output probability distribution of the local model and the global model, and adjust the aggregation weight based on the information entropy loss function, and output the adjusted aggregation weight set Will be used as input to the client's local model update module;

[0079] The client local model update module is used to aggregate the local model header parameters, perform differential weighting on the local header parameters and the global header parameters according to the adjusted aggregation weight, and output the aggregated model parameters for reporting to the server. It will serve as the input to the server aggregation and global model update module;

[0080] The server aggregation and global model update module is used to perform weighted aggregation on the local model parameters and data volume uploaded by each client, calculate the global aggregation results of the base layer and the head layer respectively, and output the new round of global model parameters. It will serve as the input of the model delivery and iteration control module;

[0081] The model distribution and iteration control module is used to broadcast the global model aggregated on the server side, control the rounds of federated training, and output the global model parameters and client configuration to be distributed.

[0082] Compared with the existing technology, it has the following beneficial effects:

[0083] 1. The present invention is universal and has a wider range of application scenarios. It does not need to assume that the client model structure is consistent, which reduces deployment complexity.

[0084] 2. The present invention can be applied to more realistic application scenarios. This method generates models that meet real-world constraints. Through layered training and mutual information regularization, it is more in line with the constraints of the communication field and adapts to the distributed computing needs of the real world.

[0085] 3. The present invention also has the advantages of high efficiency and low resource consumption. The FedDCA in the present invention reduces communication and computing overhead by integrating multiple generators and adaptive aggregation mechanisms. Compared with traditional methods, it reduces the time complexity of training, saves system resource consumption, and reduces costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1 Schematic diagram of the system architecture of the present invention;

[0087] Figure 2 It is a schematic diagram of the process of the present invention;

[0088] Figure 3 Schematic diagram of information entropy of the present invention. DETAILED DESCRIPTION

[0089] In order to better illustrate the purpose and advantages of the present invention, the following is a further description of the invention in conjunction with the accompanying drawings and examples. It should be noted that the implementation of the present invention is not limited to the following embodiments, and any form of modification or change made to the present invention will fall within the scope of protection of the present invention.

[0090] The federated learning system provided in this embodiment includes a central server and multiple clients, with the number of clients K=20, and the client devices are heterogeneous terminals such as mobile phones and edge computing devices. The server is responsible for coordinating the federated training process and aggregating and updating global model parameters. The model is divided into two modules: basic shared layer parameters Personalized header parameters The server performs weighted aggregation on these two modules separately, and the updated global parameters are broadcast to all clients.

[0091] Example

[0092] like Figure 1 As shown in FIG, an adaptive aggregation method based on heterogeneous federated learning in this embodiment is specifically implemented in the following steps:

[0093] Step 1: Configure an independent optimizer and scheduler for the client-side local model split into the base layer and the header layer, and initialize the adaptive aggregation parameters of the client's header layer;

[0094] Step 1.1: Build the local model f corresponding to any k-th client of federated learning k (·;θ k );

[0095] Step 1.2: Set model parameters The representation used to learn the data and task classification head Subsets; among them, Represents the shared base layer parameters of the model; Indicates personalized header layer parameters;

[0096] Step 1.3: The client initializes the local model parameters using the server's global model parameters;

[0097] Step 1.3.1: Get the server's global model parameters

[0098] Step 1.3.2: The client receives the global model parameters from the server and copies them to the client's local model, initializing the client's local model parameters.

[0099] Step 1.4: Configure independent optimizers and schedulers for the base layer and header layer of the client's local model respectively;

[0100] Step 1.4.1: Divide the client's local model into a base layer and a header layer;

[0101] Step 1.4.2: Configure the optimizer and scheduler as shown in formula (1) for the client's base layer;

[0102]

[0103] Among them, the base layer parameters Configuration Optimizer The learning rate is η b , the scheduler is denoted as is the exponential decay rate;

[0104] Step 1.4.3: Configure the optimizer and scheduler as shown in formula (2) for the client's header layer;

[0105]

[0106] Among them, the head layer parameters Configuration Optimizer The learning rate is η h , the scheduler is denoted as

[0107] Step 1.5: Set the initialization parameters of adaptive aggregation for the client's header layer;

[0108] Step 1.5.1: Set the aggregation weight W for the client's head layer to adjust the fusion ratio of the local model and the global model head layer k ={w k,i};

[0109] Step 1.5.2: Set the information entropy adjustment factor λ for the client's header layer to adjust the local model loss calculation MI ;

[0110] Step 1.5.3: Set the learning rate weight update factor η for the client's header layer, which measures the difference between the local model output and the global model output.

[0111] Step 1.5.4: Copy the global model Synchronize to the client's local model;

[0112] In an embodiment, Figure 2 As shown, taking client k as an example, local training includes the following steps: First, initialize the local model parameters and head layer parameters And copy them to the corresponding parts of the local model as the initial state of training; secondly, in the local staged optimization, the first stage is the personalized training stage: freeze the basic layer parameters Use local data for header parameters Training, optimization round number is E p =5, learning rate η h =0.01, batch size is 10; Phase 2 is the shared training phase: freeze head parameters Training basic parameters The number of optimization rounds is E s =5, learning rate η b =0.01, batch size 10, stochastic gradient descent is used as the optimizer, and the learning rate exponential decay parameter γ = 0.99; again, dynamic weight adjustment, after the personalized training phase is completed, the client starts the dynamic weight adjustment strategy, the specific implementation process includes: using local data to input the local model head and the global model head respectively; obtaining the output probability distribution, calculating the information entropy difference between the two, that is, the approximate mutual information loss, and setting the mutual information weight coefficient to λ MI =1.0; Use this mutual information loss to update the head aggregation weight matrix W k, update the learning rate η = 1.0. Then, fuse the head parameters and update the updated weight matrix W k , used to calculate the fused client header parameters: After fusion As the final local upload parameters; finally, upload parameters, client upload basic parameters Head parameters after fusion And the amount of local data n k to the server.

[0113] Step 2: Train the header layer and base layer of the client's local model, obtain the updated parameters of the client's header layer and base layer through the client optimizer, and then obtain the output prediction value of the client's local model;

[0114] Step 2.1: Set the client local model as the dataset shown in formula (3);

[0115]

[0116] Among them, x i is the input sample, y i For labels;

[0117] Step 2.2: Header layer parameters for the client Training is performed through the client-side header layer optimizer Head layer parameters Make updates;

[0118] Step 2.2.1: Set the number of client head layer training times E p , and the client's base layer parameters Freeze;

[0119] Step 2.2.2: Header layer parameters for the client Use formula (4) to obtain the minimum target loss after training;

[0120]

[0121] in, is the task-related cross entropy loss function;

[0122] Step 2.2.3: Utilize the client-side header optimizer The client's header layer parameters are updated using the method shown in formula (5);

[0123]

[0124] in, Denotes the minimum target loss for training;

[0125] Step 2.2.4: Execute steps 2.2.2 to 2.2.3 in a loop iteration until the number of training times E is reached. p until;

[0126] Step 2.3: Base layer parameters for the client Training is performed through the client-side header layer optimizer Base layer parameters Make updates;

[0127] Step 2.3.1: Set the client's base layer training times E s , and the client's header layer parameters Freeze;

[0128] Step 2.3.2: Base layer parameters for the client Use formula (6) to obtain the minimum target loss after training;

[0129]

[0130] in, is the task-related cross entropy loss function;

[0131] Step 2.3.3: Leverage the client-side base layer optimizer The client's base layer parameters are updated using the method shown in formula (7);

[0132]

[0133] in, Denotes the minimum target loss for training;

[0134] Step 2.3.4: Execute steps 2.3.2 to 2.3.3 in a loop iteration until the number of training times E is reached. s until;

[0135] Step 2.4: Obtain the output prediction value of the client local model as shown in formula (8);

[0136]

[0137] in, represents the base layer feature extraction function, represents the head layer classification function;

[0138] In an embodiment, Figure 3As shown in the figure, when the mutual information estimation mechanism is implemented on the client side, the head network contains two layers of full connection and the middle ReLU activation function, and the output dimension is set to 128 and the number of categories C respectively. The specific implementation method is: the model output is passed through Softmax to generate a probability vector; the information entropy of the local and global models on the same input is calculated respectively; the information entropy difference That is the approximate mutual information loss

[0139] Step 3: Use the local model output prediction value and the global model copy to calculate the probability distribution and information entropy of the model output;

[0140] Step 3.1: Output the predicted value f through the local model k (·;θ k ) and the global model copy f G (·;θ G ) to perform probability calculations and obtain the output probability distributions of the local model and the global model copies as shown in formula (9);

[0141] p k (x i )=softmax(f k (x i θ k )), p G (x i )=softmax(f G (x i θ G )) (9)

[0142] Step 3.2: Use information entropy to obtain the information entropy of the local model and the global model copy as shown in formula (10);

[0143]

[0144] Step 3.3: Evaluate the local model and the global model copy using the information entropy loss shown in Equation (11);

[0145]

[0146] Step 3.4: Using the aggregation weight W k Header layer parameters for the client The weight dimension of is fine-grained and parameter adjustment is performed to form the local model aggregation weight as shown in formula (12);

[0147]

[0148] in, is the information difference gradient in the parameter direction, η is the aggregation

[0149] The learning rate of the weight; clip(·) is used to limit the weight to the interval [0,1];

[0150] Step 3.5: Use the local model aggregation weight to aggregate and update the client's header layer parameters using formula (13);

[0151]

[0152] in, is the header parameter after aggregation;

[0153] Step 4: The server performs weighted aggregation on the local model parameters from each client and calculates the server's global model parameters based on the client data volume and the aggregation weights.

[0154] Step 4.1: The server receives the client set from the current round Local model parameters of The local model parameters include the aggregation update head parameters Base layer parameters Parameters and client data volume n k ;

[0155] Step 4.2: Use formula (14) to obtain the total amount of participating data of the client set in the current round, and calculate the aggregate weight of the client in the current round through the server;

[0156]

[0157] Among them, α k satisfy n k Indicates the amount of data participation of the current client k; N t Indicates the total amount of participation data of the client set;

[0158] Step 4.3: The server calculates the base layer parameters θ b and head layer parameters θ h Make updates;

[0159]

[0160] Among them, the header parameters use the parameters aggregated by the client

[0161] Step 4.4: Use Equation (16) to obtain the global model parameters of the server;

[0162]

[0163] Step 5: Set the global training times to E and use the global model parameters Broadcast to the client and execute steps 1 to 4 in an iterative manner until the global training times reaches E;

[0164] An adaptive aggregation system based on heterogeneous federated learning in this embodiment is used to implement the above method. An adaptive aggregation system based on heterogeneous federated learning in this embodiment includes a client model initialization module, a staged local training module, an information entropy calculation and aggregation adjustment module, a client local model update module, a server aggregation and global model update module, and a model distribution and iteration control module;

[0165] The client model initialization module is used to process the global model parameters sent by the server, including model structure separation, parameter replication and optimizer configuration, and output a structured local model parameter set. and the initial aggregation weight W k ; will serve as input to the staged local training module;

[0166] The staged local training module is used to perform staged training on the initialized local model parameters, including head personalized optimization and base layer shared optimization, and output a set of locally optimized parameters. It will serve as the input of the information entropy calculation and aggregation adjustment module;

[0167] The information entropy calculation and aggregation adjustment module is used to perform information entropy difference analysis on the output probability distribution of the local model and the global model, and adjust the aggregation weight based on the information entropy loss function, and output the adjusted aggregation weight set Will be used as input to the client's local model update module;

[0168] The client local model update module is used to aggregate the local model header parameters, perform differential weighting on the local header parameters and the global header parameters according to the adjusted aggregation weight, and output the aggregated model parameters for reporting to the server. It will serve as the input to the server aggregation and global model update module;

[0169] The server aggregation and global model update module is used to perform weighted aggregation on the local model parameters and data volume uploaded by each client, calculate the global aggregation results of the base layer and the head layer respectively, and output the new round of global model parameters. It will serve as the input of the model delivery and iteration control module;

[0170] The model distribution and iteration control module is used to broadcast the global model aggregated on the server side, control the rounds of federated training, and output the global model parameters and client configuration to be distributed.

[0171] To further illustrate the practicality of the present invention in heterogeneous transmission of personalized federated learning, the following is an illustration using a simulation method.

[0172] As shown in Table 1, by constructing the following data set and the corresponding comparison algorithm, it can be seen that the accuracy obtained by the FedDCA algorithm in the present invention when performing client training rounds for 1000 times is as follows:

[0173] Table 1 Comparison of the accuracy of various algorithms based on heterogeneous data environments

[0174]

[0175] The results show that the present invention can improve the accuracy of algorithm training results in the federated learning scenario in a heterogeneous data environment.

[0176] The above specific description further illustrates the purpose, technical solutions and beneficial effects of the invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An adaptive aggregation method based on heterogeneous federated learning, characterized by: The following steps are included: Step 1: Configure an independent optimizer and scheduler for the client-side local model split into the base layer and the header layer, and initialize the adaptive aggregation parameters of the client's header layer; Step 2: Train the header layer and base layer of the client's local model, obtain the updated parameters of the client's header layer and base layer through the client optimizer, and then obtain the output prediction value of the client's local model; Step 2.1: Set the client local model as the dataset shown in formula (3); Among them, x i is the input sample, y i For labels; Step 2.2: Header layer parameters for the client Training is performed through the client-side header layer optimizer Head layer parameters Make updates; Step 2.3: Base layer parameters for the client Training is performed through the client-side header layer optimizer Base layer parameters Make updates; Step 2.4: Obtain the output prediction value of the client local model as shown in formula (8); in, represents the base layer feature extraction function, represents the head layer classification function; Step 3: Use the local model output prediction value and the global model copy to calculate the probability distribution and information entropy of the model output; Step 3.1: Output the predicted value f through the local model k (·;θ k ) and the global model copy f G (·;θ G ) to perform probability calculation and obtain the output probability distribution of the local model and the global model copy as shown in formula (9); p k (x i )=softmax(f k (x i ;θ k )),p G (x i )=softmax(f G (x i ;θ G )) (9) Step 3.2: Use information entropy to obtain the information entropy of the local model and the global model copy as shown in formula (10); Step 3.3: Evaluate the local model and the global model copy using the information entropy loss shown in Equation (11); Step 3.4: Using the aggregation weight W k Header layer parameters for the client The weight dimension of is fine-grained and parameter adjustment is performed to form the local model aggregation weight as shown in formula (12); in, is the information difference gradient in the parameter direction, η is the learning rate of the aggregate weight; clip(·) is used to limit the weight to the interval [0,1]; Step 3.5: Use the local model aggregation weight to aggregate and update the client's header layer parameters using formula (13); in, is the header parameter after aggregation; Step 4: The server performs weighted aggregation on the local model parameters from each client and calculates the server's global model parameters based on the client data volume and the aggregation weights. Step 5: Set the global training times to E and use the global model parameters Broadcast to the client and execute steps 1 to 4 in an iterative manner until the global training times reaches E.

2. The adaptive aggregation method based on heterogeneous federated learning according to claim 1, characterized in that: Step 1 is implemented as follows: Step 1.1: Build the local model f corresponding to any k-th client of federated learning k (·;θ k ); Step 1.2: Set model parameters The representation used to learn the data and task classification head Subsets; among them, Represents the shared base layer parameters of the model; Indicates personalized header layer parameters; Step 1.3: The client initializes the local model parameters using the server's global model parameters; Step 1.4: Configure independent optimizers and schedulers for the base layer and header layer of the client's local model respectively; Step 1.5: Set the initialization parameters for adaptive aggregation at the client's header layer.

3. The adaptive aggregation method based on heterogeneous federated learning according to claim 2, characterized in that: Step 1.3 is implemented as follows: Step 1.3.1: Get the server's global model parameters Step 1.3.2: The client receives the global model parameters from the server and copies them to the client's local model, and initializes the client's local model parameters.

4. The adaptive aggregation method based on heterogeneous federated learning according to claim 2, characterized in that: Step 1.4 is implemented as follows: Step 1.4.1: Divide the client's local model into a base layer and a header layer; Step 1.4.2: Configure the optimizer and scheduler as shown in formula (1) for the client's base layer; Among them, the base layer parameters Configuration Optimizer The learning rate is η b , the scheduler is denoted as γ∈(0,1) is the exponential decay rate; Step 1.4.3: Configure the optimizer and scheduler as shown in formula (2) for the client's header layer; Among them, the head layer parameters Configuration Optimizer The learning rate is η h , the scheduler is denoted as 5. The adaptive aggregation method based on heterogeneous federated learning according to claim 2, characterized in that: Step 1.5 is implemented as follows: Step 1.5.1: Set the aggregation weight W for the client's head layer to adjust the fusion ratio of the local model and the global model head layer k ={w k,i }; Step 1.5.2: Set the information entropy adjustment factor λ for the client's header layer to adjust the local model loss calculation MI ; Step 1.5.3: Set the learning rate weight update factor η for the client's header layer, which measures the difference between the local model output and the global model output. Step 1.5.4: Copy the global model Synchronize to the client's local model.

6. The adaptive aggregation method based on heterogeneous federated learning according to claim 1, characterized in that: The implementation method of step 2.2 is: Step 2.2.1: Set the number of client head layer training times E p , and the client's base layer parameters Freeze; Step 2.2.2: Header layer parameters for the client Use formula (4) to obtain the minimum target loss after training; Where l(·,·) is the task-related cross entropy loss function; Step 2.2.3: Utilize the client-side header optimizer The client's header layer parameters are updated using the method shown in formula (5); in, Denotes the minimum target loss for training; Step 2.2.4: Execute steps 2.2.2 to 2.2.3 in a loop iteration until the number of training times E is reached. p until.

7. The adaptive aggregation method based on heterogeneous federated learning according to claim 1, characterized in that: The implementation method of step 2.3 is: Step 2.3.1: Set the client's base layer training times E s , and the client's header layer parameters Freeze; Step 2.3.2: Base layer parameters for the client Use formula (6) to obtain the minimum target loss after training; Where l(·,·) is the task-related cross entropy loss function; Step 2.3.3: Leverage the client-side base layer optimizer The client's base layer parameters are updated using the method shown in formula (7); in, Denotes the minimum target loss for training; Step 2.3.4: Execute steps 2.3.2 to 2.3.3 in a loop iteration until the number of training times E is reached. s until.

8. The adaptive aggregation method based on heterogeneous federated learning according to claim 1, characterized in that: Step 4 is implemented as follows: Step 4.1: The server receives the client set from the current round Local model parameters of The local model parameters include the aggregation update head parameters Base layer parameters Parameters and client data volume n k ; Step 4.2: Use formula (14) to obtain the total amount of participating data of the client set in the current round, and calculate the aggregate weight of the client in the current round through the server; Among them, α k satisfy n k Indicates the amount of data participation of the current client k; N t Indicates the total amount of participation data of the client set; Step 4.3: The server calculates the base layer parameters θ b and head layer parameters θ h Make updates; Among them, the header parameters use the parameters aggregated by the client Step 4.4: Use Equation (16) to obtain the global model parameters of the server.

9. An adaptive aggregation system based on heterogeneous federated learning that implements the method of claim 1, characterized in that: It includes client model initialization module, staged local training module, information entropy calculation and aggregation adjustment module, client local model update module, server aggregation and global model update module, model distribution and iteration control module; The client model initialization module is used to process the global model parameters sent by the server, including model structure separation, parameter replication and optimizer configuration, and output a structured local model parameter set. and the initial aggregation weight W k ; will serve as input to the staged local training module; The staged local training module is used to perform staged training on the initialized local model parameters, including head personalized optimization and base layer shared optimization, and output a set of locally optimized parameters. It will serve as the input of the information entropy calculation and aggregation adjustment module; The information entropy calculation and aggregation adjustment module is used to perform information entropy difference analysis on the output probability distribution of the local model and the global model, and adjust the aggregation weight based on the information entropy loss function, and output the adjusted aggregation weight set Will be used as input to the client's local model update module; The client local model update module is used to aggregate the local model header parameters, perform differential weighting on the local header parameters and the global header parameters according to the adjusted aggregation weight, and output the aggregated model parameters for reporting to the server. It will serve as the input to the server aggregation and global model update module; The server aggregation and global model update module is used to perform weighted aggregation on the local model parameters and data volume uploaded by each client, calculate the global aggregation results of the base layer and the head layer respectively, and output the new round of global model parameters. It will serve as the input of the model delivery and iteration control module; The model distribution and iteration control module is used to broadcast the global model aggregated on the server side, control the rounds of federated training, and output the global model parameters and client configuration to be distributed.

Images