A hierarchical adaptive federated learning method, device, and medium

Abstract

This invention relates to a hierarchical adaptive federated learning method, apparatus, and medium. The method includes: a server initializing global model parameters and broadcasting them to a client; a client initializing local model parameters; the client extracting scaling factors for each batch normalization (BN) layer in the local model and calculating hierarchical channel importance scores to generate a personalized channel set and a shared channel set; performing hierarchical routing fusion of local and global parameters based on the personalized and shared channel sets to update the local model parameters; training the local model and applying adaptive constraint losses to the parameters corresponding to the personalized and shared channel sets during training; after training, the client pruning and differentially privacy-processing the updated amount of the shared parameters based on the BN layer scaling factors and uploading it to the server; the server updating the global shared parameters and distributing them to the client; repeating the aforementioned steps for a preset number of rounds to obtain the local model.

Description

Technical Field

[0001] This invention relates to the field of federated learning technology, and more particularly to a hierarchical adaptive federated learning method. Background Technology

[0002] In today's digital age, numerous fields such as the Internet of Things (IoT), autonomous driving, smart healthcare, and industrial monitoring are booming at an unprecedented pace. The amount of data generated in these fields is exploding, and this data is scattered across different devices and nodes. Traditional methods of directly and centrally training models expose several serious problems when faced with this data distribution. On the one hand, there is a risk of privacy breaches, because centralizing all data for training exposes data privacy and security to a huge threat. For example, in the field of smart healthcare, the consequences of leaking sensitive patient medical data would be unimaginable. On the other hand, the communication burden is extremely heavy. Transmitting large amounts of data not only consumes significant network bandwidth resources but may also lead to transmission delays, affecting the efficiency of model training.

[0003] Federated learning has effectively alleviated these problems to some extent. By training the model locally at each participating party and then uploading only the model parameters, it effectively addresses the challenges of privacy protection and data silos. This approach allows participating parties to collaboratively train the model without sharing raw data, significantly improving data security and availability.

[0004] However, federated learning has gradually revealed some new challenges in practical applications. Among these, the issue of significant differences in data distribution among clients (Non-IID) is particularly prominent. In traditional federated learning, the shared global model often struggles to demonstrate good performance across all devices when faced with such non-independent, identically distributed data. This is especially true for high-dimensional heterogeneous tasks such as point cloud processing, medical image analysis, and speech recognition, where the sampling methods, noise levels, and environmental characteristics of each participant vary considerably. For example, in medical image analysis, different hospitals may acquire medical image data with significantly different features due to differences in equipment and shooting environments, further exacerbating the performance differences of the global model across different nodes.

[0005] Furthermore, to further enhance privacy protection, differential privacy techniques have been introduced into federated learning. While this technique does play a significant role in privacy protection, it also introduces a series of new problems. Noise interference is one of them. Since differential privacy protects data privacy by adding noise, this inevitably interferes with the model's training process, affecting its accuracy. Moreover, the presence of noise makes model convergence difficult, increasing the time and resource costs required for training.

[0006] Currently, researchers are primarily exploring two directions to address these issues in federated learning: global model aggregation and personalized federated learning. Global model aggregation aims to leverage data distributed across clients by collaboratively training a general global model by passing only model parameters rather than the original data. This approach attempts to improve the generalization ability of the global model through parameter averaging or adding regularization terms to adapt to different client data distributions. However, in practical applications, when faced with non-independent and identically distributed (Non-IID) data scenarios, the global model still struggles to adequately adapt to the local data distribution of each client, leading to a significant decline in the model's predictive performance for individual clients. Furthermore, this method may introduce bias during the aggregation process, especially when client data distributions differ significantly. In such cases, the convergence speed of the global model may decrease significantly, potentially leading to local optima and severely impacting model accuracy.

[0007] Personalized federated learning methods primarily focus on customizing a higher-performing local personalized model for each client, aiming to address the problem of insufficient adaptability of the global model to individual clients due to data heterogeneity and other factors. However, existing technologies for implementing personalized federated learning mostly employ pre-defined model architectures, such as a fixed division between shared and personalized layers. This approach lacks a dynamic partitioning mechanism based on parameter importance, failing to flexibly adjust the parameter structure according to the actual distribution of client data, thus limiting the model's flexibility and adaptability. Meanwhile, while gradient pruning and noise addition mechanisms introduced to protect privacy enhance security, they can easily interfere with the training process, leading to distorted gradient information or numerical instability, thereby causing difficulties in model convergence. Summary of the Invention

[0008] The technical problem to be solved by the present invention is to provide a hierarchical adaptive federated learning method, apparatus and medium.

[0009] To achieve the above-mentioned objectives, this invention provides a hierarchical adaptive federated learning method, comprising the following steps: S1. The server initializes the global model parameters and broadcasts the initialized global model parameters to each participating client; each client initializes its local model parameters based on the received global model parameters. S2. Before the t-th round of training, the client extracts the scaling factor of each BN layer in the local model, calculates the importance score of the layered channels based on the scaling factor, and generates a personalized channel set and a shared channel set. S3. Based on the personalized channel set and the shared channel set, perform hierarchical routing fusion on the current local model parameters and the received global model parameters to update the local model parameters and form the local model for the t-th round of training; S4. In the t-th round of communication, the client uses the local dataset to train the local model for E cycles, and applies adaptive constraint loss to the parameters corresponding to the personalized channel set and the parameters corresponding to the shared channel set during the training process; S5. After completing local training, the client performs differential privacy processing on the update amount of the parameters corresponding to the shared channel set and uploads it to the server; S6. The server updates the global shared parameters based on the update amount of the parameters corresponding to the shared channel set uploaded by all clients, and broadcasts the updated global shared parameters to each client. S7. Repeat steps S2 to S6 until the preset communication rounds are reached. Each client will eventually obtain a personalized local model for local inference tasks.

[0010] According to one aspect of the present invention, in step S2, where the client extracts the scaling factor of each BN layer in the local model before the t-th round of training, and calculates the hierarchical channel importance score based on the scaling factor, the hierarchical channel importance score for the c-th channel of the k-th BN layer is expressed as: ; in, Indicates the first The client number The first BN layer Importance score of each channel This represents the scaling factor of the corresponding BN layer. This represents the batch-normalized variance of the corresponding BN layer records. To prevent constants with a denominator of zero.

[0011] According to one aspect of the present invention, step S2, the step of generating the personalized channel set and the shared channel set, includes: S21. Perform intra-layer normalization on the channel importance scores within each BN layer, and the normalized channel importance scores are expressed as follows: ; in, This represents the normalized channel importance score. Indicates BN level index, This represents the channel index within the BN layer. Indicates the first The client number The set of importance scores for all channels within a BN layer. This represents all channels within the BN layer; S22. Generate a personalized channel set and a shared channel set based on the quantile thresholds of the normalized channel importance scores for each BN layer. The quantile thresholds are adaptively determined by the client according to the score distribution of the channel importance scores for the current BN layer. The personalized channel set is then represented as: ; in, Indicates the first The client number A set of personalized channels for each BN layer. express Quantile function; The shared channel set is represented as: ; in, Indicates the first The client number A set of shared channels in each BN layer.

[0012] According to one aspect of the present invention, in step S3, the updated local model parameters are expressed as follows: (This refers to the step of performing hierarchical routing fusion on the current local model parameters and the received global model parameters based on the personalized channel set and the shared channel set to update the local model parameters and form the local model for the t-th round of training.) ; in, Indicates an indicator function, Indicates the first Each BN layer has a fusion coefficient, ranging from 0 to 1, which controls the fusion ratio of local parameters to globally shared parameters in the shared channel parameters. No. The client number The first BN layer The local model parameters of each channel in round t-1. This indicates the number of global model parameters broadcast by the server in round t-1. The first BN layer The local model parameters corresponding to each channel Represents global model parameters; If the channel Belongs to the personalized channel collection If the channel... Belongs to the shared channel set Then according to the hierarchical fusion coefficient We perform weighted fusion of local parameters and global parameters broadcast by the server.

[0013] According to one aspect of the invention, in step S4, in the step of applying adaptive constraint loss to the parameters corresponding to the personalized channel set and the parameters corresponding to the shared channel set during training, the loss function used in the training process for the parameters corresponding to the personalized channel set is constructed from the cross-entropy loss and the BN layer scaling factor stabilization term, and the loss function is expressed as: ; in, Indicates the first The cross-entropy loss is calculated by each client based on local data. It is a regularization coefficient that controls the strength of local knowledge retention. This is the set of BN layer scaling factors corresponding to the personalized channel set. The scaling factor of the BN layer corresponding to the current summation term can also be expressed as... , For the client in the first The scaling factor of the BN layer at the start of each round of local training can also be expressed as: ; The loss function used during parameter training for the shared channel set is constructed from the cross-entropy loss and the scaling factor pruning constraint term of the BN layer, and the loss function is expressed as: ; in, To control the regularization coefficient of the loss function constraint strength, This is the set of BN layer scaling factors corresponding to the shared channel set. The scaling factor is the clipping threshold.

[0014] According to one aspect of the present invention, in step S5, the step of the client performing differential privacy processing on the update amount of the parameters corresponding to the shared channel set and uploading it to the server includes: S51. Calculate the update amount of the parameters corresponding to the shared channel set, and express it as: ; in, This indicates the update amount of the parameters corresponding to the shared channel set. Indicates the client After completing the first The parameters corresponding to the shared channel set after local training. Indicates the client In the The parameters corresponding to the shared channel set at the start of local training; S52. The update amount of the corresponding parameters of the shared channel set is pruned according to the BN layer scaling factor, and expressed as: ; in, This represents the parameter update amount corresponding to the shared channel set after pruning based on the BN layer scaling factor. This represents the scaling factor of the corresponding BN layer after the client completes this round of local training. This represents the scaling factor clipping threshold. To prevent constants with a denominator of zero, Represents the Hadamard product; S53. Add Gaussian noise to the cropping result, obtain the update amount of the corresponding parameters of the shared channel set after differential privacy processing, and upload it to the server. The update amount used to upload to the server is represented as: ; in, This indicates the update amount of the parameters corresponding to the shared channel set after differential privacy processing on the client side. Represents the noise multiplier. This indicates that the mean is 0 and the variance is 0. Gaussian random noise.

[0015] According to one aspect of the present invention, in step S6, in the step of updating the global shared parameters based on the update amounts of the parameters corresponding to the shared channel sets uploaded by all clients, the server obtains the update amounts of the parameters corresponding to the shared channel sets uploaded by all clients and performs average aggregation processing, then the updated global shared parameters are expressed as follows: ; in, This indicates the updated globally shared parameters. This indicates the globally shared parameters before the update. This indicates the number of clients participating in this round of aggregation.

[0016] According to one aspect of the invention, the client's local model is a deep learning model trained using an adaptive optimization algorithm.

[0017] To achieve the above-mentioned objectives, the present invention provides a federated learning apparatus for the aforementioned hierarchical adaptive federated learning method, comprising: a server and multiple clients; The server includes: an initialization module, an aggregation module, and a model distribution module; The initialization module is used to initialize global model parameters; The aggregation module is used to update the global shared parameters based on the update amount of the corresponding parameters of the shared channel set uploaded by all clients; The model distribution module is used to broadcast and distribute the initialized global model parameters to each participating client, and to broadcast and distribute the updated global shared parameters to each client. The client includes: a data storage module, a scaling factor evaluation module, a local model parameter update module, a local model training module, and a differential privacy processing module; Data storage module, used to store local datasets; The scaling factor evaluation module is used by the client to extract the scaling factor of each BN layer in the local model before the tth round of training, calculate the importance score of the layered channels based on the scaling factor, and generate personalized channel sets and shared channel sets. The local model parameter update module is used to perform hierarchical routing fusion of the current local model parameters and the received global model parameters based on the personalized channel set and the shared channel set, so as to update the local model parameters to form the local model in the t-th round of training. The local model training module is used by the client to train the local model for E cycles using the local dataset in the t-th round of communication, and to apply adaptive constraint loss to the parameters corresponding to the personalized channel set and the parameters corresponding to the shared channel set during the training process; The differential privacy processing module is used to perform differential privacy processing on the updated parameters of the shared channel set after local training is completed, and then upload the updated parameters to the server.

[0018] To achieve the above-mentioned objective, the present invention provides a computer storage medium storing a computer program that, when executed in a computer, causes the computer to perform the aforementioned hierarchical adaptive federated learning method.

[0019] The technical effects of this invention are as follows: According to one aspect of the present invention, this solution is effectively applicable to various distributed learning scenarios such as image recognition, point cloud analysis, speech modeling, and medical data modeling. It is particularly suitable for multi-device collaborative learning environments with significant differences in data distribution (non-independent and identically distributed, Non-IID) and privacy protection requirements.

[0020] According to one aspect of the present invention, this approach introduces a hierarchical channel selection mechanism based on the BN scaling factor and an adaptive constraint mechanism, enabling different clients to retain local features while sharing global knowledge, thereby achieving efficient and robust personalized federated learning, while also protecting privacy.

[0021] According to one aspect of the present invention, this approach constructs a hierarchical channel importance score through a BN layer scaling factor, and dynamically divides the personalized channel set and the shared channel set accordingly. Differentiated constraint terms are introduced for different types of channels to enhance the robustness of the model in a differential privacy environment. Hierarchical route fusion is used to achieve an adaptive balance between local and global knowledge.

[0022] According to one aspect of the present invention, this solution is effectively applicable to federated learning tasks such as image, speech, and point cloud, while taking into account personalized performance, privacy protection, and convergence speed, thus enabling the solution to have a wider range of application expansion capabilities.

[0023] According to one aspect of the present invention, while maintaining collaborative learning, this approach achieves dynamic adaptation to different data distributions through channel-level dynamic selection and adaptive constraint mechanisms. Personalized local models are trained for different data distributions on the client side, enabling the approach to be implemented in systems with distributed computing nodes, effectively ensuring the collaborativeness and flexibility of the approach.

[0024] According to one aspect of the present invention, the hierarchical channel selection method based on the scaling factor of the BN layer can adaptively identify the key channels of each client, add noise only to the parameters corresponding to the shared channels and perform global sharing, which significantly reduces the impact of privacy protection on model performance. At the same time, the scaling factor pruning mechanism effectively alleviates the convergence difficulty caused by update pruning, and achieves a balance between privacy and utility. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating the steps of the hierarchical adaptive federated learning method of the present invention; Figure 2 This is a flowchart of the hierarchical adaptive federated learning method of the present invention. Detailed Implementation

[0026] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the embodiments will be described in detail below.

[0027] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. The embodiments cannot be described in detail here, but the embodiments of the present invention are not limited to the following embodiments.

[0028] Combination Figure 1 and Figure 2 As shown, according to one embodiment of the present invention, a hierarchical adaptive federated learning method of the present invention includes the following steps: S1. The server initializes the global model parameters and broadcasts the initialized global model parameters to each participating client. Each client initializes its local model parameters based on the received global model parameters. S2. Before the t-th round of training, the client extracts the scaling factor of each BN layer in the local model, calculates the importance score of the layered channels based on the scaling factor, and generates a personalized channel set and a shared channel set. S3. Based on the personalized channel set and the shared channel set, perform hierarchical routing fusion on the current local model parameters and the received global model parameters to update the local model parameters and form the local model for the t-th round of training; S4. In the t-th round of communication, the client uses the local dataset to train the local model for E cycles, and applies adaptive constraint loss to the parameters corresponding to the personalized channel set and the parameters corresponding to the shared channel set during the training process; S5. After completing local training, the client performs differential privacy processing on the update amount of the parameters corresponding to the shared channel set and uploads it to the server; S6. The server updates the global shared parameters based on the update amount of the parameters corresponding to the shared channel set uploaded by all clients, and broadcasts the updated global shared parameters to each client. S7. Repeat steps S2 to S6 until the preset communication rounds are reached. Each client will eventually obtain a personalized local model for local inference tasks.

[0029] According to one embodiment of the present invention, in step S1, the server initializes global model parameters and broadcasts the initialized global model parameters to each participating client; in the step where each client initializes its local model parameters based on the received global model parameters, the server may pre-load a local model for client deployment. In this embodiment, the local model is a deep learning model trained using an adaptive optimization algorithm. Furthermore, the initial learning rate of the local model can be pre-set on the server. The selected optimizer and deep learning model are configured. Furthermore, global model parameters are initialized on the server. Afterwards, each client synchronously downloads the initialized global model parameters. After initializing the local model parameters, it can be represented as: ; in, Indicates the client Initialized local model parameters, This represents the parameters corresponding to the initial shared channel set in the local model. This represents the parameters corresponding to the initial set of personalized channels in the local model.

[0030] According to one embodiment of the present invention, in step S2, the client in the first In the step of extracting the scaling factor of each BN layer in the local model before training and calculating the layer channel importance score based on the scaling factor, the client reads the scaling factor of each BN layer in the local model. and the variance of BN layer records For the first The first BN layer Each channel has its own importance score, and the score for each channel is expressed as follows: ; in, Indicates the first The client number The first BN layer Importance score of each channel This represents the scaling factor for the corresponding BN layer. This represents the batch-normalized variance of the corresponding BN layer records. To prevent constants with a denominator of zero.

[0031] Therefore, based on the BN scaling factor and normalized variance reflecting the channel's ability to regulate the feature distribution, personalized retention and sharing selection are performed at the channel level.

[0032] According to one embodiment of the present invention, step S2, the step of generating the personalized channel set and the shared channel set, includes: S21. Perform intra-layer normalization on the channel importance scores within each BN layer, and the normalized channel importance scores are expressed as follows: ; in, This represents the normalized channel importance score. Indicates BN level index, This represents the channel index within the BN layer. Indicates the first The client number The set of importance scores for all channels within a BN layer. This represents all channels within the BN layer;

[0033] By using the above settings, the impact of differences in scaling factors of different BN layers on the channel selection results can be reduced.

[0034] S22. Generate a personalized channel set and a shared channel set based on the quantile thresholds of the normalized channel importance scores for each BN layer. The quantile thresholds are adaptively determined by the client according to the score distribution of the channel importance scores for the current BN layer. The personalized channel set is then represented as: ; in, Indicates the first The client number A set of personalized channels for each BN layer. express Quantile function; The shared channel set is represented as: ; in, Indicates the first The client number A set of shared channels in each BN layer.

[0035] According to one embodiment of the present invention, in step S3, the updated local model parameters are expressed as follows: (This refers to the step of performing hierarchical routing fusion on the current local model parameters and the received global model parameters based on the personalized channel set and the shared channel set to update the local model parameters and form the local model for the t-th round of training.) ; in, Indicates an indicator function, Indicates the first Each BN layer has a fusion coefficient, ranging from 0 to 1, which controls the fusion ratio of local parameters to globally shared parameters in the shared channel parameters. No. The client number The first BN layer The local model parameters of each channel in round t-1. This indicates the number of global model parameters broadcast by the server in round t-1. The first BN layer The local model parameters corresponding to each channel Represents global model parameters; If the channel Belongs to the personalized channel collection If the channel... Belongs to the shared channel set Then according to the hierarchical fusion coefficient We perform weighted fusion of local parameters and global parameters broadcast by the server.

[0036] The above settings allow channels with higher importance scores to be retained locally, while channels with lower importance scores are synchronized with globally shared parameters, thus achieving flexible and adaptive channel-level parameter partitioning.

[0037] According to one embodiment of the present invention, in step S4, when the client trains the local model for E cycles using the local dataset in the t-th round of communication, E can be stored in the server in a pre-set manner. Further, in step S4, during the training process, in the step of applying adaptive constraint loss to the parameters corresponding to the personalized channel set and the parameters corresponding to the shared channel set, the loss function used during the training process for the parameters corresponding to the personalized channel set is constructed from cross-entropy loss and a BN layer scaling factor stabilization term, aiming to preserve local features and suppress abnormal drift of the BN scaling factor. The loss function is then expressed as: ; in, Indicates the first The cross-entropy loss is calculated by each client based on local data. It is a regularization coefficient that controls the strength of local knowledge retention. This is the set of BN layer scaling factors corresponding to the personalized channel set. The scaling factor of the BN layer corresponding to the current summation term can also be expressed as... , For the client in the first The scaling factor of the BN layer at the start of each round of local training can also be expressed as: ; The loss function used during parameter training for the shared channel set is constructed from the cross-entropy loss and the scaling factor pruning constraint term of the BN layer, and the loss function is expressed as: ; in, To control the regularization coefficient of the loss function constraint strength, This is the set of BN layer scaling factors corresponding to the shared channel set. The scaling factor is the clipping threshold.

[0038] According to one embodiment of the present invention, in step S5, the step of the client performing differential privacy processing on the update amount of the parameters corresponding to the shared channel set and uploading it to the server includes: S51. Calculate the update amount of the parameters corresponding to the shared channel set, and express it as: ; in, This indicates the update amount of the parameters corresponding to the shared channel set. Indicates the client After completing the first The parameters corresponding to the shared channel set after local training. Indicates the client In the The parameters corresponding to the shared channel set at the start of local training; S52. The update amount of the corresponding parameters of the shared channel set is pruned according to the BN layer scaling factor, and expressed as: ; in, This represents the parameter update amount corresponding to the shared channel set after pruning based on the BN layer scaling factor. This represents the scaling factor of the corresponding BN layer after the client completes this round of local training. This represents the scaling factor clipping threshold. To prevent constants with a denominator of zero; in this embodiment, the This refers to the BN layer scaling factor corresponding to each channel in the shared channel set, i.e., the scaling factor of each channel in the shared channel set. Crop threshold determined by scaling factor Scaling factor and constant The determined cropping ratio is based on the number of channels and the update volume. Alignment or broadcasting applies to the corresponding parameter update amount. This represents the Hadamard product.

[0039] S53. Add Gaussian noise to the cropping result, obtain the update amount of the corresponding parameters of the shared channel set after differential privacy processing, and upload it to the server. The update amount used to upload to the server is represented as: ; in, This indicates the update amount of the parameters corresponding to the shared channel set after differential privacy processing on the client side. Represents the noise multiplier. This indicates that the mean is 0 and the variance is 0. Gaussian random noise. Since the update amount of the parameters corresponding to the shared channel set has been pruned according to the BN layer scaling factor, The upper bound of sensitivity after cropping is defined, therefore... As a Gaussian noise scale.

[0040] According to one embodiment of the present invention, in step S6, the step of updating the global shared parameters based on the update amounts of the parameters corresponding to the shared channel set uploaded by all clients, involves the server obtaining the update amounts of the parameters corresponding to the shared channel set uploaded by all clients and performing average aggregation processing, wherein the update amounts received by the server from client uploads are... The updated globally shared parameters are then represented as follows: ; in, This indicates the updated globally shared parameters. This indicates the globally shared parameters before the update. This indicates the number of clients participating in this round of aggregation.

[0041] Therefore, after updating the global model parameters on the server, the next round of training can begin.

[0042] According to one embodiment of the present invention, in step S7, steps S2 to S6 are repeated until a preset number of communication rounds are reached, and each client finally obtains a personalized local model for use in local inference tasks. The preset number of communication rounds is... It can be stored on the server in a preset manner, so that for each client Retain its final personalized local model parameters , This represents the final shared parameters obtained by the i-th client after completing R rounds of communication. This represents the final personalized parameters obtained by the i-th client after completing R rounds of communication.

[0043] Through the above settings, the personalized local model of this solution is iteratively optimized through hierarchical channel selection and adaptive constraint mechanism driven by BN scaling factor. Its parameter set is the result of dynamic evolution, fully integrating globally shared knowledge and retaining local data characteristics. Finally, the personalized local model of the client can be directly deployed on the client's local machine for its own inference tasks.

[0044] According to one embodiment of the present invention, the client's local model is a deep learning model trained using an adaptive optimization algorithm. In this embodiment, the deep learning model uses a ResNet network.

[0045] According to one embodiment of the present invention, a local dataset The datasets used are the FEMNIST and CIFAR-10 datasets. The FEMNIST dataset is a handwritten character dataset commonly used for federated learning, containing 62 categories. The CIFAR-10 dataset is a widely used image classification dataset, containing 10 categories.

[0046] According to one embodiment of the present invention, the present invention provides a federated learning apparatus for the aforementioned hierarchical adaptive federated learning method, comprising: a server and multiple clients; The server includes: an initialization module, an aggregation module, and a model distribution module; The initialization module is used to initialize global model parameters; The aggregation module is used to update the global shared parameters based on the update amount of the corresponding parameters of the shared channel set uploaded by all clients; The model distribution module is used to broadcast and distribute the initialized global model parameters to each participating client, and to broadcast and distribute the updated global shared parameters to each client. The client includes: a data storage module, a scaling factor evaluation module, a local model parameter update module, a local model training module, and a differential privacy processing module; Data storage module, used to store local datasets; The scaling factor evaluation module is used by the client to extract the scaling factor of each BN layer in the local model before the tth round of training, calculate the importance score of the layered channels based on the scaling factor, and generate personalized channel sets and shared channel sets. The local model parameter update module is used to perform hierarchical routing fusion of the current local model parameters and the received global model parameters based on the personalized channel set and the shared channel set, so as to update the local model parameters to form the local model in the t-th round of training. The local model training module is used by the client to train the local model for E cycles using the local dataset in the t-th round of communication, and to apply adaptive constraint loss to the parameters corresponding to the personalized channel set and the parameters corresponding to the shared channel set during the training process; The differential privacy processing module is used to perform differential privacy processing on the updated parameters of the shared channel set after local training is completed, and then upload the updated parameters to the server.

[0047] Specific limitations regarding the federated learning apparatus can be found in the above section on the limitations of the hierarchical adaptive federated learning method, and will not be repeated here. Each module in the aforementioned federated learning apparatus can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0048] In this embodiment, the memory may be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc.

[0049] In this embodiment, the processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0050] According to one embodiment of the present invention, a computer storage medium is provided on which a computer program is stored, which, when executed in a computer, causes the computer to perform the aforementioned hierarchical adaptive federated learning method.

[0051] To further illustrate this plan, an example will be provided.

[0052] Example 1 This example sets up a scenario where multiple parties' data are not independent and identically distributed to verify the model's performance under different levels of privacy protection.

[0053] In this implementation, the number of participating clients is 50, the global communication rounds are 30, the local update count is 3, and the batch size is 16. The initial learning rate is 0.001, the optimizer is Adam, and the deep learning model is a ResNet network. Differential privacy parameters are used. Set to 0.02, privacy budget The quantile parameters of the BN scaling factor are 2, 4, 8, and 16 respectively. Set to 0.6, scaling factor clipping threshold Set to 1.0, regularization coefficient Regularization coefficient .

[0054] The data distribution uses the FEMNIST dataset, which inherently exhibits non-independent and identically distributed characteristics. Each client corresponds to a writer of a handwritten character, resulting in a different data distribution for each client. The training set contains local data from 50 clients, while the test set is stored on the server for testing the global model performance.

[0055] The implementation process is as follows: First, the server initializes the global model parameters and distributes them to all participating clients. Each participating client receives the global model parameters, reads the scaling factors and corresponding variances of each BN layer in its local model, calculates the channel importance score, and performs intra-layer normalization. Based on the quantile threshold of the score distribution for each BN layer, a personalized channel set and a shared channel set are generated. The parameters corresponding to the personalized channels are stored locally, while the parameters corresponding to the shared channels are fused with the global parameters distributed by the server through hierarchical routing. Participants use adaptive constraints for local training, applying different BN scaling factor constraints to the personalized and shared channels respectively. During training, the shared parameter updates are pruned and Gaussian noise is added based on the BN layer scaling factors. Participants upload the updated shared parameters to the server, which performs federated averaging to obtain the updated global model. This process is repeated until the preset number of global communication rounds is reached.

[0056] Example 2 This example sets up different levels of data heterogeneity to verify the effectiveness of the dynamic personalization strategy under Non-IID data.

[0057] In this implementation, the number of participating clients is 10, the global communication rounds are 40, the local update count is 4, and the batch size is 64. The initial learning rate is 0.001, the optimizer is Adam, and a relatively deep CNN network is used as the deep learning model. Differential privacy parameters are used. Set to 0.1, privacy budget The value is 8. Non-IID level parameter. The quantile parameters of the BN scaling factor are taken as 1, 10, and 100 respectively. Set to 0.6, scaling factor clipping threshold Set to 1.0, regularization coefficient Regularization coefficient .

[0058] The data distribution uses the CIFAR-10 dataset, simulating different degrees of Non-IID data partitioning through Dirichlet distribution. Parameters Controlling the degree of data heterogeneity The smaller the value, the higher the degree of Non-IID. The dataset is divided into training and test sets for 10 clients, ensuring that the data distribution for each client has specific skew characteristics.

[0059] The implementation process is as follows: First, CIFAR-10 is partitioned into Non-IID values ​​using a Dirichlet distribution to generate a data distribution for each client. Each client reads the scaling factor of its local model's BN layer and calculates the channel importance score based on the BN layer variance to identify channels that strongly modulate the local feature distribution. A hierarchical channel set is used to protect high-scoring channels from noise; these channels are retained locally as personalized channels. Adaptive constraints are applied to personalized and shared channels respectively. Personalized channels retain local knowledge using a BN layer scaling factor stabilization term, while shared channels are subject to BN layer scaling factor pruning constraints to control shared updates at the channel scale. Privacy protection mechanisms include pruning shared parameter updates based on the BN layer scaling factor, adding Gaussian noise to meet differential privacy requirements, and using Renyi differential privacy for privacy accounting calculations. After each communication round, the performance of each client's personalized model is evaluated on its local test set, and the average accuracy is calculated to measure the effectiveness of the method.

[0060] The above description is merely an example of a specific solution of the present invention. For any devices and structures not described in detail herein, it should be understood that they are implemented using common devices and methods already available in the art.

[0061] The above description is merely one embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the invention by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A hierarchical adaptive federated learning method, characterized in that, Includes the following steps: S1. The server initializes the global model parameters and broadcasts the initialized global model parameters to each participating client; each client initializes its local model parameters based on the received global model parameters. S2. Before the t-th round of training, the client extracts the scaling factor of each BN layer in the local model and calculates the hierarchical channel importance score based on the scaling factor, generating a personalized channel set and a shared channel set; wherein, for the c-th channel of the k-th BN layer, the hierarchical channel importance score is expressed as: in, Indicates the first The client number The first BN layer Importance score of each channel This represents the scaling factor of the corresponding BN layer. This represents the batch-normalized variance of the corresponding BN layer records. To prevent constants with a denominator of zero; S3. Based on the personalized channel set and the shared channel set, perform hierarchical routing fusion on the current local model parameters and the received global model parameters to update the local model parameters and form the local model for the t-th round of training; S4. In the t-th round of communication, the client uses the local dataset to train the local model for E cycles, and applies adaptive constraint loss to the parameters corresponding to the personalized channel set and the parameters corresponding to the shared channel set during the training process; S5. After completing local training, the client performs differential privacy processing on the update amount of the parameters corresponding to the shared channel set and uploads it to the server; S6. The server updates the global shared parameters based on the update amount of the parameters corresponding to the shared channel set uploaded by all clients, and broadcasts the updated global shared parameters to each client. S7. Repeat steps S2 to S6 until the preset communication rounds are reached. Each client will eventually obtain a personalized local model for local inference tasks.

2. The hierarchical adaptive federated learning method according to claim 1, characterized in that, Step S2, which involves generating the personalized channel set and the shared channel set, includes: S21. Perform intra-layer normalization on the channel importance scores within each BN layer, and the normalized channel importance scores are expressed as follows: in, This represents the normalized channel importance score. Indicates BN level index, This represents the channel index within the BN layer. Indicates the first The client number The set of importance scores for all channels within a BN layer. This represents all channels within the BN layer; S22. Generate a personalized channel set and a shared channel set based on the quantile thresholds of the normalized channel importance scores for each BN layer. The quantile thresholds are adaptively determined by the client according to the score distribution of the channel importance scores for the current BN layer. The personalized channel set is then represented as: in, Indicates the first The client number A set of personalized channels for each BN layer. express Quantile function; The shared channel set is represented as: in, Indicates the first The client number A set of shared channels in each BN layer.

3. The hierarchical adaptive federated learning method according to claim 2, characterized in that, In step S3, where hierarchical routing fusion is performed on the current local model parameters and the received global model parameters based on the personalized channel set and the shared channel set to update the local model parameters and form the local model for the t-th round of training, the updated local model parameters are represented as follows: in, Indicates an indicator function, Indicates the first Each BN layer has a fusion coefficient, ranging from 0 to 1, which controls the fusion ratio of local parameters to globally shared parameters in the shared channel parameters. Indicates the first The client number The first BN layer The local model parameters of each channel in round t-1. This represents the local model parameter corresponding to the c-th channel of the k-th BN layer in the global model parameters broadcast by the server in round t-1. Represents global model parameters; If the channel Belongs to the personalized channel collection If the channel... Belongs to the shared channel set Then according to the hierarchical fusion coefficient We perform weighted fusion of local parameters and global parameters broadcast by the server.

4. The hierarchical adaptive federated learning method according to claim 3, characterized in that, In step S4, during the training process, in the step of applying adaptive constraint loss to the parameters corresponding to the personalized channel set and the parameters corresponding to the shared channel set, the loss function used in the training process for the parameters corresponding to the personalized channel set is constructed from the cross-entropy loss and the BN layer scaling factor stabilization term, and the loss function is expressed as: in, This represents the cross-entropy loss calculated by the i-th client based on local data. It is a regularization coefficient that controls the strength of local knowledge retention. This is the set of BN layer scaling factors corresponding to the personalized channel set. The scaling factor of the BN layer corresponding to the current summation term can also be expressed as... , For the client in the first The scaling factor of the BN layer at the start of each round of local training can also be expressed as: ; The loss function used during parameter training for the shared channel set is constructed from the cross-entropy loss and the scaling factor pruning constraint term of the BN layer, and the loss function is expressed as: in, To control the regularization coefficient of the loss function constraint strength, This is the set of BN layer scaling factors corresponding to the shared channel set. The scaling factor is the clipping threshold.

5. The hierarchical adaptive federated learning method according to claim 4, characterized in that, In step S5, the step of the client performing differential privacy processing on the update amount of the parameters corresponding to the shared channel set and uploading it to the server includes: S51. Calculate the update amount of the parameters corresponding to the shared channel set, and express it as: in, This indicates the update amount of the parameters corresponding to the shared channel set. Indicates the client After completing the first The parameters corresponding to the shared channel set after local training. Indicates the client In the The parameters corresponding to the shared channel set at the start of local training; S52. The update amount of the corresponding parameters of the shared channel set is pruned according to the BN layer scaling factor, and expressed as: in, This represents the parameter update amount corresponding to the shared channel set after pruning based on the BN layer scaling factor. This represents the scaling factor of the corresponding BN layer after the client completes this round of local training. This represents the scaling factor clipping threshold. To prevent constants with a denominator of zero, Represents the Hadamard product; S53. Add Gaussian noise to the cropping result, obtain the update amount of the corresponding parameters of the shared channel set after differential privacy processing, and upload it to the server. The update amount used to upload to the server is represented as: in, This indicates the update amount of the parameters corresponding to the shared channel set after differential privacy processing on the client side. Represents the noise multiplier. This indicates that the mean is 0 and the variance is 0. Gaussian random noise.

6. The hierarchical adaptive federated learning method according to claim 5, characterized in that, In step S6, where the server updates the global shared parameters based on the update amounts of the parameters corresponding to the shared channel sets uploaded by all clients, the server obtains the update amounts of the parameters corresponding to the shared channel sets uploaded by all clients and performs average aggregation processing. The updated global shared parameters are then expressed as follows: in, This indicates the updated globally shared parameters. This indicates the globally shared parameters before the update. This indicates the number of clients participating in this round of aggregation.

7. The hierarchical adaptive federated learning method according to claim 1, characterized in that, The client's local model is a deep learning model trained using an adaptive optimization algorithm.

8. A federated learning apparatus for the hierarchical adaptive federated learning method according to any one of claims 1 to 7, characterized in that, include: Server and multiple clients; The server includes: an initialization module, an aggregation module, and a model distribution module; The initialization module is used to initialize global model parameters; The aggregation module is used to update the global shared parameters based on the update amount of the parameters corresponding to the shared channel set uploaded by all clients; The model distribution module is used to broadcast and distribute the initialized global model parameters to each participating client, and to broadcast and distribute the updated global shared parameters to each client. The client includes: a data storage module, a scaling factor evaluation module, a local model parameter update module, a local model training module, and a differential privacy processing module; Data storage module, used to store local datasets; The scaling factor evaluation module is used by the client to extract the scaling factor of each BN layer in the local model before the tth round of training, calculate the importance score of the layered channels based on the scaling factor, and generate personalized channel sets and shared channel sets. The local model parameter update module is used to perform hierarchical routing fusion of the current local model parameters and the received global model parameters based on the personalized channel set and the shared channel set, so as to update the local model parameters to form the local model in the t-th round of training. The local model training module is used by the client to train the local model for E cycles using the local dataset in the t-th round of communication, and to apply adaptive constraint loss to the parameters corresponding to the personalized channel set and the parameters corresponding to the shared channel set during the training process; The differential privacy processing module is used to perform differential privacy processing on the updated parameters of the shared channel set after local training is completed, and then upload the updated parameters to the server.

9. A computer storage medium having a computer program stored thereon, characterized in that, When the computer program is executed in a computer, it causes the computer to perform the hierarchical adaptive federated learning method according to any one of claims 1-7.

Images