Federal learning dynamic hierarchical aggregation method and system based on convolution kernel importance sampling

Abstract

The invention discloses a federated learning dynamic hierarchical aggregation method and system based on convolution kernel importance sampling, and belongs to the technical field of federated learning. The method is a dynamic layered aggregation method based on convolution kernel importance sampling. The method comprises the following steps: firstly, designing a layer contribution degree evaluation mechanism guided by a super network, dynamically layering and aggregating weights, and gradually mining and utilizing potential similarity relationships among clients; secondly, a dynamic screening method based on threshold judgment is designed, the L1 norm of the convolution kernel weight of each client is used as an importance measurement index, only high-quality clients are allowed to participate in parameter aggregation of a key layer, and the negative influence of low-quality or abnormal clients on a global model is inhibited; according to the method, the problems of model convergence difficulty and generalization performance reduction in a heterogeneous data scene are effectively relieved. Experimental results show that in a Cifar100 image classification task, compared with a reference method, the recognition accuracy of the method is improved by 28.94% at most.

Description

Technical Field

[0001] The present invention belongs to the technical field of federated learning, and specifically relates to a federated learning dynamic hierarchical aggregation method and system based on convolution kernel importance sampling. Background Art

[0002] The popularity of edge devices such as mobile phones and wearable devices in modern society has led to a rapid growth of data. Although abundant data provides huge opportunities for artificial intelligence applications, most of this data is highly sensitive in nature and exists in the form of isolated islands, which is particularly important in the healthcare industry. Because medical data is highly sensitive and is often collected and stored in different healthcare institutions. This situation poses a huge challenge to the adoption of artificial intelligence. How to achieve efficient distributed machine learning while protecting user privacy has become a key challenge. Federated Learning (FL), as a privacy-preserving distributed training paradigm, has received widespread attention in recent years. It allows clients to collaboratively train global models without sharing original data.

[0003] However, the widespread statistical heterogeneity in real-world scenarios severely restricts the practical application of federated learning systems. When users' datasets are non-independently and independently distributed (Non-IID), sharing a global model for all clients can lead to slow convergence or poor inference performance, as the model may deviate significantly from its local data.

[0004] To mitigate the adverse effects of this statistical heterogeneity, researchers proposed a personalized federated learning (PFL) mechanism, which allows each client to train a customized model to adapt to their own data distribution. The current literature on implementing federated learning includes data-based methods, single-model methods, and multi-model methods.

[0005] The core idea of the data-based approach is to optimize the way local client data is processed, generated, or interacted with, intervening at the data level rather than the model or parameter level. The efficiency and performance of federated learning can be improved through data enhancement, data compression, data selection, or data generation. Representative algorithms include the FedAug algorithm and the FedGAN algorithm.

[0006] Commonly used single-model approaches include those based on regularization techniques, meta-learning techniques, and parameter decoupling. Regularization-based approaches balance the global and local models by introducing regularization terms during model training. Their core idea is to constrain the update direction or magnitude of local model parameters to maintain consistency with the global model or adapt to local data characteristics. Representative algorithms include the FedProx and Ditto algorithms. Meta-learning-based approaches are optimization methods that incorporate meta-learning principles. Their core idea is to train a global model through a meta-learning framework, enabling it to quickly adapt to the local data distribution of different clients. Representative algorithms include the Per-FedAvg and FedMeta algorithms. The core idea of parameter decoupling approaches is to separate the parameters of different parts of the model, aggregate some parameters globally on the server, and retain others locally, thus enabling more flexible model training and deployment. Typical methods include the FedPer and FedRep methods.

[0007] The core idea of the multi-model approach is to abandon the assumption of a single global model by simultaneously maintaining, training, or aggregating multiple global or local models, allowing different clients or client groups to use different models to better adapt to local data distribution or task objectives. Typical methods include clustered federated learning and multi-task federated learning. The core idea of the clustered federated learning algorithm is to divide the clients into multiple clusters according to data distribution or task similarity, and train an independent global model for each cluster. Representative algorithms include the IFCA algorithm and the FeSEM algorithm. The core idea of multi-task federated learning is to regard the local training of each client as an independent task, achieve knowledge transfer between tasks by sharing some model parameters, while retaining task-specific parameters, and the server coordinates the joint optimization of the objective functions of multiple tasks. Representative algorithms include the MOCHA algorithm.

[0008] In the non-independent and identically distributed data environment, the existing federated learning methods do not fully consider the differences in contributions of different clients in each layer of the model in model aggregation, and lack an effective client importance screening mechanism, resulting in difficulties in model convergence and decreased generalization performance in heterogeneous data scenarios. Summary of the Invention

[0009] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a dynamic hierarchical aggregation method and system for federated learning based on convolution kernel importance sampling, so as to solve the problem that the prior art does not fully consider the differences in contributions of different clients to each layer of the model, resulting in difficulty in model convergence and decreased generalization performance in heterogeneous data scenarios.

[0010] In order to achieve the above object, the present invention adopts the following technical solutions:

[0011] A dynamic hierarchical aggregation method for federated learning based on convolution kernel importance sampling includes the following steps:

[0012] The client updates the local model parameters based on private data and passes the local model parameters to the server;

[0013] The server configures a dedicated super-network for each client. Based on the local model parameters of each client, each super-network updates the corresponding weight matrix. Based on the updated weight matrix, the server performs optimization and obtains the personalized model parameters of each client based on the optimization results. During the optimization process, the server selects the clients with the highest contribution to aggregate and participate in the optimization.

[0014] The client is updated according to the personalized model parameters.

[0015] A further improvement of the present invention is:

[0016] Preferably, the client trains the model using an SGD optimizer based on private data and updates the model parameters.

[0017] Preferably, the formula of the weight matrix is:

[0018]

[0019] Among them, v i is the embedding vector of client i; is the hypernetwork parameter of client i, the v i and Obtained through local model parameters.

[0020] Preferably, the goals of the server-side optimization are:

[0021]

[0022] Where V={v1,v2,…v N}, ψ={ψ1,ψ2,…ψ N}, v i is the embedding vector of client i; is the hypernetwork parameter of client i; {θ l1 ,θ l2 ,…θ ln} is the intermediate parameter of all clients after local training; L i is the dataset D of client i i The associated loss function; m i is the size of the dataset of client i; M is the sum of the data sizes of all clients.

[0023] Preferably, the personalized model parameters of the client are:

[0024]

[0025] in, and is the aggregated model parameter of client i About the hypernetwork embedding vector v i and its parameter ψ i The gradient, and is the local loss function L i About the hypernetwork embedding vector v i and its parameter ψ i gradient; is the loss function L obtained by client i during local training i Regarding the aggregated model parameters of client i gradient; and is the super network corresponding to client i with respect to the embedding vector v i and its parameter ψ i gradient.

[0026] The client's personalized model parameters are:

[0027]

[0028] The method of selecting clients with the highest contribution to participate in the aggregation optimization is to select clients with the highest contribution to participate in the aggregation based on the convolution kernel L1 norm.

[0029] Preferably, only the convolutional layer of the client participates in the aggregation process, and the fully connected layer parameters are retained locally.

[0030] Preferably, the method of selecting the clients with the largest contribution to participate in the aggregation based on the convolution kernel L1 norm is as follows: for all layers of client i, the aggregation weights are Sort in descending order and select the largest top k layers to retain.

[0031] A dynamic hierarchical aggregation system for federated learning based on convolution kernel importance sampling, including:

[0032] The client local training module is used to update the local model parameters based on private data and pass the local model parameters to the server;

[0033] The server-side supernetwork module is used to configure a dedicated supernetwork for each client on the server side. Based on the local model parameters of each client, each supernetwork updates the corresponding weight matrix. Based on the updated weight matrix, the server side performs optimization and obtains the personalized model parameters of each client based on the optimization results. During the optimization process, the server side selects the clients with the highest contribution to aggregate and participate in the optimization.

[0034] The client parameter update module is used for the client to update the parameters according to the personalized model.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] The present invention discloses a dynamic hierarchical aggregation method for federated learning based on convolution kernel importance sampling, which aims to achieve fine-grained hierarchical aggregation by dynamically evaluating the importance of each layer in the client model. Unlike traditional federated learning methods (such as the FedAvg method) that pursue a single global model, pFedCK adopts a hyper-network-driven personalized weight generation mechanism to maintain independent model parameters for each client on the server side, and dynamically adjusts the aggregation method of the convolution layer and the fully connected layer based on its data distribution characteristics and hierarchical contribution differences. This method effectively alleviates the problems of model convergence difficulties and degraded generalization performance in heterogeneous data scenarios through convolution kernel importance screening and inter-layer differential weighting. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is the pFedCK algorithm process;

[0038] Figure 2 This is a description of the super network framework in the pFedCK algorithm;

[0039] Figure 3 It is the Top-k selection mechanism;

[0040] Figure 4 The distribution of the number of tags of 10 clients under different data heterogeneity, where (a) is the tag distribution when β = 0.1; (b) is the tag distribution when β = 0.5;

[0041] Figure 5 Comparison of the accuracy of each method under pathological conditions;

[0042] Figure 6 Comparison of the accuracy of each method in a practical environment;

[0043] Figure 7 is the aggregate weight of all layers of the target client;

[0044] Figure 8 The performance of each method under different heterogeneous environments. DETAILED DESCRIPTION

[0045] Hereinafter, the terms "first," "second," "third," and "fourth" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the technical features indicated. Thus, a feature identified as "first," "second," "third," or "fourth" may explicitly or implicitly include one or more of such features.

[0046] The co-shooting method provided in the embodiments of the present application can be applied to terminal devices such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs). The embodiments of the present application do not impose any restrictions on the specific types of terminal devices.

[0047] It should be noted that the terms "first," "second," and the like in the description and drawings of the present invention are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatus.

[0048] This paper proposes a dynamic hierarchical aggregation method for federated learning based on convolution kernel importance sampling. This method configures a dedicated hypernetwork for each client on the server side, dynamically adjusting aggregation weights through this hypernetwork to gradually discover similarities between users. Furthermore, the method measures parameter importance using the L1 norm of the convolution kernel, selecting clients that contribute significantly to model performance for aggregation. This method leverages the advantages of high-quality clients and effectively alleviates the difficulties in model convergence and degraded generalization performance in heterogeneous data scenarios.

[0049] The present invention discloses a dynamic hierarchical aggregation method for federated learning based on convolution kernel importance sampling, comprising the following steps:

[0050] S1, the client updates the local model parameters θ based on private data i →Δθi , and pass the local model parameters to the server Δθ i ;

[0051] S2, the server configures a dedicated super network for each client Based on the local model parameters HN of each client i , Δθ i , each super network updates the corresponding weight matrix α i Based on the updated weight matrix, the server performs optimization and obtains personalized model parameters for each client based on the optimization results. During the optimization process, the server selects the clients with the highest contributions to participate in the optimization.

[0052] Compared with traditional federated learning methods, the method of the present invention introduces the pFedCK algorithm (Douglas-Peucker algorithm) in federated learning. It no longer applies the same weights to the neural networks of all layers of the client model. Instead, it fully considers the utility differences of different layers in the neural network and assigns unique weights to each layer, achieving more detailed personalized aggregation. In addition, unlike traditional methods that use distance metrics between the parameters of the entire model to calculate weights, the pFedCK algorithm uses a super network to parameterize the aggregate weights during the training phase. These hierarchical weights are generated by the super network and updated alternately with the personalized model. This ensures that the aggregate weights can effectively reflect the optimization direction and that the update direction of the weights is consistent with the optimization direction of the objective function.

[0053] In some embodiments of the present invention, in S1, each client uses a common SGD optimizer to train a model based on a local private dataset and updates the local model parameters θ i →Δθ i , the client will update the parameter Δθ i Upload to the server. Each client has its own data and will not share data, only local model parameters.

[0054] In some embodiments of the present invention, in S3, the server configures a dedicated super network for each client. Each hypernetwork consists of several fully connected layers, where v i is the embedding vector of client i; φ i is the hypernetwork parameter of client i.

[0055] By HN i , Δθ i Update the aggregation weight matrix α i , α i , stored on the server side, which exploits the similarity between clients, α i The definition is as follows:

[0056]

[0057] Where N is the number of clients; n is the number of layers in the network; α i represents the aggregate weight vector of client i; represents the weight vector of the 1st, 2nd, ..., nth layers of client i; represents the aggregate weight vector of client i at layer n; Indicates the weight contributed by the jth client in the nth layer of client i; the sum of the weights contributed by all clients in the nth layer of client i is 1, that is,

[0058] A super network is a network used to generate parameters or weights of other networks. It is not the main network that directly participates in task training, but it dynamically assigns optimal weights to the main network (or client model) by learning the mapping function. Each super network consists of several fully connected layers, and its input is an embedding vector v that is automatically updated with the model parameters. i , the output is the aggregation weight matrix α i ,like Figure 2 As shown, the super network on client i is defined as follows:

[0059]

[0060] Among them, v i is the embedding vector of client i; is the hypernetwork parameter of client i. l1 ,θ l2 ,…θ ln} is the intermediate parameter after local training of all clients, is the set of parameters of all N clients on the nth layer network. The model parameters of client i are given by α i Aggregation yields:

[0061]

[0062] in It is expressed as follows:

[0063]

[0064] In federated learning, its optimization goal can be expressed as:

[0065]

[0066] in:

[0067]

[0068] Is the collection of personalized parameters of all clients; L i is the dataset D of client i i The associated loss function; L CE Represents the cross entropy loss, which is used to measure the difference between the predicted value and the true label of the data sample.

[0069] The final optimization goal can be expressed by Eq.5 as:

[0070]

[0071] Where V={v1,v2,…v N}, ψ={ψ1,ψ2,…ψ N}. Therefore, the pFedCK algorithm sets the client parameter θ i The optimization problem is converted into the embedding vector v of the hypernetwork i and parameter ψ i optimization problem.

[0072] In some embodiments, v i and ψ i The update process is as follows:

[0073]

[0074] The above update process is based on the local loss function gradient To calculate v i and ψ i The gradient update direction is is the loss function L obtained by client i during local training i About the model parameter θ i gradient; and is the super network corresponding to client i with respect to the embedding vector v i and its parameter ψ i gradient.

[0075] In some implementations, the method uses a more general method to update v i and ψ i , v i and ψ i The update process is as follows:

[0076]

[0077] The above update is based on the model parameter change Δθ i To calculate v i and ψ i The gradient update direction of this modification reduces the computational overhead and does not require additional calculation of the loss gradient Moreover, in the early stage of training or when the data distribution is uneven, the parameter change Δθ i Specific gradient More smooth and stable, where Δθ i is the change in model parameters after local training on client i; v i Used to capture the personalized features of client i; ψ i is the internal parameter of the hypernetwork; hypernetwork HN i It is a network that dynamically generates weights by embedding the vector v i and parameter ψ i Generate personalized weight matrix; It is a super network HN i About the embedding vector v i gradient; It is a super network HN i About its parameter ψ i pFedCK updates the embedding vector and parameters of the super network of client i in each communication round, and then updates the aggregation weight matrix α i .

[0078] In some embodiments of the present invention, in S5, first, the server-side aggregation process is only for the convolution layer of the client, and the fully connected layer parameters are retained locally. Secondly, based on the convolution kernel L1 norm, the client with the largest contribution is selected to participate in the aggregation, thereby generating personalized model parameters.

[0079] Compared with existing works, such as FedBN points out that the BN layer parameters do not participate in aggregation and are calculated locally, and FedRep points out that the classifier layer is learned locally, pFedCK dynamically selects some specific layers to update locally, and the remaining layers are aggregated on the server during training, selecting some layers with Top-k (ATk) aggregation weights For local updates, specifically for all layers of client i according to the aggregate weight Sort in descending order and select the largest top k layers to retain.

[0080]

[0081] Among them, AT k is the selection function, and k is a hyperparameter that can be manually modified before training. The detailed workflow of the Top-k selection mechanism is as follows Figure 3 As shown in Figure 2. A module may contain more than one layer of network, and the light blue rectangle is a module that is only updated locally.

[0082] This method retains layers with larger aggregation weights because layers with higher ranking indices contribute more to model personalization and contain more important personalized information. Therefore, retaining these layers locally, rather than participating in global aggregation, helps improve the effectiveness of the personalized model without significantly impacting the overall performance of the global model. Furthermore, by retaining some layers for local training on the client, the server does not need to transmit the parameters of these layers, which also reduces communication overhead. The server can save the cost of transmitting the parameters of the retained layers.

[0083] The algorithmic flow of the method is as follows: In each round of communication, the client first downloads the latest personalized model from the server, and then uses local SGD to train multiple rounds based on the private dataset. Afterwards, the model update of each client will be uploaded to the server to update the embedding vector v i and parameter ψ i The parameters of each client are aggregated on the server side and only the convolution kernels with larger weights are aggregated. The layers with larger weight vectors are retained. The server uses a hypernetwork to generate personalized model parameters for each client. The communication overhead of this method depends on the Δθ sent from the client. i The size of the data sent from the server Therefore, there is no additional communication cost compared with traditional federated learning methods such as FedAvg.

[0084] In some embodiments of the present invention, in S5, the client downloads the personalized model parameters from the server

[0085] A federated learning dynamic hierarchical aggregation method system based on convolution kernel importance sampling, comprising:

[0086] The client local training module is used to update the local model parameters based on private data and pass the local model parameters to the server;

[0087] The server-side supernetwork module is used to configure a dedicated supernetwork for each client on the server side. Based on the local model parameters of each client, each supernetwork updates the corresponding weight matrix. Based on the updated weight matrix, the server side performs optimization and obtains the personalized model parameters of each client based on the optimization results. During the optimization process, the server side selects the clients with the highest contribution to aggregate and participate in the optimization.

[0088] The client parameter update module is used for the client to update the parameters according to the personalized model.

[0089] It also includes a dynamic hierarchical aggregation module: the server selects high-contributing clients based on the L1 norm of the convolution kernel, performs weighted aggregation only on the convolutional layers, and retains local parameters in the fully connected layers to maintain personalized semantics. This framework achieves data distribution-aware personalized model generation through hierarchical decoupling (global sharing of convolutional layers and local optimization of fully connected layers).

[0090] The following is further described with reference to specific embodiments.

[0091] Example

[0092] In this example, the experiment used three datasets commonly used in federated learning: MNIST, CIFAR-10, and CIFAR-100. A classic four-layer convolutional neural network architecture was used to perform the classification task, with a learning rate of η = 0.05. The MNIST dataset is one of the most commonly used foundational datasets for image classification tasks, primarily used for handwritten digit recognition and serving as a classic introductory example in machine learning. CIFAR-10 is a classic computer vision dataset widely used for supervised learning of image classification tasks. This dataset contains 60,000 32×32 pixel color images, each composed of three channels: red (R), green (G), and blue (B). The dataset is divided into a training set (50,000 images) for model training and a test set (10,000 images) for evaluating model performance. Each image is annotated with a corresponding class label, representing 10 categories, represented by the digits 0-9: airplane, car, bird, cat, deer, dog, frog, horse, boat, and truck. The CIFAR-100 dataset is also mainly used for image classification tasks. It consists of 60,000 32x32 color images. The CIFAR-100 dataset contains 100 categories, with 600 images in each category. The categories are divided into 20 coarse-grained categories (such as "mammals", "vehicles") and 100 fine-grained categories (such as "bears", "whales", "motorcycles", and "buses"). Each image has a "fine" label (that is, the fine-grained category it belongs to) and a "coarse" label (that is, the coarse-grained category it belongs to). Each category has 500 training images and 100 test images.

[0093] This experiment simulates heterogeneous environments in two widely used scenarios, namely pathological environments and practical environments. A pathological environment refers to an environment in which the data is highly unbalanced or skewed. In the present invention, each client randomly extracts only 2 categories of data from a data set containing 10 categories as its local data set. The practical environment refers to a situation that is more in line with the real situation, in which the data is distributed among the clients based on the Dirichlet distribution (denoted as Dir(β)), which is a commonly used technique in federated learning. The size of β can indicate the degree of difference in the data distribution between the clients. A smaller value of β will lead to a significant tilt in the data distribution, so that the system exhibits stronger non-independent and identically distributed characteristics. Figure 4 The stacked bar chart in Figure 1 shows the distribution of the number of tags for 10 clients under different data heterogeneity.

[0094] We use β = 0.1 as the default practical setting here, which means there is some class imbalance, but not as extreme as in pathological settings. Each client receives the proportion of samples from different classes according to the Dirichlet distribution. Then, the data for each client is split into a training set (75%) and a test set (25%).

[0095] Based on the parameter settings of the FedAvg method, this experiment set the local batch size to 10, the number of local learning cycles to 1, the total number of clients to 20, and the client addition ratio ρ = 1. All tasks were run for 500 iterations until all methods converged empirically. This experiment reports the test accuracy of the best global model of the traditional federated learning method and the average test accuracy of the best personalized model of the federated learning method. All experiments were run five times, and the means and standard deviations are reported. The hardware configuration of this experiment is shown in Table 1.

[0096] Table 1 Hardware configuration

[0097] Experimental Configuration Parameter information GPU NVIDIA Tesla T4×4 GPU memory 16GB×4 operating system Ubuntu 20.04LTS (64-bit) Deep Learning Framework PyTorch CUDA version CUDA 11.3

[0098] In order to evaluate the performance advantages of the pFedCK method in a heterogeneous data environment, the method of the present invention selected the current mainstream federated learning methods for comparative experiments, the specific contents of which are as follows.

[0099] While protecting data privacy, the FedAvg method performs multiple rounds of local training on the client and aggregates model parameters on the server, thereby improving communication efficiency.

[0100] The FedProx method introduces a regularization term into the client's optimization objective to constrain the difference between the local model and the global model, thereby reducing the deviation during the update process.

[0101] The Per-FedAvg method combines the optimization idea of meta-learning, enabling the global model to quickly adapt to the personalized needs of each client through a small amount of local adjustments.

[0102] The pFedMe method uses Moreau envelope optimization and regularization constraints to decompose complex optimization problems into sub-problems that can be calculated in parallel, achieving efficient personalized model training.

[0103] The FedAMP method dynamically allocates client aggregation weights through the attention mechanism to promote collaboration between similar clients.

[0104] The Ditto method constrains the consistency between the local personalized model and the global model through regularization, thereby improving the personalization performance while maintaining the model robustness.

[0105] The FedPer method divides the model into a globally shared base layer and a locally retained personalized layer. By aggregating only the base layer parameters, it improves the generalization ability of the global model while retaining client personalization.

[0106] The FedRep method achieves a balance between global commonality and local characteristics by only sharing the parameters of the feature extraction layer and fully privatizing the parameters of the classifier layer.

[0107] like Figure 5 and Figure 6As shown in Table 2, the pFedCK method surpasses most baseline methods on the MNIST, CIFAR10, and CIFAR100 datasets. In pathological settings, pFedCK outperforms the best baseline method on the Cifar100 task by 0.69% and outperforms the traditional federated learning method FedAvg by 27.71%, as shown in Table 2. In the default real-world setting, pFedCK outperforms the best baseline method on the CIFAR100 task by 0.52% and outperforms the traditional federated learning method FedAvg by 27.41%, as shown in Table 3. Traditional federated learning methods, such as FedAvg and FedProx, suffer from poor performance due to the difficulty of adapting global models to heterogeneous client-side local data. Among federated learning methods, the Per-FedAvg method initializes the global model through meta-learning and relies on local fine-tuning on the client to adapt to heterogeneous data. However, its convergence is unstable in severe non-IID data environments. The pFedMe method constrains the difference between the local and global models through regularization, but its dynamic adaptability to heterogeneous data is limited. The FedAMP method is based on attention weight aggregation and is susceptible to interference from malicious clients. The Ditto method balances the personalized model with the global model through a fixed regularization coefficient, but the performance of the personalized model is highly dependent on the quality of the global model. Once the global model degrades due to low-quality or malicious clients, the personalized model will also be affected. The FedPer and FedRep methods fixedly split the model layers, using classifier layer personalization and feature layer aggregation. They cannot dynamically adjust the importance of different layers and are susceptible to parameter interference from malicious clients. Unlike these methods, the pFedCK approach dynamically updates hierarchical aggregation weights through a hypernetwork and uses the L1 norm to select clients with high parameter importance for aggregation. This prevents low-quality or malicious clients from interfering with the global model, improves model robustness, and enables the global model to converge to its optimal state more quickly, thus balancing model universality and personalization requirements during the aggregation process. Experimental results demonstrate that fine-grained personalization based on dynamic aggregation weights and client contribution differences is a key path to improving the effectiveness of federated learning.

[0108] Table 2 Accuracy of image classification tasks under pathological environment (β=0.1)

[0109] method MNIST Cifar10 Cifar100 FedAvg 86.84±0.06 44.11±0.29 21.85±0.47 FedProx 85.63±0.04 46.88±0.31 23.95±0.41 Per-FedAvg 97.88±0.11 65.72±0.21 30.26±0.35 pFedMe 98.50±0.08 66.11±0.32 38.35±0.45 FedAMP 99.14±0.05 78.72±0.19 47.67±0.48 Ditto 99.43±0.04 80.59±0.01 48.87±0.64 FedPer 99.25±0.02 79.22±0.33 45.63±0.54 FedRep 99.01±0.11 80.42±0.22 47.38±0.36 pFedCK 99.31±0.01 80.88±0.03 49.56±0.08

[0110] Table 3 Accuracy of image classification tasks under the default actual environment (β = 0.1)

[0111]

[0112]

[0113] To investigate the impact of retaining different numbers of local layers k in the pFedCK method on the average model accuracy and communication overhead, the following experiments were conducted. The experimental results are shown in Table 4. By applying different values of k, the experiment demonstrates the effect of retaining different numbers of local layers. The experiment uses the MNIST image classification task as an example. The number of clients is set to 10, and all clients participate in the experiment. Figure 7 The aggregate weights of all layers for the target client are shown. These weights are independently generated by the client's hypernetwork. The purpose of the aggregate weights is to measure the relative importance of each layer in the aggregation process, rather than to form a probability distribution. Due to the large weights of the classifier layers, the average accuracy of the model improves when some local layers are retained. This conclusion is also confirmed by the more advanced FedRep method, which shows that removing the classifier layer for aggregation can improve model performance in the environment of non-IID data. In other words, retaining some local layers helps to avoid the transfer of irrelevant knowledge during the aggregation process, thereby improving model performance.

[0114] Table 4 Accuracy and communication overhead under different numbers of local layers

[0115] The number of retained layers k k＝0 k＝1 k＝2 Model accuracy (%) 89.87±0.02 47.14±0.45 25.46±0.12 Communication overhead (MBytes) 99.17 99.36 99.58

[0116] In order to explore the performance of the pFedCK method in different heterogeneous environments, the pFedCK method was compared with other methods in the CIFAR100 image classification task, such as Figure 8 As shown in Table 5, the experiment controls the degree of data heterogeneity by adjusting the β parameter. The lower the β value, the higher the heterogeneity of the data distribution. The experimental results are shown in Table 5. In highly heterogeneous scenarios (i.e., when β is small), the pFedCK method significantly outperforms the baseline method due to its personalized design. However, the performance of the traditional federated learning methods FedAvg and FedProx is limited because the global model has difficulty adapting to the differentiated local data. When data heterogeneity decreases (i.e., when β is large), the advantages of most personalized methods weaken, while the FedAvg method demonstrates stronger generalization ability on Cifar100. When β = 0.5 (low heterogeneity), the per-FedAvg method fails to show a significant advantage over the traditional federated method FedAvg, highlighting the limitations of meta-learning methods in balanced data scenarios. These phenomena indicate that the effectiveness of personalized methods is highly dependent on the degree of data heterogeneity. The pFedCK method, through its hypernetwork dynamic aggregation weight generation mechanism and client screening mechanism, maintains its robustness advantage under different heterogeneity settings.

[0117] Table 5 Accuracy of heterogeneous image classification tasks

[0118]

[0119] To verify the scalability of the pFedCK method, this experiment simulated dynamic changes in the number of clients by randomly selecting subsets (N = 10, 30, and 50) from a fixed set of 50 clients, as shown in Table 6. Randomly selecting subsets ensures representativeness, avoids bias from human selection, and simulates the random availability of clients in real-world scenarios. The experimental settings were client participation ratio ρ = 1, client number N = 50, and a parameter β = 0.1, used to measure the degree of heterogeneity. Validation was performed on the CIFAR100 dataset. When the number of clients is small, the Ditto method uses a fixed regularization coefficient to force the personalized model to align with the global model, preventing the local model from deviating excessively from the global model. Furthermore, the global alignment model provides initialization guidance for the personalized model, enabling faster model convergence in scenarios with a small number of clients. However, due to the insufficient number of clients, this method's L1-norm-based client selection method prevents the global feature layer from fully learning common features, which affects the model's generalization ability and results in inferior performance to the Ditto method when the number of clients is small. As the number of clients increases, the Ditto method uses a fixed regularization value λ. However, this method cannot cope with the complex heterogeneity brought about by the dynamic increase in clients and cannot fully utilize the information of newly added clients, resulting in reduced personalization performance. In contrast, the L1 norm in the pFedCK method tends to be a sparse solution, which can effectively eliminate noise parameters from low-contributing clients and improve model robustness. Therefore, the L1 norm is more reliable in assessing parameter importance and can accurately screen out truly contributing clients. As shown in Table 6, this method outperforms most baseline methods in most cases.

[0120] Table 6 Scalability Experiment

[0121]

[0122] To verify the impact of the convolutional kernel weight screening module and the hypernetwork dynamic weight adjustment module in the pFedCK method, we removed both modules and conducted experimental evaluations on two datasets to explore their importance in improving pFedCK performance. The experimental results, shown in Table 7, show that when only the convolutional kernel weight screening module is included, the accuracy is 98.86%, which is closer to the accuracy of the combined module, indicating that this module contributes most to the performance on the MNIST task. This module effectively reduces noise interference by filtering low-quality client parameters using the L1 norm, which is particularly important for highly homogeneous data. When only the hypernetwork dynamic weight adjustment module is used, the accuracy is 97.14%, slightly lower than the result of using the weight screening module alone. This module exploits user similarity by dynamically aggregating weights, but in simple tasks like MNIST, where the differences between clients are small, the optimization space is limited, resulting in relatively weak gains. In addition, as shown in Table 7, the synergistic effect of the two modules can further optimize the model performance. The weight screening module ensures the quality of the aggregation parameters, and the super network module refines the weight distribution. The two complement each other to improve the stability and accuracy of the model.

[0123]

[0124] This method first designs a hyper-network-guided layer contribution evaluation mechanism, dynamically aggregates weights in layers, and gradually explores and utilizes potential similarity relationships between clients. Secondly, it proposes the L1 norm of convolution kernel parameters as an importance metric and designs a dynamic screening method based on threshold judgment. This method only allows high-quality clients to participate in parameter aggregation in key layers, thereby suppressing the negative impact of low-quality or abnormal clients on the global model. This method effectively alleviates the difficulties of model convergence and degraded generalization performance in heterogeneous data scenarios. Experimental results show that in the Cifar100 image classification task, this method improves recognition accuracy by up to 28.94% compared to the baseline method.

[0125] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0126] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0127] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operations S are performed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide for implementing the process described in the flow. Figure 1 a process or multiple processes and / or boxes Figure 1 S of the function specified in one or more boxes.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

[0129] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A dynamic hierarchical aggregation method for federated learning based on convolution kernel importance sampling, characterized by: The following steps are involved: The client updates the local model parameters based on private data and passes the local model parameters to the server; The server configures a dedicated super-network for each client. Based on the local model parameters of each client, each super-network updates the corresponding weight matrix. Based on the updated weight matrix, the server performs optimization and obtains the personalized model parameters for each client based on the optimization results. During the optimization process of the server, the clients with the highest contributions are selected to participate in the optimization in an aggregated manner; The client is updated according to the personalized model parameters.

2. The method for dynamic hierarchical aggregation of federated learning based on convolution kernel importance sampling according to claim 1, characterized in that: The client trains the model based on private data using the SGD optimizer and updates the model parameters.

3. The method for dynamic hierarchical aggregation of federated learning based on convolution kernel importance sampling according to claim 1, characterized in that: The formula of the weight matrix is: Among them, v i is the embedding vector of client i; is the hypernetwork parameter of client i, the v i and Obtained through local model parameters.

4. The method for dynamic hierarchical aggregation of federated learning based on convolution kernel importance sampling according to claim 1, characterized in that: The goals of the server-side optimization are: Where V={v1,v2,…v N }, ψ={ψ1,ψ2,…ψ N }, v i is the embedding vector of client i; is the hypernetwork parameter of client i; {θ l1 ,θ l2 ,…θ ln } is the intermediate parameter of all clients after local training; L i is the dataset D of client i i The associated loss function; m i is the size of the dataset of client i; M is the sum of the data sizes of all clients.

5. The method for dynamic hierarchical aggregation of federated learning based on convolution kernel importance sampling according to claim 1, characterized in that: The client's personalized model parameters are: in, and is the aggregated model parameter of client i About the hypernetwork embedding vector v i and its parameter ψ i The gradient, and is the local loss function L i About the hypernetwork embedding vector v i and its parameter ψ i gradient; is the loss function L obtained by client i during local training i Regarding the aggregated model parameters of client i gradient; and is the super network corresponding to client i with respect to the embedding vector v i and its parameter ψ i gradient.

6. The method for dynamic hierarchical aggregation of federated learning based on convolution kernel importance sampling according to claim 1, characterized in that: The client's personalized model parameters are: in, and is the aggregated model parameter of client i About the hypernetwork embedding vector v i and its parameter ψ i The gradient, and is the local loss function L i About the hypernetwork embedding vector v i and its parameter ψ i gradient; is the loss function L obtained by client i during local training i Regarding the aggregated model parameters of client i gradient; and is the super network corresponding to client i with respect to the embedding vector v i and its parameter ψ i gradient.

7. The method for dynamic hierarchical aggregation of federated learning based on convolution kernel importance sampling according to claim 1, characterized in that: The method of selecting clients with the highest contribution to participate in the aggregation optimization is to select clients with the highest contribution to participate in the aggregation based on the convolution kernel L1 norm.

8. The method for dynamic hierarchical aggregation of federated learning based on convolution kernel importance sampling according to claim 7, characterized in that: In the aggregation process, only the convolutional layer of the client is involved, and the parameters of the fully connected layer are retained locally.

9. The method for dynamic hierarchical aggregation of federated learning based on convolution kernel importance sampling according to claim 7, characterized in that: The method of selecting the clients with large contributions to participate in the aggregation based on the convolution kernel L1 norm is as follows: for all layers of client i, based on the aggregation weight Sort in descending order and select the largest top k layers to retain.

10. A federated learning dynamic hierarchical aggregation system based on convolution kernel importance sampling, characterized by: include: The client local training module is used to update the local model parameters based on private data and pass the local model parameters to the server; The server-side hypernetwork module is used to configure a dedicated hypernetwork for each client. Based on the local model parameters of each client, each hypernetwork updates the corresponding weight matrix. Based on the updated weight matrix, the server performs optimization and obtains the personalized model parameters for each client based on the optimization results. During the optimization process of the server, the clients with the highest contributions are selected to participate in the optimization in an aggregated manner; The client parameter update module is used for the client to update the parameters according to the personalized model.

Images