A personalized federated learning method and system for data heterogeneous and resource constrained environments

Abstract

The application provides a personalized federated learning method and system for data heterogeneous and resource limited environment, which is executed by a server and a plurality of clients in cooperation, for protecting client data privacy and collaboratively training a model adapted to local data distribution of each client, comprising the following steps: (1) server initialization and distribution; (2) client personalized model construction and training; (3) client uploading; (4) the server receives the local parameter ranking and the local super network parameter uploaded by each client; the local parameter ranking is aggregated in a discrete manner by using a voting aggregation method, and the global parameter consensus ranking is updated; at the same time, the local super network parameter is aggregated in a continuous manner by using a weighted average method, and the global super network parameter is updated; (5) steps (1) to (4) are repeatedly executed until the model performance meets the preset convergence condition.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of distributed machine learning, and particularly relates to a personalized federated learning method and system for data heterogeneous and resource limited environment. More particularly, it relates to a federated learning method and system for balancing global model generalization and local model personalization by decoupling model learning process and adopting hybrid aggregation strategy. BACKGROUND

[0002] With the popularization of artificial intelligence technology, how to train a model by utilizing the "data silos" distributed in different devices or institutions under the premise of protecting user data privacy has become a key challenge. Federated learning (FL) provides an effective solution for this, which allows multiple participants (clients) to collaboratively train a global model by exchanging model updates without sharing raw data.

[0003] However, traditional federated learning methods, such as federated averaging (FedAvg), face serious challenges in the non-independent and identically distributed (Non-IID) scenarios that are prevalent in the real world. Data heterogeneity can cause significant differences in the direction of local model updates for each client. If direct average aggregation is used, it can cause the "client-drift" problem of the global model, which can slow down the convergence speed of the model and reduce the final performance, and even cannot meet the individual needs of all clients.

[0004] To address the shortcomings of traditional FL in communication efficiency and robustness, federated rank learning (FRL) is proposed as a new paradigm. FRL converts the model updates of clients from continuous floating-point weight space to discrete parameter ranking space, and aggregates them through a voting mechanism. This design significantly reduces communication overhead and enhances the robustness to model poisoning attacks from the architecture. However, the existing FRL framework still has two major limitations: first, the aggregation strategy it uses treats the rankings submitted by each client equally, without considering that in the Non-IID environment, the contribution value of different client data distributions to the global model may differ greatly, affecting the efficiency of collaborative learning. Second, the single global sparse model generated by this mechanism is difficult to adapt to the local data distribution of all clients, resulting in insufficient personalization ability and limited performance in highly heterogeneous scenarios.

[0005] Therefore, there is an urgent need for a new federated learning framework that can inherit the advantages of FRL in communication efficiency and robustness, and effectively solve the personalization problem caused by data heterogeneity, to achieve comprehensive improvement in model performance, fairness, and system efficiency.

[0006] In particular, the above challenges are particularly prominent in Internet of Things (IoT) application scenarios. For example, in a smart home environment, a plurality of sensor devices (clients) such as smart cameras, microphones, etc. are deployed. These devices have varying hardware performance, and are located in different physical locations and have different observation angles, resulting in naturally highly heterogeneous data (such as activity images and sounds of family members) collected by the devices. At the same time, these devices are usually limited in computing resources, and are connected through unstable wireless networks such as Wi-Fi, and are extremely sensitive to communication overhead. Therefore, how to enable these resource-constrained heterogeneous IoT devices to efficiently collaboratively train a precise environment perception or event recognition model (such as fall detection and abnormal intrusion recognition) under the premise of protecting family privacy is a technical problem that needs to be solved in the current cross field of Internet of Things and artificial intelligence. SUMMARY

[0007] In view of the defects and deficiencies of the prior art, the present application provides a personalized federated learning method and system for data heterogeneous and resource-constrained environments, aiming to solve the technical problems that the traditional federated learning (such as federated averaging) is prone to "client drift" in the non-independent and identically distributed (Non-IID) data scenario, the model personalization capability is insufficient, the existing federated ranking learning (FRL) treats each client parameter ranking equally, the single global model is difficult to adapt to heterogeneous data, and the communication and computing overhead is too high in resource-constrained scenarios such as Internet of Things.

[0008] The core design of the present application is to decouple the federated learning model training process into two parallel tasks of global consensus learning and local personalized adaptation, and to realize efficient collaboration by using a hybrid aggregation strategy. On the global side, the server initializes a shared over-parameterized backbone network, a learnable parameter importance score corresponding to the backbone network weight, and a lightweight global hypernetwork, and distributes the global parameter consensus ranking and the global hypernetwork parameter to the clients participating in training in each round of communication; the client generates a mask based on the global parameter consensus ranking, selects a part of the weights ranked in the front to activate the shared sparse feature extractor, which not only inherits the advantage of low communication overhead of FRL, but also adapts the computing capability of resource-constrained devices through sparsification. On the local side, the client inputs the global hypernetwork with the locally independently maintained low-dimensional private embedding vector to dynamically generate a dedicated personalized classification head, and constructs a complete personalized model; during the training process, the local parameter importance score, the local hypernetwork parameter and the private embedding vector are updated synchronously through the gradient descent optimization algorithm, to ensure that the model can adapt to the local data distribution and provide effective contribution to the global consensus, and the private embedding vector is always maintained locally and not uploaded, to protect data privacy.

[0009] The server end adopts a differential aggregation mechanism: performing Borda counting method and other voting aggregations on the local parameter ranking uploaded by the client to update the global parameter consensus ranking and enhance the robustness to model poisoning attacks; performing weighted average aggregation proportional to the local data volume of the client on the local supernetwork parameter uploaded by the client to update the global supernetwork parameter and ensure that the client with more representative data distribution can affect the global model to a greater extent. The system performs the process of "server initialization distribution-client model construction training-client uploading update-server hybrid aggregation" through multiple iterations until the total communication round reaches a preset threshold (usually set to 2000 rounds) or the model performance converges (for example, the performance of the global model on the validation set fluctuates by no more than a preset standard for multiple rounds in succession).

[0010] Experimental verification shows that in the highly heterogeneous data setting based on the CIFAR-10 dataset, the average test accuracy of the method reaches 75.8%, which is significantly higher than 68.2% of federated ranking learning and 65.5% of federated averaging; at the same time, the communication overhead is low, and the data volume uploaded and downloaded per round is about 26.2MB (14MB lower than federated averaging, and only 14.4MB in the extreme sparse scenario), and the number of communication rounds required to converge to the target accuracy is reduced by about 30% compared with federated ranking learning. In addition, the lightweight supernetwork design and sparse feature extractor enable it to adapt to resource-constrained sensor devices such as intelligent cameras and microphones in the Internet of Things scenario, and efficiently solve the collaborative training problem of heterogeneous devices under the premise of protecting data privacy.

[0011] The technical solution adopted by the application to solve the technical problem is:

[0012] A personalized federated learning method for data heterogeneous and resource limited environment, which is executed by a server and multiple clients in cooperation, is used to protect the data privacy of the clients and collaboratively train a model that adapts to the local data distribution of each client, comprising the following steps:

[0013] (1) The server initializes a shared backbone network, parameter importance scores corresponding to the weights of the backbone network, and a global supernetwork; in each communication round, the global parameter consensus ranking and the global supernetwork parameter are distributed to the clients participating in the training;

[0014] (2) The client receives the global parameter consensus ranking and the global supernetwork parameter; generates a mask based on the global parameter consensus ranking to activate the shared sparse feature extractor; at the same time, inputs the locally independently maintained private embedding vector into the global supernetwork to generate a personalized classification head and build a complete personalized model; then trains the personalized model with local data and synchronously updates the local parameter importance scores, the local supernetwork parameter and the private embedding vector through back propagation;

[0015] (3) The client generates a local parameter ranking based on the updated local parameter importance score ranking, and uploads the local parameter ranking and the updated local super network parameter to the server; the private embedding vector is maintained locally by the client and is not uploaded to the server;

[0016] (4) The server receives the local parameter ranking and the local super network parameter uploaded by each client; the local parameter ranking is discretely aggregated by using a voting aggregation method to update the global parameter consensus ranking; at the same time, the local super network parameter is continuously aggregated by using a weighted average method to update the global super network parameter;

[0017] (5) Steps (1) to (4) are repeatedly executed until the model performance meets the preset convergence condition.

[0018] Further, the backbone network is an over-parameterized neural network, and the weights of the backbone network remain fixed after initialization; the parameter importance score is a learnable floating point value, which is used to evaluate the contribution of the corresponding backbone network weight to the model performance, and is the basis for generating the global parameter consensus ranking.

[0019] Further, the generation rule of the mask is: according to the global parameter consensus ranking, a part of weights with high ranking are set to 1, and the rest are set to 0, so as to activate a sparse feature extractor.

[0020] Further, the global super network is a lightweight neural network, the input of which is adapted to the dimension of the private embedding vector, and the size of the output tensor is the same as the total parameter amount of the personalized classification head.

[0021] Further, the voting aggregation method uses Borda counting method, which generates the global importance score of each weight in the backbone network by counting the sequence of the weight in the local parameter ranking of the client, to update the global parameter consensus ranking; in the weighted average method, the weight of each client is proportional to the sample number of the local data set of the client.

[0022] Further, the local data training uses a gradient descent optimization algorithm, which quantifies the model prediction error by using a loss function, and simultaneously calculates the gradient of the local parameter importance score, the local super network parameter and the private embedding vector based on the error, to realize the collaborative optimization of the three.

[0023] Further, the data heterogeneous and resource constrained environment is an Internet of Things scene, and the client is an Internet of Things sensor device, including a smart camera and a microphone.

[0024] Furthermore, the private embedding vector is a low-dimensional vector used to characterize the unique characteristics of the client's local data distribution; the private embedding vectors of different clients are different, and the private embedding vectors of the same client are dynamically updated with local training to ensure that the personalized classification head always adapts to the changes in the client's local data distribution.

[0025] Furthermore, the preset convergence condition is one of the following two cases: first, the total number of communication rounds reaches the first preset threshold; second, the fluctuation range of the global model's performance on the preset validation set does not exceed the second preset threshold after multiple consecutive communication rounds, and reaches the preset performance standard.

[0026] And, a personalized federated learning system for heterogeneous and resource-constrained environments, comprising a server and multiple clients, wherein the server and clients collaborate to protect client data privacy and collaboratively train models adapted to the local data distribution of each client;

[0027] The server includes:

[0028] An initialization module is used to initialize the shared backbone network, the parameter importance scores corresponding one-to-one with the weights of the backbone network, and the global supernetwork;

[0029] The distribution module is used to distribute the current global parameter consensus ranking and global hypernetwork parameters to the clients participating in training in each round of communication.

[0030] The aggregation module is used to receive local parameter rankings and local hypernetwork parameters uploaded by each client, and to perform discrete aggregation of the local parameter rankings using a voting aggregation method to update the global parameter consensus ranking. At the same time, it uses a weighted average method to continuously aggregate the local hypernetwork parameters to update the global hypernetwork parameters.

[0031] The client includes:

[0032] The model building module is used to receive the global parameter consensus ranking and global hypernetwork parameters, generate a mask based on the global parameter consensus ranking to activate the shared sparse feature extractor, and input the locally maintained private embedding vector into the global hypernetwork to generate a personalized classification head and build a complete personalized model.

[0033] The training update module is used to train the personalized model with local data and synchronously update the local parameter importance scores, local hypernetwork parameters, and the private embedding vector through backpropagation.

[0034] The upload module is used to generate a local parameter ranking based on the updated local parameter importance score, upload the local parameter ranking and the updated local hypernetwork parameters to the server, and keep the private embedding vector locally maintained and not uploaded.

[0035] The server and the client further jointly comprise an iteration control module configured to repeatedly execute the "distribution-model construction-training-uploading-aggregation" process until the model performance meets a preset convergence condition.

[0036] Also, an electronic device comprises a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the processor implements the method as described above when executing the program.

[0037] A non-transitory computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the steps of the method as described above.

[0038] Compared with the prior art, the present application and its preferred solutions at least include the following beneficial effects:

[0039] Effectively balances the global model generalization and local model individualization needs, solves the "client drift" problem of traditional federated learning in a heterogeneous data scenario and the defect that the existing federated ranking learning single global model is difficult to adapt to the heterogeneous data of each client. The present application decouples model training into two parallel tasks of global consensus learning and local individualization adaptation. The global side relies on parameter ranking and sparse feature extractor to form a shared model skeleton, ensuring the generalization basis for multi-client collaboration. The local side generates a dedicated classification head with the help of private embedding vectors and super networks, so that each client model can accurately adapt to the data distribution characteristics of itself, avoiding the adaptation limitations of a single global model to heterogeneous data.

[0040] The robustness and collaborative training efficiency of the federated learning system are improved. The server uses a hybrid aggregation strategy to perform voting aggregation (such as Borda counting method) on the parameter ranking uploaded by the client, generates a global parameter ranking through multi-client consensus, naturally enhances the resistance to malicious attacks such as model poisoning, and overcomes the shortcomings of existing federated ranking learning that equally treats the rankings of each client and ignores the differences in data contribution. The super network parameters are aggregated by weighted average aggregation related to the amount of data of the client, so that the client with more representative data distribution can more reasonably influence the global super network, improving the efficiency of collaborative training and the adaptation accuracy of the global model.

[0041] Significantly reduce communication and computing overhead, adapt to the landing needs of resource-constrained scenarios. The global side generates a sparse feature extractor through parameter ranking, only activates part of the network weights with high ranking, greatly reduces the computing load of the client; the client uploads only discrete parameter ranking and lightweight super network parameters, avoiding the high communication cost of uploading complete model weights in traditional federated learning, especially suitable for the actual constraints of limited hardware resources and unstable network bandwidth in Internet of Things and other scenarios, solving the problem of large computing overhead and difficulty in adapting to edge devices in existing super network personalized federated learning methods.

[0042] Strengthen the ability of data privacy protection. The private embedding vector maintained locally by the client as the core information representing its own data distribution is always updated and used locally, and is not uploaded to the server, effectively avoiding the leakage of the client's data distribution characteristics; at the same time, the global model training does not need to collect the original data of each client, continuing the privacy protection advantage of federated learning that "data does not move, model moves", further protecting user data security, and applicable to smart home, medical health and other privacy-sensitive application scenarios.

[0043] Good scene applicability and implementation flexibility. The technical solution of the present application is not bound to a specific model architecture, and the specific implementation of the backbone network and the super network can be adjusted according to actual needs, and the key links such as sparsification degree and training optimization strategy support flexible configuration; especially for Internet of Things scenarios, it can adapt to various sensor devices such as smart cameras and microphones, and efficiently solve the collaborative training problem of heterogeneous devices under the premise of protecting privacy, providing a feasible path for the large-scale application of federated learning in resource-constrained and data-heterogeneous scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0044] The present application will be further described in detail below in combination with the drawings and specific embodiments:

[0045] Figure 1 The method flowchart of the embodiment of the present application;

[0046] Figure 2 The interaction process diagram between the server and the client in one communication round of the method of the embodiment of the present application;

[0047] Figure 3 The schematic diagram of the core mechanism of the embodiment of the present application;

[0048] Figure 4 The system structure diagram of the embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the features and advantages of the present application more obvious and easy to understand, the following embodiments are described in detail as follows:

[0050] It should be noted that the following detailed description is exemplary in nature and is intended to provide further description of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0051] It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments in accordance with the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, steps, operations, elements, components, and / or groups thereof, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0052] To address the above challenges and technical problems, the present application provides a ranking and super network based personalized federated learning system and method, in particular to a personalized federated learning method in a data heterogeneous environment. More specifically, it relates to a federated learning system and method that decouples the model learning process and adopts a hybrid aggregation strategy to balance the global model generalization and local model personalization, to solve the problems of insufficient personalization ability and limited learning efficiency of existing federated ranking learning (FRL) framework in a data heterogeneous environment; and the problem of large computational overhead and insufficient robustness of existing super network personalized federated learning method; provides a distributed learning framework that can efficiently balance the global generalization and local personalization needs of the model.

[0053] The scheme aims to solve the challenge that the existing framework of federated learning in a data heterogeneous environment, especially in an Internet of Things environment, is difficult to balance the global model generalization, communication efficiency and client personalization needs. The core of the scheme is to decouple the model learning process into two parts of global sharing and local personalization, and adopt a hybrid aggregation strategy. For the global shared model skeleton, the system follows the federated ranking learning paradigm, and the clients only exchange discrete ranking information of the parameters, and the server aggregates the rankings by majority voting mechanism, thereby efficiently constructing a global sparse subnetwork. At the same time, for the personalized module, the system introduces a lightweight super network co-trained by all clients, which can dynamically generate exclusive model layer parameters for each client according to its local private embedding vector, to accurately adapt to its local data distribution. The present application overcomes the performance bottleneck of existing technologies in a heterogeneous data environment by combining robust global consensus construction and flexible local model generation, significantly improves the model personalization ability while maintaining low communication overhead and high convergence efficiency.

[0054] The main implementation of the scheme can refer to the following steps:

[0055] S101, system initialization and distribution: the server initializes a shared backbone network, a set of scores for global ranking, and a global hypernetwork; the server distributes a global consensus ranking and global hypernetwork parameters to multiple clients participating in training in each round;

[0056] S102, client model reconstruction and individualization generation: the client receives the global consensus ranking and global hypernetwork parameters; based on the global consensus ranking, a shared sparse feature extractor is reconstructed; using a locally maintained private embedding vector as input, a personalized classification head parameter is dynamically generated through the hypernetwork defined by the global hypernetwork parameters;

[0057] S103, client local joint optimization: the client trains the personalized model composed of the sparse feature extractor and the personalized classification head on its local data; during training, the scores for generating local rankings, local hypernetwork parameters, and private embedding vectors are simultaneously updated through backpropagation;

[0058] S104, client update packaging and uploading: after local training is completed, the client generates a local parameter ranking according to the updated local scores; the client uploads the local parameter ranking and the updated local hypernetwork parameters to the server;

[0059] S105, server-side hybrid aggregation: the server receives local parameter rankings and local hypernetwork parameters from multiple clients; performs voting aggregation on multiple local parameter rankings to update the global consensus ranking; at the same time, performs weighted average aggregation on multiple local hypernetwork parameters to update the global hypernetwork parameters;

[0060] S106, system iteration: the server uses the updated global consensus ranking and global hypernetwork parameters for the next round of federated learning, repeating steps S101 to S105 until the preset convergence condition is met.

[0061] Further, the detailed process of the above steps is as follows:

[0062] Step S101 specifically includes:

[0063] Before the entire federated learning task begins, the central server initializes the global model. The server first defines an over-parameterized backbone network and initializes its weights using random values which will remain fixed during subsequent training. At the same time, the server generates a corresponding, learnable initial score for each weight parameter of the backbone network, which will be used to evaluate the importance of each parameter. In addition, the server initializes a lightweight global hypernetwork with initial parameters . Lightweight Super Network can be a multi-layer perceptron (MLP) consisting of two fully connected layers. The number of nodes in its input layer is the same as the dimension of the private embedding vector , the hidden layer can contain 64 or 128 neurons and use ReLU activation function, and the number of nodes in the output layer is consistent with the total parameter amount of the personalized classification head. At the beginning of each round of communication from to , the server randomly selects a subset of clients from all clients and distributes two core components to each client in the subset i : the current global consensus ranking (which is an integer sequence obtained by sorting the global scores maintained on the server side) and the current global super network parameters .

[0064] Step S102 specifically includes:

[0065] The selected client i starts to build its local personalized model after receiving the global consensus ranking and the global super network parameters from the server. This process is divided into two parallel parts. First, the client i uses a mask generation function to convert the received global consensus ranking into a binary mask. In this mask, the positions corresponding to the top-k% parameters with the highest ranking are 1, and the remaining positions are 0. The value of k (i.e., the sparsity) here is a tunable hyperparameter, which can be set between 30% and 70% to balance the expressive power and computational efficiency of the model. The client i multiplies the mask with its locally stored fixed backbone network weights consistent with the server, element by element, thereby activating a sparse feature extractor . Second, the client i takes its locally maintained private embedding vector that is never uploaded to the server as input and sends it into the super network defined by the received global super network parameters , dynamically generating a set of personalized classification head weights i specific to the client​ Finally, the shared sparse feature extractor and the personalized classification head jointly constitute the client i The complete model used for local training in the current round .

[0066] Step S103 specifically includes:

[0067] The client i uses its built personalized model to perform one round of training on its local private dataset . In each step of the local training in each round, the client takes one batch of data from , performs forward propagation through the model to compute the predicted values, and computes the value of the loss function based on the predicted values and the true labels. Subsequently, the gradients of the loss function with respect to the three trainable components are computed by the backpropagation algorithm. The three components are: 1) the local scores which are one-to-one corresponding to the backbone network parameters, whose gradients are used to adjust the evaluation of the importance of each connection in the network; 2) the local hypernetwork parameters whose gradients are used to optimize the ability of the hypernetwork to generate personalized weights; 3) the private embedding vector of the client whose gradient is used to learn an “identity” representation that better reflects the local data distribution characteristics of the client. The client uses an optimizer (such as Adam or SGD) to update these three components synchronously based on the computed gradients. Typically, each client uses the SGD optimizer to train for 5 rounds, with a learning rate of 0.4, a momentum of 0.9, a weight decay of 1e-4, and a batch size of 32. The entire local optimization process aims to jointly find a parameter ranking that can contribute to the global consensus and a personalized model that accurately adapts to the local data.

[0068] Step S104 specifically includes:

[0069] After completing rounds of local joint optimization, the client i needs to package and upload its learning results to the server. This process strictly follows the principles of decoupling and privacy protection. The client i first applies the ranking index function to its final updated local scores , thereby generating an integer sequence, i.e., the local parameter ranking . This ranking sequence indicates which network connections the client considers most important. At the same time, the client iits final updated local super-network parameters as another part of the update. Eventually, the client i only ranks the local parameters and the local super-network parameters These two parts are uploaded to the server. The crucial private embedding vector is always kept locally on the client after the update and never disclosed, thus guaranteeing the privacy of the information strongly related to the client’s identity.

[0070] Step S105 specifically includes:

[0071] After receiving the updates from all participating clients in the current round , the server performs an innovative hybrid aggregation strategy to update the global model state. This strategy contains two parallel and independent aggregation tasks. The first task is the FRL voting aggregation for the backbone network: the server collects all the local parameter rankings uploaded by the clients. For each edge (i.e., each parameter) in the backbone network, the server computes the sum of its weighted ranks in all the rankings as its new global importance score via a voting algorithm (e.g., Borda count). Specifically, for any edge in the backbone network, the sum of its ranks in all the client rankings determines its global importance score, and the server reorders all the edges according to this score to generate a new global consensus ranking . The second task is the federated averaging aggregation for the super-network: the server collects all the local super-network parameters uploaded by the clients and performs a weighted average on them. The weight i of each client is usually proportional to the size of its local dataset. Through the weighted averaging computation, the server obtains the updated global super-network parameters .

[0072] Step S106 specifically includes:

[0073] After completing the hybrid aggregation, the server obtains a new global consensus ranking and new global super-network parameters , and a complete round of communication is completed. The system then enters the next round of communication (the th round). The server distributes and as the new global model state to the selected client subset for the next round. The entire system repeats steps S101-S105, and this iteration continues until the total number of communication rounds reaches the pre-set value , or the performance of the global model reaches the convergence criterion on the validation set. In this way, the global sparse structure and the personalized generation strategy are continuously optimized and improved in multiple rounds of distributed collaboration.

[0074] The main symbols involved in the scheme and their definitions include:

[0075] : The number of communication rounds. It represents the total number of times of model information exchange and aggregation between the server and the client. It is usually set to 2000 rounds.

[0076] : The number of local training rounds. It represents the number of iterations of model training using local data by the client in each round of communication. It is usually set to 5 rounds.

[0077] : The total number of clients. It represents the total number of client devices participating in the federated learning system. It is usually set to 1000 clients.

[0078] : The number of clients participating in each round. It represents the number of clients selected by the server to participate in training in each round of communication. It is usually set to 25 clients participating in training per round.

[0079] : The set of selected clients in the t-th round. It represents the subset of clients that actually participate in local training and upload updates in the t-th round of communication, .

[0080] : The local dataset of client i . It represents the private data owned by client i for local model training and evaluation.

[0081] : The backbone network. A fixed, randomly initialized over-parameterized neural network, serving as the basis for finding sparse subnetworks. Its parameters remain unchanged throughout the training process.

[0082] : The fixed weights of the backbone network. The specific parameter values of the backbone network are not updated after initialization.

[0083] : The learnable score of network weights. A learnable floating-point value corresponding to each weight parameter of the backbone network. This score is used to evaluate the importance of the corresponding weight and is the basis for generating parameter rankings.

[0084] : The t-th Global consensus rank of the round. An integer sequence, representing the global importance ranking of all parameters in the backbone network at the beginning of the round Global importance ranking of all parameters in the backbone network, generated by the server and distributed to clients at the beginning of the round.

[0085] : Client i Local rank of the round. An integer sequence, representing the local importance ranking of all parameters in the backbone network at the beginning of the round Local rank of the round. An integer sequence, representing the local importance ranking of all parameters in the backbone network at the beginning of the round i Local score of the client after the local training of the round is finished. Local score of the client after the local training of the round is finished. Importance ranking of all parameters in the backbone network, generated by the server and distributed to clients at the beginning of the round.

[0086] : Mask generation function. A function that takes a parameter rank as input and outputs a binary mask. In the mask, the top-k% of parameters with the highest rank are set to 1, and the rest are set to 0, which is used to activate the sparse subnetwork from the backbone network.

[0087] : Sparse feature extractor. A subnetwork formed by the backbone network and the mask , responsible for extracting features from the input data. Its effective structure is determined by the global consensus rank . The effective subnetwork is formed, that is , where represents element-wise multiplication, which is used to activate the weights of the selected edges in the subnetwork.

[0088] : Classification head. The last part of the model, responsible for mapping the feature vector output by the feature extractor to the final prediction category.

[0089] : Personalization hypernetwork. A lightweight neural network whose parameters are . Its function is to receive a private embedding vector of a client as input and generate a personalized classification head weight dedicated to that client.

[0090] : Global hypernetwork parameters of the round. Represent the hypernetwork weights generated by the server and distributed to clients at the beginning of the round Global hypernetwork parameters of the round. Represent the hypernetwork weights generated by the server and distributed to clients at the beginning of the round Global hypernetwork parameters of the round. Represent the hypernetwork weights generated by the server and distributed to clients at the beginning of the round

[0091] : Client i Local rank of the round. An integer sequence, representing the local importance ranking of all parameters in the backbone network at the beginning of the round Local super network parameters of the wheel. Represent the client i In the first Super network weights in the wheel local training process.

[0092] Private embedding vector of the client i A low-dimensional, trainable vector maintained locally by the client i , used to represent the unique characteristics of its data distribution. This vector is not uploaded to the server.

[0093] Personalized classification head weights. Generated dynamically by the super network according to the private embedding i of the client , which is unique to the client i . . .

[0094] Personalized model of the client i . The complete model of the client , which is unique to the client , is composed of a shared sparse feature extractor i and a personalized classification head .

[0095] Aggregated weights of the client i . In the federated averaging aggregation process, the weights assigned to the client i , which are usually proportional to the amount of its local data , satisfy .

[0096] Loss function. A function used to quantify the difference between model predictions and true labels, such as cross-entropy loss.

[0097] Ranking index function. A function that takes a numerical vector (such as a score ) as input and outputs an integer sequence, which represents the index position of the elements in the original vector arranged in ascending order of value.

[0098] The above scheme of the invention decouples model learning into global shared and local personalized parts, and adopts an innovative hybrid aggregation strategy, which significantly improves the comprehensive performance of the federated learning system in a heterogeneous data environment, including:

[0099] Enhancing model performance and personalization ability: The application generates a dedicated personalized model layer for each client by introducing a super network, effectively solving the problem that a single global model cannot adapt to the data distribution of all clients. In a series of comparative experiments based on the CIFAR-10 dataset, under a highly heterogeneous data setting, the average test accuracy of the application method reached 75.8%, which is significantly higher than the 68.2% of the federal ranking learning (FRL) and the 65.5% of the federal average (FedAvg). This shows that the hybrid framework of the application has outstanding advantages in improving personalized performance.

[0100] Accelerating model convergence and maintaining communication efficiency: The decoupling design of the application enables the consensus part and the personalized part of the model to learn more efficiently in parallel, avoiding aggregation conflicts caused by data heterogeneity. Experimental data show that while maintaining a low communication overhead comparable to FRL (about 26.2MB uploaded and downloaded per round, 14MB lower than FedAvg, and only 14.4MB uploaded and downloaded in the extreme sparse case), the communication rounds required for the application method to converge to the target accuracy are reduced by about 30% compared to the benchmark FRL. This effect of accelerating convergence without increasing communication burden is difficult to achieve with existing single aggregation strategies, and reflects the synergistic effect of the design of the application.

[0101] Enhancing robustness and applicability: The application inherits the core advantages of FRL, and the voting aggregation mechanism used for the model backbone makes it naturally robust to model poisoning attacks by malicious clients. At the same time, the lightweight personalized design (only a low-dimensional private embedding vector needs to be maintained by the client) and the low communication overhead make it very suitable for Internet of Things (IoT) devices and edge computing scenarios with limited computing and communication resources.

[0102] To better illustrate the technical content, purposes and effects of the application, the following embodiments are combined with the drawings to more specifically demonstrate and introduce the schemes provided by the application.

[0103] The most critical idea of the application is to decouple the model training process in federated learning into two parallel tasks: "global consensus learning" and "local personalized adaptation", and to design a special, hybrid aggregation mechanism for these two tasks. For the global model skeleton (sparse feature extractor) that requires the cooperation of all clients to form a consensus, the federated ranking learning (FRL) and its voting aggregation mechanism are used, which are communication-efficient and robust. For the personalized module (classification head) that needs to adapt to the unique data distribution of each client, a lightweight super network is introduced, which is trained collaboratively by all clients and dynamically generated based on the private embedding unique to each client.

[0104] As Figures 1-4As shown, the implementation of the personalized federated learning scheme based on ranking and super network provided by the embodiment includes the following steps:

[0105] S101, system initialization and distribution: the server initializes a shared backbone network, a set of scores for global ranking, and a global super network; the server distributes a global consensus ranking and global super network parameters to each client participating in training in each round;

[0106] S102, client model reconstruction and personalized generation: the client receives the global consensus ranking and global super network parameters; based on the global consensus ranking, a shared sparse feature extractor is reconstructed; using a locally maintained private embedding vector as input, a personalized classification head parameter is dynamically generated through the super network defined by the global super network parameter;

[0107] S103, client local joint optimization: the client trains the personalized model composed of the sparse feature extractor and the personalized classification head on its local data; during training, the scores for generating local ranking, local super network parameters, and private embedding vectors are updated simultaneously through backpropagation;

[0108] S104, client update packaging and uploading: after local training is completed, the client generates a local parameter ranking according to the updated local scores; the client uploads the local parameter ranking and the updated local super network parameters to the server;

[0109] S105, server-side hybrid aggregation: the server receives the local parameter rankings and local super network parameters from multiple clients; the multiple local parameter rankings are aggregated by voting to update the global consensus ranking; at the same time, the multiple local super network parameters are aggregated by weighted averaging to update the global super network parameters;

[0110] S106, system iteration: the server uses the updated global consensus ranking and global super network parameters for the next round of federated learning, and repeats steps S101 to S105 until the preset convergence condition is met.

[0111] Further, the detailed process of the above steps is as follows:

[0112] Step S101, system initialization and distribution, specifically includes: before the entire federated learning task begins, the central server initializes the global model. The server first defines an over-parameterized backbone network and initializes its weights using random values which will remain fixed during subsequent training. At the same time, the server generates a corresponding, learnable initial score These scores will be used to evaluate the importance of each parameter. In addition, the server also initializes a light global hypernetwork with initial parameters . The light hypernetwork can be a multi-layer perceptron (MLP) consisting of two fully connected layers. The number of input layer nodes is the same as the dimension of the private embedding vector , the hidden layer can contain 64 or 128 neurons and use the ReLU activation function, and the number of output layer nodes is consistent with the total parameter amount of the personalized classification head. At the beginning of each round of communication ( from to ), the server randomly selects a subset of clients from all clients and distributes two core components to each client in the subset: the current global consensus ranking i (the ranking is an integer sequence obtained by sorting the global scores maintained on the server side), and the current global hypernetwork parameters . The model architecture and parameters are shown in Table 1.

[0113] Table Model architecture and parameter instances

[0114]

[0115] Step S102, client model reconstruction and personalized generation, specifically including: the selected client i starts to build its local personalized model after receiving the global consensus ranking and the global hypernetwork parameters from the server. This process is divided into two parallel parts. First, the client i uses a mask generation function to convert the received global consensus ranking into a binary mask. In this mask, the positions corresponding to the top-k% parameters with the highest ranking are 1, and the remaining positions are 0. The value of k (i.e. the sparsity) here is a tunable hyperparameter, which can be set between 30% and 70% to balance the model's expression ability and computational efficiency. The client i multiplies the mask with its locally stored fixed backbone network weights consistent with the server, element by element, thereby activating a sparse feature extractor . Second, the client i ​pulls out its locally maintained private embedding vector which is never uploaded to the server , and feeds it into the global super-network defined by the received global super-network parameters . , dynamically generating a set of personalized classification head weights i that are exclusively owned by the client . Finally, the shared sparse feature extractor and the personalized classification head together form the complete model i that the client uses for local training in the current round.

[0116] Step S103, client-side local joint optimization, specifically includes: the client i uses its constructed personalized model to perform rounds of training on its local private dataset . In each step of each round of local training, the client pulls out a batch of data from , performs forward propagation through the model to calculate the predicted value, and calculates the value of the loss function based on the predicted value and the true label. Subsequently, the gradients of the loss function with respect to the three trainable components are calculated through the backpropagation algorithm. These three components are: 1) the local score corresponding to the backbone network parameters, the gradient of which is used to adjust the evaluation of the importance of each connection in the network; 2) the local super-network parameters , the gradient of which is used to optimize the ability of the super-network to generate personalized weights; 3) the client's private embedding vector , the gradient of which is used to learn an "identity" representation that more accurately reflects the distribution characteristics of its local data. The client uses an optimizer (such as Adam or SGD) to update these three components synchronously based on the calculated gradients. Typically, each client uses the SGD optimizer to train for 5 rounds, with a learning rate of 0.4, a momentum of 0.9, a weight decay of 1e-4, and a batch size of 32. The entire local optimization process aims to jointly find a parameter ranking that can contribute to the global consensus and a personalized model that accurately adapts to the local data.

[0117] Step S104, client-side update packaging and uploading, specifically includes: after completing rounds of local joint optimization, the client i needs to package and upload its learning results to the server. This process strictly follows the principles of decoupling and privacy protection. The client i first updates its final local score Application ranking index function , thereby generating an integer sequence, the local parameter ranking . This ranking sequence indicates which network connections the client considers most important. At the same time, the client i updates its final local supernetwork parameters as another part of the update. Finally, the client i uploads only the local parameter ranking and the local supernetwork parameters to the server. The crucial private embedding vector is always kept locally on the client after the update and never disclosed, thereby guaranteeing privacy information strongly related to the client's identity.

[0118] Step S105, server-side hybrid aggregation, specifically includes: after receiving the updates from all participating clients in the current round , the server updates the global model state by performing an innovative hybrid aggregation strategy. This strategy contains two parallel and independent aggregation tasks. The first task is the FRL voting aggregation for the backbone network: the server collects the local parameter rankings uploaded by all clients. For each edge (i.e., each parameter) in the backbone network, the server calculates the weighted sum of its ranks in all rankings as its new global importance score by a voting algorithm (e.g., Borda count). Specifically, for any edge in the backbone network, the sum of its ranks in all client rankings determines its global importance score, and the server reorders all edges according to this score to generate a new global consensus ranking . The second task is the federated averaging aggregation for the supernetwork: the server collects the local supernetwork parameters uploaded by all clients and performs a weighted average on them. The weight i of each client is usually proportional to the size of its local dataset. Through the weighted average calculation, the server obtains the updated global supernetwork parameters .

[0119] Step S106, system iteration, specifically includes: after completing the hybrid aggregation and obtaining the new global consensus ranking and the new global supernetwork parameters , a complete round of communication is completed. The system then enters the next round of communication (the th round). The server distributes and as the new global model state to the selected client subset The entire system will repeatedly perform steps S101-S105, and thus circulate. This iteration process continues until the total number of communication rounds reaches a preset value , or the performance of the global model on the validation set reaches a convergence criterion. In this way, the global sparse structure and the personalized generation strategy are continuously optimized and improved in multiple rounds of distributed collaboration.

[0120] Based on the same inventive concept, the present application also provides a computer device, which comprises one or more processors and a memory for storing one or more computer programs; the program comprises program instructions, and the processor is configured to execute the program instructions stored in the memory. The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., which are the computing core and control core of the terminal, and are configured to implement one or more instructions, and are specifically configured to load and execute one or more instructions in the computer storage medium to implement the above method.

[0121] It needs to be further explained that, based on the same inventive concept, the present application also provides a computer storage medium, which stores a computer program, and the computer program is executed by the processor to perform the above method. The storage medium can adopt any combination of one or more computer readable media. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium can be, but is not limited to, an electrical, magnetic, optical, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples (non-exhaustive list) of the computer readable storage medium include: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or component.

[0122] In the description of the specification, the description of the terms "one embodiment", "an example", "a specific example" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present disclosure. In the specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0123] The basic principles, main features and advantages of the present disclosure are shown and described above. Those skilled in the art should understand that the present disclosure is not limited by the above embodiments, and the above embodiments and descriptions in the specification are only to illustrate the principles of the present disclosure. Without departing from the spirit and scope of the present disclosure, various changes and improvements of the present disclosure can be made, which all fall within the scope of the claimed present disclosure.

[0124] The present application is not limited to the above best mode, and anyone can derive other various forms of a personalized federated learning method and system for data heterogeneous and resource constrained environments under the inspiration of the present application. Any equivalent changes and modifications made within the scope of the patent application of the present application shall fall within the scope of the present application.

Claims

1. A personalized federated learning method for data heterogeneous and resource constrained environment, executed by a server in cooperation with a plurality of clients, for protecting client data privacy and collaboratively training a model adapted to local data distribution of each client, characterized in that, The method comprises the following steps: (1) The server initializes a shared backbone network, parameter importance scores corresponding to weights of the backbone network, and a global hypernetwork; In each round of communication, the server distributes the current global parameter consensus ranking and the global hypernetwork parameters to the clients participating in the training; (2) The clients receive the global parameter consensus ranking and the global hypernetwork parameters; generate a mask based on the global parameter consensus ranking to activate a shared sparse feature extractor; Meanwhile, input a locally independently maintained private embedding vector into the global hypernetwork to generate a personalized classification head and build a complete personalized model; then train the personalized model using local data to synchronously update the local parameter importance scores, the local hypernetwork parameters, and the private embedding vector through back propagation; (3) The clients generate a local parameter ranking based on the updated local parameter importance scores, and upload the local parameter ranking and the updated local hypernetwork parameters to the server; The private embedding vector is maintained locally on the clients and is not uploaded to the server; (4) The server receives the local parameter rankings and the local hypernetwork parameters uploaded by the clients; discretely aggregates the local parameter rankings using a voting aggregation method to update the global parameter consensus ranking; meanwhile, continuously aggregates the local hypernetwork parameters using a weighted average method to update the global hypernetwork parameters; (5) Repeat steps (1) to (4) until the model performance meets the preset convergence condition.

2. The personalized federated learning method for data heterogeneous and resource constrained environments according to claim 1, characterized in that: The backbone network is an over-parameterized neural network, and the weights of the backbone network remain fixed after initialization; the parameter importance scores are learnable floating-point values used to evaluate the contribution of the corresponding backbone network weights to the model performance and are the basis for generating the global parameter consensus ranking.

3. The personalized federated learning method for data heterogeneous and resource constrained environments according to claim 1, characterized in that: The generation rule of the mask is as follows: according to the global parameter consensus ranking, set the positions corresponding to a part of weights with high rankings to 1 and the remaining positions to 0 to activate a sparse feature extractor.

4. The personalized federated learning method for data heterogeneous and resource constrained environments according to claim 1, characterized in that: The global hypernetwork is a lightweight neural network, the dimensions of the input of the global hypernetwork are adapted to the private embedding vector, and the size of the output tensor is the same as the total parameter amount of the personalized classification head.

5. The personalized federated learning method for data heterogeneous and resource constrained environments according to claim 1, characterized in that: The voting aggregation method uses the Borda count method to generate the global importance scores of the weights of the backbone network by counting the sequence positions of the weights in the local parameter rankings on the clients to update the global parameter consensus ranking; In the weighted average method, the weight of each client is proportional to the sample amount of the local data set of the client.

6. The personalized federated learning method for data heterogeneous and resource constrained environments according to claim 1, characterized in that: The local data training uses a gradient descent optimization algorithm to quantify the model prediction error through a loss function, and simultaneously calculates the gradients of the local parameter importance scores, the local hypernetwork parameters, and the private embedding vector based on the error to realize the collaborative optimization of the three.

7. The personalized federated learning method for data heterogeneous and resource constrained environments according to claim 1, characterized in that: The data-heterogeneous and resource-constrained environment is an Internet of Things scenario, and the clients are Internet of Things sensor devices, including intelligent cameras and microphones.

8. The personalized federated learning method for data heterogeneous and resource constrained environments according to claim 1, characterized in that: The private embedding vector is a low-dimensional vector used to represent the unique characteristics of the local data distribution of the client; the private embedding vectors of different clients are different, and the private embedding vectors of the same client are dynamically updated with the local training to ensure that the personalized classification head always adapts to the changes in the local data distribution of the client.

9. The personalized federated learning method for data heterogeneous and resource constrained environments according to claim 1, characterized in that: The preset convergence condition is one of the following two conditions: one is that the total communication round reaches a first preset threshold; Two is that the performance of the global model on the preset verification set fluctuates by no more than a second preset threshold for multiple rounds of communication, and reaches a preset performance standard.

10. A personalized federated learning system for data heterogeneous and resource constrained environments, characterized in that, The server and the plurality of clients, the server cooperates with the clients to protect the data privacy of the clients and collaboratively trains a model adapted to the local data distribution of each client; The server comprises: An initialization module configured to initialize a shared backbone network, parameter importance scores corresponding to weights of the backbone network, and a global super network; A distribution module configured to distribute, in each round of communication, a current global parameter consensus ranking and global super network parameters to the clients participating in training; An aggregation module configured to receive local parameter rankings and local super network parameters uploaded by the clients, perform discrete aggregation on the local parameter rankings by using a voting aggregation manner to update the global parameter consensus ranking, and perform continuous aggregation on the local super network parameters by using a weighted average manner to update the global super network parameters; The client comprises: A model construction module configured to receive the global parameter consensus ranking and the global super network parameters, generate a mask based on the global parameter consensus ranking to activate a shared sparse feature extractor, input a private embedding vector independently maintained locally into the global super network to generate a personalized classification head, and construct a personalized model; A training update module configured to train the personalized model with local data, and synchronously update the local parameter importance scores, the local super network parameters, and the private embedding vector by using back propagation; An uploading module configured to sort local parameter rankings based on the updated local parameter importance scores, upload the local parameter rankings and the updated local super network parameters to the server, and maintain the private embedding vector locally without uploading; The server and the client further comprise an iteration control module configured to repeatedly execute the "distribution-model construction-training-uploading-aggregation" process until the performance of the model meets the preset convergence condition.

Images