Universal model heterogeneous federal learning privacy protection method

Abstract

The invention provides a universal model heterogeneous federated learning privacy protection method, which comprises the following steps that: a cloud server randomly selects an online client, and obtains a model generation parameter union set of the client through a privacy protection union set protocol; generating heterogeneous models with different sizes and architectures according to the union set, and distributing the models to edge nodes for storage; the client downloads the local model from the corresponding edge node, and disturbs and encrypts the trained model parameters; the edge node receives the encryption model parameters uploaded by the client, and performs secure aggregation through a threshold vector aggregation protocol to obtain an aggregation model meeting the differential privacy requirement; and the cloud server receives the aggregation models from all the edge nodes, and globally updates the aggregation models by adopting a heterogeneous model aggregation method to obtain a new global model. Through an innovative privacy protection protocol and system architecture, the privacy protection problem in a model heterogeneous federated learning scene is effectively solved, and the practicability and security of federated learning are improved.

Description

Technical Field

[0001] The present invention relates to the field of cyberspace security technology, and in particular to a universal model heterogeneous federated learning privacy protection method. Background Art

[0002] Federated Learning (FL), an emerging distributed machine learning paradigm, aims to address data silos. By allowing multiple clients to train models locally and upload model parameters to a server for aggregation, FL optimizes the global model while protecting user data privacy. In recent years, with the rapid development of big data and artificial intelligence technologies, FL has been widely applied in various fields and has attracted widespread attention.

[0003] However, traditional federated learning methods often require users to train models with the same architecture as the global model, which presents numerous challenges when working with heterogeneous devices with limited resources. For example, differences in computing power, storage space, and network bandwidth between devices lead to performance bottlenecks when training large-scale global models. For example, Google's application of federated learning to Gboard boasts a model size of only 1.4MB, making it suitable for resource-constrained environments like mobile devices. Alibaba's recommendation model for Taobao, on the other hand, is a whopping 134GB. Using such a large model for federated learning is clearly impractical for most heterogeneous devices.

[0004] To address these issues, researchers have begun exploring model heterogeneous federated learning (MHL). This allows users to train local models that differ from the global model architecture based on their device capabilities and needs. This approach improves the practicality of federated learning and enables a wider range of devices to participate. However, MHL also introduces new privacy challenges.

[0005] First, to customize models with different structures based on varying capabilities and needs, the client needs to share additional information with the server, including the model structure, device capabilities, and local data types. This process can pose a risk of privacy leakage, as sensitive information can be stolen or misused by malicious attackers during transmission and storage. Traditional federated learning privacy protection methods are often designed for a unified model architecture and are difficult to directly apply to heterogeneous model scenarios.

[0006] Secondly, aggregating heterogeneous models often requires parameter alignment or the use of advanced fusion techniques (such as knowledge distillation) to achieve model parameter aggregation. The complexity of these techniques further increases the difficulty of designing universal privacy-preserving methods. Existing model-heterogeneous privacy-preserving federated learning (MH-PPFL) methods are often designed for specific algorithms and are difficult to broadly apply to different types of heterogeneous models. Therefore, designing a universal MH-PPFL method that decouples privacy-preserving protocols from specific algorithms has become a hot topic and a challenge in current research.

[0007] In summary, while model-heterogeneous federated learning improves its practicality, it also introduces new privacy protection challenges. Traditional federated learning privacy protection methods are difficult to directly apply to model-heterogeneous scenarios, necessitating the development of new privacy protection technologies to safeguard user privacy and data security. This paper addresses this need by proposing a universal privacy protection method for model-heterogeneous federated learning, aiming to address these challenges and achieve efficient, secure, and universal federated learning. Summary of the Invention

[0008] In response to the shortcomings of the existing technology, the present invention provides a universal privacy protection method for model heterogeneous federated learning. Through an innovative privacy protection protocol and system architecture, this method effectively solves the privacy protection problem in the model heterogeneous federated learning scenario, improves the practicality and security of federated learning, and can be efficient, universal, and have low model accuracy loss on the basis of privacy protection.

[0009] The present invention achieves the above-mentioned purpose through the following technical solutions: A general privacy protection method for heterogeneous federated learning of models includes the following steps: Initialization phase: The cloud server randomly selects online clients and obtains the client's model generation parameter union through a privacy-preserving union protocol. Based on the union, heterogeneous models of different sizes and architectures are generated and distributed to edge nodes for storage. Local training phase: The client downloads the local model from the corresponding edge node, uses local data for training, and perturbs and encrypts the trained model parameters. Secure aggregation phase: The edge node receives the encrypted model parameters uploaded by the client and performs secure aggregation through the threshold vector aggregation protocol to obtain an aggregated model that meets differential privacy requirements; Global update phase: The cloud server receives the aggregated model from all edge nodes and uses any heterogeneous model aggregation method to perform global updates to obtain a new global model.

[0010] According to a general privacy protection method for model heterogeneous federated learning provided by the present invention, the threshold vector aggregation protocol is used to aggregate vectors from clients while ensuring that the information of individual vectors is not leaked during the aggregation process. The aggregator can only decrypt the final aggregated vector after all encrypted vectors are aggregated. The threshold vector aggregation protocol adopts the Paillier cryptographic algorithm and Shamir secret sharing technology, and is optimized in combination with super-increasing sequences.

[0011] According to a general privacy protection method for heterogeneous federated learning provided by the present invention, the threshold vector aggregation protocol is implemented by the following steps: Key generation step: The key generation center KGC generates a key pair for encryption and decryption, including selecting two large prime numbers ,in and ,and and is also a prime number; let and , KGC randomly selects two integers and , their order is mold , and choose a random integer ; Select a sequence of integers And construct the polynomial ;make , ; The public key parameters are , client The encrypted private key is ,in and ; The aggregator's decryption private key is ; Client encryption step: The client uses a super-increasing sequence to encode the vector to be aggregated Mapped to integers , and then use the encryption key and a random value For this integer Encrypt and generate encrypted vector and upload to aggregators; Aggregator aggregation step: Receive the ciphertext sets uploaded by all clients After that, perform the aggregation operation and then use the decryption key Decrypt the aggregated encrypted vector to obtain the aggregated vector ; Finally, execute Algorithm 1 to restore the aggregation vector .

[0012] According to a general privacy protection method for heterogeneous federated learning of models provided by the present invention, the privacy-preserving union protocol is implemented by the following steps: Compute union : Each client first sets its capacity parameter Mapped to a dimensional vector ,in:

[0013] like , then the corresponding vector for ,in It is from Then, these vectors are transmitted to the cloud server for summation; the threshold vector aggregation protocol is used to implement vector summation.

[0014] According to a general model heterogeneous federated learning privacy protection method provided by the present invention, when using the threshold vector aggregation protocol to implement vector summation, the client Perform client-side encryption steps on the vector Encrypted and uploaded to the cloud server for aggregation; after receiving all encrypted vectors, the server performs the aggregator aggregation step to privately calculate the aggregate vector , then by Reconstructed ,Right now .

[0015] According to a general privacy protection method for heterogeneous federated learning of models provided by the present invention, in the initialization phase, At the beginning of a training round, the central server initializes the global model , randomly selected Online Clients , and set the parameters Send to these clients; Each client needs to Select an appropriate capability parameter ,and disclose it to the cloud server for generating heterogeneous models.,The PUP protocol is used to enable the client to disclose the,joint set of its capability parameters to the cloud server without,disclosing its privacy; After executing PUP, the cloud server can only obtain the joint set of all client capability parameters , but cannot access the specific parameter values of any single client; Server according to Generate heterogeneous models , and each model Assigned to the corresponding edge node , and share the corresponding allocation information of the model and edge nodes with the client; The client connects to the corresponding edge node to download the local model; connect to the edge node The client set is denoted as .

[0016] According to a general privacy protection method for heterogeneous federated learning of models provided by the present invention, after the initialization phase, each client establishes a connection with its corresponding edge node and performs local training through the edge node. All clients connected to the same edge node use a local model with the same structure for training. The training process follows the federated learning workflow and specifically includes the following steps: The client downloads the local model from the corresponding edge node; The client uses the local dataset to train the downloaded local model and obtain the trained model parameters; The client perturbs and encrypts the trained model parameters to protect their privacy. The encrypted model parameters are uploaded by the client to the corresponding edge node, which performs secure aggregation operations.

[0017] According to a general privacy protection method for model heterogeneous federated learning provided by the present invention, in the security aggregation stage, a hybrid security aggregation protocol H-Sagg based on the threshold vector aggregation protocol is adopted to aggregate model parameters, wherein, in the key generation step, the key for homomorphic encryption and the noise parameter for implementing Renyi differential privacy are generated simultaneously; the noise addition operation is integrated into the key generation algorithm, so that each client automatically adds noise when encrypting its model parameters.

[0018] According to a general privacy protection method for heterogeneous federated learning of models provided by the present invention, the hybrid security aggregation protocol H-Sagg is implemented by the following steps: First, edge nodes choose A random seed is used to generate a random vector and transmit it to the client , these seeds are used to generate The length is Random vectors of random matrices row; Then, the client Use its local data for local training to obtain local parameters , the execution threshold is Parameter pruning and generating a secret vector , and an error vector , whose elements come from a Gaussian distribution ;in, , is the size of the domain, is the width of the error distribution; Then, the client Generate encryption keys , and calculate the encrypted model parameters , and send it to the edge node ; After receiving all encryption vectors, the edge node It can be aggregated to obtain , the calculation process is expressed as the following formula:

[0019] According to the general privacy protection method for heterogeneous federated learning of models provided by the present invention, Decryption, edge node The aggregated secret vector needs to be calculated , and then obtain the aggregation model that satisfies differential privacy ; The threshold vector aggregation protocol is used to aggregate the client's secret vector at the edge node to obtain ; Then, the edge node Aggregate multiple encryption vectors to obtain an aggregated secret vector , and calculate the aggregated model , the model satisfies -RDP, where ; edge nodes After the aggregation is completed, the aggregation model that satisfies differential privacy is submitted to the cloud server for final aggregation; The cloud server uses a heterogeneous model aggregation method to aggregate the models from edge nodes and obtain an updated global model.

[0020] As can be seen, compared with the existing technology, this invention proposes a universal privacy-preserving model heterogeneous federated learning method, which protects user privacy while providing an efficient and secure federated learning environment, with the following beneficial effects: 1. The Threshold Vector Aggregation (TVA) protocol proposed in this paper ensures that the server can only obtain the aggregated results after aggregating the data of all clients, effectively avoiding the risk of premature decryption by the server, thereby significantly improving the level of privacy protection during the data aggregation process. Through carefully designed encryption and decryption mechanisms, the TVA protocol ensures the privacy of individual client data during the aggregation process, providing a solid privacy protection foundation for heterogeneous federated learning of models.

[0021] 2. Based on the TVA protocol, this paper constructs a privacy-preserving union protocol. This protocol effectively protects the privacy of individual data by only disclosing the union of all client-uploaded data to the server. This protocol allows clients to participate in data aggregation in federated learning without revealing the specific data content, further enhancing the system's privacy protection capabilities.

[0022] 3. Based on the TVA protocol, this paper implements the H-Sagg secure aggregation protocol, which combines LWE-based homomorphic encryption and Renyi differential privacy technology to effectively reduce the loss of model accuracy while maintaining efficient computation. By integrating the noise addition step directly into the key generation step, the H-Sagg protocol reduces the risk of privacy degradation caused by malicious clients intentionally omitting noise, further improving the security and reliability of the system.

[0023] In summary, the method proposed in this paper, by introducing the three core protocols mentioned above, achieves efficient and secure privacy protection in heterogeneous federated learning scenarios. This method not only effectively protects user privacy, but also improves computational efficiency and communication complexity, while minimizing the loss of model aggregation accuracy. This provides strong support for the application of federated learning in a wider range of scenarios.

[0024] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a basic architectural principle diagram of the UniFed system in an embodiment of a general model heterogeneous federated learning privacy protection method of the present invention.

[0026] Figure 2 This is a flowchart of an embodiment of a general model heterogeneous federated learning privacy protection method of the present invention.

[0027] Figure 3 This is a schematic diagram of an embodiment of a general model heterogeneous federated learning privacy protection method of the present invention.

[0028] Figure 4This is a schematic diagram of the principles of Algorithm 1 in an embodiment of a general model heterogeneous federated learning privacy protection method of the present invention.

[0029] Figure 5 This is a schematic diagram of the principles of Algorithm 2 in an embodiment of a general model heterogeneous federated learning privacy protection method of the present invention.

[0030] Figure 6 This is a schematic diagram of the principles of Algorithm 3 in an embodiment of a general model heterogeneous federated learning privacy protection method of the present invention. DETAILED DESCRIPTION

[0031] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0032] References to "embodiments" herein mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0033] 1. Model Heterogeneous Federated Learning: A model heterogeneous federated learning (FL) framework consists of a server and Clients Composition, each client uses a model with a different structure The goal of model heterogeneous federated learning is to utilize the client's local dataset Jointly train a global model This process usually includes three key steps: 1) Heterogeneous model generation: Each client Provide a build parameter to the server , which reflects the device capability or data distribution for generating personalized models. Generate a heterogeneous model customized for the client. 2) Local training: Each client In its dataset Train its local model on 3) Heterogeneous model aggregation: The server uses techniques such as knowledge distillation or selective averaging to aggregate heterogeneous local models and update the global model. .

[0034] 2. Renyi Differential Privacy (RDP) Definition 1 ( -RDP): If a random mechanism For any pair of adjacent data sets , both If established, the mechanism satisfies -RDP, where represents the Renyi divergence.

[0035] Definition 2 (post-processing): If satisfy -RDP, then for any random mapping , Also satisfied -RDP.

[0036] 3. Learning With Errors (LWE) LWE is a difficult problem in lattice cryptography, which is defined as follows: Definition 3 (LWE sample): Assume is a prime number Finite field of . Given a secret vector , error vector , and the matrix , the LWE sample is ,in .

[0037] Definition 4 (Decision LWE Problem): Given Independent samples , the decision LWE problem is to distinguish whether these samples are LWE samples, or whether they are A pair of samples uniformly randomly selected from .

[0038] 4. Super-increasing sequence: given a positive integer , the length is A super-increasing sequence is a vector ,in is a prime number and satisfies , and for each ,have ,and A vector ,in , can be mapped to an integer value through formula (1) . Subsequently, if the super-increasing sequence is known , then we can execute Algorithm 1 (such as Figure 4 shown) to restore the original vector.

[0039]

[0040] It should be noted that due to the characteristics of super-increasing sequences, the length is finite. Therefore, A vector of length can be split into multiple sub-vectors, each of which has a length less than , and then mapped using super-increasing sequences respectively.

[0041] See also Figure 1 This embodiment provides a system (UniFed) for implementing a universal model heterogeneous federated learning privacy protection method, including: Cloud server: Used to initialize the global model, randomly select online clients, and obtain the client's model generation parameter union through a privacy-preserving union protocol; generate heterogeneous models of different sizes and architectures based on the union, and distribute these models to edge nodes for storage; receive aggregated models from all edge nodes and perform global updates using any heterogeneous model aggregation method; Edge nodes: Located between the cloud server and the client, they maintain local models of specific structures. They receive encrypted model parameters uploaded by the client, securely aggregate them using the threshold vector aggregation protocol to obtain an aggregated model that meets differential privacy requirements, and upload the aggregated model to the cloud server. Client: Used to download the local model from the corresponding edge node, train it using local data, perturb and encrypt the trained model parameters, and upload the encrypted model parameters to the corresponding edge node.

[0042] Specifically, Figure 1 The basic architecture of the UniFed system is presented, which consists of three entities: cloud servers, edge nodes, and clients with different computing capabilities.

[0043] Cloud Server (CS): A cloud server is a server cluster with powerful computing and storage capabilities. It is responsible for maintaining the global model. Its main tasks include generating models of various structures and aggregating heterogeneous local models to update the global model.

[0044] Edge nodes (ENs): Edge nodes are located between cloud servers and clients. Each edge node maintains a local model with a specific structure. Its functions include deploying the model to connected clients and performing homogeneous federated learning (FL) with the clients.

[0045] Client: The client is a resource-constrained device whose tasks include training the local model downloaded from the edge node, perturbing and encrypting the model parameters, and uploading them to the edge node.

[0046] In addition, this embodiment also introduces a key generation center (KGC) to generate the keys required for the privacy protection protocol proposed in this article.

[0047] like Figure 2 and 3 As shown, this embodiment provides a general model heterogeneous federated learning privacy protection method, which includes the following steps: Step S1, initialization phase: The cloud server randomly selects online clients and obtains the client's model generation parameter union through the privacy-preserving union protocol; generates heterogeneous models of different sizes and architectures based on the union, and distributes these models to edge nodes for storage; Step S2, local training phase: the client downloads the local model from the corresponding edge node, trains it using local data, and perturbs and encrypts the trained model parameters; Step S3, secure aggregation phase: The edge node receives the encrypted model parameters uploaded by the client and performs secure aggregation through the threshold vector aggregation protocol to obtain an aggregated model that meets the differential privacy requirements; Step S4, global update phase: The cloud server receives the aggregated models from all edge nodes and uses any heterogeneous model aggregation method to perform global updates to obtain a new global model.

[0048] Specifically, such as Figure 3 As shown, first, the cloud server ( ) randomly selects a client and executes the PUP protocol to obtain the union of model generation parameters (step 1). Subsequently, the cloud server generates models of different sizes and architectures based on the joint parameters and distributes these models to edge nodes ( ) for storage (steps 2 and 3). Next, the client downloads the model from the edge node and trains it using local data. After training, the client encrypts the model parameters and uploads them to the corresponding edge node (steps 4-6). Subsequently, multiple edge nodes execute the H-Sagg protocol to generate -RDP aggregated model and uploaded it to the cloud server (steps 7 and 8). Finally, the cloud server aggregates the models uploaded by all edge nodes to obtain the global model for the next round of training (step 9).

[0049] In this embodiment, the Threshold Vector Aggregation (TVA) protocol is used to aggregate vectors from clients while ensuring that information about individual vectors is not leaked during the aggregation process. The aggregator can only decrypt the final aggregated vector after all encrypted vectors have been aggregated. To achieve this, this embodiment employs the Paillier cryptographic algorithm and Shamir secret sharing, combined with super-increasing sequences, to further reduce communication complexity and improve efficiency. The Threshold Vector Aggregation (TVA) protocol is implemented through the following steps: Key generation steps :The key generation center KGC generates the key pair for encryption and decryption, including the selection of two large prime numbers ,in and ,and and is also a prime number; let and , KGC randomly selects two integers and , their order is mold , and choose a random integer ; Select a sequence of integers And construct the polynomial ;make , ; The public key parameters are , client The encrypted private key is ,in and ; The aggregator’s decryption private key is ; Client encryption steps ( ): The client uses a super-incrementing sequence to aggregate the vector Mapped to integers , and then use the encryption key and a random value For this integer Encrypt and generate encrypted vector And upload it to the aggregator, the calculation process is as follows:

[0050] Aggregator Aggregation Step ( ): When the aggregator receives the ciphertext sets uploaded by all clients After that, perform the aggregation operation. The specific steps are as follows:

[0051] Then, using the decryption key Decrypt the aggregated encrypted vector to obtain the aggregated vector , the specific steps are as follows:

[0052]

[0053] Finally, the aggregator executes Algorithm 1 to recover the aggregation vector .

[0054] In this embodiment, the privacy-preserving union protocol is implemented through the following steps: To privately compute the union Each client first sets its capacity parameter Mapped to a dimensional vector (Line 3 of Algorithm 2), such as Figure 5 As shown, where:

[0055] For example, if , then the corresponding vector for ,in It is from Then, these vectors are transmitted to the cloud server for summation; the threshold vector aggregation protocol is used to implement vector summation.

[0056] Specifically, when using the threshold vector aggregation protocol to implement vector summation, the client Perform client encryption steps Pair Vector Encrypt and upload to the cloud server for aggregation (line 4 of Algorithm 2); after receiving all the encrypted vectors, the server performs the aggregator aggregation step Compute aggregate vectors privately (Line 5 of Algorithm 2), and then by Reconstructed ,Right now (Line 6 of Algorithm 2). TVA ensures The aggregate vector can only be decrypted and accessed after all the client-provided encryption vectors have been aggregated. In addition, the random numbers in the vectors mask the count information after aggregation, which means that the cloud server can only know whether a client has selected a specific , but it is not known how many clients have chosen this type .

[0057] In this embodiment, the UniFed execution process includes: The system consists of a global model Cloud Server 、 edge nodes , a client pool and a model generation parameter set Specifically, It is a set of parameters pre-defined by the server and distributed to all clients. Interpreted as a representation of a device capability score, where For example, when When , a feasible capability parameter set can be . Complete communication round The training process is as follows Figure 3 As shown in, and described in detail in Algorithm 3, Figure 6 shown.

[0058] In the initialization phase, At the beginning of a training round, the central server initializes the global model , randomly selected Online Clients , and set the parameters Send to these clients (Lines 2-3 of Algorithm 3); Each client needs to Select an appropriate capability parameter , and disclose it to the cloud server for generating heterogeneous models. However, direct transmission This is not feasible because the parameter is sensitive information and may reflect the performance of the device or even indirectly indicate the income level of the client user. Therefore, this embodiment uses the PUP protocol to enable the client to disclose the joint set of its capability parameters to the cloud server without leaking privacy; After executing PUP, the cloud server can only obtain the joint set of all client capability parameters , and cannot access the specific parameter values of any single client (Line 4 of Algorithm 3, Figure 3 Step 1).

[0059] Then, the server Generate heterogeneous models , and each model Assigned to the corresponding edge node (Line 5 of Algorithm 3, Figure 3 Steps 2-3) and the corresponding allocation information of the model and edge nodes is shared with the client.

[0060] Finally, the client connects to the corresponding edge node to download the local model; connecting to the edge node The client set is denoted as .

[0061] After the initialization phase, each client establishes a connection with its corresponding edge node and performs local training through the edge node; all clients connected to the same edge node are trained using a local model with the same structure (Lines 8-9 of Algorithm 3, Figure 3 The training process follows the federated learning workflow, which includes: The client downloads the local model from the corresponding edge node; the client uses the local data set to train the downloaded local model to obtain the trained model parameters; the client perturbs and encrypts the trained model parameters to protect the privacy of the model parameters; the encrypted model parameters are uploaded by the client to the corresponding edge node, and the edge node performs secure aggregation operations.

[0062] In the secure aggregation phase, a hybrid secure aggregation protocol (HybridSecure Aggregation Protocol, H-Sagg) based on the threshold vector aggregation protocol is used to aggregate model parameters. In the key generation step, the key for homomorphic encryption and the noise parameters for achieving Renyi differential privacy are generated simultaneously; the noise addition operation is integrated into the key generation algorithm so that each client automatically adds noise when encrypting its model parameters. Compared with existing secure aggregation methods based on differential privacy (DP), H-Sagg combines LWE-based homomorphic encryption and Renyi differential privacy, reducing the loss of aggregate model accuracy while maintaining strong privacy protection. This is because LWE-based homomorphic encryption ensures the confidentiality of model parameters during transmission and aggregation, so that the aggregator can only obtain the aggregated model and cannot access the client's individual information. Under this guarantee, the client only needs to add minimum noise to its model parameters, thereby ensuring that the cumulative noise of all clients can make the aggregated model meet -RDP. Therefore, the degradation of model accuracy can be significantly mitigated. Furthermore, the H-Sagg protocol integrates the noise addition step directly into the key generation process, reducing the risk of malicious clients intentionally omitting noise and thus compromising privacy. Next, we will describe the working principle of the H-Sagg protocol.

[0063] First, edge nodes choose A random seed is used to generate a random vector and transmit it to the client , these seeds are used to generate The length is Random vectors of random matrices row; Then, the client Use its local data for local training to obtain local parameters , the execution threshold is Parameter pruning and generating a secret vector , and an error vector , whose elements come from a Gaussian distribution ;in, , is the size of the domain, is the width of the error distribution; Then, the client Generate encryption keys , and calculate the encrypted model parameters , and send it to the edge node (Lines 10-12 of Algorithm 3; Figure 3 Steps 5-6); After receiving all encryption vectors, the edge node It can be aggregated to obtain , the calculation process is expressed as the following formula:

[0064] For the sake of Decryption, edge node The aggregated secret vector needs to be calculated , and then obtain the aggregation model that satisfies differential privacy ; For security reasons, the secret vector cannot be transmitted directly because it is crucial to maintain the confidentiality of the client model parameters. To solve this problem, a threshold vector aggregation protocol is used to aggregate the client’s secret vector at the edge node to obtain ; Therefore, in addition to the encrypted model parameters , client You also need to upload Obtained encrypted secret vector (Line 13 of Algorithm 3).

[0065] Then, the edge node Aggregate these encrypted vectors to get the aggregated secret vector , and calculate the aggregated model , the model satisfies -RDP, where (Lines 15-17 of Algorithm 3, Figure 3 Step 7).

[0066] edge nodes After the aggregation is completed, the aggregation model that satisfies differential privacy is submitted to the cloud server for final aggregation (line 18 of Algorithm 3, Figure 3 Step 8); Subsequently, the cloud server aggregates the models from the edge nodes using a heterogeneous model aggregation method and obtains an updated global model, which also satisfies the post-processing properties of RDP. -RDP (line 19 of Algorithm 3, Figure 2 Step 9).

[0067] In summary, in order to solve the various problems of the existing technology, this embodiment proposes a UniFed system and a general privacy-preserving model heterogeneous federated learning method. In response to the problems brought about by the first aspect of the existing technology, this embodiment proposes a private union protocol (PUP), which enables the client to share the union of its information with the server while maintaining privacy. Specifically, the union operation is converted into the summation of client-randomized indicator vectors. These indicator vectors are obtained by mapping the shared information into binary vectors and perturbing the '1' value with random numbers to confuse the aggregated count values. In order to privately aggregate these vectors, this embodiment designs a threshold vector aggregation (TVA) method to achieve efficient and privacy-preserving vector summation while significantly reducing communication complexity.

[0068] In response to the problem of decoupling the privacy protection protocol from the specific model heterogeneous FL method mentioned in the second aspect of the problem brought about by the existing technology, a simple idea is to adopt differential privacy (DP) technology. Local differential privacy (LDP) protects privacy by locally adding random noise to the model parameters without affecting the aggregation process on the server side, making it naturally suitable for various model heterogeneous FL methods. However, traditional LDP methods usually add a lot of noise to the model parameters to protect privacy, which often significantly reduces the accuracy of the model during transmission and aggregation. In UniFed, this embodiment combines homomorphic encryption (HE) and differential privacy technology to implement a hybrid secure aggregation (H-Sagg) protocol, in which HE ensures the confidentiality of the model parameters during transmission and aggregation, so that the aggregator can only obtain the aggregated model but cannot access individual information. Under this guarantee, the client only needs to add minimal noise to its model parameters, thereby ensuring that the noise accumulated on all clients can achieve centralized differential privacy for the aggregated model. Subsequently, as Figure 1 As shown, edge nodes are introduced to facilitate communication between the client and the server. Each edge node uses the H-Sagg protocol to aggregate models with the same structure (as done in traditional FL) to obtain a differentially private edge model. Ultimately, the server can aggregate these edge models using any heterogeneous model aggregation method (such as parameter alignment and knowledge distillation) to obtain a new global model. Therefore, the method provided in this embodiment is applicable to a variety of model-heterogeneous FL algorithms.

[0069] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0070] The above embodiments are only preferred embodiments of the present invention and cannot be used to limit the scope of protection of the present invention. Any non-substantial changes and replacements made by technicians in this field on the basis of the present invention fall within the scope of protection required by the present invention.

Claims

1. A general privacy protection method for heterogeneous federated learning, characterized by: The following steps are involved: Initialization phase: The cloud server randomly selects online clients and obtains the client's model generation parameter union through a privacy-preserving union protocol. Based on the union, heterogeneous models of different sizes and architectures are generated and distributed to edge nodes for storage. Local training phase: The client downloads the local model from the corresponding edge node, uses local data for training, and perturbs and encrypts the trained model parameters. Secure aggregation phase: The edge node receives the encrypted model parameters uploaded by the client and performs secure aggregation through the threshold vector aggregation protocol to obtain an aggregated model that meets differential privacy requirements; Global update phase: The cloud server receives the aggregated model from all edge nodes and uses any heterogeneous model aggregation method to perform global updates to obtain a new global model.

2. The method according to claim 1, wherein: The threshold vector aggregation protocol is used to aggregate vectors from clients while ensuring that the information of individual vectors is not leaked during the aggregation process. The aggregator can only decrypt and obtain the final aggregated vector after all encrypted vectors are aggregated. The threshold vector aggregation protocol uses the Paillier cryptographic algorithm and Shamir secret sharing technology, and is optimized in combination with super-increasing sequences.

3. The method according to claim 2, characterized in that The threshold vector aggregation protocol is implemented by the following steps: Key generation step: The key generation center KGC generates a key pair for encryption and decryption, including selecting two large prime numbers ,in and ,and and is also a prime number; let and , KGC randomly selects two integers and , their order is mold , and choose a random integer ; Select a sequence of integers And construct the polynomial ;make , ; The public key parameters are , client The encrypted private key is ,in and ; The aggregator’s decryption private key is ; Client encryption step: The client uses a super-increasing sequence to encode the vector to be aggregated Mapped to integers , and then use the encryption key and a random value For this integer Encrypt and generate encrypted vector and upload to aggregators; Aggregator aggregation step: Receive the ciphertext sets uploaded by all clients After that, perform the aggregation operation and then use the decryption key Decrypt the aggregated encrypted vector to obtain the aggregated vector ; Finally, execute Algorithm 1 to restore the aggregation vector .

4. The method according to claim 3, characterized in that The privacy-preserving union protocol is implemented by the following steps: Compute union : Each client first sets its capacity parameter Mapped to a dimensional vector ,in: like , then the corresponding vector for ,in It is from Then, these vectors are transmitted to the cloud server for summation; the threshold vector aggregation protocol is used to implement vector summation.

5. The method according to claim 4, characterized in that: When using the threshold vector aggregation protocol to implement vector summation, the client Perform client-side encryption steps on the vector Encrypted and uploaded to the cloud server for aggregation; after receiving all encrypted vectors, the server performs the aggregator aggregation step to privately calculate the aggregate vector , then by Reconstructed ,Right now .

6. The method according to claim 1, wherein: In the initialization phase, At the beginning of a training round, the central server initializes the global model , randomly selected Online Clients , and set the parameters Send to these clients; Each client needs to Select an appropriate capability parameter ,and disclose it to the cloud server for generating heterogeneous models.,The PUP protocol is used to enable the client to disclose the,joint set of its capability parameters to the cloud server without,disclosing its privacy; After executing PUP, the cloud server can only obtain the joint set of all client capability parameters , but cannot access the specific parameter values of any single client; Server according to Generate heterogeneous models , and each model Assigned to the corresponding edge node ,At the same time, the corresponding allocation information of the model and edge nodes is shared with the client; The client connects to the corresponding edge node to download the local model; connect to the edge node The client set is denoted as .

7. The method according to claim 6, characterized in that: After the initialization phase, each client establishes a connection with its corresponding edge node and performs local training through the edge node. All clients connected to the same edge node are trained using a local model with the same structure. The training process follows the federated learning workflow, which includes: The client downloads the local model from the corresponding edge node; The client uses the local dataset to train the downloaded local model and obtain the trained model parameters; The client perturbs and encrypts the trained model parameters to protect their privacy. The encrypted model parameters are uploaded by the client to the corresponding edge node, which performs secure aggregation operations.

8. The method according to any one of claims 1 to 7, characterized in that: In the secure aggregation stage, a hybrid secure aggregation protocol H-Sagg based on the threshold vector aggregation protocol is used to aggregate model parameters. In the key generation step, the key for homomorphic encryption and the noise parameters for implementing Renyi differential privacy are generated simultaneously; the noise addition operation is integrated into the key generation algorithm, so that each client automatically adds noise when encrypting its model parameters.

9. The method according to claim 8, characterized in that The hybrid security aggregation protocol H-Sagg is implemented by the following steps: First, edge nodes choose A random seed is used to generate a random vector and transmit it to the client , these seeds are used to generate The length is Random vectors of random matrices row; Then, the client Use its local data for local training to obtain local parameters , the execution threshold is Parameter pruning and generating a secret vector , and an error vector , whose elements come from a Gaussian distribution ;in, , is the size of the domain, is the width of the error distribution; Then, the client Generate encryption keys , and calculate the encrypted model parameters , and send it to the edge node ; After receiving all encryption vectors, the edge node It can be aggregated to obtain , the calculation process is expressed as the following formula: 。 10. The method according to claim 9, characterized in that: right Decryption, edge node The aggregated secret vector needs to be calculated , and then obtain the aggregation model that satisfies differential privacy ; The threshold vector aggregation protocol is used to aggregate the client's secret vector at the edge node to obtain ; Then, the edge node Aggregate multiple encryption vectors to obtain an aggregated secret vector , and calculate the aggregated model , the model satisfies -RDP, where ; edge nodes After the aggregation is completed, the aggregation model that satisfies differential privacy is submitted to the cloud server for final aggregation; The cloud server uses a heterogeneous model aggregation method to aggregate the models from edge nodes and obtain an updated global model.

Images