A Method and Device for Constructing a Cross-Island Heterogeneous Federated Learning System Based on Homomorphic Encryption

Abstract

This invention discloses a method and device for constructing a cross-island heterogeneous federated learning system based on homomorphic encryption. After local training on the client, the local model is packaged and sparsified at the packet granularity. The sparsed packet data is homomorphically encrypted, and the ciphertext and mask are sent to a parameter server. The parameter server aggregates the data according to the client's ciphertext weights and sends the aggregated ciphertext global model and mask to the client. The client decrypts the global ciphertext and unpacks it to obtain the global model, and updates its local model. The client calculates and returns a sketch of its local model. By clustering the sketch, the fastest client in each cluster is selected as a participant in the next round of training, and the contribution is calculated based on the sketch to obtain the aggregation weights for the next round. The trained local model is then applied to the cross-island heterogeneous federated learning system. This invention can improve the efficiency of homomorphic encryption while protecting privacy, and simultaneously resist the negative impact of lagging on model synchronization.

Description

Technical Field

[0001] This invention belongs to the fields of computer network security technology and cryptography technology, and relates to a method and device for constructing a cross-silo heterogeneous federated learning system. Specifically, it relates to a method and device for constructing a cross-silo federated learning system based on sketch-based client selection and Packed Homomorphic Encryption (PHE). Background Technology

[0002] Federated learning across isolated systems enables multiple institutions and clients to collaboratively build a global model without leaking private data. To avoid privacy leaks during aggregation, homomorphic encryption, a widely used cryptographic technique, is used to encrypt model updates, allowing direct aggregation of model updates on ciphertext. However, the computationally intensive cryptographic operations of homomorphic encryption lead to high computational and communication overhead. Packetized homomorphic encryption can package and encrypt multiple plaintexts into a single ciphertext, performing parallel encryption and decryption operations on multiple plaintexts to reduce overhead. However, existing packetized homomorphic encryption does not consider the heterogeneity of different clients, making it difficult to deploy in practice. On the one hand, data is not independently and identically distributed across different clients, i.e., statistical heterogeneity, which leads to differences in local models and affects convergence performance. The encrypted transmission brought by homomorphic encryption further reduces training efficiency. On the other hand, clients have different computing capabilities and communication bandwidths, i.e., system heterogeneity, which leads to the straggler problem, significantly reducing training efficiency under computationally intensive encryption and decryption operations.

[0003] Currently, two common methods for addressing heterogeneity issues are weight aggregation and client selection. Weight aggregation assigns different aggregation weights based on the contribution of each client's local model. Since not all clients are equally important, a subset of clients with faster training speeds and better data quality can be selected as participants in the training process. This can significantly alleviate the problem of lagging models without sacrificing model accuracy. However, these methods operate in plaintext without any encryption to protect privacy. Therefore, how to apply packetized homomorphic encryption to protect privacy in heterogeneous federated learning systems is a pressing challenge. Summary of the Invention

[0004] The purpose of this invention is to provide a method and device for building a secure and efficient cross-island federated learning system using packaged homomorphic encryption, in order to solve the problem of client heterogeneity.

[0005] The technical solution adopted by the method of the present invention is: a method for constructing a cross-island heterogeneous federated learning system based on homomorphic encryption, wherein the client-side local model of the cross-island heterogeneous federated learning system is obtained through training; the training process specifically includes the following steps:

[0006] Step 1: At the start of each round of training of the local model on the client side, each client trains and updates the local model multiple times;

[0007] Step 2: The client expands and packages its local model, performs sparsification at the package granularity, encrypts the sparsified package data, and sends the ciphertext and mask to the parameter server for aggregation;

[0008] Step 3: The parameter server aggregates the received encrypted ground model and corresponding encrypted weights, and sends the encrypted global model and corresponding mask to the client.

[0009] Step 4: The client decrypts and unpacks the global ciphertext to obtain the global model and updates its local model;

[0010] Step 5: The client calculates and sends a sketch of its local model to the parameter server;

[0011] Step 6: By clustering the sketches, select the fastest client from each cluster, and select a subset of all clients as participants in the next round of training. Calculate the contribution based on the sketches to obtain the aggregate weights for the next round.

[0012] Step 7: Once the preset conditions are met, training ends, and the trained local client model is obtained and applied to the cross-island heterogeneous federated learning system.

[0013] Preferably, in step 2, the data is first packaged and then sparsified. The client updates the package on the local model and calculates the gate threshold θ based on the sparsity ratio s%. When the mean of the package data is greater than θ, the corresponding package mask is set to 1. Then, the selected package data is encrypted, and the corresponding ciphertext is... and mask Send to the parameter server.

[0014] Preferably, in step 3, the parameter server assigns different aggregation weights based on the client's contribution to the global model during aggregation, where the contribution is determined by the similarity of the client's sketches; aggregation weights β is a positive number. For the selected client, For client parameters, This represents the similarity between the two local model updates.

[0015] Preferably, in step 5, the client uses the LSH function to reduce the dimensionality and obtain a sketch of the local model.

[0016] Preferably, in step 6, the parameter server automatically determines the number of clusters based on the sketch information using the GapStatistic interval statistic during clustering. Where G(·) is the interval statistic function, For client collection, For the client's sketch, γ is the threshold value that limits the maximum number of clusters;

[0017] After obtaining the optimal number of clusters C, K-means clustering is performed. If clients in the same class have similar sketches, then a representative client from that class is selected to participate.

[0018] As a preferred option, in step 6, the parameter server prioritizes the client that trains the fastest. The parameter represents the order in which the server receives the i-th client, taking into account the historical performance of the previous t-1 rounds. The client with the shortest synthesis time is selected as the representative of the current class. α is the influence factor; for a class with only one client, it is directly added to the candidate set.

[0019] The technical solution adopted by the device of the present invention is: a device for building a cross-island heterogeneous federated learning system based on homomorphic encryption, comprising:

[0020] One or more processors;

[0021] A storage device for storing one or more programs, which, when executed by one or more processors, enable the one or more processors to implement the method for constructing a cross-island heterogeneous federated learning system based on homomorphic encryption.

[0022] The technical solution adopted by the medium of the present invention is: a storage medium storing a program, which, when executed by a processor, implements the method for constructing a cross-island heterogeneous federated learning system based on homomorphic encryption.

[0023] Compared with the prior art, the beneficial effects of the present invention are:

[0024] (1) The present invention provides a high-efficiency and lag-resistant homomorphic encrypted heterogeneous federated learning system (FedPHE). Existing technologies do not fully consider the heterogeneity of federated learning, lack solutions for lag during training, and struggle to achieve both efficiency and security while protecting the privacy of client data. The present invention performs secure client selection and ciphertext weight aggregation in encrypted mode, while considering the use of packing and sparsification techniques to address the computational and time overhead caused by homomorphic encryption. It also solves the challenges caused by the heterogeneity of client data and the system, achieving the advantages of both privacy protection and efficiency.

[0025] (2) This invention proposes an efficient, encrypted, contribution-aware weight aggregation method that assigns different aggregation weights based on the client's contribution to the global model. Existing technologies typically use plaintext weights, leaking client privacy information, and the weights are determined by the client's sample size ratio, remaining a constant value during iterative training without considering changes in the model's contribution to global updates. This invention considers the contribution to be determined by the similarity between the sketches sent by the client to the parameter server in two rounds. The parameter server calculates the probability of sketch collisions between the two rounds to approximate the Jaccard similarity of the local models in the two rounds; the lower the similarity, the more beneficial it is to the global model. The process of calculating model similarity using dimensionality-reduced sketches is much more efficient than the commonly used cosine similarity. This invention utilizes the CKKS feature supporting encrypted multiplication to complete encrypted weight aggregation.

[0026] (3) This invention proposes packet-level sparsity to further optimize data transmission efficiency. Existing technologies directly use homomorphic encryption to protect privacy, which leads to significant communication overhead under bandwidth-constrained conditions. This invention fully considers communication load, first packing the data, then sparsifying the packets, and finally performing encryption. It combines the advantages of packet encryption and sparsity, solving the problem of high ciphertext overhead caused by CKKS, and making homomorphic encryption practically valuable in federated learning.

[0027] (4) This invention utilizes the Local-Sensitive Hashing (LSH) algorithm to calculate a sketch of the local model, achieving efficient client selection in a closed state, and comprehensively considering data distribution and resource availability. Existing technologies directly transmit plaintext data to calculate similarity, which, on the one hand, leaks privacy information, and on the other hand, involves a large amount of data transmission and high communication overhead. This invention reduces the dimensionality of the local model using multiple hash functions, mapping high-dimensional model parameters to low dimensions. The use of hash functions is highly efficient, and the reduced-dimensional content has high privacy, which can be considered as a closed-state transmission, without leaking model information during transmission. At the same time, selecting the fastest client from each category can greatly alleviate the problem of lagging without sacrificing accuracy.

[0028] (5) This invention does not require a parameter server to fix the number of clusters, while existing technologies usually require initial determination of the number of clusters, but it is difficult to find the optimal number of clusters. This invention automatically determines the optimal number of clusters based on the sketch using the gap statistic method, and uses K-means to complete the clustering algorithm, which has good robustness. Attached Figure Description

[0029] The technical solutions described herein are further illustrated below using examples and specific implementation methods. Additionally, accompanying drawings are used in the description of the technical solutions. Those skilled in the art can, without any creative effort, obtain other drawings and the intent of the present invention based on these drawings.

[0030] Figure 1 This is a deployment diagram of a cross-island heterogeneous federated learning system according to an embodiment of the present invention;

[0031] Figure 2 This is a schematic diagram of the CKKS packaging mode according to an embodiment of the present invention;

[0032] Figure 3 This is a flowchart of the Top-k sparsity process for homomorphic encryption in an embodiment of the present invention. Detailed Implementation

[0033] To facilitate understanding and implementation of the present invention by those skilled in the art, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0034] Cross-island federated learning enables multiple clients to collaboratively train a global model without sharing local data. To avoid privacy leaks during aggregation, homomorphic encryption is widely used in federated learning to encrypt local model updates, but this leads to significant computational and communication overhead. To reduce this overhead, packed homomorphic encryption (PHE) encrypts multiple plaintexts into a single ciphertext. However, PHE ignores the heterogeneity between clients, resulting in decreased training efficiency and lag issues.

[0035] This embodiment proposes an efficient packed homomorphic encryption scheme that incorporates sparsity, ciphertext weight aggregation, and client selection to address the heterogeneity problem. The technologies involved in this embodiment include packed homomorphic encryption (PHE), pack-level sparsity techniques, contribution-aware weight aggregation mechanisms, and sketching-based client selection mechanisms.

[0036] First, we give the formal definition of the symbol for round t in this system:

[0037] Homomorphic encryption function E(·);

[0038] Locality-sensitive hash function

[0039] Interval statistic function G(·);

[0040] Client Collection

[0041] Client parameters

[0042] Client sketch

[0043] Client-side aggregate weight

[0044] Selected clients

[0045] ciphertext received by the parameter server

[0046] The mask received by the parameter server

[0047] Global ciphertext and mask C aggregated by parameter server t M t ;

[0048] The sparsity ratio is s%.

[0049] The threshold value θ for sparsification.

[0050] Please see Figure 1 This embodiment provides a method for constructing a cross-island heterogeneous federated learning system based on homomorphic encryption. The client-side local model of the cross-island heterogeneous federated learning system (FedPHE) is obtained through training. The training process specifically includes the following steps:

[0051] Step 1: At the start of each round of training of the local model on the client side, each client trains and updates the local model multiple times;

[0052] Step 2: The client expands and packages its local model, performs sparsification at the package granularity, encrypts the sparsified package data, and sends the ciphertext and mask to the parameter server for aggregation;

[0053] Step 3: The parameter server aggregates the received encrypted ground model and corresponding encrypted weights, and sends the encrypted global model and corresponding mask to the client.

[0054] Step 4: The client decrypts and unpacks the global ciphertext to obtain the global model and updates its local model;

[0055] Step 5: The client calculates and sends a sketch of its local model to the parameter server;

[0056] Step 6: By clustering the sketches, select the fastest client from each cluster, and select a subset of all clients as participants in the next round of training. Calculate the contribution based on the sketches to obtain the aggregate weights for the next round.

[0057] Step 7: Once the preset conditions are met, training ends, and the trained local client model is obtained and applied to the cross-island heterogeneous federated learning system.

[0058]

[0059]

[0060] Another specific implementation method is also provided here. In order to improve the efficiency of the general homomorphic encryption mechanism, packed homomorphic encryption (PHE) is used, such as... Figure 2As shown, multiple plaintexts can be packaged and encrypted into a single ciphertext simultaneously, enabling parallel encryption and decryption operations on multiple plaintexts. CKKS, as a fully homomorphic encryption mechanism, is superior to Paillier and BFV homomorphic encryption mechanisms. It supports encryption directly on real number vectors, while Paillier and BFV only support encryption on integer plaintexts, requiring additional quantization mechanisms to convert floating-point numbers to integers. Furthermore, CKKS supports homomorphic multiplication, enabling secure weight aggregation on the ciphertext, whereas Paillier does not support ciphertext multiplication, and BFV can cause negative number overflow when using ciphertext multiplication after quantization. Therefore, using CKKS in conjunction with packaged homomorphic encryption allows for homomorphic aggregation on the ciphertext to be completed on the parameter server side, offering both security and efficiency.

[0061] Furthermore, sparsity, as a commonly used communication optimization method, is exemplified by Top-k sparsity, where each client selects the k largest parameters for transmission. However, sparsity is typically used at the scalar level. When data is packaged and encrypted, the discontinuous mask and the invisible internal structure can cause the parameter server to misalign the ciphertext, resulting in incompatibility between sparsity and packaged homomorphic encryption, making it impossible to simultaneously achieve the optimization effects of both. This invention, as described above... Figure 3 As shown, the process involves first packaging and then sparsifying. The client updates the packaged data of the local model and calculates the gate threshold θ based on the sparsity ratio s%. When the mean of the package data is greater than θ, the corresponding package mask is set to 1. Subsequently, the selected package data is encrypted, and the corresponding ciphertext is... and mask Send it to the parameter server. The parameter server can align the ciphertext according to the mask; the ciphertext here is the result of encrypting multiple plaintext packets.

[0062] Another specific implementation is provided here: during aggregation, the parameter server assigns different aggregation weights based on the client's contribution to the global model. The contribution is determined by the similarity of the client's sketches, and Locality-Sensitive Hashing (LSH) is used to approximate Jaccard similarity (JS). The parameter server calculates the probability of collision between client sketches in two rounds to estimate the similarity between two local updates. The lower the similarity, the greater the potential performance improvement for the global model, and therefore the higher the aggregation weight should be assigned. Right now β is a positive number used to correct the exponential curve. Next, the parameter server performs weight aggregation on the received ciphertext and corresponding mask. The weights can be calculated based on the contribution. Leveraging the CKKS algorithm's property of supporting ciphertext multiplication, the global model is obtained directly by aggregation on the ciphertext. The aggregated global ciphertext model C is obtained.t and the corresponding global mask M t It is then sent to the client for decryption and model updates.

[0063]

[0064] Another specific implementation is provided here, addressing the lag problem caused by client heterogeneity through client selection. In practice, different clients may send similar model updates to the parameter server, leading to unnecessary communication overhead. Existing client selection methods are all performed in plaintext, posing a risk of privacy leakage. This invention, based on client sketches, enables client selection in a closed state. In each round, the client calculates and sends a sketch of its local model to the parameter server, which then clusters these sketches, grouping similar sketches into the same class and selecting the fastest client from each class.

[0065] Locality Sensitive Hash (LSH) is a type of hash function that indicates a higher similarity during a collision when two inputs have the same hash value. The client uses the LSH function to reduce dimensionality and obtain a sketch of the local model. This sketch provides a more compact representation of the local model, ensuring that information from the original model is not leaked, making it an efficient communication method. When clustering, the parameter server does not need to specify the number of clusters; instead, it automatically determines the number of clusters based on the sketch information using the GapStatistic interval statistic. The interval statistic is calculated by comparing the expected value of the true clusters with that of a distribution generated by Monte Carlo simulation, where γ is a threshold value limiting the maximum number of clusters. After obtaining the optimal number of clusters C, K-means clustering is performed. Clients within the same class have similar sketches. According to the properties of the LSH function, similar client sketches indicate that clients within the same class have similar local models. Therefore, a representative client can be selected from one class to participate. However, the selection process does not involve randomly choosing a client from a class; the parameter server prioritizes clients that train faster. The parameter represents the order in which the server receives the i-th client, taking into account the historical performance of the previous t-1 rounds. The client with the shorter synthesis time is selected as the representative of the current class. α is the influence factor. If a class has only one client, it is directly added to the candidate set. Compared with traditional similarity methods, such as cosine similarity, the sketch method is not only more efficient but also operates in a dense state. It uses a hash function to map the high-dimensional model update to a low-dimensional one, ensuring that information is not leaked.

[0066]

[0067] This embodiment also provides a device for building a cross-island heterogeneous federated learning system based on homomorphic encryption, including:

[0068] One or more processors;

[0069] A storage device for storing one or more programs, which, when executed by one or more processors, enable the one or more processors to implement the method for constructing a cross-island heterogeneous federated learning system based on homomorphic encryption.

[0070] This embodiment also provides a storage medium storing a program, which, when executed by a processor, implements the method for constructing a cross-island heterogeneous federated learning system based on homomorphic encryption.

[0071] The packet encryption sparsity technique proposed in this invention is not only applicable to the CKKS homomorphic encryption algorithm, but can also be extended to other homomorphic encryption algorithms, such as Paillier and BFV, and has good scalability.

[0072] It should be understood that the above description of the preferred embodiments is quite detailed, but it should not be considered as a limitation on the scope of protection of this invention. Those skilled in the art, under the guidance of this invention, can make substitutions or modifications without departing from the scope of protection of the claims of this invention, and all such substitutions or modifications fall within the scope of protection of this invention. The scope of protection of this invention should be determined by the appended claims.

Claims

1. A method for constructing a cross-island heterogeneous federated learning system based on homomorphic encryption, characterized in that: The cross-island heterogeneous federated learning system obtains its client-side local model through training. The training process specifically includes the following steps: Step 1: At the start of each round of training of the local model on the client side, each client trains and updates the local model multiple times; Step 2: The client expands and packages its local model, performs sparsification at the package granularity, encrypts the sparsified package data, and sends the ciphertext and mask to the parameter server for aggregation; Step 3: The parameter server aggregates the received encrypted ground model and corresponding encrypted weights, and sends the encrypted global model and corresponding mask to the client. During aggregation, the parameter server assigns different aggregation weights based on the client's contribution to the global model; the contribution is determined by the similarity of the client's sketches. , It is a positive number. For the selected client, For client parameters, The similarity between the two local model updates; Step 4: The client decrypts and unpacks the global ciphertext to obtain the global model and updates its local model; Step 5: The client calculates and sends a sketch of its local model to the parameter server; Step 6: By clustering the sketches, select the fastest client from each cluster to select a subset of all clients as participants in the next round of training, and calculate the contribution based on the sketches to obtain the aggregation weights for the next round. Step 7: Once the preset conditions are met, training ends, and the trained local client model is obtained and applied to the cross-island heterogeneous federated learning system.

2. The method for constructing a cross-island heterogeneous federated learning system based on homomorphic encryption according to claim 1, characterized in that: In step 2, the model is first packaged and then sparsified. The client updates the package on the local model and calculates the threshold based on the sparsity ratio s%. The average value of the package data is greater than At that time, the corresponding packet mask is set to 1, and then the selected packet data is encrypted, and the corresponding ciphertext is generated. and mask Send to the parameter server.

3. The method for constructing a cross-island heterogeneous federated learning system based on homomorphic encryption according to claim 1, characterized in that: In step 5, the client uses the LSH function to reduce the dimensionality and obtain a sketch of the local model.

4. The method for constructing a cross-island heterogeneous federated learning system based on homomorphic encryption according to claim 1, characterized in that: In step 6, the parameter server automatically determines the number of clusters based on the sketch information using the GapStatistic interval statistic during clustering. ;in, For interval statistics functions, For client collection, For the client's sketch, It is the threshold value that limits the maximum number of clusters; After obtaining the optimal number of clusters C, K-means clustering is performed. If clients in the same class have similar sketches, then a representative client from that class is selected to participate.

5. The method for constructing a cross-island heterogeneous federated learning system based on homomorphic encryption according to any one of claims 1-4, characterized in that: In step 6, the parameter server prioritizes the client that trains the fastest. Representative parameter server received the first i The order of the clients, taking into account their historical performance in the previous t-1 rounds. The client with the shortest overall time is selected as the representative of the current class. , It is an influence factor; for a class with only one client, it is directly added to the candidate set. .

6. A device for building a cross-island heterogeneous federated learning system based on homomorphic encryption, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the method for constructing a cross-island heterogeneous federated learning system based on homomorphic encryption as described in any one of claims 1 to 5.

7. A storage medium storing a program, characterized in that, When the program is executed by the processor, it implements the method for constructing a cross-island heterogeneous federated learning system based on homomorphic encryption as described in any one of claims 1 to 5.

Images