A vertical federated learning acceleration method based on an FPGA chip

Abstract

The application discloses a longitudinal federal learning acceleration method based on an FPGA chip, and provides a longitudinal federal learning method without a trusted third party and accelerates federal learning through the FPGA chip.The application accelerates a decryption algorithm and ciphertext scalar multiplication of a Paillier encryption protocol based on the FPGA, and embeds the module into the whole framework, so that the decryption algorithm and the ciphertext scalar multiplication of the Paillier encryption protocol are decoupled to be executed on the FPGA, and the CPU is only responsible for precalculation, forward propagation, data encryption and model updating of data such as a key and a random mask, so as to support efficient longitudinal federal learning based on the Paillier encryption protocol.Under the premise of not losing the data security of the two parties, the FPGA is embedded into the federal learning system, so that the speed of federal modeling can be significantly improved, and the iteration time of training is reduced.

Description

Technical Field

[0001] This invention belongs to the field of federated learning and edge computing technology, and relates to a vertical federated learning acceleration method based on FPGA chip. Background Technology

[0002] In recent years, the rapid development of artificial intelligence has led to its widespread application across various industries. In the financial sector, an increasing number of companies are applying AI technology to various business operations, such as fraud prevention and detection, and risk management, to increase the reliability of corporate decision-making.

[0003] However, a reliable machine learning model requires a large amount of data for training. Only with a sufficient number of training samples and a sufficiently broad feature dimension can the model's high reliability be guaranteed. Unfortunately, the problem of "data silos" is particularly prominent in the financial sector. For large enterprises, the number of samples is generally not a problem, but because the businesses they are responsible for are diverse, the feature dimensions of the data they collect also differ, leading to a lack of comprehensive feature dimensions. For small enterprises, they not only face the problem of insufficient feature dimensions from large enterprises, but also the problem of small sample sizes due to a small customer base. Faced with this reality, many enterprises have chosen data sharing to solve this problem. How to resolve the contradiction between data sharing and privacy protection has become a huge opportunity and challenge. Federated learning is an effective solution to the data sharing problem while protecting data security and privacy. Vertical federation, as a typical federated learning paradigm, can solve the problem of supplementing and expanding feature dimensions and richness in actual business operations, and is therefore widely used in scenarios such as joint risk control and joint marketing.

[0004] Currently, commonly used vertical federated learning frameworks generally divide participants into three roles: service requester, data provider, and coordinator. The service requester is typically the initiator of the business and possesses some features and labels; the data provider possesses other features but no labels. The service requester and data provider have significant overlap in sample IDs, and are collectively referred to as participants. The coordinator is usually a trusted third party, often referred to as a trusted third party. The coordinator has two functions: first, to assist participants in generating public and private keys; and second, to aggregate the results from both sides (e.g., gradients, losses). However, this current vertical federated learning framework has the following problems:

[0005] Question 1: In real-world scenarios, it is difficult to find a third party that all parties involved can trust;

[0006] Question 2: When performing aggregation operations, it is usually based on the aggregation of multi-party data in the ciphertext space. The encryption algorithm is inefficient, and the modeling speed is 4-5 orders of magnitude slower than the traditional method of directly aggregating data together. Summary of the Invention

[0007] To address the shortcomings of existing technologies, the purpose of this invention is to provide a vertical federated learning acceleration method based on FPGA chips. This invention provides a vertical federated learning method that does not require a trusted third party and accelerates federated learning through FPGA chips. Compared with other similar methods, the difference lies in the fact that this invention introduces FPGA-accelerated decryption algorithms and ciphertext scalar multiplication of the Paillier encryption protocol, and embeds this module into the entire hardware and software framework to support efficient federated learning based on the Paillier encryption protocol.

[0008] To address the difficulty in finding a trustworthy third party for all participants in existing technologies, this solution adopts an architecture without a trusted third party, employing a dual-FPGA architecture. The two FPGAs belong to two participants, A and B, respectively. The FPGA chip belonging to participant A is used for data decryption, while the FPGA chip belonging to participant B is used for homomorphic operations on the ciphertext. To address the issue of low encryption algorithm efficiency in multi-party data aggregation based on ciphertext space, this invention: 1) decouples data decryption and homomorphic operations, moving them to the FPGA. The CPU is only responsible for partial data pre-computation (e.g., key, random mask), forward propagation, and model updates; 2) to accelerate the execution efficiency of homomorphic encryption algorithms on the FPGA, this invention combines Montgomery modular multiplication and fast exponentiation to optimize modular multiplication and exponentiation operations; 3) while maximizing the utilization of FPGA resources (logic resources and DSP), multiple identical circuits are generated on the FPGA to perform some identical operations in parallel.

[0009] Logistic Regression (LR) is a common algorithm in machine learning, and due to its strong interpretability, it is widely used in modeling tasks. This invention takes longitudinal LR as an example. The overall framework diagram of the longitudinal federated learning framework of this invention is as follows: Figure 1 As shown, participant A and participant B each possess an FPGA chip. Functionally, the FPGA chip belonging to participant A is used to decrypt the encrypted gradient value transmitted from participant B; the FPGA chip belonging to participant B is used to calculate the gradient in the ciphertext domain. After participant A and participant B each perform forward propagation, participant B will use its predicted value... The data is transmitted to participant A. Participant A aggregates the predicted values ​​and calculates the difference (diff) between them and the labels. Then, participant A encrypts the difference and transmits the encrypted difference. The encrypted difference is given to participant B. Participant B transmits the encrypted difference to its subordinate FPGA chip, which then calculates the encrypted gradient and adds a random mask. After the calculation is complete, the ciphertext gradient is sent back to the CPU of participant B, and then transmitted by participant B to participant A. Participant A transmits the ciphertext gradient to its subordinate FPGA chip, which decrypts it to obtain the decrypted gradient grad+R. Finally, participant A transmits the plaintext gradient grad+R back to participant B.

[0010] In vertical LR scenarios, participants typically have datasets with the same sample space but different feature spaces.

[0011] The sample space refers to the samples used by the participants in performing logistic regression, with each sample containing different features. Participants having the same sample space means that the samples used for logistic regression are the same, i.e., they have the same sample ID.

[0012] The feature space refers to the features possessed by the samples, i.e., the input of the LR model. Different feature spaces for the participants mean that the samples have different feature dimensions and there is little or no overlap in the features between the samples.

[0013] Logistic Regression (LR) algorithm assumes that the data in the dataset follows a Bernoulli distribution and uses the cross-entropy loss function as the loss function. The form of the cross-entropy loss function is as follows: Where N represents the number of samples, y i Let p be the label of the i-th sample. i The prediction result is for the i-th sample; the loss function is used to represent the deviation between the predicted output and the true value, and finally the optimization algorithm is used to solve the parameters to achieve the purpose of classifying the data.

[0014] Vertical federated learning methods based on the logistic regression (LR) algorithm generally consist of two parts: sample secure alignment and modeling. Sample alignment can be accomplished using RSA, ECC, or OT-based secure alignment protocols. Assuming that all participants have completed sample alignment, the modeling process for vertical LR federated learning is as follows:

[0015] Step 1: The participants, including the service requester and the data provider, perform forward propagation to obtain their own prediction values. The data provider transmits its prediction values ​​to the service requester, and the business initiator (i.e. the service requester) aggregates the prediction results and calculates the difference with the labels. The labels are information that the service requester pre-generates and stores in its CPU memory before logistic regression.

[0016] Step 2: The service requester encrypts the difference obtained after calculating the aggregated predicted values ​​and transmits it to the data provider.

[0017] Step 3: The service requester directly uses the unencrypted difference to calculate the gradients of the weights and biases on its local CPU. The data provider directly performs ciphertext multiplication and ciphertext addition operations on the encrypted data to calculate the gradients of the LR model weights and biases and sends them back to the service requester. The ciphertext addition is calculated as follows: Where m1 and m2 are the plaintexts involved in the ciphertext addition. This represents the result of addition on the ciphertext field, where c1 and c2 are the ciphertexts of m1 and m2 respectively, g and n are public key parameters, and r is the confusion parameter; the ciphertext multiplication is calculated as follows: Where m1 and m2 are the plaintexts involved in the ciphertext multiplication. This represents the result of multiplication over the ciphertext field, where c1 is the ciphertext after encrypting m1, g and n are public key parameters, and r is the obfuscation parameter.

[0018] Step 4: The service requester decrypts the ciphertext gradient and returns it to the data provider. The data provider subtracts the corresponding random mask from the received plaintext gradient to obtain the final gradient value.

[0019] Step 5: Each participant updates the model according to the gradient; if the termination condition is not met, continue to repeat steps 2 to 5; otherwise, terminate training and obtain the final model.

[0020] After obtaining the final trained model through the above modeling process, training participants can use the model to predict new input data and classify the data based on the prediction results. For example, in risk detection, participants can use the trained model to predict customer risk and determine whether there is a fraud risk based on whether the output result is 0 or 1. The specific representation of 0 and 1 will be determined based on the actual training model.

[0021] The present invention summarizes the above process according to the roles of the participating parties as shown in Table 1.

[0022] Table 1. Modeling process of LR algorithm

[0023]

[0024]

[0025] According to Table 1, the modeling flowchart is as follows: Figure 3 As shown.

[0026] Analysis of the modeling process reveals that the Logistic Regression (LR) model typically consists of linear layers and softmax layers. The output of the linear layer is fed into the softmax layer to obtain the final predicted value. Because the cross-entropy loss function has a fast convergence speed and is simple to differentiate, it is generally used as the loss function. During backpropagation, since the cross-entropy loss function is used, the gradient calculation result is simple to represent, and can be expressed using scalar multiplication and addition of ciphertext. The Paillier encryption protocol supports the required computations and has a simple algorithm; therefore, this invention adopts the Paillier encryption protocol.

[0027] The Paillier encryption protocol is a semi-homomorphic encryption protocol that supports two types of ciphertext space operations: scalar addition of two ciphertexts and scalar multiplication of a plaintext and a ciphertext. Assuming N parties participate in the computation, the Paillier encryption protocol mainly consists of the following four steps: First, any computation party A randomly generates two large prime numbers (usually 1024 or 2048 bits) to generate a public key and a private key, and publishes the public key to all computation parties; Second, each participating computation party uses the public key to encrypt the data and sends the ciphertext to any computation party B except the private key holder; Third, party B directly operates on the ciphertext and then sends the result back to computation party A; Finally, computation party A decrypts the result.

[0028] Specifically, the modeling and application process of this invention is mainly divided into the following six steps, of which steps 1 to 5 are model training and establishment, and step 6 is using the model for prediction.

[0029] Step 1: Forward Propagation

[0030] LR models typically consist of a linear layer and a softmax layer, as shown in the model below. Figure 2 As shown. The output z of the linear layer can be expressed as z = W T x+b, where W and b are the weights and biases of the linear layer, respectively, and x is the input feature vector of the participating party. The Softmax layer can be represented as... For an LR model with only one linear layer, its output can be represented as: Based on the above calculation method, participant A and participant B respectively input their private data (feature vectors) into the model for forward propagation and obtain their respective predicted values. Then, participant B uses the predicted value... The prediction is transmitted to participant A via the channel. Next, participant A submits their prediction. And participant B will predict the value Aggregate to obtain aggregated predicted values. And calculate the difference between the label and the aggregated predicted value. The label y is information that is pre-generated by the service requester and stored in its CPU memory before logistic regression.

[0031] Step 2: Encrypt the difference

[0032] During system initialization, participant A's CPU pre-generates a public key (n, g) and a private key (λ, μ). The generation of both public and private keys is implemented using the GMP large number library in C++. Before key generation, two random large prime numbers p and q need to be generated. Using these two prime numbers, the public key is generated as follows: n = p * q, g = n + 1; the private key is generated as follows: λ = (p - 1) * (q - 1), μ = λ - 1. The public key is publicly disclosed and stored in the CPU memory of both participants A and B, as well as on the FPGA chip belonging to participant B. The private key is only stored on the FPGA chip belonging to participant A. Participant A uses the public key to encrypt the difference (diff) calculated in step 1 to protect its own privacy data (tag value). The encryption algorithm is as follows: in `diff` is the difference after encryption, `g` and `n` are the public key parameters, and `r` is a randomly generated obfuscation parameter coprime to `n`. Although encryption is also an inefficient computation, and there are related technologies dedicated to improving its efficiency, the amount of data to be encrypted is much smaller than that of decryption, so encryption is still performed on the CPU.

[0033] Step 3: Gradient Calculation

[0034] LR models typically use the cross-entropy loss function as the loss function. The cross-entropy loss function in this invention is: Where N represents the number of samples, y i Let p be the label of the i-th sample. i Let be the prediction result for the i-th sample. Then the gradients of the LR model weights and biases can be calculated as follows:

[0035]

[0036]

[0037] Where input is the sample input feature, L is the loss function, w is the weight, and b is the bias.

[0038] ①Party A

[0039] Participant A directly uses the unencrypted difference (diff) on its local CPU to calculate the gradients of the weights and biases.

[0040] ②Party B

[0041] During the system initialization phase, participant B will pre-generate a set of random masks on the CPU to avoid unnecessary delays caused by generating random masks in subsequent operations, and encrypt them on the CPU using a public key; the random mask is a random number randomly generated using the C++GMP large number library.

[0042] Calculating the gradient of participant B requires scalar multiplication of the ciphertext. The scalar multiplication of the ciphertext is calculated as follows: in, This represents the encrypted result of plaintext m multiplied by plaintext a. d' is the ciphertext, 'a' is the plaintext, and 'n' is the public key parameter. This operation involves modular exponentiation. This invention combines Montgomery modular multiplication and fast exponentiation to accelerate modular exponentiation. Montgomery modular multiplication is an algorithm used to accelerate modular multiplication of large numbers. It replaces the original resource- and time-consuming division modulo operation with a shift operation, thus accelerating the algorithm. Its algorithm flow is shown in Algorithm 1. Here, 'a' and 'b' are the two elements involved in the multiplication, 'n' is the modulus of the public key parameter, 'd'' is the output of the Montgomery modular multiplication, 'r' is 2 to the power of k, and 'k' is a positive integer.

[0043]

[0044] This invention combines fast exponentiation and Montgomery modular multiplication to achieve modular exponentiation, and its algorithm flow is shown in Algorithm 2. Since Montgomery modular multiplication accelerates modular multiplication operations, and modular exponentiation is composed of modular multiplication, the combination of fast exponentiation and Montgomery modular multiplication speeds up modular exponentiation operations.

[0045]

[0046]

[0047] Furthermore, since the Montgomery modular multiplication unit requires a large amount of logic resources and DSP, this invention reuses the Montgomery modular multiplication unit required to implement the Montgomery exponentiation module on an FPGA, reducing the resources (logic resources and DSP) required for modular exponentiation operations. This allows multiple Montgomery exponentiation modules to be implemented on a single FPGA, enabling parallel computation of ciphertext scalar multiplication. After the gradient calculation is complete, the result is transmitted to the CPU of participant B via the output buffer on the FPGA chip belonging to participant B. Participant B then adds an encrypted random mask to the ciphertext gradient. The encryption method is the same as the difference encryption method to protect the privacy data of participant B. The calculation result is... Whenever a set Participant B then transmits the result to Participant A for decryption without waiting for all computations to complete. This approach allows for parallel gradient calculation and decryption operations, further improving the efficiency of federated learning. The computation flow of the FPGA chip belonging to Participant B is as follows: Figure 4 As shown. Participant B will encrypt the difference. Features of the i-th sample of participant B The data is transmitted to the input buffer on the FPGA and temporarily stored there. When the Montgomery modular multiplication module corresponding to the input buffer is idle, the Montgomery modular multiplication module reads the data from the input buffer for processing, according to the formula... For the encrypted gradient, The encrypted difference is given by input, where input is the sample input feature, and n is the public key parameter. Substituting the input, the calculated result is: The calculation results will be temporarily stored in the output buffer, waiting for participant B to read them.

[0048] Step 4: Gradient Decryption

[0049] This step only involves participant B. Once the FPGA chip belonging to participant B has calculated a set of encryption gradients, it sends the result to participant A. When the FPGA chip belonging to participant A is idle, the CPU of participant A inputs the result into the input buffer on the FPGA chip belonging to participant A. The FPGA chip belonging to participant A uses the decryption module to decrypt the gradients in parallel, using the decryption method m = L(c λ mod n 2 )mod n 2 ,in, c is the ciphertext, n is the public key parameter, and λ is the private key parameter. The calculation result, i.e., the plaintext m, is first temporarily stored in the output buffer on the FPGA chip and then sent back to the CPU memory of participant A. Then, participant A sends the decrypted gradient grad+R back to participant B. Participant B subtracts the corresponding random mask R from the received decrypted gradient to obtain the final gradient value. This operation is repeated until all gradients are decrypted. The parallel operation configuration of participants A and B allows the FPGA chip belonging to participant B to perform gradient calculation while the FPGA chip belonging to participant A can perform gradient decryption, minimizing the idle state of any FPGA chip and achieving parallel operation. Through the dual FPGA chip architecture, this invention realizes the above pipeline operation. The operation flow of the FPGA chip belonging to participant A is as follows: Figure 5 As shown.

[0050] Step 5: Gradient Update

[0051] To reduce the frequency of ciphertext operations, this invention selects the minibatch-SGD method for gradient updates. The minibatch-SGD method involves processing one batch of samples at a time, calculating the gradient for each sample, accumulating the gradients, and then updating the gradient. The gradient of this method can be calculated as follows: Where grad represents the total gradient, grad i Let represent the gradient of the i-th sample, and N represent the number of samples in a batch. The updated weights are represented as w = w - η·grad, and the updated biases are represented as b = b - η·grad, where η is the learning rate. Before training begins, a training epoch is pre-selected, representing the maximum number of training epochs. Training stops when the maximum number of training epochs is reached. Simultaneously, an expected training accuracy (prediction accuracy) is set. Training stops even if the maximum number of training epochs has not been reached when the expected training accuracy is reached. Training stops when the expected number of training epochs or the expected accuracy is reached. Otherwise, steps 2 to 5 are repeated.

[0052] Step 6: Model Prediction

[0053] Training stops once the model has reached the expected number of epochs or achieved the expected accuracy, yielding the final prediction model, which consists of the weights and biases obtained from the last gradient update. Participants in the training process can then use this model for prediction. Similar to training, participants input their private data (i.e., features) into the model, and the prediction model outputs different predictions based on different inputs. Participants can then make appropriate decisions based on these predictions.

[0054] The present invention also provides an FPGA chip for use in the above method, wherein the FPGA chip has different functions depending on the participating parties; the participating parties include service requesters and data providers;

[0055] The FPGA chip of the service requester includes an input buffer, an output buffer, and a decryption unit; wherein the decryption unit includes one or more decryption modules;

[0056] The FPGA chip of the data provider includes an input buffer, an output buffer, and a ciphertext multiplication unit; wherein, the ciphertext multiplication unit includes one or more Montgomery power module modules for implementing ciphertext scalar multiplication operations; the Montgomery power module module includes a Montgomery modular multiplication unit.

[0057] The FPGA-based method for accelerating vertical federated learning designed in this invention has implications for the field of vertical federated learning and other individual classification fields.

[0058] The beneficial effects of this invention include: without compromising the data security of both participating parties, embedding an FPGA into a federated learning system can significantly improve the speed of federated modeling, thereby reducing training iteration time. Simultaneously, because FPGA chips are inexpensive, far smaller than dedicated hardware acceleration chips, using a dual-FPGA chip encryption method does not incur excessive additional costs, but offers significant time savings.

[0059] This invention, based on a financial application scenario, contrasts with existing implementations entirely on CPUs. In the same application scenario, this invention's FPGA-based vertical federated learning acceleration method can improve the speed of calculating the ciphertext gradient (participant B) and decrypting the ciphertext gradient (participant A), reducing the time required for one iteration and thus reducing the overall training time. In this scenario, this invention implements four ciphertext multiplication units on participant B's FPGA chip. The average time for each ciphertext multiplication unit to perform one ciphertext multiplication is similar to the time required on a CPU. However, because multiple multiplication units are implemented, ciphertext multiplication can be computed in parallel, thus reducing the required time to [a fraction of the original time]. This invention implements two decryption units on the FPGA chip of participant A, reducing the average decryption time of one data point by 40.5%. Although it increases the transmission latency from the CPU to the corresponding FPGA chip compared to the version implemented entirely by the CPU, the time required for one iteration is still significantly reduced, by approximately 41%. Furthermore, compared to the CPU-implemented version, the FPGA used in this invention is less expensive than the CPU, effectively reducing the system cost. Attached Figure Description

[0060] Figure 1 This is a diagram of the overall hardware and software framework of the present invention.

[0061] Figure 2 This is a structural diagram of the LR model used in this invention.

[0062] Figure 3 This is a flowchart for modeling federated learning based on FPGA chips.

[0063] Figure 4 This is a flowchart of the FPGA chip operation belonging to participant B in this invention.

[0064] Figure 5 This is a flowchart of the FPAG chip operation belonging to participant A in this invention.

[0065] Figure 6 The flowcharts for steps 2 to 8 of this embodiment of the invention are shown below. Detailed Implementation

[0066] The invention will be further described in detail below with reference to the specific embodiments and accompanying drawings. Except for the contents specifically mentioned below, the processes, conditions, and experimental methods for implementing the invention are all common knowledge and general knowledge in the art, and the invention does not have any particular limitations.

[0067] This invention presents a vertical federated learning method that does not require a trusted third party. Unlike other similar methods, this invention provides a heterogeneous vertical federated learning framework consisting of a CPU and an FPGA. This invention accelerates the decryption algorithm and ciphertext scalar multiplication of the Paillier encryption protocol using the FPGA and embeds this module into the entire framework. Thus, the decryption algorithm and ciphertext scalar multiplication of the Paillier encryption protocol are decoupled and executed on the FPGA. The CPU is only responsible for data pre-computation (such as keys, random masks, etc.), forward propagation, data encryption, and model updates, supporting efficient vertical federated learning based on the Paillier encryption protocol.

[0068] like Figure 1 As shown, this invention proposes a vertical federated learning acceleration method based on FPGA chips. The structure of this vertical federated learning model mainly includes three parts: a service initiator (participant A), a data provider (participant B), and FPGA chips belonging to the service initiator and data provider, respectively. The FPGA chip of the data provider and its computational flow are as follows... Figure 4 As shown, the FPGA chip and computing process of the service initiator are as follows: Figure 5 As shown. The Logistic Regression (LR) algorithm is a common algorithm in machine learning, and due to its strong interpretability, it is widely used in modeling tasks. Therefore, this method uses the longitudinal LR algorithm as the model for modeling and application. The modeling and application mainly includes the following six steps (steps 1 to 5 are model training and building, and step 6 is using the model for prediction):

[0069] Step (1): The service requester and the data provider conduct forward propagation to obtain their own prediction values. The data provider transmits its prediction values ​​to the service requester. Then, the service requester aggregates the prediction results and calculates the difference between the prediction results and the labels.

[0070] Step (2): The service requester uses the public key to encrypt the difference calculated in step (1) and transmits it to the data provider.

[0071] Step (3): The service requester directly uses the unencrypted difference to calculate the gradients of the weight and bias on its local CPU, while the data provider calculates the gradients using an FPGA chip in the encrypted domain. Each time the data provider completes the calculation of a set of gradients, it adds an encrypted random mask to the encrypted gradients and sends the result to the service requester.

[0072] Step (4): Whenever the data provider's FPGA chip completes the calculation of a set of encryption gradients, it sends the result to the service requester for decryption. The service initiator sends the decrypted data back to the data provider. The data provider subtracts the corresponding random mask from the received plaintext gradient to obtain the final gradient value. This operation is repeated until all gradients have been decrypted.

[0073] Step (5): The participants update the model using the calculated gradients. If the expected number of training epochs or the expected accuracy are achieved, training stops. Otherwise, repeat steps 2 through 5.

[0074] Step (6): Once the model training reaches the expected number of rounds or meets the expected accuracy, training stops and the final prediction model is obtained. Training participants can use the obtained model to make predictions and make corresponding decisions based on the prediction results.

[0075] The above is only a brief overview of the main process of the present invention; detailed steps will be presented in the embodiments.

[0076] Example 1

[0077] The embodiments of the present invention specifically describe a vertical federated learning acceleration method based on FPGA chips, and the detailed steps of the method are described below:

[0078] Step 1: System Initialization

[0079] ①Key generation

[0080] Participant A uses the GMP library on its CPU to randomly generate two large prime numbers p and q, each with a bit width of 1024 bits or 2048 bits. Then, it uses these two prime numbers to generate a public key (n, g) and a private key (λ, μ). The public key is transmitted to the CPUs of both participants for storage. The private key is only transmitted to the FPGA chip belonging to participant A for storage. The calculation methods for the public and private keys are as follows:

[0081] n = p * q

[0082] g = n + 1

[0083] λ=(p-1)*(q-1)

[0084] μ=λ-1

[0085] To randomly generate a 1024-bit key, you first need to generate two large 512-bit prime numbers:

[0086] p=10241590792327466746611424239096600108336373506003618747798620009538378827 72341992080940764938264863865795095465034195657521952279593194787169456943501467 64701373638181860484720938637096299507980491789336739916682030898636890818310804 887989276579810268016797575722706881064210200376996596428223025750327435671

[0087] q=53201136836516487354517119191688100205501220111176124040567964379393326889 11806851888768032212560782765555100690505333977350211520569471184758722425384733 98493721047793111516640232196268779129039737126463924005643849241516261430160113 13232527912544818099438352716110967793795106472880402101598629975280461803

[0088] Based on the above calculation method for public and private keys, the 1024-bit public and private keys can be calculated as follows:

[0089] Public key:

[0090] n＝54486427316622086944209022021863439569734533281415852508329238656927355502 37838930706257556082093484242793313918169759973249523131869146749016942447956257 13809724930243636979596060272332011969689874338982813382574990736267520015309373 45449370511352745995954252515036254087234254695249374131304565758984179419411643 11278824556920072426669873362627212846417192479842546557815717084779189643260207 27120987404930757822092209880970110930253531827431094463094161455905664658659885 14154606881748565878047726022985529587644569330194695102042362284552371475366324 68090400600669348726406554977122873156403716597098673655174813

[0091] g=54486427316622086944209022021863439569734533281415852508329238656927355502 37838930706257556082093484242793313918169759973249523131869146749016942447956257 13809724930243636979596060272332011969689874338982813382574990736267520015309373 45449370511352745995954252515036254087234254695249374131304565758984179419411643 11278824556920072426669873362627212846417192479842546557815717084779189643260207 27120987404930757822092209880970110930253531827431094463094161455905664658659885 14154606881748565878047726022985529587644569330194695102042362284552371475366324 68090400600669348726406554977122873156403716597098673655174814

[0092] Private key:

[0093] λ=54486427316622086944209022021863439569734533281415852508329238656927355502 37838930706257556082093484242793313918169759973249523131869146749016942447956257 13809724930243636979596060272332011969689874338982813382574990736267520015309373 45449370511352745995954252515036254087234254695249374131304565758984179417855472 66519033402099441065087219261338347891245980168323992393340939969612837375533225 51439392195509334316036869033679558360519215326012038009802301056495209584085589 02438243032129998646273517178330509756321396866036467216872749016390359257321401 13004284364740909908557696971816023279405186775442948047277340

[0094] μ=5448642731662208694420902202186343956973453328141585250832923865692735550 23783893070625755608209348424279331391816975997324952313186914674901694244795625 71380972493024363697959606027233201196968987433898281338257499073626752001530937 34544937051135274599595425251503625408723425469524937413130456575898417941785547 26651903340209944106508721926133834789124598016832399239334093996961283737553322 55143939219550933431603686903367955836051921532601203800980230105649520958408558 90243824303212999864627351717833050975632139686603646721687274901639035925732140 113004284364740909908557696971816023279405186775442948047277339

[0095] ② Random mask generation

[0096] Participant B uses the GMP library on the CPU to pre-generate a set of random masks R, and then encrypts this set of random masks using a public key to obtain an encrypted random mask. It is stored in the CPU memory of participant B.

[0097] Step 2: Forward Propagation

[0098] Participant A and Participant B use their private data to perform forward propagation on their respective models to obtain their respective predicted values. and After completing the forward propagation of a batch, participant B will submit the predicted values. Transmitted to participant A via the channel.

[0099] Step 3: Predicted value aggregation

[0100] Participant A will receive the predicted value from Participant B. and their own predicted values Perform aggregation to obtain the aggregated predicted value. Based on the aggregated predicted values, the difference between the labels and the predicted values ​​is calculated.

[0101] Step 4: Encrypt the difference

[0102] To prevent participant B from learning participant A's private data, participant A needs to encrypt the difference. Since the Paillier encryption protocol can only encrypt unsigned numbers, the difference must first be encoded. The encoding method is shown in Algorithm 3. Based on the encoding method in Algorithm 1, assuming the values ​​to be encrypted are a negative number -0.176348 and a positive number 0.8644, the results after encoding using the n generated in "Step 1" are shown in Table 2 below. Participant A encrypts the encoded difference enDiff using the public key to obtain the ciphertext. Using the operands in Table 2, the ciphertext obtained by encrypting the encoded data according to the encryption algorithm is shown in Table 3 below. Then, the ciphertext... It is transmitted to participant B.

[0103]

[0104] Table 2 Examples of Encoding Results

[0105]

[0106] Table 3 Examples of Encryption Results

[0107]

[0108]

[0109]

[0110] Step 5: Gradient Calculation

[0111] ①Party A

[0112] Participant A directly uses the unencrypted difference to calculate the gradients of the weights and biases on the CPU.

[0113] ②Party B

[0114] Participant B received the ciphertext Then, it is transmitted to the input buffer on the FPGA chip belonging to participant B. Multiple ciphertext multiplication units are implemented on the FPGA chip belonging to participant B. These ciphertext multiplication units read data from their respective input buffers and compute the gradient in parallel using the public key. The formula for calculating the ciphertext gradient using ciphertext multiplication is as follows: Where `input` represents the encoded input feature. Assuming a difference of -0.176348 and an input feature of 0.8644, the encoded and encrypted results are shown in Tables 2 and 3 above, respectively. The ciphertext gradient can then be obtained:

[0115]

[0116] To prevent participant A from learning participant B's private data (participant B's sample features), a random mask encrypted with a public key will be added to the calculation result. The calculation formula is: Encrypted random mask The calculated output is pre-calculated and stored in the memory of participant B during system initialization. The data will be continuously transmitted to participant B via the output buffer on the chip. Then, participant B will also continuously transmit the data obtained from the output buffer to participant A. As long as the FPGA chip belonging to participant A is idle, participant A will... The data is transmitted to the FPGA chip belonging to participant A for decryption.

[0117] Step 6: Gradient Decryption

[0118] As long as the FPGA chip belonging to participant A is idle, participant A will... The data is transmitted to the input buffer on the FPGA chip belonging to participant A. The decryption module on this FPGA chip retrieves the data from the corresponding input buffer and then decrypts it using the private key (λ, μ). Since there are multiple decryption modules on the FPGA chip belonging to participant A, multiple... It is decrypted in parallel. The decrypted data grad+R is sent back to participant A via the output buffer. Then, participant A sends the result to participant B. Participant B receives the decrypted gradient value grad+R and subtracts the random mask R to obtain the final gradient. The decryption formula is as follows:

[0119] m = L(c λ mod n 2 )mod n 2

[0120]

[0121] Step 7: Gradient Update

[0122] Participant A and Participant B update their own models using their respective update strategies and gradients.

[0123] Step 8: Iteration

[0124] If the expected accuracy or the maximum number of training epochs is reached, training stops; otherwise, steps 2 through 7 are repeated until the expected accuracy or the maximum number of training epochs is reached. The flowcharts for steps 2 through 8 are as follows: Figure 6 As shown.

[0125] Step 9: Model Prediction

[0126] Training stops once the model has reached the expected number of epochs or achieved the expected accuracy, resulting in the final prediction model. Participants can then use this model to make predictions. They input their private data (i.e., features) into the model, which outputs different predictions based on the input. Participants can then make appropriate decisions based on these predictions.

[0127] Example 2

[0128] Let's analyze this invention using a specific financial scenario:

[0129] The dataset used in this scenario is a credit dataset from a credit company. The training set contains 730,000 samples, and the test set contains 110,000 samples, with each sample containing 100 features. This embodiment aims to train a reliable Logistic Regression (LR) model to determine whether samples in the test set pose a fraud risk. To simulate a two-party longitudinal federated learning scenario, the dataset is divided into two parties, A and B. Party A simulates the service requester, and party B simulates the data provider. The dataset is then partitioned, with party A possessing 50 features and labels from the 730,000 samples, and party B possessing the remaining 50 features from the 730,000 samples. The two parties cannot exchange plaintext features and labels. If the label is 0, it indicates that the sample is risk-free; otherwise, it indicates that the sample is risky. In this scenario, the batch size is set to 256, meaning each batch contains 256 customer records.

[0130] Step 1: System Initialization

[0131] 1) Key generation

[0132] Participant A uses the GMP library on its CPU to randomly generate two large prime numbers p and q, each with a bit width of 512 bits. Then, it uses these two prime numbers to generate a public key (n, g) and a private key (λ, μ). The public key is distributed and stored in the CPU memory of both Participant A and Participant B. The private key is only distributed and stored on the FPGA chip belonging to Participant A. The calculation methods for the public and private keys are as follows:

[0133] n = p * q

[0134] g = n + 1

[0135] λ=(p-1)*(q-1)

[0136] μ=λ-1

[0137] 2) Random mask generation

[0138] Participant B uses the GMP library on the CPU to pre-generate a set of random masks R, and then encrypts this set of random masks using a public key to obtain an encrypted random mask. It is stored in the CPU memory of participant B.

[0139] 3) Model initialization

[0140] Participant A and Participant B each generate a weight matrix of dimension 50×2 and a bias matrix of dimension 1×2. The elements of the weight matrix are initialized to uniformly distributed random numbers in the interval [-3, 3]. The elements of the bias matrix are initialized to 0.

[0141] Step 2: Forward Propagation

[0142] Participants A and B each select 32 samples with the same ID from their respective datasets as a batch. They then combine the 50 features of these 32 samples into a 32×50 sample matrix. This matrix is ​​then multiplied by their respective weight matrices to obtain a 32×2 matrix. Finally, their respective bias matrices are added to yield the final 32×2 predicted value matrix. and This completes the forward propagation of one batch. Then, participant B will transfer its prediction matrix to participant A's CPU memory.

[0143] Step 3: Predicted value aggregation

[0144] Participant A will receive the predicted value from Participant B. and their own predicted values The aggregation is performed to obtain the aggregated prediction matrix. The addition is a matrix addition. Based on the aggregated predicted values, the difference matrix between the labels and the predicted values ​​is calculated. The dimensions are 32×2.

[0145] Step 4: Encrypt the difference

[0146] To prevent participant B from learning participant A's label data, participant A needs to encrypt the difference matrix. Since the Paillier encryption protocol can only encrypt unsigned numbers, the difference must first be encoded. The encoding method is shown in Algorithm 3. The input is the elements of the difference matrix from step 3. Based on the encoding method in Algorithm 1, an encoded matrix `enDiff` is obtained. Then, the encoded matrix is ​​encrypted using the encryption algorithm to obtain a ciphertext matrix with dimensions 32×2. It is transmitted to participant B.

[0147] Step 5: Gradient Calculation

[0148] ①Party A

[0149] Participant A directly uses the unencrypted difference matrix diff to calculate the gradients of the weights and biases on the CPU.

[0150] ②Party B

[0151] Participant B receives the ciphertext matrix Then, the data is transmitted in row-major order to the input buffer on the FPGA chip belonging to participant B. The FPGA chip belonging to participant B implements multiple ciphertext multiplication units, each with its own input buffer. These ciphertext multiplication units read data from their respective input buffers and compute the gradient in parallel using the public key. The formula for calculating the ciphertext gradient using ciphertext multiplication is: in is a matrix element in the ciphertext matrix, and input is the input feature corresponding to participant B after encoding.

[0152] To prevent participant A from learning participant B's private data (participant B's sample features), a random mask encrypted with a public key will be added to the calculation result. The calculation formula is: Encrypted random mask This is pre-calculated and stored in the memory of participant B during system initialization. The calculated output matrix has a dimension of 50×2. The data will be continuously transmitted to participant B via the output buffer on the chip. Then, participant B will also continuously transmit the data obtained from the output buffer to participant A. As long as the FPGA chip belonging to participant A is idle, participant A will... The data is transmitted to the FPGA chip belonging to participant A for decryption.

[0153] Step 6: Gradient Decryption

[0154] As long as the FPGA chip belonging to participant A is idle, participant A will... The matrix is ​​transmitted to the input buffer on the FPGA chip belonging to participant A. The decryption module on this FPGA chip retrieves the data from the corresponding input buffer and then decrypts the data using the private key (λ, μ). Since there are multiple decryption modules on the FPGA chip belonging to participant A, multiple... It is decrypted in parallel. The decrypted matrix grad+R is sent back to participant A via the output buffer. Then, participant A transmits the result to participant B. After receiving the decrypted gradient matrix grad+R, participant B subtracts the corresponding random mask R from each matrix element to obtain the final gradient matrix. The decryption formula is as follows:

[0155] m = L(c λ mod n 2 )mod n 2

[0156]

[0157] Step 7: Gradient Update

[0158] Participant A and Participant B use their respective application matrix addition methods to add the weight gradient matrix and the weight matrix, and the bias gradient and the bias matrix, to obtain a new weight matrix and a new bias matrix, which is the new LR model.

[0159] Step 8: Iteration

[0160] If the expected accuracy or the maximum number of training epochs is reached, training stops; otherwise, steps 2 through 7 are repeated until the expected accuracy or the maximum number of training epochs is reached. The flowcharts for steps 2 through 8 are as follows: Figure 6 As shown.

[0161] Step 9: Model Prediction

[0162] Training stops once the model has reached the expected number of epochs or achieved the expected accuracy, resulting in the final prediction model. Participants can then use this model for prediction. They input new, private data (i.e., features) into the model, and the prediction model produces different outputs based on the input. If the model's output is 0, it means the input sample poses no risk; otherwise, it indicates a risk.

[0163] Since existing technologies lack a two-partition architecture, the comparative method is an experiment conducted by this invention on a CPU in the same scenario. Compared to existing cases implemented entirely on a CPU, in the same application scenario, the vertical federated learning acceleration method based on an FPGA chip in this invention can improve the speed of calculating the ciphertext gradient (participant B) and decrypting the ciphertext gradient (participant A), reducing the time required for one iteration and thus reducing the overall training time. In this scenario, this invention implements four ciphertext multiplication units on the FPGA chip of participant B. The average time for each ciphertext multiplication unit to perform one ciphertext multiplication is similar to the time required on a CPU. However, because multiple multiplication units are implemented, ciphertext multiplication can be computed in parallel, thus reducing the required time to [a fraction of the original time]. This invention implements two decryption units on the FPGA chip of participant A, reducing the average decryption time of one data point by 40.5%. Although it increases the transmission latency from the CPU to the corresponding FPGA chip compared to the version implemented entirely by the CPU, the time required for one iteration still achieves a significant reduction, approximately 41%.

[0164] The scope of protection of this invention is not limited to the above embodiments. Any variations and advantages that can be conceived by those skilled in the art without departing from the spirit and scope of the inventive concept are included in this invention and are protected by the appended claims.

Claims

1. A method for accelerating vertical federated learning based on FPGA chips, characterized in that, The method includes the following steps: Step (1): The service requester and the data provider each perform forward propagation to obtain their respective prediction values. The data provider transmits its prediction value to the service requester. Then, the service requester aggregates the prediction results and calculates the difference between the prediction value and the label. Step (2): The service requester uses the public key to encrypt the difference calculated in step (1) and transmits it to the data provider; Step (3): The service requester directly uses the unencrypted difference to calculate the gradient of weight and bias on the local CPU. The data provider uses the encrypted difference from step (2) to calculate the gradient using the FPGA chip. Whenever the data provider completes the calculation of a set of gradients, it adds an encrypted random mask to the ciphertext gradient and sends the result to the service requester. Step (4): After the service requester receives the encryption gradient, it decrypts the encryption gradient and sends the decrypted data back to the data provider. The data provider subtracts the corresponding random mask from the received plaintext gradient to obtain the final gradient value; this operation is repeated until all gradients are decrypted. Step (5): Each participant updates the model using the calculated gradient. If the expected number of training rounds or the expected accuracy are reached, training is stopped; otherwise, steps (2) to (5) are repeated. Step (6): When the model training reaches the expected number of rounds or meets the expected accuracy, the training stops and the final prediction model is obtained; the training participants use the obtained model to make predictions and make corresponding decisions based on the prediction results.

2. The method according to claim 1, characterized in that, The service requester's functions include key generation, forward propagation, encryption difference calculation, gradient calculation, and model prediction. The key includes a public key (n, g) and a private key (λ, μ). Key generation is performed using the GMP large number library in C++. Before key generation, two random large prime numbers p and q need to be generated. Using these two large prime numbers, the public key is generated as follows: n = p * q, g = n + 1. The private key is generated as follows: λ = (p - 1) * (q - 1), μ = λ - 1. The public key is made public to all participants, including the service requester and the data provider, and is stored in the CPU memory of the service requester and the data provider, as well as in the memory of the data provider's FPGA chip. The private key is only stored in the memory of the service requester's FPGA chip.

3. The method according to claim 1, characterized in that, The data provider's functions include generating a random mask, encrypting the random mask, forward propagation, adding an encrypted random mask to the gradient in the ciphertext domain, and model prediction. The random mask is a random number generated using the C++ GMP large number library. The data provider encrypts the random mask using a public key on the CPU and stores it in the CPU's memory.

4. The method according to claim 1, characterized in that, The longitudinal federated learning uses an LR model, which consists of a linear layer and a softmax layer; the output z of the linear layer is represented as z = W. T x+b, where W and b are the weights and biases of the linear layer, respectively, and x is the input feature vector of the participant; the output of the Softmax layer is represented as The output of the entire LR model is represented as The LR model uses the cross-entropy loss function as the loss function, and the form of the cross-entropy loss function is as follows: Where N represents the number of samples, y i Let p be the label of the i-th sample. i This is the prediction result for the i-th sample.

5. The method according to claim 1, characterized in that, In step (1), the method by which the service requester aggregates the prediction results is defined as follows: in, The aggregated prediction results and These are the prediction results for the service requester and the data provider, respectively; the difference is calculated as follows: Where y is a label that is pre-generated by the service requester and stored in its CPU memory before logistic regression.

6. The method according to claim 1, characterized in that, In step (2), the encryption method for the difference is as follows: in The value represents the difference after encryption, diff is the difference before encryption, g and n are public key parameters, and r is a randomly generated obfuscation parameter that is coprime to n.

7. The method according to claim 1, characterized in that, In step (3), the service requester calculates the gradients of the weights and biases on its local CPU using the following formulas: and Where input is the sample input feature, L is the loss function, w is the weight, and b is the bias; The data provider calculates the gradient of the encryption difference in the ciphertext domain using an FPGA chip. The calculation method is as follows: For the encrypted gradient, The encrypted difference is the input parameter, where input is the sample input feature and n is the public key parameter. The data provider transmits the encrypted difference. The sample input features from the data provider are transmitted to the input buffer of the FPGA chip. The Montgomery exponentiation module in the ciphertext multiplication unit reads the data from the corresponding input buffer for calculation. After each set of gradients is calculated, the FPGA chip from the data provider transmits it to the CPU of the data provider through the corresponding output buffer. The data provider adds a random mask to the gradient in the ciphertext domain to obtain the result. It is transmitted to the service requester for decryption.

8. The method according to claim 1, characterized in that, In step (4), the CPU chip of the service requester calculates the gradient of the ciphertext field received from the data provider. The data transmitted to the input buffer of the FPGA chip is decrypted by the decryption module in the decryption unit on the FPGA chip of the service requester. The decryption method is m = L(c λ mod n 2 )mod n 2 ,in c is the ciphertext, n is the public key parameter, and λ is the private key parameter. After decryption, the plaintext m will be temporarily stored in the output buffer on the FPGA chip of the service requester, and then sent back to the CPU memory of the service requester.

9. The method according to claim 1, characterized in that, In the ciphertext gradient calculation, the decryption operation of the FPGA chip of the service requester and the gradient calculation of the FPGA chip of the data provider are performed simultaneously. The data provider will directly transmit the calculated ciphertext gradient to the service requester without waiting for all gradient calculations to be completed, thus realizing a pipelined operation of gradient calculation and decryption.

10. The method according to claim 1, characterized in that, In step (5), the gradient update uses the minibatch-SGD method; the minibatch-SGD method refers to processing one batch of samples at a time, calculating the gradient for each sample, accumulating the gradients, and then updating the gradient. The gradient update calculation method is as follows: Where grad represents the total gradient, grad i represents the gradient of the i-th sample, and N represents the number of samples in a batch; the updated weight is represented as w = w - η·grad, and the updated bias is represented as b = b - η·grad, where η is the learning rate.

11. The method according to claim 1, characterized in that, In step (5), before training begins, a training epoch is pre-selected to represent the maximum number of training epochs. When the maximum number of training epochs is reached, training stops. At the same time, an expected training precision is set. When the expected training precision is reached, training stops even if the maximum number of training epochs has not been reached.

12. An FPGA chip applied to the method according to any one of claims 1-11, characterized in that, The FPGA chip has different functions depending on the participating parties; the participating parties include service requesters and data providers. The FPGA chip of the service requester includes an input buffer, an output buffer, and a decryption unit; wherein the decryption unit includes one or more decryption modules; The FPGA chip of the data provider includes an input buffer, an output buffer, and a ciphertext multiplication unit; wherein, the ciphertext multiplication unit includes one or more Montgomery power module modules for implementing ciphertext scalar multiplication operations; the Montgomery power module module includes a Montgomery modular multiplication unit.

Images