Vertical federal learning accelerated training method

Abstract

The invention provides a vertical federated learning accelerated training method. The method comprises the following steps: carrying out parallel processing on an assembly line of vertical federated learning VFL training; according to the model training stage, the model updating obsolessness is controlled; a feedback-based mechanism is used to adjust the number of local training iterations and pre-calculated batches in real time as a function of obsolescence. According to the invention, the training efficiency of the VFL in a high-delay scene can be improved; different calculation and communication conditions can be adapted, and VFL training is accelerated in a heterogeneous environment; according to the invention, the idle time of the passive party and the time for receiving the embedding by the active party are reduced by adaptively and rapidly transmitting multiple embedding; according to the method, the scale factor is introduced, so that the negative influence of the obsolessness in the training process is avoided; the accuracy and the training speed in the training process are ensured through a feedback mechanism.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed machine learning, and in particular to a vertical federated learning accelerated training method. Background Art

[0002] Vertical Federated Learning (VFL) enables multiple parties to jointly build machine learning models while protecting privacy. Unlike horizontal federated learning (HFL), the parties in VFL hold different and complementary features of the same training samples. Labels are usually held by only one party, called the active party, while the other parties are called passive parties. VFL can enhance cooperation between different industries by facilitating the exchange of domain knowledge without exposing their private data. This feature makes VFL widely used in many privacy-conscious fields, including collaborative medical research, financial risk assessment, and inter-organizational data collaboration.

[0003] Different VFL participants must exchange their forward activations and reverse gradients, which results in more frequent cross-party communication and longer training time compared to HFL. In addition, in practice, different participants are often geographically unevenly distributed, have heterogeneous network infrastructures, and usually communicate through wide area network connections. This makes the communication process of VFL training limited in transmission bandwidth and high in network latency. The suboptimal communication efficiency limits the wider implementation of VFL systems. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides a vertical federated learning accelerated training method, which can improve the training efficiency of VFL in high-latency scenarios, and can adapt to different computing and communication conditions to accelerate the training of VFL in a heterogeneous environment.

[0005] The technical solution of the present invention is: a vertical federated learning accelerated training method, comprising the following steps:

[0006] S1), parallelize the pipeline of vertical federated learning VFL training;

[0007] S2), controlling the obsolescence of the model update in step S1) according to the model training stage;

[0008] S3), using a feedback-based mechanism to adjust the number of local training iterations and pre-computed batches in step S1) in real time according to the staleness in step S2).

[0009] Preferably, in step S1), the training of each passive party in the VFL training is divided into three stages, namely, a forward calculation stage, a local training stage, and a backward update stage; each active party is divided into four stages, namely, a forward calculation stage, a local training stage, a backward update stage, and an aggregation stage.

[0010] Preferably, in step S1), during the working process of the pipeline, when the embeddings of all passive parties arrive, the active party uses its bottom model to calculate its own embedding, then aggregates all embeddings, and calculates the corresponding gradients according to their labels, and sends the corresponding gradients to the passive parties, and the active party updates its own bottom model and top model.

[0011] Preferably, in step S1), before starting the next round of training, the active party uses the old gradient to perform local training until the embeddings of all passive parties are collected.

[0012] Preferably, in step S1), in VFL training, the passive party enables pre-computation to reduce its idle time, and transmits multiple embeddings quickly by adaptively, thereby reducing the idle time of the passive party and the time for the active party to receive the embeddings.

[0013] Preferably, in step S1), the adaptive fast transmission of multiple embeddings includes the following two cases:

[0014] Case 1: Gradient Received from the active party and stored in the not yet calculated The gradient queue Q of the passive party k k Where l represents the training loss, h represents the embedding, k represents the index of the passive party, and t represents the tth batch. represents the passive party k sending the tth batch of embeddings;

[0015] Case 2: Passive party k has not received However, the maximum allowed number of iterations for stale local training has been reached.

[0016] As a preference, for case 1, assuming that the number of embedded And define the number of batches u for which embeddings are actually sent but the corresponding gradients are not used k ; The passive side switches to the forward calculation process and checks the control signal The control signal lie in In the same data packet;

[0017] The passive party k will send embedding, that is, if Calculate and send and And embed the quantity Increase to 2;

[0018] And if the control signal Then only calculate and send To make

[0019] Then the passive party k switches to backward update, using the gradient queue Q k The gradient in Perform one backpropagation iteration on the underlying model and update the model parameters accordingly;

[0020] If the gradient If it has been used, it will be removed from the gradient queue Q k Discarded;

[0021] If the queue is still not empty, the active side has at least one newer gradient has also arrived at the queue; the passive party switches to forward calculation again and calculates according to The relevant control signals are computed and embedded.

[0022] As a preference, for case 2, switch to forward calculation and calculate according to determines the number of new embeddings to compute; if If it has not arrived yet, the gradient message queue Q k If is empty, the passive party k enters idle time, otherwise it continues to perform backward update.

[0023] Preferably, in step S2), in the early training stage, the stale local training algorithm FedBCD is used to perform staleness control, and the metric factor S is introduced in the later training process. e To quantify obsolescence, the metric factor S e The expression is:

[0024]

[0025] Where, e is the current training epoch; k is the index of the passive party; t is the tth data batch;

[0026] In the formula, Indicates the maximum number of stale local training iterations using the same gradient; is the number of batches sent in communications with the active party.

[0027] Preferably, in step S2), in each training round, the staleness parameter is updated as follows:

[0028]

[0029] Where, e is the number of completed training epochs; s max is the initial hyperparameter, indicating the maximum staleness; S e+1 is the staleness of the e+1th training epoch.

[0030] Preferably, in step S3), for the active party, the active party controls the control signal Adjustment If two or more embeddings are received, the active party will send a control signal C to the passive party. k =-1, means stop sending embedding;

[0031] If the active party does not receive the expected embedding and the computing power is sufficient, the active party sends a control signal C k =1 to prompt the passive party, where the control signal will be sent simultaneously with the gradient package.

[0032] Preferably, in step S3), for the passive party, if the passive party does not receive the active party's change The passive party sets

[0033] If the passive party receives the control signal C k if an indication is given, then evaluate whether to adjust the batch accordingly.

[0034] As a preferred embodiment, in step S3), if the control signal C received by the passive party k =1, there are two cases:

[0035] (i) If passive party k cannot process additional batches due to local constraints, then the passive party applies

[0036] (ii) The passive party k is able to handle the additional batches of computation, as shown by performing local training in the previous round, May increase.

[0037] As a preferred embodiment, in step S3), if the control signal C received by the passive party k =-1, then is set to The maximum value between 0 and 1.

[0038] The beneficial effects of the present invention are:

[0039] 1. The present invention can improve the training efficiency of VFL in high-latency scenarios; it can adapt to different computing and communication conditions and accelerate the training of VFL in heterogeneous environments;

[0040] 2. The present invention transmits multiple embeddings quickly and adaptively, thereby reducing the idle time of the passive party and the time for the active party to receive the embeddings;

[0041] 3. The present invention avoids the negative impact of staleness during the training process by introducing a scale factor;

[0042] 4. The present invention ensures the accuracy and speed of training in the training process through a feedback mechanism. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0044] The specific implementation of the present invention will be further described below in conjunction with the accompanying drawings:

[0045] Example 1

[0046] like Figure 1 As shown, this embodiment provides a vertical federated learning accelerated training method, including the following steps:

[0047] S1), parallelize the pipeline of vertical federated learning VFL training;

[0048] The training of each passive party in the vertical federated learning VFL training is divided into three stages, namely, the forward calculation stage, the local training stage, and the backward update stage; each active party is divided into four stages, namely, the forward calculation stage, the local training stage, the backward update stage, and the aggregation stage.

[0049] During the working process of the pipeline, when the embeddings of all passive parties arrive, the active party uses its bottom model to calculate its own embedding, then aggregates all embeddings and calculates the corresponding gradients according to their labels, and sends the corresponding gradients to the passive parties, and the active party updates its own bottom model and top model.

[0050] Before starting the next round of training, the active side performs local training using old gradients until the embeddings of all passive sides are collected.

[0051] In VFL training, the passive side enables pre-computation to reduce its idle time, by adaptively and quickly transmitting multiple embeddings, thereby reducing the passive side's idle time and the active side's time to receive embeddings.

[0052] The adaptive fast transmission of multiple embeddings includes the following two cases:

[0053] Case 1: Gradient Received from the active party and stored in the not yet calculated The gradient queue Q of the passive party k kWhere l represents the training loss, h represents the embedding, k represents the index of the passive party, and t represents the tth batch. represents the passive party k sending the tth batch of embeddings;

[0054] Assuming the previously determined number of embedded processes And define the number of batches u for which embeddings are actually sent but the corresponding gradients are not used k ; The passive side switches to the forward calculation process and checks the control signal The control signal lie in In the same data packet;

[0055] The passive party k will send embedding, that is, if Calculate and send and And embed the quantity Increase to 2;

[0056] And if the control signal Then only calculate and send To make

[0057] Then the passive party k switches to backward update, using the gradient queue Q k The gradient in Perform one backpropagation iteration on the underlying model and update the model parameters accordingly;

[0058] If the gradient If it has been used, it will be removed from the gradient queue Q k Discarded;

[0059] If the queue is still not empty, the active side has at least one newer gradient has also arrived at the queue; the passive party switches to forward calculation again and calculates according to The relevant control signals are computed and embedded.

[0060] Case 2: Passive party k has not received However, the maximum number of allowed iterations of stale local training has been reached. Switch to forward calculation and calculate according to determines the number of new embeddings to compute; if If it has not arrived yet, the gradient message queue Q k If is empty, the passive party k enters idle time, otherwise it continues to perform backward update.

[0061] S2), controlling the obsolescence of the model update in step S1) according to the model training stage;

[0062] In the early training stage, this embodiment uses the stale local training algorithm FedBCD to perform staleness control multiple times, and introduces the measurement factor S in the later training process. e To quantify obsolescence, the metric factor S e The expression is:

[0063]

[0064] Where, e is the current training epoch; k is the index of the passive party; t is the tth data batch;

[0065] In the formula, Indicates the maximum number of stale local training iterations using the same gradient; is the number of batches sent in communications with the active party.

[0066] In each training epoch, the staleness parameter is updated as:

[0067]

[0068] Where, e is the number of completed training epochs; s max is the initial hyperparameter, indicating the maximum staleness; S e+1 is the staleness of the e+1th training epoch.

[0069] S3), using a feedback-based mechanism to adjust the number of local training iterations and pre-computed batches in step S1) in real time according to the staleness in step S2).

[0070] In this embodiment, for the active party, the active party Adjustment If two or more embeddings are received, the active party will send a control signal C to the passive party. k =-1, means stop sending embedding;

[0071] If the active party does not receive the expected embedding and the computing power is sufficient, the active party sends a control signal C k =1 to prompt the passive party, where the control signal will be sent simultaneously with the gradient package.

[0072] For the passive party, if the passive party does not receive the active party's changes The passive party sets

[0073] If the passive party receives the control signal C k if an indication is given, then evaluate whether to adjust the batch accordingly.

[0074] If the passive party receives the control signal C k =1, there are two cases:

[0075] (i) If passive party k cannot process additional batches due to local constraints, then the passive party applies

[0076] (ii) The passive party k is able to handle the additional batches of computation, as shown by performing local training in the previous round, May increase.

[0077] If the passive party receives the control signal C k =-1, then is set to The maximum value between 0 and 1.

[0078] Example 2

[0079] This example uses a series of representative datasets and appropriate models for evaluation, with the following specific settings. MNIST: This example uses all the data in the experiment, evenly divides each image into two parts, each with 392 features, sets the underlying model to a two-layer MLP, and trains for 30 epochs. Fashion-MNIST (FMNIST): The dataset size and division of FMNIST are the same as MNIST. The underlying model is set to CNN, and trained for 30 epochs.

[0080] CIFAR10: Images are evenly distributed among the parties, with each party having 1536 features in the two-player setting and 768 features in the four-player setting. The underlying model is set to VGG13 with BN layers, and we set the total training epochs to 30.

[0081] NUSWIDE: This dataset contains five categories of network images with different image features and labels. The active side uses low-level images with a total of 634 features. The passive side has the top 1000 popular labels. The underlying model is a two-layer MLP, which converges after 10 epochs.

[0082] VFL baseline methods: ① Each party in FedSGD updates its local model once on each training batch. ② FedBCD is a synchronous training method that uses stale gradients for multiple updates. In this embodiment, each party in FedBCD performs one standard local update and four local updates per round. ③ FlexVFL-s: In order to ensure that the embeddings arrive at the active party as simultaneously as possible, this embodiment implements a simplified baseline called FlexVFL-s based on FlexVFL. This embodiment sets a timeout for local training while taking into account communication delays, but the adaptive learning rate extension has been removed. Since the focus of this embodiment is to compare the scheduling of training, other parameters such as learning rate and optimizer remain consistent between different methods.

[0083] To evaluate the impact of whether the top model is trainable in the active side, this example considers two different VFL settings for all methods: SplitVFL: The top model is trainable. For MLP or CNN, the top model is defined as a fully connected layer. For VGG13, the top model is the classification layer of VGG13. AggVFL: The top model is not trainable, and the active side directly aggregates the outputs of the underlying models and uses the aggregated output as the prediction.

[0084] Simulation experiment training settings: There are two participants, one is the active party and the other is the passive party. The forward and backward times are set to 10 and 20 time units respectively, and the aggregation time is 10 time units. The communication delay between the passive party and the active party is set to 5, 50, 100 and 1000 time units respectively. max and max The parameters were set to 3 and 4, respectively. Each experiment was repeated 5 times.

[0085] As can be seen from Tables 1 and 2, in low-latency scenarios (5 or 50 time units), the final accuracy of the algorithm of this embodiment is optimal. This is because the model of this embodiment is updated more frequently than FedSGD because there is local training during 30 training cycles. In addition, compared with FedBCD and FlexVFL-s, which perform more local training, the present invention can better control the negative impact of staleness, as evidenced by its superior final accuracy. As the communication delay increases (100, 1000 time units), the final convergence accuracy of this embodiment will decrease due to certain operations (such as sending embeddings in advance). However, this slight decrease in accuracy is acceptable. When the communication delay increases, the algorithm of this embodiment is significantly better than other algorithms in terms of speedup ratio. The speedup ratio under SplitVFL is similar to that of AggVFL. This acceleration can improve the convergence performance between different data sets.

[0086] In high communication latency scenarios, this embodiment can bring significant training acceleration compared to the VFL baseline method without sacrificing model convergence and test accuracy.

[0087] Table 1 shows the final model accuracy results of the simulation experiments of the method of this embodiment and other benchmark algorithms under the AggVFL setting in multiple data sets and scenarios, as well as the simulation time and speedup ratio to achieve the target accuracy (75%, 96%, 89%, 76%).

[0088]

[0089] Among them, '-' indicates the missing data due to the inability to achieve the target accuracy.

[0090] Table 2 Final model accuracy results of simulation experiments of the method of this embodiment and other benchmark algorithms under the SplitVFL setting in multiple data sets and scenarios, and simulation time and speedup ratio to achieve target accuracy (75%, 97%, 90%, 77%)

[0091]

[0092] Among them, '-' indicates the missing data due to the inability to achieve the target accuracy.

[0093] The real machine test training settings of this embodiment are as follows: the test platform consists of four 4090 GPUs connected to a switch, and each participant is a running process on a GPU. This embodiment uses the Linux traffic control (TC) program to configure the network interface card (NIC) and records the communication delay accordingly. The transmission rates between the passive and active parties are approximately 100Mbps, 50Mbps, and 5Mbps, representing the communication heterogeneity.

[0094] The above embodiments and descriptions are only for illustrating the principles and best embodiments of the present invention. Without departing from the spirit and scope of the present invention, the present invention may be subject to various changes and improvements, all of which fall within the scope of the present invention to be protected.

Claims

1. A vertical federated learning accelerated training method, characterized in that: The steps include: S1), parallelize the pipeline of vertical federated learning VFL training; S2), controlling the obsolescence of the model update in step S1) according to the model training stage; S3), using a feedback-based mechanism to adjust the number of local training iterations and pre-computed batches in step S1) in real time according to the staleness in step S2).

2. A vertical federated learning accelerated training method according to claim 1, characterized in that: In step S1), the training of each passive party in the VFL training is divided into three stages, namely, a forward calculation stage, a local training stage, and a backward update stage; each active party is divided into four stages, namely, a forward calculation stage, a local training stage, a backward update stage, and an aggregation stage.

3. A vertical federated learning accelerated training method according to claim 2, characterized in that: In step S1), during the working process of the pipeline, when the embeddings of all passive parties arrive, the active party uses its bottom model to calculate its own embedding, then aggregates all embeddings, and calculates the corresponding gradients according to their labels, and sends the corresponding gradients to the passive parties, and the active party updates its own bottom model and top model; Before starting the next round of training, the active side performs local training using old gradients until the embeddings of all passive sides are collected.

4. A vertical federated learning accelerated training method according to claim 3, characterized in that: In step S1), in VFL training, the passive party enables pre-computation to reduce its idle time, and transmits multiple embeddings quickly by adaptively, thereby reducing the idle time of the passive party and the time for the active party to receive the embeddings.

5. A vertical federated learning accelerated training method according to claim 4, characterized in that: In step S1), the adaptive fast transmission of multiple embeddings includes the following two cases: Case 1: Gradient Received from the active party and stored in the not yet calculated The gradient queue Q of the passive party k k Where l represents the training loss, h represents the embedding, k represents the index of the passive party, and t represents the tth batch. represents the passive party k sending the tth batch of embeddings; Case 2: Passive party k has not received However, the maximum allowed number of iterations for stale local training has been reached.

6. A vertical federated learning accelerated training method according to claim 5, characterized in that: For case 1, assume that the number of embedded processes determined previously And define the number of batches u for which embeddings are actually sent but the corresponding gradients are not used k ; The passive side switches to the forward calculation process and checks the control signal The control signal lie in In the same data packet; The passive party k will send embedding, that is, if Calculate and send and And embed the quantity Increase to 2; And if the control signal Then only calculate and send To make Then the passive party k switches to backward update, using the gradient queue Q k The gradient in Perform one backpropagation iteration on the underlying model and update the model parameters accordingly; If the gradient If it has been used, it will be removed from the gradient queue Q k Discarded; If the queue is still not empty, the active side has at least one newer gradient has also arrived in the queue; the passive party switches to forward calculation again and calculates according to The relevant control signals are computed and embedded.

7. The vertical federated learning accelerated training method according to claim 5, characterized in that: For case 2, switch to forward calculation and according to Decide how many new embeddings to compute; if If it has not arrived yet, the gradient message queue Q k If is empty, the passive party k enters idle time, otherwise it continues to perform backward update.

8. The vertical federated learning accelerated training method according to claim 1, characterized in that: In step S2), in the early training stage, the staleness control is performed by using the multiple execution of the stale local training algorithm FedBCD, and the metric factor S is introduced in the later training process. e To quantify obsolescence, the metric factor S e The expression is: Where, e is the current training epoch; k is the index of the passive party; t is the tth data batch; In the formula, Indicates the maximum number of stale local training iterations using the same gradient; is the number of batches sent in the communication with the active party; In each training round, the staleness parameter is updated as: Where, e is the number of completed training epochs; s max is the initial hyperparameter, indicating the maximum staleness; S e+1 is the staleness of the e+1th training epoch.

9. A vertical federated learning accelerated training method according to claim 6, characterized in that: In step S3), for the active party, the active party Adjustment If two or more embeddings are received, the active party will send a control signal C to the passive party. k =-1, means stop sending embedding; If the active party does not receive the expected embedding and the computing power is sufficient, the active party sends a control signal C k =1 to prompt the passive party, where the control signal will be sent simultaneously with the gradient package; For the passive party, if the passive party does not receive the active party's changes The passive party sets If the passive party receives the control signal C k if an indication is given, then evaluate whether to adjust the batch accordingly.

10. A vertical federated learning accelerated training method according to claim 9, characterized in that: In step S3), if the passive party receives the control signal C k =1, there are two cases: (i) If passive party k cannot process additional batches due to local constraints, then the passive party applies (ii) The passive party k is able to handle the additional batches of computation, as shown by performing local training in the previous round, It may increase; If the passive party receives the control signal C k =-1, then is set to The maximum value between 0 and 1.

Images