Longitudinal federated learning training acceleration method, electronic device, storage medium and product

Abstract

This invention discloses a method, electronic device, storage medium, and product for accelerating longitudinal federated learning training. The entity includes a server and several clients. A feature selector is added to the local model of each client. The feature selector quantifies the influence of corresponding features based on its model weights, and then filters out unimportant features using a binary mask through sigmoid approximation, reshaping the local training dataset. The model weights of the feature selector and the model weights of the client's local model are updated together during gradient backpropagation. The technologies involved in this invention include feature weight evaluation technology, feature selection technology, and adaptive feature quantity adjustment technology based on the concept of transmission control protocol. This invention can improve training efficiency with almost no impact on model accuracy, while resisting the negative impact of lag on model synchronization.

Description

Technical Field

[0001] This invention belongs to the field of computer distributed learning technology, and relates to a method, electronic device, storage medium and product for accelerating vertical federated learning training. Specifically, it relates to a method, electronic device, storage medium and product for accelerating vertical federated learning training based on vertical federated learning and feature selection technology. Background Technology

[0002] To address issues such as privacy protection and "data silos," federated learning (FL) has been proposed. It allows clients to train a global model without sharing the original training data, generating an efficient global model. FL can be categorized into horizontal federated learning (HFL) and vertical federated learning (VFL) based on data distribution. In HFL, the training data of each client is partitioned by the sample IDs of the dataset; that is, the sample spaces are different, but the data feature spaces are the same. In VFL, different clients hold training data with the same sample IDs, but their feature subsets are disjoint; that is, the data feature spaces are different, but the sample spaces are the same. A VFL deployment diagram is shown below. Figure 1 As shown. Unlike HFL, VFL is particularly suitable for joint modeling scenarios with significant user overlap and complementary features, and is widely used in the financial and healthcare fields.

[0003] However, a key challenge facing VFL is client heterogeneity. On the one hand, due to hardware differences (i.e., system heterogeneity), clients may have different computing capabilities and communication bandwidths, leading to a laggard effect and prolonging training time. On the other hand, existing VFL work mainly focuses on uniform feature distribution among clients, but in practice, data features are often distributed in an unbalanced manner among clients (i.e., statistical heterogeneity), which can adversely affect the model's convergence behavior.

[0004] To improve the training efficiency of VFL in heterogeneous scenarios, current solutions typically involve adjusting the number of local updates or the learning rate on each client. However, different local update counts among clients may cause the global model to converge to a stable point that does not match the objective function, resulting in inconsistency in the objective and thus affecting model performance. Therefore, how to improve training efficiency in heterogeneous longitudinal federated learning systems with almost no impact on model accuracy is a pressing challenge. Summary of the Invention

[0005] To address the technical challenge of selecting appropriate feature subsets for heterogeneous clients to achieve a balance between training efficiency and model accuracy, this invention provides a method, system, and device for accelerating vertical federated learning training.

[0006] The technical solution adopted by the present invention is: a longitudinal federated learning training acceleration method, the entity including a server and several clients; a feature selector is added to the local model of each client; the feature selector quantifies the influence of the corresponding feature according to its model weight, and then filters out unimportant features by means of binary mask through sigmoid approximation, and reshapes the local training dataset; the model weight of the feature selector and the model weight of the local model of the client are updated together during the gradient backpropagation process.

[0007] Preferably, the addition of a feature selector to the local model for each client involves adding an input layer that is one-to-one with the data feature x, the output of which is w⊙x; where w is the weight vector of the data feature x, and ⊙ represents the vector dot product operation; by updating the weight vector w during training, it is transformed into a sparse vector according to the value of the vector, and only the features corresponding to the non-zero weight values ​​in w will be selected for model training.

[0008] As a preferred option, the feature selector F for client k k Using binary Bernoulli variables hard-coded as feature selector F k weight value F k,i , D k Denotes the feature dimension of client k; where, Where w i This represents the weight of the i-th feature, which is optimized during training.

[0009] As a preferred option, in the feature selector F for client k k After optimization, feature subset selection is performed based on probability, for each feature x k,i Assign a probability p k,i The likelihood of feature selection is directly proportional to the probability.

[0010] Preferably, during training, the server caches the training performance metrics of each client, adaptively adjusts the feature dimensions for the client based on the cached information, and sends the results to the client.

[0011] Among them, the feature dimension D selected for client k k Adjustments should be made according to the following strategies:

[0012]

[0013] Among them, fk This represents the rate of decrease in feature dimension, where K represents the number of clients; Let a represent the time required for client k in the i-th global round. o Representing the former T ex The performance metrics for a longitudinal federated learning global model across global rounds, a n This represents the performance metric of the longitudinal federated learning global model in the current global round, where T is a pre-set threshold.

[0014] The technical solution adopted by the electronic device of the present invention is: an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the longitudinal federated learning training acceleration method when executing the program.

[0015] The technical solution adopted by the storage medium of the present invention is: a non-transitory computer-readable storage medium storing a computer program thereon, wherein the computer program implements the longitudinal federated learning training acceleration method when executed by a processor.

[0016] The technical solution adopted by the product of the present invention is: a computer program product, including a computer program, which, when executed by a processor, implements the longitudinal federated learning training acceleration method.

[0017] Compared with the prior art, the beneficial effects of the present invention include:

[0018] (1) Based on the characteristics of unbalanced distribution of client data features and differences in computing power, this invention improves model training efficiency without sacrificing model performance.

[0019] (2) The important feature selection algorithm proposed in this invention adds a feature selector to the local model of each client. The feature selector quantifies the influence of the corresponding feature according to its model weights, and then filters out unimportant features by means of a binary mask through sigmoid approximation. The model weights of the feature selector are updated together with the model weights of the client's local model during the gradient backpropagation process, and the additional computational cost is negligible.

[0020] (3) This invention uses an adaptive adjustment algorithm to further optimize the balance between model accuracy and training efficiency. The algorithm adaptively adjusts the number of selected features. When the server detects different computation times of heterogeneous clients, the algorithm slowly reduces the feature dimensions of the heterogeneous clients, thereby improving training efficiency and client resource utilization while ensuring the stability of model training performance.

[0021] (4) The longitudinal federated learning training acceleration method proposed in this invention is not only applicable to heterogeneous distributed architectures, but can also be extended to other distributed architectures, such as homogeneous and asynchronous architectures, and has good scalability. Attached Figure Description

[0022] The technical solutions of the present invention will be further illustrated below using embodiments and specific implementation methods. In addition, some accompanying drawings are used in the description of the technical solutions. Those skilled in the art can obtain other drawings and the intent of the present invention from these drawings without any creative effort.

[0023] Figure 1 This is a schematic diagram of a general vertical federated learning deployment in existing technologies;

[0024] Figure 2 This is a schematic diagram illustrating the method principle of an embodiment of the present invention;

[0025] Figure 3 This is a schematic diagram illustrating the experimental results of an embodiment of the present invention. Detailed Implementation

[0026] To facilitate understanding and implementation of the present invention by those skilled in the art, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0027] First, we give the formal definitions of the symbols involved in the t-th round of training of this system:

[0028] Client Collection

[0029] Disjoint feature subsets {X} of the client k |i=1,2,3,…,K};

[0030] Local (linear or nonlinear) embedding functions k (·);

[0031] Local feature selector {F i |i=1,2,3,…,K};

[0032] Client-side low-dimensional embedding vector

[0033] Please see Figure 2This embodiment provides a method for accelerating training in longitudinal federated learning, comprising a server and several clients; a feature selector is added to the local model of each client; the feature selector quantifies the influence of the corresponding features according to its model weights, and then filters out unimportant features by means of binary masks through sparsification processing, thereby reshaping the local training dataset; the model weights of the feature selector and the model weights of the client's local model are updated together during gradient backpropagation.

[0034] In one implementation, to improve the efficiency of the feature weight evaluation process, a feature selection layer is embedded in the neural network architecture to select important features at a lower cost, thus reshaping the local training dataset. The basic principle of this method is to add an input layer with a one-to-one correspondence with the data features, so that the output of this layer is w⊙x; where w is the weight vector of the data feature x, and ⊙ represents the vector dot product operation. By updating the weight vector w during training, it can be transformed into a sparse vector based on its values, and only features corresponding to non-zero weight values ​​in w will be selected for model training.

[0035] In one implementation, to reduce the cost of model training, including computational and communication costs, the feature selector F must be updated simultaneously with the training of the VFL model. k Considering F k Using binary Bernoulli variables hard-coded, i.e. For feature selector F k The binary Bernoulli variable F k,i Smooth it to approximate it as Where w i The weight of the i-th feature can be optimized during training. k This represents the feature dimension of client k. To ensure the selector is initialized to... Choose a large constant vector as the initial value of w;

[0036] In one implementation, the feature selector F for client k k After optimization, feature subset selection is performed based on probability, for each feature x k,i Assign a probability p k,i Considering the weight value F k,i The higher the value, the more important the corresponding feature may be, therefore the probability p of selecting that feature will increase. k,i The higher the value, the better.

[0037] Due to the trade-off between training efficiency and model accuracy resulting from feature selection, one implementation devises an adaptive feature quantity adjustment algorithm to achieve a balance between model accuracy and training efficiency. Considering that only the server can obtain the time spent by the client in each global training round for training and communication, as well as the accuracy of the global model, the server adaptively adjusts the selected feature dimensions for each client participating in global model training through the following two processes:

[0038] 1) Cache the average training performance of the client. The training process for each global training round is divided into two phases: the client computation phase and the client-server communication phase. In the i-th global round, use... Indicates the calculation time. This represents the communication time required for client k to transmit the locally computed low-dimensional embedding vector to the server. Typically, With feature dimension D k There is a certain proportional relationship: And for Since the dimensions of the low-dimensional embedding vectors ultimately computed by each client are not significantly different, it is assumed that the communication time of each client is the same in each global round i. To improve training efficiency and client resource utilization during model training, the following objectives are aimed at achieving: After all clients have communicated with the server, the server can cache the training performance of each client.

[0039] 2) Adaptive adjustment of selected feature dimensions. When the server detects different computation times from heterogeneous clients, the algorithm slowly reduces the feature dimensions of the heterogeneous clients, improving training efficiency and client resource utilization while ensuring the stability of model training performance.

[0040] Specifically, during the training process, each T ex The feature dimension D selected by client k in each global round k Adjustments should be made according to the following strategies:

[0041]

[0042] Among them, f k This represents the rate of decrease in feature dimension, where K represents the number of clients; Let a represent the time required for client k in the i-th global round. o Representing the former T ex The performance metrics for a longitudinal federated learning global model across global rounds, a nThis represents the performance metric of the longitudinal federated learning global model in the current global round, where T is a pre-defined threshold. When the client... When there are significant differences in values, in order to make the rate of reduction of client feature dimensions more gradual and thus ensure the stability of the global model, a larger value for f should be determined. k If the server observes a performance degradation in the model exceeding a pre-set threshold T, the client can appropriately increase the feature dimensions to ensure the performance of the final global model. This adaptive feature dimension adjustment process not only guarantees model performance but also significantly improves training efficiency, while reducing client idle time during training, thereby increasing the utilization of client computing resources.

[0043] In one implementation, the method includes the following steps:

[0044] Step 1: Local training and feature weight evaluation;

[0045] At the start of the t-th round of training, each client receives the gradients of the feature dimension and the low-dimensional embedding vector from the server, and updates its local model E times based on the gradients; at the same time, it updates the local feature selector to evaluate the weights of the features.

[0046] Step 2: The client selects an important subset of features through a local feature selector for forward propagation, calculates the local low-dimensional embedding vector, and sends it to the server; in each round of training, the server caches the training performance metrics of each client, adaptively adjusts the feature dimensions for the client based on the cached information, and sends the results to the client.

[0047] Step 3: Aggregate the low-dimensional embedding vectors and compute;

[0048] After receiving the low-dimensional embedding vector from the client, the server performs a forward pass to calculate the loss l(Θ) for model training, and then performs a backward pass to calculate the server model. The gradient of the embedded vector is calculated and sent to the client, and the model parameters are updated E times based on the gradient of the embedded vector.

[0049]

[0050]

[0051] This embodiment also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the longitudinal federated learning training acceleration method.

[0052] This embodiment also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the longitudinal federated learning training acceleration method.

[0053] This embodiment also provides a computer program product, including a computer program that, when executed by a processor, implements the longitudinal federated learning training acceleration method.

[0054] The technologies involved in this invention include feature weight evaluation techniques, feature selection techniques, and adaptive feature quantity adjustment techniques. The embodiments of this invention are tested on the CIFAR10 dataset, and the benchmark algorithm is as follows:

[0055] FedBCD: Federated Random Block Coordinate Descent Training Algorithm, which uses all features for model training when the number of client features is unbalanced.

[0056] Please see Figure 3 The embodiments of the present invention can resist the negative impact of lag on model training and improve training efficiency (achieving end-to-end acceleration of 55%-78%) without affecting model accuracy (the reduction in model performance does not exceed 4%).

[0057] It should be understood that the embodiments described above are only some, not all, of the embodiments of the present invention. Furthermore, the technical features of the various embodiments or individual embodiments provided by the present invention can be arbitrarily combined to form feasible technical solutions. Such combinations are not constrained by the order of steps and / or structural composition patterns, but must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0058] It should be understood that the above description of the preferred embodiments is quite detailed, but it should not be considered as a limitation on the scope of protection of this invention. Those skilled in the art, under the guidance of this invention, can make substitutions or modifications without departing from the scope of protection of the claims of this invention, and all such substitutions or modifications fall within the scope of protection of this invention. The scope of protection of this invention should be determined by the appended claims.

Claims

1. A method for accelerating training in vertical federated learning, comprising a server and several clients; characterized in that: Add a feature selector to the local model for each client; The feature selector quantifies the influence of the corresponding feature based on its model weights, and then filters out unimportant features by means of a binary mask through sigmoid approximation, thus reshaping the local training dataset; the model weights of the feature selector and the model weights of the client's local model are updated together during the gradient backpropagation process. Among them, for the client Feature selector Using binary Bernoulli variables hard-coded as feature selectors weight value , , Indicates the client The feature dimensions; where, ,in Indicates the first The weights of each feature are optimized during training; During training, the server caches the training performance metrics for each client, adaptively adjusts the feature dimensions for each client based on the cached information, and sends the results back to the client; among these, for the client... k Selected feature dimensions Adjustments should be made according to the following strategies: ； in, This represents the rate of decrease in feature dimension. K Indicates the number of clients; Indicates the client In the Time required for each global round Representative before Performance metrics for a longitudinal federated learning global model per global round. This represents the performance metric of the longitudinal federated learning global model in the current global round. For a pre-set threshold; The method includes the following steps: Step 1: Local training and feature weight evaluation; In the t At the start of each training round, each client receives the gradients of the feature dimension and the low-dimensional embedding vector from the server, and then performs training based on these gradients. The local model is updated; simultaneously, the local feature selector is updated to evaluate the weights of the features. Step 2: The client selects an important subset of features through a local feature selector for forward propagation, calculates the local low-dimensional embedding vector, and sends it to the server; in each round of training, the server caches the training performance metrics of each client, adaptively adjusts the feature dimensions for the client based on the cached information, and sends the results to the client. Step 3: Aggregate the low-dimensional embedding vectors and compute; After receiving the low-dimensional embedding vector from the client, the server performs a forward pass to calculate the loss during model training. Then, the server model is calculated by reverse propagation. The gradient of the embedding vector is calculated and sent to the client, which then performs [further processing] based on the gradient of the model parameters. This is the second update.

2. The method for accelerating training in vertical federated learning according to claim 1, characterized in that: The addition of a feature selector to the local model for each client involves adding a feature selector that matches the data features. A one-to-one input layer, the output of which is ;in, For data features The weight vector, This represents the vector dot product operation; it is performed by adjusting the weight vector during training. To update, transform the vector into a sparse vector based on its values, only... Features with non-zero weight values ​​will be selected for model training.

3. The method for accelerating training in vertical federated learning according to claim 1, characterized in that: In the client Feature selector After optimization, feature subset selection is performed based on probability, for each feature Assign a probability The likelihood of feature selection is directly proportional to the probability.

4. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it implements the longitudinal federated learning training acceleration method as described in any one of claims 1 to 3.

5. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the longitudinal federated learning training acceleration method as described in any one of claims 1 to 3.

6. A computer program product, comprising a computer program, characterized in that: When the computer program is executed by the processor, it implements the longitudinal federated learning training acceleration method as described in any one of claims 1 to 3.

Images