A batch-based parallel split federated learning method

Abstract

The present invention discloses a batch-based parallel split federated learning method, which includes: global model partitioning, local data batch partitioning, intermediate value grouping, efficient parallel computing of server models, and client model aggregation. The method divides the global model into server models and client models, divides the client data into batches, groups them according to the data batches, each group contains all the client data, and distributes the server models by group. In addition, the method realizes parallel training of server clusters, divides the server model into multiple sequences, and distributes the sequences to the server cluster. The servers realize parallel computing by scheduling different client data. The method converges faster without reducing the accuracy, and the average training time per round is accelerated by 3.5%-46.5%.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a batch-based parallel splitting federated learning method. Background Art

[0002] With the widespread application of neural networks, vast amounts of data are being used to train neural network models, driving technological advancements in areas such as computer vision, natural language processing, and edge computing. However, this data often contains sensitive personal information, such as medical records, consumer behavior, and location data. While the large-scale use of this data improves model training effectiveness, it also poses a serious risk of privacy breaches. Federated learning, a privacy-preserving distributed machine learning framework, allows data to remain locally distributed, allowing distributed model training to be conducted by simply sharing model parameters. However, with the advancement of neural networks, model sizes are increasing, and the computing nodes involved in training vary in computing power and storage speed. Despite the large number of clients, their processing power is limited, making it impossible to fully scale large models.

[0003] In order to train large models on clients with limited resources, SplitFed proposes a split federated learning framework (SFL). In SFL, the model w is usually split into two sub-models, namely ,in Represents the client model assigned to the client, Indicates the server model assigned to the server. Client model Since data needs to be input, the first few layers of the model are usually included, and the server model The remaining model layers are included. For example, taking the classic convolutional neural network (CNN) architecture as an example, the client model usually includes the input layer and the convolution layer, while the server model includes the fully connected layer and the output layer. After the forward propagation, the intermediate activation values are sent to the split learning server. The split learning server is responsible for the forward and backward propagation of subsequent layers, and then sends it back to the client to complete the final backward propagation of the model, thus completing the forward and backward propagation of the entire model. Finally, the federated server aggregates the model updates of all clients.

[0004] The goal of SFL is to minimize the loss function :

[0005] ;

[0006] in is the loss function of the server model, is the loss function of the client model. SFL typically has two execution strategies. The first strategy requires a separate server model for each client. The global model is split into a client model and a server model, with each client model having a corresponding server model replica. After the client completes model training, it transmits the intermediate values computed to its corresponding server model through one-to-one parallel communication for further computation. While this approach allows for parallel processing of multiple client requests, multiple model replicas increase server memory usage and computing resource requirements. The second solution maintains only a shared server model on the server side, requiring client data requests to queue and be processed one by one by the server. Clients can only proceed with the next data transfer and update after the previous task has completed. While this approach uses less memory, due to its serial processing, multiple client requests create a waiting queue, impacting system processing efficiency and response speed. Both strategies have their own trade-offs between memory usage and response speed, making it difficult to simultaneously meet the requirements of efficiency and resource utilization. To achieve a balance between memory usage and computational efficiency, a grouping strategy is often introduced. Clients are divided into groups based on the allocation of computing resources or the similarity of data features. Each client in a group shares a single server model copy, which reduces the number of server model copies and memory usage while enabling parallel computing within the group and improving computational efficiency. However, most grouping methods only include a subset of client data within a group, preventing the server model corresponding to that group from learning global information, thus affecting the generalization performance and accuracy of the aggregated global model.

[0007] To further improve training efficiency, training can be optimized by optimizing aggregation strategies, reducing redundant computations, and implementing appropriate communication scheduling. By optimizing aggregation and bandwidth allocation strategies, client computation and communication time can be reduced, thereby improving computational efficiency. To reduce redundant computations during training, clients can share intermediate activation values or features. To improve parallel efficiency, synchronous or asynchronous scheduling strategies can be implemented to achieve efficient computation and communication between clients and servers. However, with the development of neural networks, model sizes are increasing, with many models now having tens of billions or even hundreds of billions of parameters, requiring hundreds of GB to over 1 TB of RAM for training. For example, ChatGPT-3 has 175 billion parameters and requires over 350 GB of graphics memory to run. The Gemini Nano-2 large language model, widely used on edge devices, has 325 million parameters and requires at least 4 GB of graphics memory for inference. As models continue to grow in the future, training these large models will require even more graphics memory and other distributed computing resources. To meet this demand, server clusters are often used to collaboratively train large models. In this training, the model is partitioned and distributed across different servers for separate computations. However, both forward and backward propagation require computation layer by layer, and each layer must wait for the output of the previous layer. This computational dependency causes the server to remain idle during this waiting period, making it underutilized. When training large models, SFL offloads parts of the model that cannot be handled by the client to the server for computation. As the model continues to grow, the computation time of the server cluster becomes a significant bottleneck.

[0008] Combining the above technical solutions, it is found that there is a split federated learning solution, which may cause the server to work serially or occupy memory, and does not take into account the parallel optimization of the server cluster, resulting in inefficient utilization of server resources, resulting in low computing efficiency or resource utilization. Summary of the Invention

[0009] The purpose of this invention is to solve the above problems and design a batch-based parallel split federated learning method.

[0010] To achieve the above-mentioned purpose, the technical solution of the present invention is that, further, in the above-mentioned batch-based parallel split federated learning method, the parallel split federated learning method includes the following steps:

[0011] Divide the global model into two parts, including at least part of the input layer model. Train one part of the model on the client, and train the remaining part on the server cluster.

[0012] Divide the local dataset into batches of fixed size, train the data in batches, and send the intermediate values to the controller after a batch of data training is completed;

[0013] A controller converts the sequential sequence of the intermediate values into a sequence that can be trained in parallel, groups the data using the controller, each group containing a batch of intermediate values of all clients, and assigns a server model to each group;

[0014] The server model is divided into multiple sequences, each of which contains a series of layers. The sequences are assigned to different servers, and a server cluster parallel strategy based on client batch data is implemented. After the server calculates the intermediate gradient, it sends it to the corresponding client for calculation;

[0015] The client receives the intermediate gradients calculated by the server, updates the client model, and sends the model gradients to the federated server;

[0016] The federated server receives model gradients from the client and aggregates all gradients, updates the full client model, and sends it to the client.

[0017] Furthermore, in the above-mentioned batch-based parallel split federated learning method, the global model is divided into two parts, including at least a part of the input layer model, and the part of the model is placed on the client for training, and the remaining part is placed on the server cluster for training, including:

[0018] The original unified global model is divided into two parts, including the client model and the server model;

[0019] The client model is deployed on each terminal device and is used to perform forward calculations on local input data and generate corresponding intermediate activation values. Some models include the first few layers of neural network structures to extract preliminary features. After the client performs preliminary calculations locally, the intermediate results are sent to the server in the form of activation values.

[0020] The server model is deployed on a server cluster to receive the intermediate activation values uploaded by the client and continue to complete the remaining forward calculation and back propagation processes.

[0021] Furthermore, in the above batch-based parallel split federated learning method, the local dataset is divided into batches of fixed size, the data is trained in batches, and after a batch of data is trained, the intermediate value is sent to the controller, including:

[0022] Divide the local dataset into batches of fixed size. For each client i’s dataset, divide it into multiple small batches of data.

[0023] After calculating each small batch of data, the client generates a small batch of intermediate activation values for subsequent model updates and transmissions.

[0024] Furthermore, in the above batch-based parallel split federated learning method, the local dataset is divided into batches of fixed size, the data is trained in batches, and after a batch of data is trained, the intermediate value is sent to the controller, further comprising:

[0025] For each client i’s dataset, it is divided into A small piece of data, represented as ;

[0026] in, Indicates the data, and Represents the input data and corresponding labels of the batch;

[0027] After client i calculates each small batch of data, it generates the intermediate activation value of the small batch , used for subsequent model updates and transmission.

[0028] Furthermore, in the above batch-based parallel split federated learning method, the local dataset is divided into batches of fixed size, the data is trained in batches, and after a batch of data is trained, the intermediate value is sent to the controller, including:

[0029] The controller is used to receive the intermediate activation values uploaded by each client, and uniformly schedule and distribute them according to the batch number to build several parallel training groups;

[0030] Each training group consists of the median values of the same batch index from all clients, and each group contains complete cross-client sample information;

[0031] The controller assigns an independently replicated server-side model copy to each training group for calculation with the server, and the groups execute calculation tasks in parallel.

[0032] Furthermore, in the batch-based parallel split federated learning method, after the server calculates the intermediate gradient, it sends it to the corresponding client for calculation, including:

[0033] Split the server model of each group into multiple sequences. Each sequence contains a series of layers, and multiple servers in the group are responsible for the calculation of the sequence.

[0034] While one sequence is computing a client's mini-batch data, the remaining sequences are computing mini-batch data for other clients. The server fills the gaps in model computation by interleaving forward and backward propagation tasks.

[0035] After the server calculates the forward and backward propagation of the server model, it sends the corresponding gradients asynchronously to the corresponding client.

[0036] Furthermore, in the batch-based parallel split federated learning method, the federated server receives model gradients from the client and aggregates all gradients, updates the full client model, and sends it to the client, including:

[0037] The client receives the intermediate gradient data and continues the backward propagation of the client model, and updates the model;

[0038] The client sends the model update to the federated server, which aggregates the model updates from all clients and updates the global model.

[0039] Furthermore, in a system for implementing the above-mentioned batch-based parallel split federated learning method, the system includes the following modules:

[0040] The model initialization module is used to divide the global model into two parts, including at least part of the input layer model. Part of the model is placed on the client for training, and the remaining part is placed on the server cluster for training;

[0041] The batch computing module is used to divide the local dataset into batches of fixed size, train the data in batches, and send the intermediate values to the controller after the training of a batch of data is completed;

[0042] an intermediate activation value partitioning module, configured to convert the sequential sequence of the intermediate values into a sequence that can be trained in parallel using a controller, group the data using the controller, each group containing a batch of intermediate values from all clients, and assign a server model to each group;

[0043] The parallel computing module is used to divide the server model into multiple sequences, each of which contains a series of layers. The sequences are assigned to different servers, and a server cluster parallel strategy based on client batch data is implemented. After the server calculates the intermediate gradient, it sends it to the corresponding client for calculation;

[0044] The backpropagation calculation module is used to update the client model using the intermediate gradients calculated by the client and send the model gradients to the federated server;

[0045] The gradient aggregation module is used by the federated server to receive model gradients from the client and aggregate all gradients, update the full client model, and send it to the client.

[0046] Furthermore, in a system implementing the above-mentioned batch-based parallel split federated learning method, the batch computing module further includes the following submodules:

[0047] The partitioning submodule is used to divide the local dataset into batches of fixed size. For each client i’s dataset, it is divided into multiple small batches of data.

[0048] The generation submodule is used by the client to generate small batch intermediate activation values after calculating each small batch of data, which is used for subsequent model updates and transmission.

[0049] Furthermore, in a system implementing the above-mentioned batch-based parallel split federated learning method, the parallel computing module further includes the following submodules:

[0050] The split submodule is used to split the server model of each group into multiple sequences. Each sequence contains a series of layers, and multiple servers in the group are responsible for the calculation of the sequence.

[0051] The filling submodule is used to calculate the mini-batch data of a client in one sequence while the remaining sequences calculate the mini-batch data of other clients. The server fills the gaps in model calculation by cross-executing forward and backward propagation tasks.

[0052] The transmission submodule is used to asynchronously send the corresponding gradients to the corresponding clients after the server calculates the forward and backward propagation of the server model.

[0053] The beneficial effects lie in the use of a batch-based parallel split federated learning method. These include: a batch-based parallel framework (BPSFL) is proposed, which balances computational efficiency and memory usage, enabling parallel computation between and within groups on the server, accelerating SFL training time; a batch-based grouping strategy is designed. Client data is divided into multiple batches, which are then divided into groups based on the number of batches. Each group contains batch data from all clients. A server model is replicated for each group, enabling parallel computation between groups while minimizing memory usage; and a server cluster parallel strategy based on client batch data is proposed. This method further splits the server model and computes client batch data in parallel. After the server calculates the gradient, it directly sends it to the client for subsequent computation. This method enables parallel computation within the server cluster, further utilizing computing resources. The effectiveness of the BPSFL algorithm has been verified on three models and four datasets. Experimental results show that BPSFL requires significantly shorter training time per round than four other algorithms, with average training time improvements of 3.5%-46.5%. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Various other advantages and benefits will become apparent to those skilled in the art by reading the following detailed description of the preferred embodiment.The accompanying drawings are only for the purpose of illustrating the preferred embodiment and are not to be considered as limiting the present invention.

[0055] Figure 1 Schematic diagram of a first embodiment of a batch-based parallel split federated learning method in an embodiment of the present invention;

[0056] Figure 2 Schematic diagram of a process of a batch-based parallel split federated learning method in an embodiment of the present invention;

[0057] Figure 3 Schematic diagram of a batch data parallel strategy for a batch-based parallel split federated learning method in an embodiment of the present invention;

[0058] Figure 4 Schematic diagram of server model splitting of a batch-based parallel split federated learning method in an embodiment of the present invention;

[0059] Figure 5 Schematic diagram of a server serial processing computing task of a batch-based parallel split federated learning method in an embodiment of the present invention;

[0060] Figure 6 Schematic diagram of a serial parallel processing computing task of a batch-based parallel split federated learning method in an embodiment of the present invention;

[0061] Figure 7 Schematic diagram of a pipeline parallel execution of forward and backward alternating execution of a batch-based parallel split federated learning method in an embodiment of the present invention;

[0062] Figure 8 This figure is a schematic diagram showing the comparison of rounds and accuracy in the ResNet50 algorithm using a batch-based parallel split federated learning method according to an embodiment of the present invention;

[0063] Figure 9 Schematic diagram showing the comparison of the next round and accuracy of a batch-based parallel split federated learning method in the VGG16 algorithm according to an embodiment of the present invention;

[0064] Figure 10 This is a schematic diagram comparing rounds and accuracy in the AlexNet algorithm using a batch-based parallel split federated learning method according to an embodiment of the present invention;

[0065] Figure 11 Schematic diagram showing a comparison of the average running time per round in the ResNet50 algorithm using a batch-based parallel split federated learning method according to an embodiment of the present invention;

[0066] Figure 12 Schematic diagram showing a comparison of the average running time per round in the VGG16 algorithm using a batch-based parallel split federated learning method according to an embodiment of the present invention;

[0067] Figure 13Schematic diagram showing a comparison of the average running time per round in the AlexNet algorithm using a batch-based parallel split federated learning method according to an embodiment of the present invention;

[0068] Figure 14 This is a schematic diagram of the impact of different batch numbers on computational efficiency in a batch-based parallel split federated learning method in an embodiment of the present invention. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0070] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a," "an," "said," and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the stated features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0071] Example 1

[0072] The present invention will be described in detail below with reference to the accompanying drawings. Figure 1 As shown, a batch-based parallel split federated learning method includes the following steps:

[0073] Step 101: Divide the global model into two parts, including at least a part of the input layer model, train the part on the client, and train the remaining part on the server cluster;

[0074] Specifically, in this embodiment, the original unified global model is divided into two parts, including a client model and a server model;

[0075] The client model is deployed on each terminal device and is used to perform forward calculations on local input data and generate corresponding intermediate activation values. Some models include the first few layers of neural network structures to extract preliminary features. After the client performs preliminary calculations locally, the intermediate results are sent to the server in the form of activation values.

[0076] The server model is deployed on a server cluster to receive the intermediate activation values uploaded by the client and continue to complete the remaining forward calculation and back propagation processes.

[0077] Step 102: Divide the local data set into batches of fixed size, train the data in batches, and send the intermediate values to the controller after the training of a batch of data is completed;

[0078] Specifically, in this embodiment, the local dataset is divided into batches of fixed size, and for each client i's dataset, it is divided into multiple small batches of data;

[0079] After calculating each small batch of data, the client generates a small batch of intermediate activation values for subsequent model updates and transmissions.

[0080] For each client i’s dataset, it is divided into A small piece of data, represented as ;

[0081] in, Indicates the data, and Represents the input data and corresponding labels of the batch;

[0082] After client i calculates each small batch of data, it generates the intermediate activation value of the small batch , used for subsequent model updates and transmission.

[0083] Step 103: The controller converts the sequential sequence of the intermediate values into a sequence that can be trained in parallel, and uses the controller to group the data, each group containing a batch of intermediate values of all clients, and assigns a server model to each group;

[0084] Specifically, in this embodiment, the controller is used to receive intermediate activation values uploaded from various clients, and uniformly schedule and distribute them according to batch numbers to construct several parallel training groups;

[0085] Each training group consists of the median values of the same batch index from all clients, and each group contains complete cross-client sample information;

[0086] The controller assigns an independently replicated server-side model copy to each training group for calculation with the server, and the groups execute calculation tasks in parallel.

[0087] Step 104: Divide the server model into multiple sequences, each of which contains a series of layers. Assign the sequences to different servers and implement a server cluster parallel strategy based on client batch data. When the server calculates the intermediate gradient, it sends it to the corresponding client for calculation.

[0088] Specifically, in this embodiment, the server model of each group is split into multiple sequences, each sequence contains a series of layers, and multiple servers in the group are responsible for the calculation of the sequence;

[0089] While one sequence is computing a client's mini-batch data, the remaining sequences are computing mini-batch data for other clients. The server fills the gaps in model computation by interleaving forward and backward propagation tasks.

[0090] After the server calculates the forward and backward propagation of the server model, it sends the corresponding gradients asynchronously to the corresponding client.

[0091] Step 105: The client receives the intermediate gradient calculated by the server, updates the client model, and sends the model gradient to the federated server.

[0092] Step 106: The federated server receives the model gradients from the client and aggregates all gradients, updates the full client model, and sends it to the client.

[0093] Specifically, in this embodiment, the client receives the intermediate gradient data to continue the backward propagation of the client model and update the model;

[0094] The client sends the model update to the federated server, which aggregates the model updates from all clients and updates the global model.

[0095] The beneficial effects lie in the use of a batch-based parallel split federated learning method. These include: a batch-based parallel framework (BPSFL) is proposed, which balances computational efficiency and memory usage, enabling parallel computation between and within groups on the server, accelerating SFL training time; a batch-based grouping strategy is designed. Client data is divided into multiple batches, which are then divided into groups based on the number of batches. Each group contains batch data from all clients. A server model is replicated for each group, enabling parallel computation between groups while minimizing memory usage; and a server cluster parallel strategy based on client batch data is proposed. This method further splits the server model and computes client batch data in parallel. After the server calculates the gradient, it directly sends it to the client for subsequent computation. This method enables parallel computation within the server cluster, further utilizing computing resources. The effectiveness of the BPSFL algorithm has been verified on three models and four datasets. Experimental results show that BPSFL requires significantly shorter training time per round than four other algorithms, with average training time improvements of 3.5%-46.5%.

[0096] Example 2

[0097] Specifically, in order to balance computing efficiency and memory usage and improve the parallel efficiency of split federated learning, the BPSFL solution is proposed.

[0098] like Figure 2 As shown in Figure 2, the core idea of BPSFL is to divide the client data into multiple batches. The controller groups the intermediate values calculated by the client into batches and then sends them to the server cluster for parallel training. The components of BPSFL are:

[0099] The client trains the data in batches. After completing a batch of training, the intermediate values are sent to the controller. After the server completes the computation and receives the intermediate gradients, the client model is updated and the model gradients are sent to the federated server.

[0100] Controller: To enable parallel computation, the controller transforms the sequential sequence into a sequence that can be trained in parallel. The controller groups the data into batches, with the number of groups equal to the number of batches. Each group contains the median values of a batch from all clients, and a server model is assigned to each group.

[0101] Splitting the server cluster: To reduce idle time in model training caused by dependencies between model layers, we further partition the server model into multiple sequences, each containing a series of layers. After partitioning the model, the sequences are assigned to servers. A server cluster parallelization strategy based on client batch data is implemented, computing client batches in parallel and alternating between forward and backward passes. Once the server calculates intermediate gradients, it directly sends them to the corresponding client for computation.

[0102] Federated server: Receives model gradients from clients and aggregates all gradients, updates the full client model, and sends it to the client.

[0103] Initially, BPSFL partitions the global model into client and server models and distributes them to clients and split server clusters. After dividing the data into batches of appropriate size, the clients compute the forward propagation in batches and upload the intermediate results to the controller. The controller groups the intermediate values by batch and assigns a server model to each group. Within the group, the server model is further split, and a parallel strategy is implemented to efficiently compute the forward and backward propagation of the server model. After the computation is complete, the intermediate gradient values are distributed to the corresponding clients, who then continue the backward propagation of their models. Finally, the federated server aggregates the gradients of all client models and updates them.

[0104] The specific method is:

[0105] Dividing data into small batches is a common optimization method for client-side model training. Dividing data into multiple small batches for gradual updates during training not only effectively reduces model training latency but also facilitates parallel scheduling across server clusters. Therefore, we divide client data into batches for training and utilize batch processing to achieve parallel computing.

[0106] We divide the local dataset into batches of fixed size, and for each client The data set is divided into B small data, expressed as ,in Indicates the data, of which and Represents the input data and corresponding labels of the batch. Accordingly, client i generates the intermediate activation value of the small batch after calculating each small batch of data , used for subsequent model updates and transmission. To reduce the client's waiting time while waiting for the server to process the backward computation, we set up groups equal to the number of batches and distribute the batched intermediate values output by the client to these groups. Each group contains the intermediate activation values of a batch of clients. After grouping, a copy of the server model is assigned to each group, allowing the server to compute data for all clients in parallel.

[0107] For each client i, the training process is to divide the data into B batches , calculate the data and output the corresponding intermediate activation value, recorded as:

[0108] ;

[0109] in It is the mapping relationship between the input data x and the predicted value of the model parameter w.

[0110] The controller distributes the intermediate values of the batch to multiple groups. Each group contains a batch of data for all clients. There will be a total of the same number of groups as the number of batches, and a copy of the server model is made for each group. Clients within each group are trained sequentially, and clients between groups are trained in parallel. We assign a server model to each group, denoted as For each server model, the update process in the hth round of training is expressed as:

[0111] ;

[0112] in is the gradient of the server model.

[0113] After the server sends the calculated intermediate gradient to the client, the client model is updated as follows:

[0114] ;

[0115] in is the gradient of a batch of client i.

[0116] Finally, the federation server aggregates all client models:

[0117] .

[0118] like Figure 3 The figure below illustrates the specific process of the batch data parallel strategy. Five local clients evenly divide the training data into three batches. The model then performs forward computation on each batch and sends the calculated intermediate activation values to the controller. The controller then divides the received sequential sequence into three groups, each containing a mini-batch of client data. Each group is assigned a server model. Within each group, the server model is computed forward and backward according to the sequential sequence, with computations performed in parallel across groups. Finally, the server sends the calculated intermediate gradients to the corresponding client. The client then performs backward propagation and calculates the error to update the model gradient.

[0119] The batch data parallel strategy avoids allocating a server model to each client, but instead allocates it according to the number of batches, which saves memory and achieves parallel computing.

[0120] While the above method achieves parallel computing to a certain extent, the computational efficiency of the server after splitting is still limited. The main reasons are as follows: 1) Computation tasks within a group are still executed sequentially: Although we assign computation tasks to multiple groups through grouping and achieve parallel computing between groups, clients within each group still execute computation tasks sequentially. This means that even if some clients' computation tasks have completed, other waiting clients still need to wait for the previous client's task to complete, resulting in a computational bottleneck. Especially when the number of clients is large, if the number of batches is small (far smaller than the number of clients), the waiting time caused by sequential execution within a group is further exacerbated, thus affecting the parallel efficiency of the training process. 2) Computation tasks have dependencies: During the model splitting process, most model parameters and computation tasks are typically processed on the server. When executing a computation task for a client, due to the dependencies between model layers, the output of the previous layer must be waited for before the next layer can be computed. This results in a large number of layers being idle during the computation. During this process, other clients cannot effectively use the idle time for forward propagation computation. To address the above issues, we draw on the pipeline parallel strategy and combine it with the batch data parallel strategy to propose a server cluster parallel strategy based on client batch data to achieve parallel computing within the group.

[0121] Pipeline parallelism is an effective parallelization method in distributed parallel computing. To improve computational efficiency, data parallelism is often used in distributed computing. Data is divided into multiple batches and distributed to various compute nodes for parallel computation. However, this approach has a drawback: compute nodes may have heterogeneous resources, meaning not all nodes can store the complete model. Therefore, a model parallel approach has been proposed. This approach divides the model into different parts based on network conditions and the computing power of the compute nodes, with each node computing a portion of the model. While this approach addresses resource heterogeneity, model computations must wait for the results of the previous layer before continuing, making model parallelism essentially a serial computation. To achieve parallel computation based on model parallelism, prior art approaches have proposed further dividing batches of data into smaller batches. After computing a smaller batch, compute node 1 transmits it to compute node 2. At this point, compute node 1 can compute the next smaller batch while compute node 2 simultaneously computes the received smaller batch. After all forward computations are completed, backward computation begins, and upon completion, the model is updated. This approach achieves parallel computation through pipeline parallelism. To further improve parallel efficiency, existing techniques have proposed prioritizing backward propagation calculations over the remaining forward propagation when the backward propagation can be calculated. This alternating forward and backward calculations further reduces idle time between worker nodes. However, because this method involves asynchronous updates, it requires storing multiple outdated weights, which incurs additional storage overhead.

[0122] Our approach further decomposes the server model, leveraging grouped client batches for parallel computation. We optimize the scheduling and computational process by implementing a parallel strategy that alternates forward and backward propagation. Because parallel training operates on grouped batches of data, each containing a batch of data for each client. Each client's intermediate gradients have no dependencies on other clients' computations, so we directly distribute the calculated intermediate weights to the server without requiring storage. This means that each client only needs to perform forward and backward propagation based on the current input data and the distributed weights, avoiding frequent data transfer and storage operations.

[0123] Specifically, after grouping, the server cluster is evenly divided. We split the server model of each group into multiple sequences, each of which contains a series of layers. Multiple servers within the group are responsible for the calculation of the sequence.

[0124] like Figure 4 As shown, the server model is split into three sequences, each sequence contains two layers of the server model, and the sequences are distributed among the server clusters within the group for calculation.

[0125] Assume that within a group, the server model is divided into 3 sequences and there are 3 servers, each responsible for calculating one sequence. And the intermediate values of the existing 5 clients need to be calculated, represented by 1, 2, 3, 4 and 5. If there is no optimization strategy, the intermediate values will be calculated in series.

[0126] like Figure 5 The figure below illustrates a traditional serial computation method. Because sequences have computational dependencies, similar to model training, computation of the previous sequence must complete before the next one can be computed. After Server 1 completes the forward propagation for Client 1, it must send the forward propagation to Server 2. Server 1 can only compute the backward propagation after Server 2 completes the backward propagation. Consequently, this method generates significant idle time and fails to efficiently utilize the model's computational resources.

[0127] To improve computational efficiency, we leverage this split server model to compute client intermediate values in parallel on each server, implementing a server cluster parallelization strategy based on client batch data. While servers have computational dependencies on one client's data, they don't have dependencies on other clients' data. Therefore, servers can run computations for other clients during idle time.

[0128] like Figure 6 As shown in the figure, when server 2 is computing client 1's forward propagation, server 1 can now compute client 2's forward propagation in parallel during the time that server 1 would have been idle, turning the previously serial computation into a parallel one. After all forward propagation computations are complete, the backward propagation computation begins in parallel. This allows multiple computations to be executed simultaneously. However, this approach does not maximize parallel efficiency, and the device still has a lot of idle time.

[0129] To further reduce device idle time, we implement a server cluster parallelization strategy based on client batch data. After the last server completes the forward propagation calculation, it immediately begins processing the backward propagation calculation for the current mini-batch, and other servers also prioritize backward propagation. After completing the forward propagation, the server asynchronously sends the gradient to the corresponding client and no longer stores the weights. Furthermore, after completing the calculation of a mini-batch, the server begins processing the next mini-batch of data without waiting for other computation tasks to complete.

[0130] like Figure 7As shown, after server 3 completes the forward calculation for client 1, it immediately calculates the backward calculation for client 1 and transmits the result directly to the client. By interleaving forward and backward calculations, each server fills the idle time in the pipeline, thereby improving device utilization.

[0131] In pipeline parallel computing, resource utilization efficiency is often limited by the slowest server. If a server takes too long to compute, other servers in the pipeline may become idle, preventing faster sequences from being computed, and thus reducing resource utilization. Therefore, to maximize the efficiency of pipeline parallelization, the model should be partitioned as evenly as possible, ensuring that all servers spend a relatively even amount of time computing.

[0132] It should be explained that the above parallel split federated learning method has the following specific process:

[0133] Initial model partitioning: The global model is divided into two parts. The part containing the input layer is trained on the client, and the remaining part is trained on the server. Client batch training: The client divides the dataset into two batches and trains these batches of data in sequence, generating the same number of intermediate activation values. When the client calculates the intermediate activation values, it immediately sends them to the controller. Controller grouping: After receiving the intermediate activation values from each client, the controller groups them. Each training group consists of intermediate values from the same batch index from all clients, ensuring that each group contains complete cross-client sample information. The controller assigns an independently replicated server-side model copy and server to each training group, and the forward and backward propagation of the server model is completed in parallel between the groups. Split server cluster parallel training: In split server, the server cluster is evenly divided into groups, and the server model of each group is further split into multiple sequences. The size of the split sequences is the same as the number of batches, and the servers within the group are assigned sequences for calculation. Forward and backward propagation are interleaved across multiple servers. After the backward gradient calculation is complete, the intermediate gradients are sent to the client for further calculation. Client model backward calculation: The client continues to calculate the backward propagation and updates the model. The federated server aggregates the client model updates: The client sends the model updates to the federated server, which aggregates the model updates from all clients and updates the global model. A new global model is then issued, and the client and server models are updated separately.

[0134] All algorithms used in the embodiments of the present invention were implemented on an NVIDIA GeForce GTX 4090D using NVIDIA CUDA 11.7. The software environment was Ubuntu 22.04, Python version 3.10, and PyTorch version 2.1.2.

[0135] In a specific implementation, the effectiveness of the BPSFL algorithm was verified using four public, real-world datasets: CIFAR-10, CIFAR-100, MNIST, and FashionMNIST (FMNIST). CIFAR-10 consists of 60,000 color images divided into 10 categories. CIFAR-100 expands the image categories of CIFAR-10, also with 60,000 images divided into 100 categories. MNIST and FMNIST consist of 70,000 grayscale images divided into 10 categories. FMNIST is closer to real-world application scenarios than MNIST. We evaluated the BPSFL algorithm on these four datasets using three different convolutional neural network models: ResNet50, VGG16, and AlexNet.

[0136] During training, we set up a server and 20 clients, with each client's data divided into two batches. The model was evenly divided into three sequences: the sequence containing the input layer was run on the client, and the remaining two sequences were run on the server for parallel computation. We evaluated BPSFL's computational speed and convergence rate from five perspectives: the impact of different batch sizes on computational efficiency, the effectiveness of BPSFL on different models, convergence on different datasets, a comparison of rounds and accuracy for different algorithms, and a comparison of average run time per round for different algorithms. We compared BPSFL with four algorithms: SFLV1, SFLV2, FedSL, and EPSL, in terms of convergence speed and computational time. SFLV1, a SplitFed variant, assigns a server model to each client during training for parallel training. SFLV2, another SplitFed variant, uses a single server model during training and processes client computations serially. FedSL, for multi-segment sequential data on different clients, combines the advantages of both the FL and SL architectures to propose a new split federated learning framework for processing multi-segment sequential data. This method divides the recurrent neural network into multiple sub-networks and assigns them to the client for training. The server aggregates the sub-networks to generate a global model. PSFL is a split federated learning framework for resource-constrained devices. This method divides the model into three parts (client front layer, server middle layer, client back layer), and divides the training into two stages. In the first stage, a generalized model is trained to provide initial weights, while in the second stage, each client trains the back-layer weights individually based on the global model obtained in the first stage, allowing the client to perform personalized fine-tuning asynchronously.

[0137] like Figure 8 、 Figure 9 、 Figure 10As shown in the figure, the round and accuracy comparison of different algorithms are shown. We use three models to compare the convergence speed of BPSFL and SFLV1, SFLV2, PSFL, and FedSL on different data sets.

[0138] like Figure 8 As shown in the figure, on ResNet-50, BPSFL converges significantly faster than other algorithms on CIFAR-10 and CIFAR-100, without sacrificing model accuracy. Due to the high accuracy of FMNIST and MNIST, with accuracy exceeding 80% in the first few rounds, the convergence speed on these two datasets is similar to that of SFLV2. However, subsequent experiments demonstrate that BPSFL outperforms other algorithms in average training time per round.

[0139] like Figure 9 As shown in Figure 2, on VGG16, BPSFL converges fastest on all four datasets, especially on CIFAR-10 and CIFAR-100. Since segmentation learning cannot converge on CIFAR10 and CIFAR100 on VGG, and SFLV1 and SFLV2 inherit the characteristics of segmentation learning, they cannot converge either

[14] .

[0140] like Figure 10 As shown in Figure 2, this phenomenon is also observed on AlexNet. SFLV1 and SFLV2 fail to converge on CIFAR-10 and CIFAR-100, but converge well on FMNIST and MNIST. BPSFL, on the other hand, converges quickly on all four datasets. Experiments show that BPSFL achieves rapid convergence without compromising model accuracy, achieving top performance across all datasets and models.

[0141] like Figure 11 、 Figure 12 、 Figure 13 The figure shows the average running time per round for different algorithms. We use three models to compare the average training speed of BPSFL and four algorithms, SFLV1, SFLV2, FedSL, and PSFL, on different datasets.

[0142] like Figure 11 As shown in the figure, BPSFL requires the shortest training time per round on ResNet50, and compared with other algorithms on CIFAR-100, BPSFL's training time advantage is even more significant. On ResNet50, BPSFL's average training time is accelerated by 22.5%, 17.5%, 46.5%, and 32.5% compared to SFLV1, SFLV2, FedSL, and PSFL, respectively.

[0143] like Figure 12As shown in the figure, on VGG16, BPSFL has the shortest average training time per round, while SFLV1 has the longest average training time per round. Furthermore, compared to the other four algorithms, BPSFL's average training time is accelerated by 38.5%, 21.9%, 33.5%, and 27.4%, respectively.

[0144] like Figure 13 As shown, on AlexNet, both BPSFL and FedSL have shorter average training times, while PSFL has the longest average training time, outperforming FedSL. Compared to the other four algorithms, BPSFL's average training time is 8.4%, 9.6%, 23.4%, and 3.5% faster. Experiments demonstrate that BPSFL improves training efficiency across multiple datasets and models, with average per-round training time significantly lower than that of the other algorithms.

[0145] like Figure 14 Figure 2 shows the impact of different batch sizes on computational efficiency. We first verify the impact of different batch sizes on computational time. Because the batch size used to divide client data can affect parallelization efficiency, we divide each client data into 2, 4, 5, and 10 batches. We then use ResNet50, AlexNet, and VGG16 datasets to verify the batch size that achieves the fastest computational speed. Figure 10 The average training time for different batch sizes on four datasets is shown. Experiments show that splitting the data into two and four batches performs better. On the ResNet50 model, the average training time per round for splitting the data into two batches is 91.7 seconds and 92.5 seconds on CIFAR-10 and CIFAR-10, respectively, while the average training time for splitting the data into four batches is 91.4 seconds and 92.6 seconds. Splitting the data into two and four batches requires similar training time, but splitting the data into two batches takes less time across all datasets. Splitting the data into two batches takes the shortest training time across all datasets on the VGG16 model. Splitting the data into four and five batches takes similar training times, but both take longer than splitting the data into two batches. Splitting the data into 10 batches takes the longest training time. Splitting the data into two batches also results in the longest training time on AlexNet. Overall, splitting the data into two batches results in the shortest average training time per round. Therefore, in the following experiments, we split the client data into two batches and use this to verify the effectiveness of the BPSFL algorithm.

[0146] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely preferred examples of the present invention and are not intended to limit the present invention. Various changes and improvements may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and improvements fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. A batch-based parallel split federated learning method, characterized in that: The parallel split federated learning method comprises the following steps: Divide the global model into two parts, including at least part of the input layer model. Train one part of the model on the client, and train the remaining part on the server cluster. Divide the local dataset into batches of fixed size, train the data in batches, and send the intermediate values to the controller after a batch of data training is completed; A controller converts the sequential sequence of the intermediate values into a sequence that can be trained in parallel, groups the data using the controller, each group containing a batch of intermediate values of all clients, and assigns a server model to each group; The server model is divided into multiple sequences, each of which contains a series of layers. The sequences are assigned to different servers, and a parallel strategy based on client-server clusters is implemented. After the server calculates the intermediate gradient, it sends it to the corresponding client for calculation; The client receives the intermediate gradients calculated by the server, updates the client model, and sends the model gradients to the federated server; The federated server receives model gradients from the client and aggregates all gradients, updates the full client model, and sends it to the client.

2. The batch-based parallel split federated learning method according to claim 1, characterized in that: The global model is divided into two parts, including at least part of the input layer model, and part of the model is placed on the client for training, and the remaining part is placed on the server cluster for training, including: The original unified global model is divided into two parts, including the client model and the server model; The client model is deployed on each terminal device and is used to perform forward calculations on local input data and generate corresponding intermediate activation values. Some models include the first few layers of neural network structures to extract preliminary features. After the client performs preliminary calculations locally, the intermediate results are sent to the server in the form of activation values. The server model is deployed on a server cluster to receive the intermediate activation values uploaded by the client and continue to complete the remaining forward calculation and back propagation processes.

3. The batch-based parallel split federated learning method according to claim 1, characterized in that: The method divides the local dataset into batches of fixed size, trains the data in batches, and sends the intermediate values to the controller after the training of a batch of data is completed, including: Divide the local dataset into batches of fixed size. For each client i’s dataset, divide it into multiple small batches of data. After calculating each small batch of data, the client generates a small batch of intermediate activation values for subsequent model updates and transmissions.

4. The batch-based parallel split federated learning method according to claim 1, wherein: The method further includes dividing the local data set into batches of fixed size, training the data in batches, and sending the intermediate values to the controller after the training of a batch of data is completed. For each client i’s dataset, it is divided into A small piece of data, represented as ; in, Indicates the data, and Represents the input data and corresponding labels of the batch; After client i calculates each small batch of data, it generates the intermediate activation value of the small batch , used for subsequent model updates and transmission.

5. The batch-based parallel split federated learning method according to claim 1, characterized in that: The method divides the local dataset into batches of fixed size, trains the data in batches, and sends the intermediate values to the controller after the training of a batch of data is completed, including: The controller is used to receive the intermediate activation values uploaded by each client, and uniformly schedule and distribute them according to the batch number to build several parallel training groups; Each training group consists of the median values of the same batch index from all clients, and each group contains complete cross-client sample information; The controller assigns an independently replicated server-side model copy to each training group for calculation with the server, and the groups execute calculation tasks in parallel.

6. The batch-based parallel split federated learning method according to claim 1, characterized in that: After the server calculates the intermediate gradient, it sends it to the corresponding client for calculation, including: Split the server model of each group into multiple sequences. Each sequence contains a series of layers, and multiple servers in the group are responsible for the calculation of the sequence. While one sequence is computing a client's mini-batch data, the remaining sequences are computing mini-batch data for other clients. The server fills the gaps in model computation by interleaving forward and backward propagation tasks. After the server calculates the forward and backward propagation of the server model, it sends the corresponding gradients asynchronously to the corresponding client.

7. The batch-based parallel split federated learning method according to claim 1, characterized in that: The federated server receives model gradients from the client and aggregates all gradients, updates the full client model, and sends it to the client, including: The client receives the intermediate gradient data and continues the backward propagation of the client model, and updates the model; The client sends the model update to the federated server, which aggregates the model updates from all clients and updates the global model.

8. A system for implementing the batch-based parallel split federated learning method as claimed in claim 1, characterized in that: The system includes the following modules: The model initialization module is used to divide the global model into two parts, including at least part of the input layer model. Part of the model is placed on the client for training, and the remaining part is placed on the server cluster for training; The batch computing module is used to divide the local dataset into batches of fixed size, train the data in batches, and send the intermediate values to the controller after the training of a batch of data is completed; an intermediate activation value partitioning module, configured to convert the sequential sequence of the intermediate values into a sequence that can be trained in parallel using a controller, group the data using the controller, each group containing a batch of intermediate values from all clients, and assign a server model to each group; The parallel computing module is used to divide the server model into multiple sequences, each of which contains a series of layers. The sequences are assigned to different servers and a parallel strategy based on the client-server cluster is implemented. When the server calculates the intermediate gradient, it sends it to the corresponding client for calculation. The backpropagation calculation module is used to update the client model using the intermediate gradients calculated by the client and send the model gradients to the federated server; The gradient aggregation module is used by the federated server to receive model gradients from the client and aggregate all gradients, update the full client model, and send it to the client.

9. The system of the batch-based parallel split federated learning method according to claim 8, characterized in that: The batch calculation module also includes the following submodules: The partitioning submodule is used to divide the local dataset into batches of fixed size. For each client i’s dataset, it is divided into multiple small batches of data. The generation submodule is used by the client to generate small batch intermediate activation values after calculating each small batch of data, which is used for subsequent model updates and transmission.

10. The system of the batch-based parallel split federated learning method according to claim 8, characterized in that: The parallel computing module also includes the following submodules: The split submodule is used to split the server model of each group into multiple sequences. Each sequence contains a series of layers, and multiple servers in the group are responsible for the calculation of the sequence. The filling submodule is used to calculate the mini-batch data of a client in one sequence while the remaining sequences calculate the mini-batch data of other clients. The server fills the gaps in model calculation by cross-executing forward and backward propagation tasks. The transmission submodule is used to asynchronously send the corresponding gradients to the corresponding clients after the server calculates the forward and backward propagation of the server model.

Images