Batch-based parallel splitting federated learning method

Abstract

The invention discloses a batch-based parallel splitting federated learning method. The method comprises the steps of global model division, local data batch division, calculation of intermediate value grouping, efficient parallel calculation of a server model and client model aggregation. According to the method, a global model is divided into server models and client models, client data is divided into batches and grouped according to the data batches, each group comprises all client data, and the server models are distributed according to groups. In addition, according to the method, parallel training of a server cluster is achieved, a server model is divided into a plurality of sequences, the sequences are distributed to the server cluster, and parallel computing is achieved by scheduling data of different clients among servers. According to the method, the convergence is faster while the precision is not reduced, and the average training time of each round is accelerated by 3.5%-46.5%.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence, and particularly to a batch-based parallel split federated learning method. Background Art

[0002] With the wide application of neural networks, a large amount of data is used to train neural network models, promoting the development of technologies in fields such as computer vision, natural language processing, and edge computing. However, this data often contains personal sensitive information, such as medical records, consumption behaviors, and location data. While the large-scale use of this data improves the model training effect, it also poses a serious risk of privacy leakage. As a privacy-preserving distributed machine learning framework, federated learning can enable data to stay local, and only share model parameters to participate in distributed model training. However, with the development of neural networks, the model scale is increasing day by day, and there are differences in computing power, storage speed, etc. among the computing nodes participating in training. Although there are a large number of clients, their processing capabilities are limited and they cannot accommodate a complete large model.

[0003] To be able to train large models on resource-limited clients, SplitFed proposed the split federated learning framework (SFL). In SFL, the model w is usually split into two sub-models, namely , where represents the client model assigned to the client, represents the server model assigned to the server. The client model usually includes the first few layers of the model because it needs to input data, while the server model includes the remaining model layers. For example, taking the classic convolutional neural network (CNN) architecture as an example, usually the client model includes the input layer and the convolutional layer, while the server model includes the fully connected layer and the output layer. After the forward propagation of the client computing model , the intermediate activation values are sent to the split learning server. The split learning server is responsible for the forward and backward propagation of the subsequent layers, and then sends them back to the client to complete the final backward propagation of the model, thus completing the forward and backward propagation of the entire model. Finally, the federated server aggregates the model updates of all clients.

[0004] The goal of SFL is to minimize the loss function : ; where is the loss function of the server model, The loss function for the client model; SFL usually has two execution strategies. The first strategy requires allocating a separate server model for each client. The global model is split into two parts: the client model and the server model. Each client model has a corresponding copy of the server model. After the client model training is completed on the client side, the calculated intermediate values are transmitted through one-to-one parallel communication to its corresponding server model for subsequent calculations. Although this method allows parallel processing of requests from multiple clients, multiple model copies will increase the memory occupancy and computational resource requirements on the server side. The second solution is to only maintain a shared server model on the server side, and the data requests from clients need to queue up and wait to be processed by the server one by one. The client can only perform the next data transmission and update after waiting for the previous task to complete. Although this method has a lower memory occupancy, due to the serial processing method, multiple client requests will cause a waiting queue, affecting the processing efficiency and response speed of the system. These two strategies have their own advantages and disadvantages in terms of memory occupancy and response speed, and it is difficult to meet the requirements of both high efficiency and resource utilization at the same time. To achieve a balance between memory occupancy and computational efficiency, a grouping strategy is usually introduced. According to the allocation of computational resources or the similarity of data characteristics, the clients are divided into several groups. The clients within each group share a copy of the server model. This can not only reduce the number of server model copies and lower the memory occupancy, but also enable parallel computing within the group to improve the computational efficiency. However, most of the grouping methods only include a part of the client data within the group, resulting in the server model corresponding to this group not being able to learn global information, thus affecting the generalization performance and accuracy of the aggregated global model.

[0005] To further improve the training efficiency, training can be optimized by optimizing the aggregation strategy, reducing redundant calculations, and performing reasonable communication scheduling. By optimizing the aggregation strategy and bandwidth allocation strategy, the computing and communication time of the client is reduced to improve the computing efficiency. To reduce redundant calculations in training, the client shares intermediate activation values or features. To improve parallel efficiency, through synchronous or asynchronous scheduling strategies, efficient computing and communication between the client and the server are achieved. However, with the development of neural networks, the model scale is increasing day by day, and the number of parameters of many models has reached tens of billions or even hundreds of billions, requiring hundreds of GB to more than 1TB of running memory for training. For example, ChatGPT-3 has 175 billion parameters and requires more than 350GB of video memory to run. The large language model Gemini Nano-2 widely used on end devices has 325 million parameters and requires at least 4GB of video memory for inference. In the future, the model will continue to grow, and the training of these large models requires more video memory and other distributed computing resources. To meet this demand, server clusters are usually used to cooperate in training large models. In such training, the model is divided and allocated to different servers for calculation respectively. However, both the forward propagation and the backward propagation need to be calculated layer by layer, and the latter layer must wait for the output of the previous layer. This computational dependency causes the server to be idle during the waiting process and not fully utilized. When SFL trains large models, it calculates the parts of the model that cannot be accommodated by the client on the server side. As the model continues to grow, the computing time of the server cluster becomes a bottleneck that cannot be ignored.

[0006] Combined with the above technical solutions, it is found that there is a split federated learning solution that may cause the server to work serially or occupy memory, and does not consider the parallel optimization of the server cluster, resulting in inefficient utilization of server resources and low computing efficiency or resource utilization. Summary of the Invention

[0007] The purpose of the present invention is to solve the above problems and design a batch-based parallel split federated learning method.

[0008] To achieve the above object, the technical solution of the present invention is as follows. Further, in the above batch-based parallel split federated learning method, the parallel split federated learning method includes the following steps: Divide the global model into two parts, at least including a partial model of the input layer, place the partial model for training on the client, and place the remaining part for training on the server cluster; Divide the local dataset into batches of fixed size, train the data in batches, and send the intermediate value to the controller when the training of a batch of data is completed; The controller converts the sequential sequence in the intermediate values into a sequence that can be trained in parallel, groups the data using the controller, with each group containing a batch of intermediate values from all clients, and assigns a server model to each group; The server model is divided into multiple sequences, each sequence containing a series of layers. The sequences are assigned to different servers, implementing a server cluster parallel strategy based on client batch data. After the server calculates the intermediate gradients, they are sent to the corresponding clients for calculation; The client receives the intermediate gradients calculated by the server, updates the client model, and sends the model gradients to the federated server; The federated server receives the model gradients from the clients and aggregates all the gradients, updates the full client model, and distributes it to the clients.

[0009] Furthermore, in the above batch-based parallel split federated learning method, dividing the global model into two parts, at least including the partial model of the input layer, placing the partial model for training on the client, and placing the remaining part for training on the server cluster, includes: Dividing the original unified global model into two parts, including the client model and the server model; Among them, the client model is deployed on each terminal device, used to perform forward calculation on local input data and generate corresponding intermediate activation values; the partial model includes the first few neural network structures, used to extract preliminary features; after the client performs preliminary calculations locally, the intermediate results are sent to the server side in the form of activation values; Among them, the server model is deployed on the server cluster, used to receive the intermediate activation values uploaded by the client and continue to complete the remaining forward calculation and backpropagation process.

[0010] Furthermore, in the above batch-based parallel split federated learning method, dividing the local dataset into batches of fixed size, training the data in batches, and sending the intermediate values to the controller after a batch of data training is completed, includes: Dividing the local dataset into batches of fixed size. For the dataset of each client i, it is divided into multiple small batches of data; After the client calculates each small batch of data, it generates small batch intermediate activation values for subsequent model updates and transmissions.

[0011] Furthermore, in the above batch-based parallel split federated learning method, dividing the local dataset into batches of fixed size, training the data in batches, and sending the intermediate values to the controller after a batch of data training is completed, also includes: For the dataset of each client i, it is divided into small data portions, denoted as ; Among them, represents the th data, and represent the input data and corresponding labels of this batch; After the client i calculates each mini-batch of data, it generates mini-batch intermediate activation values , which are used for subsequent model updates and transmissions.

[0012] Furthermore, in the above batch-based parallel split federated learning method, the dividing the local dataset into batches of fixed size, training the data in batches, and when a batch of data training is completed, sending the intermediate value to the controller includes: The controller is used to receive the intermediate activation values uploaded from each client, and uniformly schedule and allocate them according to the batch numbers to construct several parallel training groups; Each training group consists of intermediate values with the same batch index from all clients, and each group contains complete cross-client sample information; The controller allocates an independently replicated server-side model copy for each training group to perform calculations with the server, and the calculation tasks are executed in parallel between groups.

[0013] Furthermore, in the above batch-based parallel split federated learning method, the when the server calculates the intermediate gradient and then sends it to the corresponding client for calculation includes: Split the server model of each group into multiple sequences, each sequence contains a series of layers, and multiple servers within the group are responsible for the calculation of the sequence; When calculating the mini-batch data of a certain client in a sequence, the remaining sequences calculate the mini-batch data of other clients, and the server fills the idle time in the model calculation by cross-executing the forward and backward propagation tasks; After the server calculates the forward and backward propagation of the server model, it asynchronously sends the corresponding gradients to the corresponding clients.

[0014] Furthermore, in the above batch-based parallel split federated learning method, the federated server receives the model gradients from the clients and aggregates all the gradients, updates the full-client model, and sends it to the clients, including: The client receives the intermediate gradient data and continues the backward propagation of the client model, and updates the model; The client sends the update of the model to the federated server, and the federated server aggregates the model updates of all clients and updates the global model.

[0015] Further, in the system for implementing the above batch-based parallel split federated learning method, the system includes the following modules: A model initialization module, configured to divide the global model into two parts, at least including a partial model of the input layer, place the partial model on the client for training, and place the remaining part on the server cluster for training; A batch calculation module, configured to divide the local dataset into batches of a fixed size, train the data in batches, and send the intermediate value to the controller when the training of a batch of data is completed; An intermediate activation value division module, configured to use the controller to convert the sequential sequence in the intermediate value into a sequence that can be trained in parallel, use the controller to group the data, each group contains the intermediate values of one batch of all clients, and assign a server model to each group; A parallel calculation module, configured to divide the server model into multiple sequences, each sequence contains a series of layers, assign the sequences to different servers, implement a parallel strategy for the server cluster based on the client batch data, and send the intermediate gradients calculated by the server to the corresponding clients for calculation when they are calculated; A backpropagation calculation module, configured to use the client to receive the intermediate gradients calculated by the server, update the client model, and send the model gradients to the federated server; A gradient aggregation module, configured to receive the model gradients from the clients by the federated server and aggregate all the gradients, update the full client model, and send it to the clients.

[0016] Further, in the system for implementing the above batch-based parallel split federated learning method, the batch calculation module further includes the following sub-modules: A division sub-module, configured to divide the local dataset into batches of a fixed size, and divide the dataset of each client i into multiple small batches of data; A generation sub-module, configured to generate small batch intermediate activation values after each client calculates each small batch of data, for subsequent model update and transmission.

[0017] Further, in the system for implementing the above batch-based parallel split federated learning method, the parallel calculation module further includes the following sub-modules: A split sub-module, configured to split the server model of each group, divide it into multiple sequences, each sequence contains a series of layers, and multiple servers within the group are responsible for the calculation of the sequences; A filling sub-module, configured to when a sequence calculates the small batch data of a certain client, the remaining sequences calculate the small batch data of other clients, and the server fills the idle time in the model calculation by cross-executing the forward and backward propagation tasks; A transmission sub-module, which is used to asynchronously send the corresponding gradients to the corresponding clients after the server has completed the forward and backward propagation of the server model.

[0018] Its beneficial effects are as follows. Through the batch-based parallel splitting federated learning method, including: proposing a batch-based BPSFL parallel framework, which can balance computational efficiency and memory occupancy, and implement inter-group and intra-group parallel computing in the server to accelerate the training time of SFL; designing a batch-based grouping strategy. Divide the client data into multiple batches, divide the groups according to the number of batches, and each group contains the batch data of all clients. And replicate a server model for each group to avoid a large amount of memory occupancy while implementing parallel computing between groups; proposing a server cluster parallel strategy based on client batch data. This method further splits the server model and calculates the client batch data in parallel. After the server calculates the gradients, it directly sends them to the clients for subsequent calculations. This method realizes the parallel computing of the server cluster and further utilizes the computing resources; we verified the effectiveness of the BPSFL algorithm on three models and four datasets. The experimental results show that compared with the other four algorithms, BPSFL requires less time for each round of training on average, and the average training time per round is accelerated by 3.5% - 46.5%. Description of the Drawings

[0019] By reading the detailed description of the preferred embodiments below, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of showing the preferred embodiments and are not considered to be a limitation of the present invention.

[0020] Figure 1 Schematic diagram of the first embodiment of a batch-based parallel splitting federated learning method in an embodiment of the present invention; Figure 2 Flow chart of a batch-based parallel splitting federated learning method in an embodiment of the present invention; Figure 3 Schematic diagram of the batch data parallel strategy of a batch-based parallel splitting federated learning method in an embodiment of the present invention; Figure 4 Schematic diagram of the server model splitting of a batch-based parallel splitting federated learning method in an embodiment of the present invention; Figure 5 Schematic diagram of the server serial processing of computing tasks of a batch-based parallel splitting federated learning method in an embodiment of the present invention; Figure 6 Schematic diagram of the sequence parallel processing of computing tasks of a batch-based parallel splitting federated learning method in an embodiment of the present invention; Figure 7Schematic diagram of the pipelined parallel execution of the forward and backward alternating execution of a batch-based parallel split federated learning method in an embodiment of the present invention; Figure 8 Schematic diagram of the comparison of rounds and accuracy of a batch-based parallel split federated learning method in the ResNet50 algorithm in an embodiment of the present invention; Figure 9 Schematic diagram of the comparison of the next round and accuracy of a batch-based parallel split federated learning method in the VGG16 algorithm in an embodiment of the present invention; Figure 10 Schematic diagram of the comparison of rounds and accuracy of a batch-based parallel split federated learning method in the AlexNet algorithm in an embodiment of the present invention; Figure 11 Schematic diagram of the comparison of the average running time per round of a batch-based parallel split federated learning method in the ResNet50 algorithm in an embodiment of the present invention; Figure 12 Schematic diagram of the comparison of the average running time per round of a batch-based parallel split federated learning method in the VGG16 algorithm in an embodiment of the present invention; Figure 13 Schematic diagram of the comparison of the average running time per round of a batch-based parallel split federated learning method in the AlexNet algorithm in an embodiment of the present invention; Figure 14 Schematic diagram of the influence of different batch numbers of a batch-based parallel split federated learning method in an embodiment of the present invention on the calculation efficiency. Detailed implementation manners

[0021] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0022] Those skilled in the art of the present technology can understand that unless specifically stated otherwise, the singular forms "a", "an", "the" and "said" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present invention means the presence of the described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or their groups.

[0023] Embodiment 1 The present invention will be specifically described below with reference to the accompanying drawings. As Figure 1 shown, a batch-based parallel split federated learning method, the parallel split federated learning method includes the following steps: Step 101: Divide the global model into two parts, at least including a partial model of the input layer. Place the partial model on the client side for training, and place the remaining part on the server cluster for training; Specifically, in this embodiment, the original unified global model is divided into two parts, including a client model and a server model; Among them, the client model is deployed on each terminal device and is used to perform forward calculation on local input data and generate corresponding intermediate activation values; the partial model includes the first few neural network structures and is used to extract preliminary features; after the client performs preliminary calculations locally, it sends the intermediate results to the server side in the form of activation values; Among them, the server model is deployed on the server cluster and is used to receive the intermediate activation values uploaded by the client and continue to complete the remaining forward calculation and backpropagation process.

[0024] Step 102: Divide the local dataset into batches of fixed size, train the data in batches, and send the intermediate values to the controller after a batch of data training is completed; Specifically, in this embodiment, the local dataset is divided into batches of fixed size. For the dataset of each client i, it is divided into multiple small batches of data; After the client calculates each small batch of data, it generates small batch intermediate activation values for subsequent model updates and transmissions.

[0025] For the dataset of each client i, it is divided into small pieces of data, denoted as ; Among them, represents the th piece of data, and represent the input data and the corresponding labels of this batch; After the client i calculates each small batch of data, it generates small batch intermediate activation values for subsequent model updates and transmissions.

[0026] Step 103: The controller converts the sequential sequence in the intermediate values into a sequence that can be trained in parallel, groups the data using the controller, each group contains the intermediate values of one batch from all clients, and assigns a server model to each group; Specifically, in this embodiment, the controller is used to receive the intermediate activation values uploaded by each client, uniformly schedule and allocate them according to the batch number, and construct several parallel training groups; Each training group is composed of the intermediate values with the same batch index from all clients, and each group contains complete cross-client sample information; The controller allocates an independent replicated copy of the server-side model for each training group to perform calculations with the server, and the calculation tasks are executed in parallel between groups.

[0027] Step 104: Divide the server model into multiple sequences, each sequence containing a series of layers. Allocate the sequences to different servers and implement a server cluster parallel strategy based on client batch data. After the server calculates the intermediate gradients, send them to the corresponding clients for calculation. Specifically, in this embodiment, the server model of each group is split into multiple sequences, each sequence containing a series of layers, and multiple servers within the group are responsible for the calculation of the sequences. When a sequence calculates the mini-batch data of a certain client, the remaining sequences calculate the mini-batch data of other clients. The server fills the idle time in model calculation by cross-executing forward and backward propagation tasks. After the server calculates the forward and backward propagation of the server model, it asynchronously sends the corresponding gradients to the corresponding clients.

[0028] Step 105: The client receives the intermediate gradients calculated by the server, updates the client model, and sends the model gradients to the federated server. Step 106: The federated server receives the model gradients from the clients and aggregates all the gradients to update the global client model, and then sends it to the clients.

[0029] Specifically, in this embodiment, the client receives the intermediate gradient data to continue the backward propagation of the client model and updates the model. The client sends the update of the model to the federated server. The federated server aggregates the model updates of all clients and updates the global model.

[0030] The beneficial effects are as follows. Through the batch-based parallel split federated learning method, including: a batch-based BPSFL parallel framework is proposed, which can balance computational efficiency and memory occupancy, and implement inter-group and intra-group parallel computing in the server to accelerate the training time of SFL; a batch-based grouping strategy is designed. The client data is divided into multiple batches, grouped according to the number of batches, and each group contains the batch data of all clients. And a copy of the server model is made for each group to avoid a large amount of memory occupancy while implementing parallel computing between groups; a server cluster parallel strategy based on client batch data is proposed. This method further splits the server model and calculates the client batch data in parallel. After the server calculates the gradient, it is directly sent to the client for subsequent calculations. This method realizes the parallel computing of the server cluster and further utilizes the computing resources; we verified the effectiveness of the BPSFL algorithm on three models and four datasets. The experimental results show that compared with the other four algorithms, BPSFL requires less time for each round of training on average, and the average training time per round is accelerated by 3.5% - 46.5%.

[0031] Embodiment 2 Specifically for split federated learning, in order to balance computational efficiency and memory occupancy and improve the parallel efficiency of split federated learning, the BPSFL scheme is proposed.

[0032] As Figure 2 shown, the core idea of BPSFL is to divide the client data into multiple batches, and the controller groups the intermediate values calculated by the clients according to the batches and then sends them to the server cluster for parallel training. The BPSFL components are respectively: Client: Train the data in batches. After a batch of data is trained, send the intermediate value to the controller. And after the server calculation is completed. After receiving the intermediate gradient, update the client model and send the model gradient to the federated server.

[0033] Controller: In order to achieve parallel computing, the controller changes the sequential sequence into a sequence that can be parallelly trained. The controller groups the data, and the number of groups is the same as the number of batches. After grouping, each group contains the intermediate value of one batch of all clients, and a server model is assigned to each small group.

[0034] Split server cluster: In order to reduce the idle time of model training caused by the dependency relationship between model layers, we further divide the server model into multiple sequences, each sequence containing a series of layers. After dividing the model, assign the sequences to the servers, implement a server cluster parallel strategy based on client batch data, calculate the client batch data in parallel, and realize the alternating execution between forward and backward passes. When the server calculates the intermediate gradient, it is directly sent to the corresponding client for calculation.

[0035] Federated Server: Receive model gradients from clients, aggregate all gradients, update the global client model, and distribute it to clients.

[0036] At the beginning, BPSFL divides the global model into client models and server models, and distributes them to clients and the split server cluster. After dividing the data into appropriate batch sizes, the clients calculate the forward propagation in batches and upload the intermediate results to the controller. The controller groups the intermediate values according to batches and assigns a server model to each group. Within the group, the server model is further split, and a parallel strategy is executed to efficiently calculate the forward and backward propagations of the server model. After the calculation is completed, the intermediate values of the gradients are sent to the corresponding clients, and the clients continue the backward propagation of the client model. Finally, the federated server aggregates the gradients of all client models and updates them.

[0037] The specific method is as follows: In the training of the client model, dividing the data into small batches is a common optimization method. During the training process, dividing the data into multiple small batches for gradual update can not only effectively reduce the training waiting time of the model, but also make this small batch of data more convenient for parallel scheduling of the server cluster. Therefore, we divide the client data into batches for training and use batch processing to achieve parallel computing.

[0038] We divide the local dataset into batches of a fixed size. For each client 's dataset, it is divided into B small data portions, denoted as , where represents the st data portion, where and represent the input data and the corresponding label of this batch. Correspondingly, after each client i calculates each small batch of data, it generates small batch intermediate activation values , which are used for subsequent model updates and transmissions. To reduce the queuing time of the client while waiting for the server to process the backward calculation, we set the same number of groups as the number of batches, and distribute the batch intermediate values output by the client to these groups. Each group contains the intermediate activation values of one batch of the client. After grouping, a server model is assigned to each group, enabling the server to calculate all client data in parallel.

[0039] For each client i, the training process is to divide the data into B batches , calculate the data and output the corresponding intermediate activation values, denoted as: ; where It is the mapping relationship between the input data x and the predicted value of the model parameter w.

[0040] The controller distributes a batch of intermediate values into multiple groups. Each group contains a batch of data from all clients, and the total number of groups formed is the same as the number of batches. And a copy of the server model is made for each group. The clients within each group are trained in sequence, and the clients between groups are trained in parallel. We allocate a copy of the server model to each group, denoted as For the update process of each server model in the h-th round of training, it is expressed as: ; where is the gradient of the server model.

[0041] After the server sends the calculated intermediate gradients to the clients, the update of the client model is: ; where is the gradient of a batch of client i.

[0042] Finally, the federated server aggregates all client models: .

[0043] As Figure 3 shown, it is the specific process of the batch data parallel strategy. There are five clients locally, which evenly divide the training data into 3 batches respectively. Then the model performs forward calculations on each batch respectively and sends the calculated intermediate activation values to the controller. The controller divides the received sequential sequence into three groups according to batches, and each group contains a small batch of data from the clients. A copy of the server model is allocated to each group, and the forward and backward propagations of the server model are calculated within the group according to the sequential sequence, and the calculations between groups are parallel. Finally, the server sends the calculated intermediate gradients to the corresponding clients, and the clients perform backward propagation and calculate the error to update the model gradients.

[0044] The batch data parallel strategy avoids allocating a copy of the server model to each client, but allocates according to the number of batches, which can save memory and achieve parallel computing.

[0045] Although the above method realizes parallel computing to a certain extent, the computing efficiency of the server after splitting is still restricted by some factors. The main reasons are as follows: 1) The computing tasks within a group are still executed sequentially: Although we allocate computing tasks to multiple groups through grouping and achieve inter-group parallel computing, within each group, the clients still execute the computing tasks in sequence. This means that even if the computing tasks of some clients have been completed, other clients still waiting need to wait for the previous client's task to complete, resulting in a computing bottleneck. Especially when the number of clients is large, if the number of batch divisions is small and the number of batches is much smaller than the number of clients, the waiting time caused by sequential execution within the group will be further aggravated, thus affecting the parallel efficiency of the training process. 2) There are dependencies among computing tasks: During the model splitting process, most model parameters and computing tasks are usually processed on the server side. When executing the computing task of a certain client, due to the dependencies between layers of the model, it is necessary to wait for the output of the previous layer to perform the calculation of the next layer, which makes a large number of layers of the model idle during the calculation. During this process, other clients cannot effectively utilize the idle time to perform the forward propagation calculation. To address the above problems, we draw on the pipeline parallel strategy and combine it with the batch data parallel strategy to propose a parallel strategy for server clusters based on client batch data to achieve parallel computing within a group.

[0046] Pipeline parallelism is an effective parallelization method in distributed parallel computing. In distributed computing, to improve computing efficiency, data parallelism is usually used, where data is divided into multiple batches and distributed to each computing node for parallel computing. However, this method has a drawback that there is resource heterogeneity among computing nodes, and not all computing nodes can have a complete model. Therefore, the model parallelism method is proposed. This method divides the model into different parts according to the network condition and the computing power of the computing nodes, and each node calculates a part of it. Although this method solves the problem of resource heterogeneity, due to the dependence relationship in model calculation, it must wait for the result of the previous layer to continue the calculation. So, model parallelism is essentially serial computing. To achieve parallel computing on the basis of model parallelism, in the prior art, it is proposed to further divide the batch data into small batch data. After computing a small batch of data, computing node 1 passes it to computing node 2. At this time, computing node 1 can calculate the next small batch of data, while computing node 2 calculates the received small batch of data simultaneously. After all forward computations are completed, backward computation starts, and the model is updated after backward computation is completed. This method realizes parallel computing through this pipeline parallelism method. To further improve the parallel efficiency, in the prior art, it is also proposed that when backward propagation can be computed, backward propagation is computed first, and then the remaining forward propagation is computed. This method of alternating forward and backward computations further reduces the idle time between worker nodes. However, since this method belongs to asynchronous update, multiple obsolete weights need to be stored, resulting in additional storage overhead.

[0047] In our method, the server model is further split, and parallel computing is achieved by using the grouped client batch data. Moreover, the parallel strategy of alternating forward propagation and backward propagation is used to optimize the scheduling of computing tasks and the computing process. Since the parallel training is on the grouped batch data, this data contains one batch of data from each client. The intermediate gradients of each client have no dependence on the computations of other clients. Therefore, we directly send down the computed intermediate weights without the server storing them. This means that when each client performs computations, it only needs to perform forward propagation and backward propagation computations based on the current input data and the sent-down weights, thus avoiding frequent data transmission and storage operations.

[0048] Specifically, after grouping, the server cluster is also evenly divided. We split the server model of each group into multiple sequences, and each sequence contains a series of layers. Multiple servers within the group are responsible for the computations of the sequences.

[0049] As Figure 4 shown, the server model is split into three sequences, and each sequence contains two layers of the server model. Sequences are assigned to the server cluster within the group for computations.

[0050] Suppose that within a certain group, the server model is divided into 3 sequences, and there are 3 servers, with each server responsible for the calculation of one sequence. And there are currently 5 client intermediate values to be calculated, represented by 1, 2, 3, 4, and 5. Without any optimization strategy, the intermediate values will be calculated serially. As Figure 5 shown, it is a traditional serial calculation method. Since there are computational dependencies between sequences just like in model training, the next sequence cannot be calculated until the previous sequence calculation is completed. When Server 1 finishes the forward propagation of Client 1, it needs to send it to Server 2 to continue the forward propagation calculation, and Server 1 can only calculate the backward propagation after the backward calculation of Server 2 is completed. Therefore, this method will generate a large amount of idle time and cannot efficiently utilize the computational resources of the model.

[0051] To improve the computational efficiency, using the split server model, we calculate the client intermediate values in parallel on each server to implement a server cluster parallel strategy based on client batch data. Although there are computational dependencies between servers for one client's data, there are no dependencies for other clients' data. Therefore, during the idle time of the servers, the computational tasks of other clients can be calculated.

[0052] As Figure 6 shown, when Server 2 calculates the forward propagation of Client 1, during the time when Server 1 was originally idle, now Server 1 can calculate the forward propagation of Client 2 in parallel, which changes the original serial calculation to a parallel calculation. After all the forward propagation calculations are completed, the backward propagation is calculated in parallel. In this way, multiple computational tasks can be executed in parallel at the same time. However, this method does not maximize the parallel efficiency, and there is still a lot of idle time on the device.

[0053] To further reduce the device idle time, we execute a server cluster parallel strategy based on client batch data. After the last server finishes the forward propagation calculation, it immediately starts to process the backward propagation calculation of the current small batch, and other servers also give priority to processing the backward propagation. After completing the backward propagation, the servers will asynchronously send the gradients to the corresponding clients, and the servers no longer store the weights. At the same time, after the servers complete the calculation of one small batch, they start to process the data of the next small batch without waiting for the completion of other computational tasks.

[0054] As Figure 7 shown, when Server 3 finishes the forward calculation of Client 1, it immediately calculates the backward calculation of Client 1 and directly transmits the calculation result to the client. By interspersing the execution of forward and backward calculations on each server, the idle time in the pipeline is filled, thereby improving the utilization efficiency of the device.

[0055] In pipelined parallel computing, the utilization efficiency of computing resources is usually limited by the slowest server. If a server takes too long to compute, it may cause other servers in the pipeline to be idle while waiting, preventing faster sequences from being computed and reducing resource utilization. Therefore, to maximize the efficiency of pipelined parallelism, the model is preferably divided as evenly as possible to ensure that all servers have relatively uniform computing times.

[0056] It should be explained that the above parallel split federated learning method has the following specific process: Initial model partitioning: The global model is divided into two parts. The partial model containing the input layer is trained on the client side, and the remaining part is trained on the server side. Client-side batch training: The client divides the dataset into two batches and trains these batches of data sequentially to generate the same number of intermediate activation values as the batches. When the client calculates the intermediate activation values, they are immediately sent to the controller. Controller grouping: After receiving the intermediate activation values from each client, the controller groups them. Each training group consists of the intermediate values with the same batch index from all clients, ensuring that each group contains complete cross-client sample information. The controller assigns an independently replicated server-side model copy and a server to each training group, and the forward and backward propagations of the server model are completed in parallel between groups. Split server cluster parallel training: In the split servers, the server cluster is evenly divided into groups, and further, the server models in each group are split into multiple sequences. The size of the split sequences is the same as the number of batches, and the servers within the group are assigned sequences for calculation. The forward and backward propagations are executed interleaved on multiple servers, and after the backward gradient calculation is completed, the intermediate gradients are sent to the client for further calculation. Client-side model backward calculation: The client continues to calculate the backward propagation and updates the model. Federated server aggregates client models: The client sends the model updates to the federated server, which aggregates all the client model updates and updates the global model. The new round of the global model is distributed, and the client model and the server model are updated separately after splitting.

[0057] All algorithms adopted in the embodiments of the present invention are implemented on NVIDIA GeForce GTX 4090D environment using NVIDIA's CUDA 11.7. The software environment is Ubuntu 22.04, the Python version is 3.10, and the PyTorch version is 2.1.2.

[0058] In a specific implementation, four publicly available real-world datasets are used to verify the effectiveness of the BPSFL algorithm, where the datasets include CIFAR-10, CIFAR-100, MNIST, and FashionMNIST (FMNIST). CIFAR-10 consists of 60,000 color images divided into 10 categories. CIFAR-100 extends the image categories on the basis of CIFAR-10 with the same 60,000 images, divided into 100 categories. MNIST and FMNIST include 70,000 grayscale images divided into 10 categories, and FMNIST is closer to the actual application scenario than MNIST. We run these four datasets on three different convolutional neural network models to evaluate the BPSFL algorithm, and the models include ResNet50, VGG16, and AlexNet.

[0059] During the training process, we set up one server and 20 clients, and each client's data is divided into 2 batches. We evenly divide the model into 3 sequences. The sequence containing the input layer is run on the client, and the other two sequences are implemented on the server for parallel computing. We evaluate the computing speed and convergence speed of BPSFL from five aspects: the impact of different batch numbers on computing efficiency, the performance of BPSFL on different models, the convergence of different datasets, the comparison of rounds and accuracy under different algorithms, and the comparison of the average running time per round under different algorithms. We compare the convergence speed and computing time of BPSFL with four algorithms, namely SFLV1, SFLV2, FedSL, and EPSL: SFLV1 is a variant of SplitFed. During the training process, a server model is assigned to each client for parallel training; SFLV2 is another variant of SplitFed. During the training process, there is only one server model, and the computing tasks of the clients are processed serially; FedSL combines the advantages of the FL and SL architectures for multi-segment sequential data on different clients and proposes a new split federated learning framework to process multi-segment sequential data. This method divides the recurrent neural network into multiple sub-networks and assigns them to the clients for training, and the server aggregates the sub-networks to generate a global model; PSFL is a split federated learning framework for resource-constrained devices. This method divides the model into three parts (client front layer, server middle layer, client back layer), and divides the training into two stages. In the first stage, a generalized model is trained to provide initial weights, and in the second stage, each client separately trains the back layer weights on the basis of the global model obtained in the first stage, allowing the clients to perform personalized fine-tuning asynchronously.

[0060] As Figure 8 , Figure 9 , Figure 10As shown in the figure, the comparison of rounds and accuracy under different algorithms is presented. We use three models to compare the convergence speeds of BPSFL and four algorithms, namely SFLV1, SFLV2, PSFL, and FedSL, on different datasets. As Figure 8 shown in the figure, on ResNet50, BPSFL converges significantly faster than other algorithms on CIFAR-10 and CIFAR-100, and the model accuracy does not decrease. Since the accuracies of FMNIST and MNIST are relatively high and reach over 80% accuracy in the first few rounds, the convergence speed of BPSFL is similar to that of SFLV2 on these two datasets. However, in subsequent experiments, it is proven that BPSFL is superior to other algorithms in terms of the average training time per round.

[0061] As Figure 9 shown in the figure, on VGG16, BPSFL converges the fastest on all four datasets, especially on CIFAR-10 and CIFAR-100. Since split learning cannot converge on CIFAR10 and CIFAR100 in VGG, and SFLV1 and SFLV2 inherit the characteristics of split learning, they also cannot converge

[14] .

[0062] As Figure 10 shown in the figure, on AlexNet, the same phenomenon is also observed. SFLV1 and SFLV2 cannot converge on CIFAR-10 and CIFAR-100, but converge well on FMNIST and MNIST. While BPSFL converges quickly on all four datasets. The experiments show that BPSFL can converge quickly without reducing the model accuracy and performs best on all datasets and models.

[0063] As Figure 11 、 Figure 12 、 Figure 13 shown in the figure, it is the comparison of the average running time per round under different algorithms. We use three models to compare the average training speeds of BPSFL and four algorithms, namely SFLV1, SFLV2, FedSL, and PSFL, on different datasets. As Figure 11 shown in the figure, on ResNet50, the training time required by BPSFL per round is the shortest, and the training time advantage of BPSFL is more prominent when compared with other algorithms on CIFAR-100. On ResNet50, compared with the four algorithms SFLV1, SFLV2, FedSL, and PSFL, the average training time of BPSFL is speeded up by 22.5%, 17.5%, 46.5%, and 32.5%.

[0064] As Figure 12As shown in the figure, on VGG16, BPSFL has the shortest average training time per round, while SFLV1 has the longest average training time per round. And compared with the four algorithms, BPSFL's average training time is speeded up by 38.5%, 21.9%, 33.5%, and 27.4%.

[0065] As Figure 13 shown in the figure, on AlexNet, both BPSFL and FedSL have relatively short average training times, while PSFL has the longest average training time, and BPSFL's training time is better than FedSL. Compared with the four algorithms, BPSFL's average training time is speeded up by 8.4%, 9.6%, 23.4%, and 3.5%. Experiments show that BPSFL shows an improvement in training efficiency on multiple datasets and models, and the average training time per round is significantly lower than that of other algorithms.

[0066] As Figure 14 shown in the figure, it is the impact of different batch numbers on the computing efficiency. We first verify the impact of different batch numbers on the computing time. Since the batch size of client data partitioning may affect the parallel efficiency, we divide each client data into 2 batches, 4 batches, 5 batches, and 10 batches, and use ResNet50, AlexNet, VGG16, and four datasets to verify the batch number with the fastest computing speed. Figure 10 The figure shows the average time of one-round training of different batch numbers on four datasets. Experiments show that dividing the data into 2 batches and 4 batches has better effects. On the ResNet50 model, the average training time of one round for CIFAR-10 and CIFAR-10 divided into 2 batches is 91.7 seconds and 92.5 seconds, while the average training time for dividing into 4 batches is 91.4 seconds and 92.6 seconds. The training times required for dividing into 2 batches and 4 batches are about the same, but on all datasets, the training time for dividing into 2 batches is shorter. On the VGG16 model, the training time for dividing into 2 batches is the shortest on all datasets. The training times for dividing into 4 batches and 5 batches are not much different, but both are longer than the training time for dividing into 2 batches. The training time is the longest after dividing into 10 batches. It is also found that the training time for dividing into 2 batches on AlexNet. Generally, the average training time per round is the shortest when dividing into 2 batches. Therefore, in the following experiments, we will divide the client data into 2 batches and verify the effectiveness of the BPSFL algorithm on this basis.

[0067] The foregoing has shown and described the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments, and the above embodiments and the descriptions in the specification are only preferred examples of the present invention and are not used to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements all fall within the scope of the present invention claimed. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims

1. A batch-based parallel splitting federated learning method, characterized in that The described parallel split federated learning method includes the following steps: Divide the global model into two parts, including at least a partial model of the input layer, place the partial model on the client for training, and place the remaining part on the server cluster for training; Divide the local dataset into batches of fixed size, train the data in batches, and send the intermediate value to the controller after a batch of data is trained; The controller converts the sequential sequence in the intermediate value into a sequence that can be trained in parallel, groups the data using the controller, each group contains an intermediate value of a batch from all clients, and assigns a server model to each group; Divide the server model into multiple sequences, each sequence contains a series of layers, assign the sequences to different servers, implement a parallel strategy for the server cluster based on client batch data, and send the intermediate gradients calculated by the server to the corresponding clients for calculation after they are calculated; The client receives the intermediate gradients calculated by the server, updates the client model, and sends the model gradients to the federated server; The federated server receives the model gradients from the clients and aggregates all the gradients, updates the full client model, and sends it to the clients.

2. The parallel splitting federated learning method based on batches according to claim 1, wherein The step of dividing the global model into two parts, including at least a partial model of the input layer, placing the partial model on the client for training, and placing the remaining part on the server cluster for training, includes: Divide the original unified global model into two parts, including a client model and a server model; Among them, the client model is deployed on each terminal device, used to perform forward calculation on local input data, and generate corresponding intermediate activation values; the partial model includes the first few neural network structures, used to extract preliminary features; after the client performs preliminary calculations locally, it sends the intermediate results to the server side in the form of activation values; Among them, the server model is deployed on the server cluster, used to receive the intermediate activation values uploaded by the clients, and continue to complete the remaining forward calculation and backpropagation process.

3. The parallel splitting federated learning method based on batches according to claim 1, wherein, The step of dividing the local dataset into batches of fixed size, training the data in batches, and sending the intermediate value to the controller after a batch of data is trained, includes: Divide the local dataset into batches of fixed size, and divide the dataset of each client i into multiple small batches of data; After the client calculates each small batch of data, it generates small batch intermediate activation values for subsequent model updates and transmissions.

4. The parallel split federated learning method based on batches as claimed in claim 1, wherein The step of dividing the local dataset into batches of fixed size, training the data in batches, and sending the intermediate value to the controller after a batch of data is trained, further includes: For the dataset of each client i, divide it into small data portions, denoted as ; Among them, represents the th data, and represent the input data and corresponding labels of this batch; After the client i finishes calculating each small batch of data, it generates small batch intermediate activation values , which are used for subsequent model updates and transmissions.

5. The parallel splitting federated learning method based on batches according to claim 1, characterized in that The step of dividing the local dataset into batches of fixed size, training the data in batches, and sending the intermediate value to the controller after a batch of data is trained, includes: The controller is used to receive the intermediate activation values uploaded by each client, and uniformly schedule and allocate them according to the batch numbers to construct several parallel training groups; Each training group consists of intermediate values with the same batch index from all clients, and each group contains complete cross-client sample information; The controller allocates an independently replicated copy of the server-side model to each training group for calculation with the server, and the calculation tasks are executed in parallel between groups.

6. The parallel splitting federated learning method based on batches according to claim 1, characterized in that When the server calculates the intermediate gradients and distributes them to the corresponding clients for calculation, it includes: The server model of each group is split into multiple sequences, each sequence contains a series of layers, and multiple servers within the group are responsible for the calculation of the sequences; When calculating the mini-batch data of a certain client in a sequence, the remaining sequences calculate the mini-batch data of other clients, and the server fills the idle time in model calculation by cross-executing forward and backward propagation tasks; After the server calculates the forward and backward propagation of the server model, it asynchronously sends the corresponding gradients to the corresponding clients.

7. A batch-based parallel splitting federated learning method as claimed in claim 1, wherein, The federated server receives the model gradients from the clients and aggregates all the gradients, updates the global client model, and distributes it to the clients, including: The client receives the intermediate gradient data and continues the backward propagation of the client model, and updates the model; The client sends the update of the model to the federated server, and the federated server aggregates the model updates of all clients and updates the global model.

8. A system for implementing a batch-based parallel split federated learning method as claimed in claim 1, wherein, The system includes the following modules: The model initialization module is used to divide the global model into two parts, at least including the partial model of the input layer, place the partial model on the client for training, and place the remaining part on the server cluster for training; The batch calculation module is used to divide the local data set into batches of fixed size, train the data in batches, and send the intermediate value to the controller when a batch of data training is completed; The intermediate activation value division module is used to use the controller to convert the sequential sequence in the intermediate value into a sequence that can be trained in parallel, use the controller to group the data, each group contains the intermediate value of one batch of all clients, and allocate a server model to each small group; The parallel calculation module is used to divide the server model into multiple sequences, each sequence contains a series of layers, allocate the sequences to different servers, implement a parallel strategy for the server cluster based on client batch data, and when the server calculates the intermediate gradients, distribute them to the corresponding clients for calculation; The backward propagation calculation module is used to use the client to receive the intermediate gradients calculated by the server, update the client model, and send the model gradients to the federated server; The gradient aggregation module is used for the federated server to receive the model gradients from the clients and aggregate all the gradients, update the global client model, and distribute it to the clients.

9. The system of a batch-based parallel splitting federated learning method according to claim 8, characterized in that, The batch calculation module further includes the following sub-modules: The division sub-module is used to divide the local data set into batches of fixed size, and divide the data set of each client i into multiple small batches of data; The generation sub-module is used for the client to generate mini-batch intermediate activation values after calculating each mini-batch of data, for subsequent model update and transmission.

10. The system of a batch-based parallel splitting federated learning method according to claim 8, wherein The parallel calculation module further includes the following sub-modules: The splitting sub-module is used to split the server model of each group into multiple sequences, each sequence contains a series of layers, and multiple servers within the group are responsible for the calculation of the sequences; The filling sub-module is used to calculate the mini-batch data of other clients with the remaining sequence when calculating the mini-batch data of a certain client in a sequence. The server fills the gap time in model calculation by cross-executing the forward and backward propagation tasks. The transmission sub-module is used to asynchronously send the corresponding gradients to the corresponding clients after the server calculates the forward and backward propagation of the server model.

Images