A multi-server method to accelerate the training speed of segmentation learning model

By distributing client data to multiple servers for parallel training, the method accelerates segmentation learning model training speed and enhances privacy, addressing efficiency and security concerns in existing models.

CN116702883BActive Publication Date: 2024-12-03HAINAN HANGQI CULTURE TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310684834.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-06-09
Publication Date
2024-12-03
Estimated Expiration
2043-06-09

AI Technical Summary

Technical Problem

Existing segmentation learning models suffer from reduced operating efficiency and privacy security issues due to sequential client interactions with a single server, which limits training speed and exposes model data to potential privacy threats.

Method used

Client data is segmented and distributed to multiple servers for parallel training, where each server interacts with different parts of the client data, allowing for accelerated convergence while maintaining privacy through reduced direct contact with the client model.

Benefits of technology

The method significantly improves training speed and convergence efficiency while ensuring model privacy, outperforming existing methods in terms of time efficiency and maintaining accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116702883B_ABST
    Figure CN116702883B_ABST
Patent Text Reader

Abstract

The present invention proposes a method for accelerating the training speed of a segmentation learning model using multiple servers, which belongs to the field of distributed deep learning; specifically, each client divides the data into data 1 and data 2; client a uses its data 1 to train with server s1, and updates the model to client b, and s1 updates its model; client b uses its data 1 and s1 to train according to the above method, and at the same time, client a uses its data 2 to train with server s2 according to the above method; client c continues to train according to the above method, and at the same time, client b aggregates the two client models obtained, and based on this, continues training using its data 2 and s2. The above training process continues until the end of this round of training, after which the server obtains a global server model through the client's aggregation forwarding, and the client obtains the latest model through model forwarding for the next round of training; until the number of training rounds set in the experiment is completed. The present invention accelerates the training speed while protecting the privacy of the model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of distributed deep learning, and relates to the model efficiency and privacy security of segmentation learning. Specifically, it is a method for accelerating the training speed of segmentation learning models by using multiple servers. Background Art

[0002] One of the characteristics of split learning models is that the model is updated in a way that causes multiple clients to execute sequentially.

[0003] In the traditional multi-user segmented learning system, only one client interacts with the server for training at the same time; compared with the model in which multiple clients are trained in parallel in federated learning, the overall operating efficiency of the model is greatly reduced.

[0004] At present, the solution to the problem of split learning operation efficiency is mainly through server aggregation, client local parallel or ensemble learning, etc. For example, reference 1: Thapa C, Arachchige PCM, Camtepe S, et al. Splitfed: When federated learning meets split learning [J]. arXiv preprint arXiv: 2004.12088, 2020. It proposes to use a separate aggregation server to aggregate the client-side model. Its essence is a framework that combines split learning with federated learning, called SplitFed.

[0005] In this framework, the client and the main server jointly perform the training process by splitting the model. In addition, a Fed server is introduced to aggregate the client's model, so as to combine the advantages of split learning and federated learning. By introducing split learning to split the complete model, some clients with insufficient computing power can participate in training, and at the same time, there is better model privacy. By introducing a Fed server to aggregate the client model, the client part can be trained in parallel, retaining the advantages of federated learning. However, this framework violates the main goal of split learning, which is to avoid model sharing between server and client.

[0006] Reference 2: Gao Y, Kim M, Thapa C, et al. Evaluation and optimization of distributed machine learning techniques for internet of things [J]. IEEE Transactions on Computers, 2021, 71(10): 2538-2552. A more generalized splitfed learning (SFLG) framework was proposed for SplitFed; SFLG is an improvement based on SplitFed's SFLV2.

[0007] The main improvement is to group the data on the main server side, specifying that a group receives data from several clients and trains them. The training is carried out sequentially within each group, and then the models are aggregated in parallel between groups to improve the training speed. Like the SplitFed framework, this method still uses the server to receive the client's model, which is not conducive to the privacy security of the model.

[0008] Reference 3: Jeon J, Kim J. Privacy-sensitive parallel split learning [A]. / / 2020 International Conference on Information Networking (ICOIN) [C]. IEEE, 2020: 7-9.) It is believed that the sequential training method of split learning is prone to overfitting, so a parallel training framework is proposed to prevent overfitting caused by differences in training order and data set size of each client.

[0009] This method splits the client's local data into multiple small batches, where the size of each batch is proportional to the size of its local data set. All clients first initialize the model synchronously. In each round of training, each client trains on a small batch of data and propagates forward to the segmentation layer, and sends the segmentation layer output to the server. The server performs forward and backward propagation, and each client also performs backward propagation based on the loss value transmitted by the server to obtain the gradient update of the local model. Then each client sends the local gradient update to the server, the server aggregates the received gradients, and sends the aggregated gradient information to the client for model update.

[0010] After such a process, the model data of all clients will be the same. However, this method still uses the method of server-aggregated client models, which will pose a certain threat to the privacy security of the model. Summary of the invention

[0011] In response to the above problems, the present invention proposes a method for accelerating the training speed of a segmented learning model using multiple servers. First, the client data is segmented, and different client data is sent to different servers for training. This speeds up the training of the model while ensuring privacy, thereby achieving the technical effect of improving the convergence speed.

[0012] The method for accelerating the training speed of the segmentation learning model by using multiple servers has the following specific steps:

[0013] Step 1: Build a communication scenario including a client and a server, and each client randomly divides its data into a first part of data and a second part of data;

[0014] The number of servers is the same as the number of data copies of the client;

[0015] Step 2: For the first client a, first use its first part of data a_1 to initialize the model W on the local client C Forward propagation training is performed on the network, and the output data of the segmentation layer is sent to the first server s1.

[0016] Step 3: The first server s1 initializes the model W on the local server S Continue training the output data of the segmentation layer, and back propagate, and send the gradient returned to the segmentation layer to client a;

[0017] Step 4: Client a continues to back-propagate and updates its initial model W based on the returned gradient C for And send it to the next client b.

[0018] At the same time, server s1 updates its initial model W S for

[0019] Step 5: Client B gets the model After that, the first part of data b_1 is used for forward propagation training, and the segmentation layer output data is sent to server s1; at the same time, the model of client a Use the second part of data a_2 for forward propagation training, and get the segmentation layer output data to send to server s2.

[0020] Step 6: Server s1 uses the segmentation layer output data sent by client b to generate the local server model Perform forward propagation and back propagation on the client, send the gradient to client b, and client b continues to train to obtain the updated model And send this model to client c, server s1 continues to send the model it has At the same time, server s2 and client a perform back propagation. After training, client a obtains the model and sends it to client b, server s2 updates its model

[0021] Step 7: Client c uses data c_1 in the model Client b will get two client models through interactive training with server s1: With model Perform aggregation, and use data b_2 and server s2 to continue training on the aggregated model to obtain the model And send this model to the next client.

[0022] Step 8: Continue the above training process until server s1 sends the model After K updates, server s2 will update the model After K updates, the two servers send their models to the client, and the client aggregates the server models to obtain the global server model W. S , and then forwarded to the two servers for the next round of training. The last client also sends its final model Sent to client a for the next round of training.

[0023] The value of K should be consistent with the number of clients.

[0024] Step 9: Continue until the number of training rounds set in the experiment is completed, ultimately accelerating the training speed of the segmentation learning model.

[0025] The advantages of the present invention are:

[0026] A method for accelerating the training speed of a segmentation learning model by using multiple servers improves the convergence efficiency of the model while avoiding direct contact between the server and the client model. In addition to accelerating the training speed, compared with other existing methods for increasing the training speed, the present invention protects the privacy of the model to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A schematic diagram of a method for accelerating the training speed of a segmentation learning model using multiple servers according to the present invention;

[0028] Figure 2 A flow chart of a method for accelerating the training speed of a segmentation learning model using multiple servers according to the present invention;

[0029] Figure 3The figure is a schematic diagram comparing the accuracy of the present invention and the traditional segmentation learning model. DETAILED DESCRIPTION

[0030] The present invention will be further described in detail below with reference to the accompanying drawings and implementation examples.

[0031] The present invention is a method for accelerating the training speed of a segmented learning model by using multiple servers. It adopts a solution of segmenting client data. By sending different client data to different servers for training, the training speed of the client model and the server model is accelerated while ensuring privacy, achieving the technical effect of improving the convergence speed, and having an advantage in training speed while ensuring the privacy of the model.

[0032] The method of accelerating the training speed of the segmentation learning model by using multiple servers is as follows: Figure 1 and Figure 2 As shown, the following steps are taken:

[0033] Step 1: Build a communication scenario including a client and a server, and each client randomly divides its data into a first part of data and a second part of data;

[0034] The number of servers is the same as the number of data copies of the client;

[0035] This example selects 5 clients and 2 servers.

[0036] Step 2: In the first round of training, for the first client a, first use its first part of data a_1 to initialize the local client model W C Forward propagation training is performed on the network, and the output data of the segmentation layer is sent to the first server s1.

[0037] Step 3: The first server s1 uses the split layer output data to initialize the model W on the local server S Continue training and back propagate, sending the gradient returned to the segmentation layer to client a;

[0038] Step 4: Client a continues to back-propagate and updates its initial model W based on the returned gradient C for And send it to the next client b; at the same time, server s1 updates its initial model W S for

[0039] Step 5: Client B gets the model After that, the first part of data b_1 is used for forward propagation training, and the segmentation layer output data is sent to server s1; at the same time, the model of client a Use the second part of data a_2 for forward propagation training, and get the segmentation layer output data to send to server s2.

[0040] From this step, parallel training begins. At this time, there are two sets of clients and servers running simultaneously, corresponding to Figure 1 There are two "③" in the first group. Client b gets the model After that, the first part of the data b_1 is used to perform forward propagation training on the model, and finally the segmentation layer output data is obtained and sent to the server s1. In the second group, client a still has the model It uses its second part of data a_2 for forward propagation training and sends the resulting segmentation layer output data to server s2.

[0041] Step 6: Server s1 uses the segmentation layer output data sent by client b to generate the local server model Perform forward propagation and back propagation on the client, send the gradient to client b, and client b continues to train to obtain the updated model And send this model to client c, server s1 continues to send the model it has At the same time, server s2 and client a perform back propagation. After training, client a obtains the model and sends it to client b, server s2 updates its model

[0042] There are still two sets of clients and servers running simultaneously in the system, corresponding to Figure 1 There are two “④” in it.

[0043] In the first group, server s1 uses the segmentation layer output data sent by client b to generate the local server model. Perform forward propagation and back propagation on the client, send the gradient to client b, and client b continues to train to obtain the updated model And send this model to client c, server s1 continues to send the model it has to update.

[0044] The second group is the back propagation between server s2 and client a. The process is the same as the back propagation of the first group. After the training is completed, client a obtains the model and sends it to client b, server s2 updates its model

[0045] Step 7: Client c uses data c_1 in the model Client b will get two client models through interactive training with server s1: With model Perform aggregation, and use data b_2 and server s2 to continue training on the aggregated model to obtain the model And send this model to the next client.

[0046] Step 8: Continue the above training process until server s1 sends the model Updated 5 times, server s2 will model After 5 updates, the two servers send their models to the client, and the client aggregates the server models to obtain the global server model W. S , and then forwarded to the two servers for the next round of training. The last client also sends its final model Sent to client a for the next round of training.

[0047] Step 9: Continue until the number of training rounds set in the experiment is completed, ultimately accelerating the training speed of the segmentation learning model.

[0048] In this embodiment, the number of training rounds is selected as 50 times.

[0049] The present invention adopts a solution of segmenting client data. By submitting different client data to different servers for training, the model training speed is accelerated while ensuring privacy, achieving the technical effect of improving the convergence speed, and having an advantage in training speed while ensuring model privacy.

[0050] The present invention uses different data sets, network models and hardware devices to conduct relevant experiments. The experiment uses Python 3.9.12 to run on the PyTorch framework. The specific experimental configuration of the experiment is as follows:

[0051]

[0052] The results show that:

[0053] like Figure 3As shown, the accuracy of the traditional segmentation learning model and the multi-server accelerated segmentation learning training speed model proposed in the present invention is compared. This experiment was conducted on the HAM10000 dataset based on the ResNet18 model. It can be observed from the figure that throughout the training process, the multi-server accelerated segmentation learning training speed model proposed in the present invention is not much different from the traditional segmentation learning model in terms of accuracy. The overall trends of the curves of the two are roughly the same, and there are also small fluctuations during the training process. During the 20th to 30th rounds of training, the model accuracy of both fluctuated around 75%. After the 30th round, the accuracy of both was basically stable at above 75%. Finally, after 50 rounds of training, the accuracy of the traditional segmentation learning model was 74.8%, and the accuracy of the multi-server parallel segmentation learning model proposed in the present invention was 75.9%, which was slightly higher than the traditional segmentation learning model.

[0054] In terms of model efficiency, the traditional segmentation learning time in this experiment was 127.48 minutes, while the model proposed in the present invention took 74.64 minutes, which saved about 46.02% of the time compared with the traditional segmentation learning model. The experimental results show that the multi-server parallel segmentation learning model proposed in the present invention improves the model training speed of segmentation learning.

[0055] The following two tables show the comparison data of model accuracy and training time of the model proposed in this invention and the traditional segmentation learning model under different data sets and network models.

[0056] In terms of model accuracy, the four groups of experiments have little difference in model accuracy. This is because the multi-server parallel segmentation learning model proposed in the present invention is essentially a change in the model training order, that is, for any client, all the data it has is not trained with the server at one time. For this experiment, the number of servers is set to 2, and the client's data is divided into two parts. The client first interactively trains half of the data with the server, and then interactively trains the other half of the data with another server after the training is completed. In this process, the client's model will be updated in sequence according to the client's training order, and the server's model will also be aggregated and updated after each round of training.

[0057] Comparison of accuracy of different data sets under different network models

[0058] Dataset Network Model SL This model SplitFed FL HAM10000 ResNet18 74.8% 75.9% 75.07% 75.6% MNIST ResNet18 95.6% 94.7% 95.2% 94.8% CIFAR10 LeNet 66.26% 66.37% 64.1% 65.62% CIFAR10 VGG16 85.19% 83.67% 83.8% 85.53%

[0059] Comparison of time consumption of different data sets under different network models

[0060] Dataset Network Model SL time (min) Time taken for this model (min) SplitFed(min) FL(min) HAM10000 ResNet18 127.48 74.64 35.88 33.75 MNIST ResNet18 42.15 25.58 10.12 11.53 CIFAR10 LeNet 13.01 8.11 3.87 3.28 CIFAR10 VGG16 69.74 42.64 17.02 14.42

Claims

1. A multi-server method to accelerate the training speed of segmentation learning models. It is characterized in that The specific steps are as follows: Step 1: Build a communication scenario including a client and a server, and each client randomly divides its data into a first part of data and a second part of data; Step 2: For the first client a, first use its first part of data a_1 to initialize the model W on the local client C Perform forward propagation training on the s1 layer, and get the output data of the segmentation layer and send it to the first server s1; Step 3: The first server s1 initializes the model W on the local server S Continue training the output data of the segmentation layer, and back propagate, and send the gradient returned to the segmentation layer to client a; Step 4: Client a continues to back-propagate and updates its initial model W based on the returned gradient C for And send it to the next client b; at the same time, server s1 updates its initial model W S for Step 5: Client B gets the model After that, the first part of data b_1 is used for forward propagation training, and the segmentation layer output data is sent to server s1; at the same time, the model of client a Use the second part of data a_2 for forward propagation training, and get the segmentation layer output data to send to server s2; Step 6: Server s1 uses the segmentation layer output data sent by client b to generate the local server model Perform forward propagation and back propagation on the client, send the gradient to client b, and client b continues training to obtain the updated model And send this model to client c, server s1 continues to send the model it has At the same time, the server s2 and the client a perform back propagation. After the training is completed, the client a obtains the model and sends it to client b, server s2 updates its model Step 7: Client c uses data c_1 in the model Client b will get two client models through interactive training with server s1: With model Perform aggregation, and use data b_2 and server s2 to continue training on the aggregated model to obtain the model And send this model to the next client; Step 8: Return to step 2 and continue training until server s1 sends the model After K updates, server s2 will update the model After K updates, the two servers send their models to the client, and the client aggregates the server models to obtain the global server model W. S , and then forwarded to the two servers for the next round of training; the last client also sent its final model Sent to client a for the next round of training; The value of K is consistent with the number of clients, and K is less than or equal to 5; Step 9: Continue until the number of training rounds set in the experiment is completed, ultimately accelerating the training speed of the segmentation learning model.

2. A method for accelerating the training speed of a segmentation learning model using multiple servers as claimed in claim 1, It is characterized in that In step 1, the number of servers is the same as the number of data copies of the client.