A personalized split learning method for distributed systems and applications

Abstract

The application discloses a personalized split learning method and application for a distributed system, and belongs to the technical field of distributed collaborative machine learning. In each round of training, all clients first train a global server model, on the basis of which, the clients are clustered according to data similarity, and a personalized local server model is set for each cluster, and the local server model is fine-tuned based on the global server model in the training process, so that the local server model can be more in line with the local data characteristics of the clients in the cluster compared with the global server model, and the client devices in the cluster can obtain higher prediction accuracy after local training. The global model and the local model are considered at the same time, the model can be trained by using more samples, and the problem of non-independent and identically distributed data of different clients can be solved to a certain extent, so that the prediction accuracy of the model is improved.

Description

Technical Field

[0001] This invention belongs to the field of distributed collaborative machine learning technology, and more specifically, relates to a personalized split learning method and its application for distributed systems. Background Technology

[0002] With the development of the internet, the amount of data is constantly increasing, and data is playing an increasingly important role. Distributed collaborative machine learning allows multiple client devices to participate in training while ensuring data privacy to a certain extent, thus leading to its rapid rise. Federated learning and split learning are its two main implementation methods. Federated learning synchronizes the global model to various clients for training, and after local training is completed, it is aggregated on the server. In federated learning, each client needs to run the complete machine learning model, but resource-constrained clients (such as those available in the Internet of Things) cannot run the complete machine learning model. Furthermore, during training, privacy issues arise because the server and clients have full access to both the local and global models. Split learning, on the other hand, divides the global model into layers, with each client and server storing only a portion of the complete neural network layer, performing computation and feedback separately. In split learning, each client and server train sequentially, and only intermediate results are transmitted, not raw data, resulting in better privacy. Therefore, compared to federated learning, split learning can better alleviate the problem of limited client computing resources and offers better privacy.

[0003] The goal of user participation in split learning is to improve prediction accuracy through participation in global training. However, the non-independent and identically distributed (IID) nature of the client device data involved in training can negatively impact the training model, resulting in limited improvement in prediction accuracy after client participation. To address the challenges posed by statistical heterogeneity and non-independent and identically distributed (Non-IID) data, an effective approach is to personalize the device, data, and model to mitigate heterogeneity and obtain a high-quality personalized model for each device. Therefore, researching a personalized split learning method is of great significance, particularly regarding the personalization techniques introduced by split learning.

[0004] Most personalization techniques typically involve two steps: in the first step, a global model is built collaboratively; in the second step, a personalized local model is trained for each client using the client's private data. Optimizing only the global model results in a model that is more difficult to personalize. For federated learning personalization to be useful in practice, the following three goals must be achieved simultaneously, rather than independently: (1) training a personalized model that benefits the majority of clients; (2) training an accurate global model that benefits those clients with less private data; and (3) achieving fast model convergence in a small number of training rounds.

[0005] Several techniques have been proposed in the field of federated learning to achieve personalized global models, such as personalization based on adding contextual features, personalization based on transfer learning, personalization based on multi-task learning, personalization based on meta-learning, personalization based on knowledge distillation, and personalization that combines global and local models, etc., to better train individual clients. However, existing personalization techniques are mainly designed for federated learning and are not suitable for the framework of split learning. The main reason is that in split learning, the client does not possess the complete neural network. Specifically, the main characteristics of split learning are: 1. It breaks down the complete neural network, placing a small part on the client and the majority on the server, alleviating the client's lack of computing resources. 2. It does not directly share the original data and model, but instead transmits the intermediate results of the neural network computation on the client through layer-by-layer splitting. This is the fundamental reason why existing personalization techniques are not suitable for split learning. For example, in personalization techniques that combine global and local models, in federated learning, the client can learn some or all parameters from the global model to obtain a personalized model. However, in split learning, since neither the client nor the server possesses the complete neural network, the client cannot learn some or all parameters from the global model. In other words, existing personalization techniques cannot be directly used in split learning. Summary of the Invention

[0006] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a personalized split learning method and application for distributed systems, which solves the technical problem of low model training accuracy caused by the inability of existing personalized technology to be applied to split learning and the inability to solve the problem of non-independent and identically distributed data of client devices participating in training.

[0007] To achieve the above objectives, in a first aspect, the present invention provides a personalized splitting learning method for distributed systems, comprising the following steps:

[0008] S1. Interact and train the local model within each client and the global server model within the global server in turn, and repeat the training for s rounds to obtain the global server model on the global server.

[0009] The interactive training process between the local model within the i-th client and the global server model within the global server includes: the i-th client inputs its local training samples into its local model for training, generates intermediate results, and sends them to the global server; after receiving the intermediate results from the i-th client, the global server inputs the intermediate results from the i-th client into its global server model for further training, performs backpropagation after training, and transmits the backpropagation results to the i-th client; after receiving the backpropagation results, the i-th client updates its local model.

[0010] S2. Cluster the clients based on the feature similarity of the local training samples on the clients to obtain M client clusters; and set up M local server models on the global server that correspond one-to-one with the M client clusters.

[0011] S3. Determine whether the current iteration is the first iteration. If so, initialize the M local server models as global server models respectively. Otherwise, perform global regularization fine-tuning on the M local server models under the previous iteration based on the global server model to obtain the M local server models under the current iteration.

[0012] S4. In each client cluster, the local model within each client and its corresponding local server model are interactively trained sequentially according to the training method in step S1, and the training is repeated for s rounds.

[0013] S5. Repeat steps S1-S4 for iteration until the current iteration reaches the preset iteration number.

[0014] More preferably, in step S2 above, a clustering algorithm is used to group each client based on the weighted average of the intermediate results generated by each client, resulting in M ​​client clusters; wherein, the weighted average of the intermediate results generated by the i-th client is:

[0015]

[0016] z i Let r be the number of local training samples in the i-th client; when the current iteration is the first iteration, r il This represents the intermediate result of the l-th round of interactive training between the local model within the i-th client and the global server model within the global server; when the current iteration is not the first iteration, r il This represents the intermediate result of the l-th round of interactive training between the local model within the i-th client and its corresponding local server model in the previous iteration round.

[0017] More preferably, in step S2 above, clients with the same number of categories of local training sample labels are grouped into the same client cluster, resulting in M ​​client clusters.

[0018] More preferably, in step S2 above, the sample label distribution of the local training samples of the client is used to cluster each client using a clustering algorithm to obtain M client clusters; wherein, the sample label distribution of the local training samples of the client is the proportion of the number of local training samples of each category in the client to the total number of training samples of that category in all clients.

[0019] More preferably, in the two preferred embodiments described above, step S2 can also be performed before step S1.

[0020] More preferably, in step S3 above, the m-th local server model in the current iteration round is:

[0021]

[0022] in, This represents the m-th local server model from the previous iteration round; η and λ are regularization parameters. ω represents the gradient of the loss function of the m-th local server model in the previous iteration; t-1 This is the global server-side model from the previous iteration round.

[0023] More preferably, in the m-th client cluster, the local model within the j-th client and the corresponding local server model v are... m The interactive training process includes:

[0024] The j-th client trains its local model based on its internal local training samples, generates intermediate results, and sends them to the global server. Upon receiving the intermediate results from the j-th client, the global server inputs these results into its internal local server model v. m Training continues in the middle, and backpropagation is performed after training is completed. The result of backpropagation is passed to the j-th client. After receiving the result of backpropagation, the j-th client updates its internal local model.

[0025] More preferably, the local model inside the client includes partial layers in the neural network and slicing layers connected after the partial layers; the global server model inside the global server includes slicing layers and the remaining layers in the neural network connected after the slicing layers; any local model inside the client and the global server model inside the global server constitute a complete neural network; the slicing layers are used to transfer intermediate results between the client and the global server.

[0026] In a second aspect, the present invention provides a distributed system, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor executes the personalized splitting learning method provided in the first aspect of the present invention when executing the computer program.

[0027] Thirdly, the present invention also provides a computer-readable storage medium comprising a stored computer program, wherein the computer program, when executed by a processor, controls the device where the storage medium is located to execute the personalized splitting learning method provided in the first aspect of the present invention.

[0028] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects:

[0029] 1. This invention provides a personalized split-learning method for distributed systems. In each training round, all clients first train a global server-side model. Based on this, clients are clustered according to data similarity, and a personalized local server-side model is set up in each cluster. During training, the local server-side models are fine-tuned based on the global server-side model through regularization. This makes the local server-side models more consistent with the local data characteristics of the clients in the cluster compared to the global server-side model. As a result, the client devices in the cluster can achieve higher prediction accuracy after local training. This invention considers both global and local models. A global model with a small loss function value is first trained, and then a personalized local model is maintained for each client device by fine-tuning the global model. The global objective is that the global server-side model trained through split-learning can help client devices achieve high prediction accuracy. The local objective is that, when the global server-side model achieves greater benefits, personalized training is performed in the pre-grouped clusters. This can utilize more samples to train the model and can also solve the problem of non-independent and identically distributed (Non-IID) data among different clients to a certain extent. The model training accuracy is high, which greatly improves the prediction accuracy of the model.

[0030] 2. The personalized splitting learning method for distributed systems provided by this invention combines the characteristics of splitting learning for dynamic clustering. During training via client-server communication, splitting learning does not directly transmit raw data. Instead, it sends the intermediate results obtained from client-side training to the server after splitting into layers for further training. These intermediate results, obtained by training the neural network on the client side, reflect the characteristics of the local samples to a certain extent for each client device. Intermediate results are generated in each training round, allowing for dynamic clustering in each round without leaking the original data, thus offering better privacy.

[0031] 3. The personalized splitting learning method for distributed systems provided by this invention considers the distribution probability of client sample label categories and combines clustering algorithms to perform fixed clustering before splitting learning begins. This not only considers the number of sample label categories on different client devices, but also further considers the possible large differences in the distribution of different category labels of samples on different client devices, which greatly improves the accuracy of model training. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the structure for personalized split learning in a distributed system provided in Embodiment 1 of the present invention;

[0033] Figure 2 This is a schematic diagram of the training process of the personalized splitting learning method based on global regularization provided in Embodiment 1 of the present invention;

[0034] Figure 3 This is a flowchart illustrating the personalized splitting learning method under optional embodiment two of the present invention;

[0035] Figure 4 This is a structural diagram of the clustering module corresponding to the third optional embodiment of the present invention;

[0036] Figure 5 This is a flowchart illustrating the personalized splitting learning method under optional embodiment three of the present invention. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0038] In split learning, the client only trains / tests the client-side network, while the server only trains / tests the server-side network. Training and testing of the entire model are accomplished by performing sequential (forward / backward) propagation between the client and server. In its simplest form, first, forward propagation occurs as follows: the client performs forward propagation, taking input data from the input layer, training through intermediate layers until a specific layer of the network (called the split layer) covers the original data, and then transmitting the activations of the split layer (called shredded data, i.e., the intermediate results output by the client) to the server. Afterward, the server treats the intermediate results received from the client as its input and performs forward propagation on the remaining layers. At this point, a single forward propagation on the complete model has been completed. Now, backpropagation occurs as follows: after calculating the loss, the server begins backpropagation, calculating the weight gradients and layer activations until the split layer, and then transmitting the gradients of the intermediate results back to the client. For the received gradients, the client performs its backpropagation on its client-side network. At this point, one backpropagation between the client and server has been completed. During model training, forward and backward propagation continue until the model is trained on all participating clients and reaches a suitable convergence point (e.g., high prediction accuracy). In split learning, the server and clients are confined to a specified portion of the complete model and cannot access the full model; therefore, SL provides a degree of privacy protection for training the model.

[0039] In the split-learning framework, due to the Non-IID problem of data, the sample label categories in different client devices may differ greatly, which may negatively affect the clients participating in training and thus reduce the prediction accuracy. In extreme cases, different clients may have completely different sample label categories, and the model after split-learning may even reduce its ability to predict new local samples. To improve this situation, this invention proposes a personalized split-learning method for distributed systems, focusing on a scalable and personalized method adapted to the split-learning training process. The following detailed description includes specific embodiments:

[0040] Example 1

[0041] A personalized decomposition learning method, such as Figure 1 As shown, this is used in distributed systems and applied to distributed training scenarios. By breaking down a complete ML model into multiple smaller neural network parts, training them with local data on the client, and alternately executing the training on distributed clients and servers, a globally shared model is generated for all clients.

[0042] Traditionally, split learning aims to consider all client devices participating in training across the network, continuously updating a single global server model ω during training. Split learning allows clients to complete the entire training process by sending intermediate results to the server, enabling the trained model to achieve better prediction results. Its purpose is to achieve the objective function (global objective). Among them, F k (ω) is the local prediction target for client k, and G(·) is used to measure the local prediction target {F} from each client. k (ω)} k∈[K] The overall benefit function measures the average loss function values ​​of each device after training. The goal is to train a global model that minimizes the average loss function value when all client devices are tested.

[0043] However, generally speaking, each device can be distributed through a different distribution. Generate data x k ,Right now To better explain this heterogeneity, it is often necessary to consider techniques for learning personalized, device-specific models within the network. In this invention, personalization is explored through a simple split-learning framework. Two objectives are considered holistically: a global objective and a local objective F. k (v k The goal is to enable a specified client device to learn a personalized model of the server device k. To link these tasks, this invention introduces a regularization term to make the personalized model approximate the optimal global model. The bi-level optimization problem obtained for each client device k∈[K] is as follows:

[0044]

[0045]

[0046] Here, the hyperparameter λ controls the interpolation between the local and global models. When λ is set to 0, the above equation simplifies to training the local model; as λ gradually increases, the equation gradually reverts to solving for the global model objective. An appropriate value of λ provides a trade-off between these two extremes: the smaller λ is, the better the personalized model v... m A larger λ is more prone to deviating from the global model ω, which can be corrupted by significant differences across multiple clients. This might provide robustness at the expense of generalization. Conversely, a larger λ allows learning from the global model ω trained on a large number of samples, but this could sacrifice robustness and be detrimental to predicting local samples for some client devices. In heterogeneous scenarios, there exists a suitable λ to provide both robustness and fairness.

[0047] To improve on the basis of the traditional split learning training process, personalized split learning based on global regularization considers training the personalized local model after training the global model. The local model is fine-tuned from the global model, which can bring better prediction performance improvement to the client device compared to the global model. The whole process is as follows: First, train a global model with a smaller loss function value, and then fine-tune from the global model to maintain a personalized local model for each client device and continue training. However, when the local samples of the client device are few, this configuration will overly rely on the global model, resulting in limited improvement in the personalized prediction accuracy. To solve the above problem, considering that there are some similarity features in the client data, all clients can be divided into several clusters for personalized training, which can, to a certain extent, solve the problem of non-independent and identically distributed data of different clients and can also train as much as possible in the local models trained from more samples to further improve the prediction accuracy.

[0048] Similarly, considering both the global objective and the local objective F k (v k ), the purpose is to enable the specified client device k to learn the personalized server v m 's model, where v m is the local server model in the m-th cluster, m ∈ [M], and M represents the M clusters into which the client devices are divided; for the number of clusters, the relationship M < K is satisfied, and there is at least one client device in each cluster. The bi-level optimization problem obtained by each device k ∈ [K] is shown as follows:

[0049]

[0050]

[0051] To implement personalized split learning based on global regularization, the present invention proposes to jointly solve the global model ω * and the personalized models {v m} m∈[M] in an alternating manner. Specifically, in each round of the training process of the personalized split learning method, the client device first participates in the training of the global server model, and then fine-tunes on the basis of the current round of the global server model combined with the previous round of personalized local server model to generate a new round of local server model. At this time, the client devices in the cluster participate in the personalized training again.

[0052] Specifically, the personalized split learning method for a distributed system in this embodiment includes the following steps:

[0053] S1. Interact and train the local model within each client and the global server model within the global server in turn, and repeat the training for s rounds to obtain the global server model on the global server.

[0054] The interactive training process between the local model within the i-th client and the global server model within the global server includes: the i-th client inputs its local training samples into its local model for training, generates intermediate results, and sends them to the global server; after receiving the intermediate results from the i-th client, the global server inputs the intermediate results from the i-th client into its global server model for further training, performs backpropagation after training, and transmits the backpropagation results to the i-th client; after receiving the backpropagation results, the i-th client updates its local model.

[0055] It should be noted that at the beginning of each interactive training session, the first client performs local training. The local model in the client only contains a portion of the entire neural network, while the other layers reside on the server. After training, the client transmits the intermediate results to the server through its layer splitting. The server receives the results from its layer splitting and uses them as input to its neural network to continue training. After training, backpropagation is performed, and the backpropagation results are transmitted to the first client through the splitting layers. Then, the second client trains, and so on, until all clients have performed gradient transmission with the server, which constitutes a complete round of interactive training.

[0056] S2. Cluster the clients based on the feature similarity of the local training samples on the clients to obtain M client clusters; and set up M local server models on the global server that correspond one-to-one with the M client clusters.

[0057] S3. Determine whether the current iteration is the first iteration. If so, initialize the M local server models as the global server model. Otherwise, perform global regularization fine-tuning on each local server model in the previous iteration based on the global server model to obtain each local server model in the current iteration.

[0058] Specifically, when the current iteration is not the first iteration, the m-th local server model in the current iteration round is:

[0059]

[0060] in, This represents the m-th local server model in the previous iteration round t-1; η and λ are regularization parameters. ω represents the gradient of the loss function of the m-th local server model in the previous iteration; t-1 This is the global server model from the previous iteration round; m = 1, 2, ..., M.

[0061] Specifically, one implementation method for the above-mentioned global regularization fine-tuning is as follows: To measure the degree of global regularization, a regularization parameter α is introduced, and the personalized model update method derived from the global model fine-tuning is shown in the following equation:

[0062]

[0063] in, This represents the parameters of the local server-side model in the t-th training round; Let α represent the parameters of the global server-side model in training round t. To allow clusters with fewer devices (i.e., fewer samples) to learn more from global regularization, we consider making the value of α relatively large; while for clusters with more devices, we consider allowing them to be trained individually to better fit local samples, so the value of α is relatively small.

[0064] S4. In each client cluster, the local model within each client and its corresponding local server model are interactively trained sequentially according to the training method in step S1, and the training is repeated for s rounds.

[0065] In the m-th client cluster, the local model and the corresponding local server model v within the j-th client are defined. m The interactive training process includes:

[0066] The j-th client trains its local model based on its internal local training samples, generates intermediate results, and sends them to the global server. Upon receiving the intermediate results from the j-th client, the global server inputs these results into its internal local server model v. m Training continues in the middle, and backpropagation is performed after training is completed. The result of backpropagation is passed to the j-th client. After receiving the result of backpropagation, the j-th client updates its internal local model.

[0067] S5. Repeat steps S1-S4 for iteration until the current iteration round reaches the preset iteration round. In this embodiment, the preset iteration round is set to 20.

[0068] It should be noted that, due to the presence of numerous labels not present in other clients, or significant differences in distribution between other clients and the original client, the portion trained on the server-side may not accurately predict local data. Therefore, this invention divides all clients participating in split learning into multiple clusters, with one server set up in each cluster. This server is used to pass the splitting data to all clients within the cluster, completing the split learning training. However, simply performing split learning in this way somewhat contradicts the original intention of client participation in split learning, as one of the purposes is for clients with fewer local samples to benefit from the global model. Clustering actually reduces the number of samples participating in the global model. Therefore, this invention considers both global and local models. A global server is set up to train the global server-side model; a separate local server-side model is set up in each cluster to train a personalized local server-side model. The local server-side model in each cluster is not a new server, but resides on the same server as the global server-side model. Multiple local server-side models are maintained for all clusters, thus not increasing hardware overhead compared to traditional split learning. By introducing a regularization term, the local server-side model in each cluster is fine-tuned based on the global server-side model, enabling the clients in the cluster to achieve better training results. Specifically, in each training round, the traditional SL process is followed, allowing each client and the global server to train. At this time, both the client and the global server update their neural network parameters. Then, based on the neural network trained in the previous round of the cluster server (if it is the first round, the global server-side model is synchronized to the local server-side model), combined with the results of the current round of global server-side model training, regularization fine-tuning is performed. At this time, the local server-side model benefits from the global server-side model trained on a large number of global samples, and also benefits from the samples of the cluster that better match the local data distribution characteristics, thus achieving better prediction of local samples.

[0069] Specifically, such as Figure 2 As shown, the training process of a personalized split learning method based on global regularization, with 4 clients divided into 2 clusters, is illustrated as an example. At the start of the first round of training, steps 1-8 in the diagram represent the 4 clients performing traditional SL training with the global model, updating the global server-side model, and simultaneously performing backpropagation to the clients. Step 9 involves updating the personalized local server-side model of each cluster using the regularized global server-side model. Specifically, in the first round, the global server-side model is directly synchronized to the local server-side model; in other rounds, the local server-side model is updated by combining the global server-side model and the personalized local server-side model from the previous round. Steps 10-13 involve the clients completing SL training with their personalized local server-side models within their clusters. At the end of training, the model of each client and the local server-side model are updated. This process continues until all rounds of training are completed.

[0070] It's important to further clarify that in split learning, neither the client nor the server possesses a complete model; instead, each has a portion. The client's local model includes some layers in the neural network and the splitting layers connected to these layers. The global server model includes the splitting layers and the remaining layers in the neural network connected to them. The local model within any client and the global server model together constitute a complete neural network. Both the client and server employ a special design called the splitting layer, used by the client to pass intermediate results and the gradients used for backpropagation by the server. In split learning, each client and server are trained sequentially, and only intermediate results are passed, without directly transmitting the original data, thus offering better privacy.

[0071] It should be further noted that the aforementioned local training samples are local private data, determined by the specific task of training the model, such as image classification, object detection, image segmentation, and other visual tasks.

[0072] This invention builds upon the traditional split learning model, which has only one global server-side model, by maintaining several local server-side models. Clients are clustered based on data similarity, with local training provided within each cluster to further improve prediction performance. Utilizing pixel features from local sample images of the client, a clustering algorithm groups similar clients into the same cluster, resulting in clusters composed of clients with similar sample features. Based on global regularization and considering the characteristics of split learning, dynamic and privacy-preserving clustering is performed using intermediate results from each training round. Local training is conducted after global training in traditional split learning, resulting in a personalized split learning method based on global regularization.

[0073] Specifically, in one optional implementation, in step S2 above, clients with the same number of categories in their local training sample labels are grouped into the same client cluster, resulting in M ​​client clusters. For example, clients with 8 category labels are grouped into one cluster, and clients with 7 category labels are grouped into another cluster, thus pre-assigning clients to different clusters. It should be noted that in this implementation, step S2 does not necessarily need to be performed between steps S1 and S3; it can also be performed before step S1.

[0074] Furthermore, while the number of labels on local samples is the primary factor determining Non-IID, other factors such as label categories and the different distributions of each category across devices also influence the degree of Non-IID. Considering only the number of label categories limits the similarity between clients within the resulting clusters, potentially failing to achieve ideal training results. Therefore, it is necessary to more fully consider the factors influencing Non-IID and incorporate more feature information as the criteria for cluster partitioning.

[0075] To address the issue of insufficient consideration of conditions in manual clustering, in an optional second implementation, in step S2 above, a clustering algorithm is used to cluster each client based on the sample label distribution of the client's local training samples, resulting in M ​​client clusters. The sample label distribution of a client's local training samples is the proportion of the number of local training samples of each category in that client to the total number of training samples of that category in all clients. It should be noted that in this second implementation, step S2 does not necessarily need to be executed between steps S1 and S3; it can be executed before step S1.

[0076] In real life, factors such as different regions and user characteristics significantly influence the distribution of tags. Even tags within the same category can vary greatly in quantity across different clients. For clustering, the goal is to obtain data that represents the characteristics of the samples. In machine learning, the images and tags used for training and testing are in a one-to-one correspondence. This means that the data distribution of samples and the distribution of tags are strongly correlated. Once pre-labeled tags are obtained, their distribution can be statistically analyzed to understand the data distribution characteristics of the samples. Specifically, before the split learning process, the client-side sample data is statistically analyzed. First, the proportion of samples of a particular category for each client within the total number of samples of that category is calculated, and this proportion is used as one dimension of the clustering vector. Then, the distribution of other categories is statistically analyzed sequentially, and the label distribution of samples from all client devices is used as the basis for clustering. Specifically, for the same category, such as cat images, within each client, the number of cat samples on that client device is first counted. Then, the total number of samples labeled "cat" across all client devices is counted, and the proportion is calculated. This proportion is used as the first dimension of the current client's probability vector. The next category, such as samples labeled "dog," is then counted, and this becomes the second dimension of the probability vector. This process continues until the client has counted the proportion of all categories, resulting in a complete probability vector for that client. This process is then repeated for other clients. Clustering is performed based on the probability vectors of all client samples. Compared to the clustering method in the optional implementation based on global regularization in personalized splitting learning, this approach not only considers the number of sample label categories on different client devices but also further considers the potentially large differences in the distribution of different category labels across different client devices. Therefore, it achieves more reasonable clustering, and the effect on personalized splitting learning is reflected in a further improvement in prediction performance. To more effectively cluster clients participating in the split learning training process, the silhouette coefficient method is first used to determine the optimal number of clusters K based on the probability distribution of the class labels of the local training samples contained in each client. Then, the K-Means algorithm is combined to perform clustering, grouping client devices with high similarity in their local training samples into one cluster. This is where the silhouette coefficient comes in. Where a represents the average distance between the sample point and all other points in the same cluster, i.e., the similarity between the sample point and other points in the same cluster; b represents the average distance between the sample point and all points in the next nearest cluster, i.e., the similarity between the sample point and other points in the next nearest cluster.

[0077] Specifically, the process of the personalized split learning method under the above optional implementation method two is as follows: Figure 3As shown, specifically, before training, the label category distribution probability of the client devices to be trained is collected and statistically analyzed. First, the silhouette coefficient method is used to determine the most suitable K value, and then the K-Means algorithm is used to calculate based on this distribution probability, dividing all client devices into K clusters. Compared with the first optional implementation method, the clustering determined at this time is not only more flexible in setting the number of clusters, allowing more similar devices to be added to a cluster, but also more fully considers the similarity features of client device samples. After fixing the clusters, personalized split learning training, which includes both global training and local training, begins. At the beginning of each training round, all client devices first participate in the training of the global model. Training samples are input from the client to train the local part of the neural network. The intermediate results calculated by the client are sent from the split layer to the global server. The global server receives the intermediate results and performs the next training. After the global model is updated, gradient backpropagation is performed, and the split layer from the server is sent to the client to update the part of the neural network of the client device. Other clients perform this process in sequence until the training of the entire global model is completed. After the global model training for the current round is completed, the client devices will participate in the local training of each cluster. That is, in each cluster, the client devices and the local personalized server will be split for learning and training. In this round of local training, since the local server model has been fine-tuned from the global model, it has a better training effect on the devices within the cluster.

[0078] However, the personalized splitting learning method under Option 2 has two problems: First, the fixed clustering means that once training begins, client devices within a cluster cannot be changed, and it is impossible to re-divide each client device into a more suitable cluster in each round. Even if there is an imprecise division, it cannot be corrected after training begins. Second, in the personalized splitting learning scheme based on fixed clustering of clusters, each client needs to communicate with other clients and the server before clustering to obtain the total number of samples from all client devices and the distribution of labels for different categories, and to statistically analyze the distribution probability of the samples. Exchanging information obtained after statistical analysis of label distribution poses a certain risk of privacy leakage, which also reduces the privacy of splitting learning to some extent and cannot meet the needs of some use cases with high confidentiality requirements.

[0079] To address the aforementioned issues, an alternative implementation method (3) considers leveraging the characteristics of split learning to achieve dynamic clustering. During training via client-server communication, split learning does not directly transmit raw data. Instead, it slices the intermediate results obtained from client-side training and sends them to the server for further training. These sliced ​​intermediate results are derived from the client-side neural network training, thus reflecting the characteristics of the local samples for each client device. It's important to note that by using sample vectors as input and having the client's local neural network calculate intermediate activation values, the distribution of these activation value vectors across different dimensions shows a similarity to the different dimensions of the samples, exhibiting a certain proportional relationship. Flattening the obtained intermediate activation value vectors into an array structure allows this array to serve as a clustering basis for a client device; that is, the intermediate activation values ​​calculated from the samples represent the client's data characteristics. For each client device, all its samples are trained using a local model, outputting intermediate results. These intermediate results are vectors calculated from the sample images, with different dimensions representing different pixel information. The pixel distribution of all sample images on the client device exhibits regularity. Different categories often have similar features near similar pixels. Therefore, by averaging the intermediate results obtained after calculating all samples, a vector representing the local sample features of the client device can be obtained, which can be used as the basis for clustering.

[0080] Specifically, in this optional implementation method three, in step S2 above, a clustering algorithm is used to group each client based on the weighted average of the intermediate results generated by each client, resulting in M ​​client clusters; specifically, the weighted average of the intermediate results generated by the i-th client is:

[0081]

[0082] Among them, z i Let r be the number of local training samples in the i-th client; when the current iteration is the first iteration, r il This represents the intermediate result of the l-th round of interactive training between the local model within the i-th client and the global server model within the global server; when the current iteration is not the first iteration, r il This represents the intermediate result of the l-th round of interactive training between the local model within the i-th client and its corresponding local server model in the previous iteration round. The corresponding clustering module structure diagram is as follows: Figure 4 As shown. In this embodiment, s is set to 10.

[0083] Specifically, the process of the personalized split learning method under the above optional implementation method three is as follows: Figure 5As shown, at the start of the first round of training, similar to the previous personalized approach, all client devices participating in personalized split learning undergo traditional split learning training to train a global server-side model. In global training, the client first trains its local model, inputting sample images, recording the intermediate outputs, and then sending these intermediate results from the splitting layer to the global server. The global model is then trained, and gradients are backpropagated, sent back to the client via the splitting layer, and then the global training continues for the next client. This process continues until all clients and the global server have completed this step, resulting in the global model for this round. Based on the intermediate results recorded in the first round of global training, before starting local personalized training, a clustering algorithm is used to cluster the client devices. A local server-side model is set for each cluster, obtained by fine-tuning the global server-side model. Within each cluster, the client and the local server perform local training. The client device first trains its local model, outputs and records the intermediate results, and uses this as input for the next round of clustering. The client passes the intermediate output to the local server model for further training by splitting the layers. The local server then backpropagates the gradients back to the client through the splitting layers, allowing the client to update its local neural network. Other clients within the cluster sequentially complete this process, thus completing this round of local personalized training. The second round of training follows the same format as the first. Before performing local personalized training, the client performs clustering, but the input for clustering is the intermediate output from each client device in the previous round of local personalized training. Each round of training uses clustering based on different intermediate results, making the training more flexible.

[0084] Specifically, in this optional implementation method three, the overall process of the personalized splitting learning method based on global regularization is shown in Table 1, where K represents the number of client devices, T represents the total number of global training rounds, s represents the number of times the client and server perform training each time in global / local training, λ and η are regularization parameters, ω is the global model, {v m} m∈[M] Represents a personalized server-side model, {d k} k∈[K] This represents the client-side model. It's important to note that the intermediate activation values ​​used in the first round are calculated by the client during global training, while the intermediate activation values ​​used in other training rounds are the intermediate activation values ​​output by the client during the previous round of local training.

[0085] Table 1

[0086]

[0087]

[0088] It should be noted that in this optional implementation method three, the number of clusters K is predetermined in the first round of iteration (K is 2 in this implementation method), and the number of clusters in each round of iteration is K. The difference is that the number of clients in the cluster may change.

[0089] To further illustrate the personalized split learning method for distributed systems provided by this invention, the CIFAR-10 dataset was used as the base dataset. The Dirichlet algorithm was employed to generate 20 different label distribution probabilities, creating a Non-IID dataset. Twenty client devices and one global server were set up, with 20 training and test sets allocated to each of the 20 client devices. The training and test sets for each client device followed the same data distribution. Image classification models were trained using local training, the traditional split learning method (hereinafter referred to as traditional SL), the split learning method provided in optional embodiment one of this invention (hereinafter referred to as GR-PSL), the split learning method provided in optional embodiment two of this invention (hereinafter referred to as FC-PSL), and the split learning method provided in optional embodiment three of this invention (hereinafter referred to as DC-PSL). Image classification prediction was then performed, and the experimental results are shown in Table 2. Local training involved each of the 20 client devices training independently within the complete neural network; that is, each client device independently completed training and made predictions on local samples. This confirmed that the performance improvement gained by the client devices from the global model when participating in split training outweighed the negative impact of different client devices' Non-IID data.

[0090] Table 2

[0091]

[0092] As shown in Table 2, the prediction accuracy of local training is 56.81%, the traditional SL prediction accuracy is 60.93%, the GR-PSL prediction accuracy provided by this invention is 62.41%, the FC-PSL prediction accuracy provided by this invention is 65.32%, and the DC-PSL prediction accuracy provided by this invention reaches 64.09%. This indicates that the local training experiment performed the worst among the four experiments because only a small number of samples were used as the training set on the device, making it impossible to achieve good prediction results. This also illustrates the necessity of participating in split learning training, i.e., using data from multiple global clients to train the server-side model, allowing clients to improve prediction performance during global training. The traditional SL experiment improved prediction accuracy by 4.12% compared to the local training experiment; however, the time cost of participating in SL training is significant, and the 4.12% improvement in prediction accuracy is not ideal. This situation also leads to the initial problem of this invention: SL involving many client devices does not necessarily lead to a significant improvement in prediction performance because the sample label categories and distributions of each device may differ greatly, causing mutual interference. A single global server device cannot provide a satisfactory improvement for all participants. In the GR-PSL personalization experiment, simple clustering was performed manually based on the number of sample label types possessed by the device, satisfying some similarity requirements. In this case, two clusters were created, each maintaining a personalized local model to fine-tune the global model trained in traditional SL, making it more suitable for clusters with certain characteristics. Experimental results show that the GR-PSL method improves prediction accuracy by 1.48% compared to the traditional SL method, preliminarily verifying that split learning personalization can improve prediction performance. The FC-PSL personalization experiment further improved clustering. Based on the statistical distribution probability of sample labels, the silhouette coefficient method was used to determine the optimal number of clusters, and then clustering was performed using a clustering algorithm, resulting in greater similarity of sample features within the resulting clusters. Therefore, compared to the GR-PSL method, a further 2.91% improvement in prediction accuracy was achieved. The DC-PSL personalization experiment offered more flexible and secure clustering implementation. Clustering was performed in each training round based on previously recorded intermediate results. After 20 rounds of training, the prediction accuracy reached 64.09%, a not significant improvement compared to the FC-PSL method. However, the DC-PSL method clusters based on its intermediate results from the previous round. Compared to the GR-PSL method, which requires collecting client sample distribution information, it does not need to communicate or transmit sample distribution information, thus offering better privacy protection. Based on the above experimental results, the participation of client devices in split learning can improve training performance to some extent due to insufficient local sample quantities. However, the Non-IID problem limits performance improvement. Therefore, this paper introduces a personalization technique to address this issue.The GR-PSL method provided by this invention improves the prediction accuracy from 60.93% in traditional split learning to 62.41% by fine-tuning the global model to obtain a local model and then performing personalized training, representing a 1.48% improvement. The FC-PSL method provided by this invention first uses the silhouette coefficient method to determine the optimal number of clusters based on the sample distribution probability, and then combines it with a clustering algorithm for clustering, achieving a prediction accuracy of 65.32%, a 4.39% improvement compared to traditional split learning, making it the best performing of the three personalized schemes. The DC-PSL method provided by this invention, compared to the former, allows for more flexible dynamic clustering in each training round and solves the privacy leakage problem that may occur when statistically analyzing distribution probabilities. Compared to the previous two schemes, it provides better privacy and has good performance, achieving a prediction accuracy of 64.09%, a 3.16% improvement compared to traditional split learning. Analysis of the experimental results leads to the conclusion that the three personalized split learning methods proposed in this invention all improve prediction accuracy on the basis of traditional split learning, and improve the limited improvement in training accuracy caused by the Non-IID problem of local data from multiple client devices participating in training.

[0093] Example 2

[0094] A distributed system includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the personalized splitting learning method provided in Embodiment 1 of the present invention.

[0095] The relevant technical solutions are the same as in Embodiment 1, and will not be repeated here.

[0096] Example 3

[0097] A computer-readable storage medium includes a stored computer program, wherein the computer program, when executed by a processor, controls the device where the storage medium is located to execute the personalized splitting learning method provided in Embodiment 1 of the present invention.

[0098] The relevant technical solutions are the same as in Embodiment 1, and will not be repeated here.

[0099] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A personalized splitting learning method for distributed systems, characterized in that, The method comprises the following steps: S1, sequentially performing interactive training on the local model in each client and the global server model in the global server, and repeating the training for s rounds, so as to obtain the global server model in the global server; Wherein, the process of performing interactive training on the local model in the i-th client and the global server model in the global server comprises: inputting the local training sample in the i-th client into the local model in the i-th client for training, generating an intermediate result, and sending the intermediate result to the global server; after the global server receives the intermediate result of the i-th client, inputting the intermediate result of the i-th client into the global server model in the global server for further training, and after the training is completed, performing back propagation and transmitting the back propagation result to the i-th client; after the i-th client receives the back propagation result, updating the local model in the i-th client; S2, clustering the clients based on the feature similarity of the local training samples of the clients to obtain M client clusters; and setting M local server models corresponding to the M client clusters in the global server; S3, determining whether the current iteration is the first iteration, if yes, initializing the M local server models as the global server model respectively; otherwise, performing global regularization fine-tuning on the M local server models in the last iteration round based on the global server model, so as to obtain the M local server models in the current iteration round; S4, in each client cluster, sequentially performing interactive training on the local model in each client and the corresponding local server model according to the training method in step S1, and repeating the training for s rounds; S5, repeating steps S1-S4 for iteration until the current iteration round reaches the preset iteration round. 2.The personalized split learning method of claim 1, wherein, In the step S2, the clustering algorithm is used based on the weighted average value of the intermediate result generated by each client to cluster the clients to obtain M client clusters; wherein, the weighted average value of the intermediate result generated by the i-th client is: z i is the number of local training samples in the ith client; when the current iteration is the first iteration, r il is the intermediate result in the lth round of s-round interactive training of the local model in the ith client and the global server model in the global server; when the current iteration is not the first iteration, r il is the intermediate result in the lth round of s-round interactive training of the local model in the ith client and the local server model corresponding thereto. 3.The personalized split learning method of claim 1, wherein, In the step S2, the clients containing the same number of class labels of the local training sample are divided into the same client cluster to obtain M client clusters. 4.The personalized split learning method of claim 1, wherein, In the step S2, the clustering algorithm is used based on the sample label distribution of the local training sample of the client to cluster the clients to obtain M client clusters; wherein, the sample label distribution of the local training sample of the client is the proportion of the number of local training samples of each class in the client in the total number of training samples of the class in all clients.

5. The personalized split learning method of claim 3 or 4, wherein, The step S2 is performed before the step S1.

6. The personalized split learning method according to any one of claims 1-4, characterized in that, In the step S3, the m-th local server model in the current iteration round t is: wherein, is the m-th local server model in the last iteration round; η and λ are regularization parameters; is the gradient value of the loss function value of the m-th local server model in the last iteration round; ω t-1 is the global server model in the last iteration round.

7. The personalized split learning method according to any one of claims 1-4, characterized in that, In the mth client cluster, the local model inside the jth client and the corresponding local server model v m The process of interactive training includes: The jth client trains the local model inside it based on the local training samples inside it, generates an intermediate result, and sends it to the global server end; after the global server end receives the intermediate result of the jth client, the intermediate result of the jth client is input into the local service end model v m The training continues, and after the training is completed, back propagation is performed, and the result of the back propagation is transmitted to the jth client; after the jth client receives the result of the back propagation, the local model inside it is updated.

8. The personalized split learning method of any one of claims 1-4, wherein, The local model inside the client includes part of layers and a split layer connected after the part of layers in the neural network; the global server-side model inside the global server includes the split layer and the rest of layers connected after the split layer in the neural network; the local model inside any one of the clients and the global server-side model inside the global server constitute a complete neural network; and the split layer is used for transmitting intermediate results between the client and the global server.

9. A distributed system, characterized by Comprise: A memory and a processor, the memory stores a computer program, and the processor executes the computer program to execute the personalized split learning method in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium includes a stored computer program, wherein when the computer program is run by a processor, it controls the device where the storage medium is located to execute the personalized split learning method in any one of claims 1-8.

Images