Split federated learning method and device based on adaptive pipeline parallelism

Abstract

The application discloses a kind of based on adaptive flow parallel segmented federal learning method and device. The method comprises: for each device: the device divides bottom submodel into multiple segmented models, sequentially respectively to each segmented model is carried out forward propagation, and the intermediate activation data corresponding to the segmented model after forward propagation is uploaded to server;After server receives intermediate activation data, forward propagation and back propagation are carried out using top submodel, and the gradient of corresponding segmented model is obtained and is issued to corresponding device;For each device: after the gradient is received, the gradient is updated using its set weight, and the set weight is inversely proportional to the update frequency of the device in the current round, and the local bottom submodel is updated using the updated gradient;Forward and backward propagation, data upload, gradient issuing, model updating are carried out in the way of pipeline parallelly. The application accelerates the training of large model on resource-limited edge device and maintains accuracy.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of federated learning and edge computing, and more particularly, relates to a split federated learning method and device based on adaptive pipeline parallelism. BACKGROUND

[0002] Federated learning realizes distributed training through the principle of "data not leaving the end", but when the model size rises to a large model, the end-side computing power and memory often cannot bear the complete model. Split federated learning (SFL) realizes the compromise between privacy and computing power by reserving a bottom sub-model on the end-side and training a top sub-model on the parameter server side, with an "intermediate state" as the interface. However, the mainstream synchronous SFL (SSFL) and asynchronous SFL (ASFL) mostly use the sequential execution paradigm of "end-side forward→upload activation→server forward / backward→downlink activation gradient→end-side backward".

[0003] On the one hand, the intermediate activation volume of modern large models is large, and the link upload time is significantly long, so the sequential paradigm will have a long communication bottleneck and computing idle time, and the end-side / server / link cannot be utilized in parallel. On the other hand, system heterogeneity (bandwidth and computing power differences of different end-sides) causes SSFL to be constrained by the "slowest node", and ASFL, although it alleviates the synchronization block, is prone to convergence instability and final accuracy degradation under non-IID data conditions. The existing technical solutions still have two weaknesses: (1) insufficient parallel overlap of communication and computation, even in the asynchronous framework, there is often a lack of fine-grained pipeline connection between end-side upload and local forward, server forward / backward and backhaul, resulting in a significant "bubble period"; (2) poor update stability and fairness for non-IID, under asynchronous accumulation, fast nodes (high bandwidth / high computing power) produce gradients more frequently, which can dominate the accumulation and introduce a bias direction, weakening the final accuracy and stability.

[0004] Therefore, there is an urgent need for an SFL training framework that can systematically overlap communication and computation while maintaining update stability and fairness under heterogeneous and non-IID conditions, so that end-cloud resources can be efficiently utilized and verifiable convergence and accuracy performance can be obtained. SUMMARY

[0005] The main purpose of the present application is to provide a split federated learning method and device based on adaptive pipeline parallelism to overcome the shortcomings of the prior art.

[0006] To achieve the above application purposes, the present application adopts the following technical solutions:

[0007] The first aspect of the present application provides a split federated learning method based on adaptive pipeline parallelism, which is applied to a federated learning framework including a server and a plurality of devices, and the method comprises: for each device, the device splits a local bottom sub-model into a plurality of split models of a set size, sequentially performs forward propagation on each split model, and uploads the intermediate activation data of the split model after forward propagation to the server; after the server receives the intermediate activation data, the server performs forward propagation and back propagation using a top sub-model to obtain the gradient of the corresponding split model and sends it to the corresponding device; wherein the forward propagation, back propagation and gradient sending in the server are performed in parallel in a pipeline manner; for each device, after receiving the gradient, the device updates the gradient using a set weight, the set weight is inversely proportional to the update frequency of the device in the current round, and the updated gradient is used to update the local bottom sub-model; wherein the forward propagation, data uploading and model updating in the device are performed in parallel in a pipeline manner.

[0008] Preferably, the set size is the smallest integer satisfying the following constraint: for each split model of each device, the time delay required by the device to upload the intermediate activation data corresponding to the split model is not less than wherein, is the calculation overhead of the forward propagation of the split model by the device, is the calculation overhead of the forward propagation and back propagation of the split model by the server.

[0009] Preferably, the set weight is:

[0010] ;

[0011] wherein, is the set weight of device i, ; is the update frequency of device j in the current round, ; is the number of devices in the federated learning architecture.

[0012] Preferably, the updated bottom sub-model is:

[0013] ;

[0014] wherein, , are the bottom sub-models before and after the update of device i in the kth round of training, respectively; is the first learning rate; is the set weight of device i; denotes the gradient received by device i for local computation; , is the number of devices in the federated learning architecture.

[0015] Preferably, the method further comprises: the server calculates a current weighted cumulative label distribution according to the set weight of each device, and the number of samples and label distribution contained in all the intermediate activation data currently received by the device; the server calculates the KL divergence between the current weighted cumulative label distribution and a preset ideal distribution; when the KL divergence is lower than a first preset threshold or the cumulative number of samples received by the server reaches a second preset threshold, the server updates the top sub-model thereof.

[0016] Preferably, the current weighted cumulative label distribution is:

[0017] ;

[0018] wherein, is the current weighted cumulative label distribution, , is the number of samples and label distribution contained in the intermediate activation data uploaded by device i respectively, is the set weight of device i, , is the number of devices in the federated learning architecture.

[0019] Preferably, the updated top sub-model is:

[0020] ;

[0021] wherein, , is the top sub-model before and after being updated by the server in the kth round of training respectively, is a second learning rate, is the number of samples contained in the intermediate activation data uploaded by device i, is the set weight of device i, denotes the gradient received by the server for local computation after receiving the intermediate activation data uploaded by device i, , is the number of devices in the federated learning architecture.

[0022] The second aspect of the present application provides a device for adaptive flow parallel segmented federated learning, which is applied to a federated learning framework including a server and a plurality of devices, and the device comprises: a segmentation and processing module for each device; for each device: the segmentation and processing module is used for segmenting a bottom sub-model locally in the device into a plurality of segmented models with a set size, sequentially performing forward propagation on each segmented model, and uploading intermediate activation data corresponding to the segmented model after forward propagation to the server; a processing and delivery module is used for performing forward propagation and back propagation on the server using a top sub-model after receiving the intermediate activation data, obtaining the gradient of the corresponding segmented model and delivering it to the corresponding device; wherein the forward propagation, back propagation and gradient delivery in the server are performed in parallel in a pipeline manner; an update module for each device; for each device: the update module is used for updating the gradient using a set weight after the device receives the gradient, the set weight being inversely proportional to the update frequency of the device in the current round, and updating the bottom sub-model locally using the updated gradient; wherein the forward propagation, data uploading and model updating in the device are performed in parallel in a pipeline manner; and the device is used to implement the adaptive flow parallel segmented federated learning method as described above.

[0023] The third aspect of the present application provides an electronic device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the adaptive flow parallel segmented federated learning method as described above when executing the program.

[0024] The fourth aspect of the present application provides a non-transitory computer readable storage medium, which stores computer instructions for causing a computer to execute the adaptive flow parallel segmented federated learning method as described above.

[0025] Compared with the prior art, the present application has the following beneficial effects:

[0026] (1) A split federated learning method based on adaptive flow parallelism is provided. On the one hand, the device end divides the local bottom sub-model into multiple segmentation models of a certain size, adopts the pipeline parallelization technology, breaks the serial mode of "transmission first and then calculation", divides the data into finer granularity "micro-batches (i.e. segmentation models)", and enables the device to upload the current micro-batch data while calculating the next micro-batch data in parallel, thereby realizing the overlap of communication and calculation, greatly improving the throughput, and significantly shortening the time required to reach the target accuracy. On the other hand, adaptive collaborative updating is adopted, the device is adaptively assigned a weight according to its update frequency, the device updates the gradient according to the weight, and the bottom sub-model is optimized according to the updated gradient, thereby solving the problem that in an asynchronous scenario, the calculation capability of different devices is different, resulting in that the upload frequency of fast devices is much higher than that of slow devices.

[0027] (2) In order to ensure the convergence direction of the global model, the server does not immediately update the top sub-model, but accumulates the weighted gradient first, and monitors the label distribution of the accumulated gradient to determine the update time, so that the global model is updated only when the gradient is representative, thereby ensuring the convergence accuracy under non-independent and identically distributed data. BRIEF DESCRIPTION OF DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiment or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments described in the present application, and those skilled in the art can also obtain other drawings according to these drawings without creating any creative labor.

[0029] Figure 1 A flowchart of the split federated learning method based on adaptive flow parallelism provided by the embodiment of the present application.

[0030] Figure 2 A framework diagram of the split federated learning system based on adaptive flow parallelism provided by the embodiment of the present application.

[0031] Figure 3 A working process schematic diagram of the split federated learning system based on adaptive flow parallelism provided by the embodiment of the present application.

[0032] Figure 4 For the case where the data distribution of each device is basically consistent, a comparison result schematic diagram between the present method and the prior art.

[0033] Figure 5 For the case where the label distribution on each device has obvious deviation, a comparison result schematic diagram between the present method and the prior art.

[0034] Figure 6A block diagram of a split federated learning device based on adaptive pipeline parallelism is provided for an embodiment of the present application.

[0035] Figure 7 A schematic diagram of an electronic device is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0036] In view of the deficiencies in the prior art, the present inventors have, through long-term research and a large number of practices, come up with the technical solution of the present application. The technical solution, its implementation process and principles will be further explained as follows.

[0037] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, it will be apparent to one skilled in the art that the present application can be practiced without the specific details presented in this description. Therefore, the scope of the present application is not limited to the specific embodiments disclosed in this description.

[0038] In addition, in the description of the present application, it should be understood that the terms “upper”, “lower”, “inner”, “outer”, “horizontal”, “vertical” and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present application.

[0039] In the description of the present application, the description of the terms “one embodiment”, “an embodiment”, “the embodiment” and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present description, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples.

[0040] Figure 1 A flowchart of a split federated learning method based on adaptive pipeline parallelism is provided for an embodiment of the present application. The method is applied to a federated learning framework, which includes a server (cloud) and a plurality of devices (clients). The method includes operations S1-S3.

[0041] In operation S1, for each device, the device divides the local bottom sub-model into a plurality of split models of a set size, and sequentially performs forward propagation on each split model and uploads the intermediate activation data corresponding to the split model after forward propagation to the server.

[0042] Preferably, the size is set to be the minimum integer satisfying the following constraint: for each device and each split model: the latency required by the device to upload the intermediate activation data corresponding to the split model is not less than wherein, is the computation overhead of the device for the forward propagation of the split model, is the computation overhead of the server for the forward and backward propagation of the split model. The set size corresponding to different devices can be the same or different.

[0043] Operation S2, after the server receives the intermediate activation data, the top sub-model is used for forward propagation and backward propagation, and the gradient of the corresponding split model is obtained and sent to the corresponding device; wherein the forward propagation, backward propagation and gradient sending in the server are performed in parallel in a pipeline manner.

[0044] Operation S3, for each device: after the device receives the gradient, the set weight is used to update the gradient, the set weight is inversely proportional to the update frequency of the device in the current round, and the updated gradient is used to update the local bottom sub-model; wherein the forward propagation, data uploading and model updating in the device are performed in parallel in a pipeline manner.

[0045] Preferably, the set weight is:

[0046] ;

[0047] wherein, is the set weight of device i, ; is the update frequency of device j in the current round, ; is the number of devices in the federated learning architecture.

[0048] Preferably, the updated bottom sub-model is:

[0049] ;

[0050] wherein, , respectively, the updated bottom sub-model before and after the update of device i in the kth round of training; is the first learning rate; is the set weight of device i; represents the gradient received by device i for local calculation; , is the number of devices in the federated learning architecture.

[0051] Preferably, the method further comprises the following operations S1'-S3'.

[0052] Operation S1', the server calculates a current weighted cumulative label distribution according to the set weight of each device and the sample quantity and label distribution contained in all intermediate activation data currently received by the server.

[0053] Operation S2', the server calculates the KL divergence between the current weighted cumulative label distribution and the preset ideal distribution.

[0054] Operation S3', when the KL divergence is lower than a first preset threshold or the cumulative quantity of samples received by the server reaches a second preset threshold, the server updates the top sub-model.

[0055] Preferably, the current weighted cumulative label distribution is:

[0056] ;

[0057] wherein, is the current weighted cumulative label distribution, , is the sample quantity and label distribution contained in the intermediate activation data uploaded by the device i respectively, is the set weight of the device i, , is the number of devices in the federated learning architecture.

[0058] Preferably, the updated top sub-model is:

[0059] ;

[0060] wherein, , is the top sub-model before and after being updated by the server in the kth round of training respectively, is the second learning rate, is the sample quantity contained in the intermediate activation data uploaded by the device i, is the set weight of the device i, represents the gradient used for local calculation by the server after receiving the intermediate activation data uploaded by the device i, , is the number of devices in the federated learning architecture.

[0061] The present application is based on a split federated learning architecture, which is a distributed training method of "vertically cutting" a deep learning model. Specifically, the complete neural network model is split into a bottom sub-model and a top sub-model: the bottom sub-model is deployed on a resource-limited edge client, and the top sub-model is deployed on a cloud server with strong computing power. This architecture not only protects user privacy (original data does not leave the local), but also utilizes the computing power of the cloud.

[0062] In traditional SFL training, the device first performs forward calculation on a batch of local data, extracts intermediate features, then transmits these data to the server through the network, after receiving, the server continues to complete the forward calculation and back propagation of the remaining network layer, calculates the gradient, finally, the server returns the bottom gradient to the device, and the device updates the local model accordingly. This "calculation-transmission-waiting" mode causes the communication delay to seriously slow down the overall training speed.

[0063] To solve the above technical bottlenecks, on the one hand, the pipeline parallelization technology is adopted to break the serial mode of "calculation after transmission", the data is divided into finer granularity "micro-batch (i.e. split model)", and the device calculates the next micro-batch data in parallel while uploading the current micro-batch data, realizing the overlap of communication and calculation, greatly improving the throughput, and significantly shortening the time required to reach the target accuracy; on the other hand, adaptive collaborative update is adopted to solve the model deviation problem caused by "fast and slow uneven", the server no longer blindly updates, but adaptively allocates weights according to the update frequency of the device, and dynamically determines the update time of the server model according to the statistical distribution of the cumulative gradient, ensuring that the model is only updated when the gradient is representative, and ensuring the convergence accuracy under non-independent and identically distributed data.

[0064] The technical solutions of the present application will be further described in detail below in combination with several preferred embodiments and drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor belong to the scope of protection of the present application. The test method is not specified in the following embodiments, which is usually carried out under conventional conditions.

[0065] The following will be combined Figures 2-5 The implementation process and effects of the split federated learning method based on adaptive flow parallel provided by the present application will be further described in detail.

[0066] Please refer to Figure 2The system for implementing the adaptive pipeline parallelism-based split federated learning method mainly includes the following four functional modules: a state monitoring module, a micro-batch size optimization module, a gradient normalization module, and a collaborative training module. The state monitoring module is used to continuously collect the computing capacity, bandwidth condition, update frequency, and local data distribution of each device during the training process. The micro-batch size optimization module allocates a respective micro-batch size for different devices according to the above information, so as to improve the end-to-cloud pipeline parallelism and reduce idling. The gradient normalization module allocates a corresponding weight for different devices, which is used to accumulate the returned gradient on the server side, so as to avoid that the high-performance device dominates the global update in the asynchronous training. The collaborative training module is responsible for the specific end-to-cloud joint training process, that is, the device cuts the local batch according to the allocated micro-batch and uploads the intermediate activation data in turn, the server asynchronously completes the forward and reverse of the top sub-model, returns the gradient, and then triggers the global model update after accumulating enough gradient representing the global distribution. The data distribution of each device is basically consistent (IID data), which can still maintain stable convergence and high precision.

[0067] Please refer to Figure 3 , which shows an end-to-cloud collaborative pipeline schematic diagram in the training phase. In a typical implementation, the system is composed of a server with a high-performance graphics processing unit and N edge devices with different computing power and bandwidth, wherein the edge device saves the bottom sub-model of the model, and the server saves the top sub-model of the model.

[0068] When the training starts, the server first allocates a micro-batch size for each device i according to the bandwidth , the local forward calculation overhead , the forward and reverse overhead of the server side for the micro-batch . . The system takes the minimum integer satisfying the constraint, and the size of the model segmented in the device i is :

[0069] ;

[0070] wherein, is the activation size of each sample.

[0071] Thus, each device can enter the pipeline with the most suitable granularity, thereby maximizing the overlap of communication and calculation and minimizing the pipeline bubbles. For each device i, the device i does not upload the local micro-batch in batches, but uploads the intermediate activation data according to splitting it into a series of microbatches: perform bottom forward pass on the q-th microbatch, send the intermediate activation data of this microbatch to the server immediately, while locally start the forward pass of the q+1-th microbatch in parallel; the server performs forward and backward pass with the top submodel immediately after receiving the intermediate activation data of the q-th microbatch, gets the gradient of this microbatch and directly sends it back to the device i. To solve the statistical heterogeneity problem in asynchronous training, the embodiment performs an adaptive update strategy on the server side and the device side. Specifically, to solve the problem that the upload frequency of fast devices is much higher than that of slow devices due to the difference in computing capacity of different devices in the asynchronous scene, the system introduces a gradient normalization mechanism. The update frequency of device i in the current round is , in order to balance the contribution, the system calculates an update weight for device i which is inversely proportional to the update frequency, and the calculation formula is:

[0072] ;

[0073] When device i receives the activation gradient returned by the server, it will scale the gradient by the weight when updating the local bottom submodel , and the update formula is as follows:

[0074] ;

[0075] On this basis, in order to ensure the convergence direction of the global model, the server side does not update the top submodel immediately, but accumulates the weighted gradient first, and monitors the label distribution of the accumulated gradient to determine the update opportunity. Let be the ideal global balanced data distribution (for example, a uniform distribution), for each microbatch arriving, the server calculates the current weighted cumulative label distribution according to the number of samples and the label distribution contained in it:

[0076] ;

[0077] Then, the server calculates the KL divergence between the current cumulative distribution and the ideal distribution :

[0078] ;

[0079] Where M is the number of labels in the label distribution , is the j-th label in the label distribution , is the proportion of recorded by the server in real time, is The proportion in the case of independent and identically distributed distributions.

[0080] When the KL divergence is lower than the first preset threshold Alternatively, when the accumulated number of samples reaches the second preset threshold, the system determines that the currently accumulated gradient is representative and has a small deviation, at which point the top sub-model is triggered. Global update:

[0081] ;

[0082] Through the aforementioned collaborative mechanism, the system simulates synchronous large-batch balanced training in an asynchronous pipeline, eliminating model bias caused by non-independent and identically distributed data. Finally, after completing several rounds of training, the server aggregates the underlying sub-models from each device to avoid excessive drift of underlying parameters between devices.

[0083] This invention, based on traditional partitioned federated learning, introduces a pipelined parallel training mechanism based on micro-batches and an adaptive gradient accumulation and update strategy. Specifically, on the device side, a training batch is divided into multiple micro-batches, and the three stages of "forward computation - intermediate activation upload - server gradient return" are overlapped in a pipeline manner, thereby significantly improving the utilization of communication and computing resources. At the same time, the server does not immediately update globally upon receiving each gradient, but first accumulates the gradients according to the update frequency of the devices, and then triggers the upper-level update of the model according to the adaptive criterion. This can suppress the bias of "fast devices dominating global updates" in asynchronous training and alleviate the convergence degradation caused by statistical heterogeneity.

[0084] The following experiments illustrate the effectiveness of this invention. Existing solutions include PipeSFL (Pipe-Split Federated Learning), Activation Cache Split Federated Learning (GAS), and MergeSFL (Merge-Split Federated Learning). PipeSFL, a type of ASFL, improves resource utilization by assigning different priorities to heterogeneous workers; GAS attempts to balance training efficiency and model accuracy by maintaining an activation buffer and sampling and generating additional activations on demand, thus mitigating statistical heterogeneity to some extent; MergeSFL, a type of SSFL, adaptively selects suitable workers, merges activations, and adjusts batch size, enabling the server to better balance the arrival time and label distribution of different workers, thereby reducing the impact of statistical heterogeneity.

[0085] In terms of experimental setup, the VGG16 (VGG for short) model was used on the Image100 dataset to perform image classification tasks. Image100 is a subset of 100 categories selected from ImageNet, used to evaluate training efficiency on resource-constrained edge devices. The experiment compared our method with four other methods: PipeSFL, GAS, and MergeSFL, and evaluated them under the following two typical scenarios: (1) when the data distribution of each terminal is basically consistent (i.e., the IID scenario); and (2) when there is a significant deviation in the label distribution on each terminal (i.e., the non-IID scenario). In this embodiment, p=10 is used to represent a strong non-IID level. The comparison results under these two scenarios are as follows: Figure 4 and Figure 5 As shown.

[0086] See Figure 4 As can be seen, in the IID scenario, all four methods ultimately achieve similar test accuracy, but this method significantly leads in convergence speed, followed by PipeSFL, while GAS and MergeSFL lag further behind. For example, this method only requires 3.34 hours to achieve a test accuracy of 66.8%; PipeSFL, GAS, and MergeSFL require 12.16 hours, 16.81 hours, and 18.95 hours respectively to reach the corresponding comparable accuracy levels. This demonstrates the core value of micro-batch pipeline parallelism: after communication and computation are overlapped, the wall clock time for edge-cloud collaborative training is significantly shortened, instead of the serial waiting of "upload first, then compute, then download" as in traditional SFL, thereby significantly improving training throughput and reducing the time cost to reach the target accuracy.

[0087] See Figure 5 It can be seen that in the non-IID scenario, the convergence speed of existing solutions is similar to that in the IID scenario, but both will experience varying degrees of accuracy degradation; while our proposed method, by simultaneously considering pipeline parallelism and adaptive updates, can still maintain both high accuracy and short training time. Figure 5 As shown, our method achieved a test accuracy of 60.5% within 7.60 hours; while PipeSFL, GAS, and MergeSFL required 12.06 hours, 17.04 hours, and 24.68 hours respectively, and at that time they could only achieve 52.2%, 57.4%, and 60.3% accuracy respectively. In other words, under strong non-IID conditions, our method not only trains faster but also achieves higher final accuracy. This indicates that unlike traditional ASFL or SSFL, it does not make a one-way compromise between "system heterogeneity" and "statistical heterogeneity." Instead, it stabilizes the global model by reweighting and uniformly triggering gradients contributed by different terminals on the server side through gradient accumulation and adaptive updates.

[0088] CombinationFigure 4 and Figure 5 It can be seen that, compared with the existing representative SFL method, the following advantages are obtained under two typical real scenarios: first, the training time is significantly shortened, that is, the target precision is reached in a shorter wall clock time; second, higher final precision is maintained under statistical heterogeneity. This "high throughput + stable precision" dual advantage is a direct manifestation of the synergistic effect of micro-batch pipeline parallelism (improving system resource utilization) and gradient accumulation and adaptive update (resisting statistical heterogeneity).

[0089] Based on the same inventive concept, the present application also provides a split federated learning device based on adaptive flow parallelism, which is applied to a federated learning framework including a server and a plurality of devices. Referring to Figure 6 , the split federated learning device based on adaptive flow parallelism 600 includes a splitting and processing module 610, a processing and issuing module 620, and an updating module 630.

[0090] The splitting and processing module 610, for example, performs operation S1 for each device; for each device: the splitting and processing module is configured to split the local bottom sub-model of the device into a plurality of split models of a set size, sequentially perform forward propagation on each split model, and upload the intermediate activation data corresponding to the split model after forward propagation to the server.

[0091] The processing and issuing module 620, for example, performs operation S2 for performing forward propagation and backward propagation using the top sub-model after the server receives the intermediate activation data, obtaining the gradient of the corresponding split model and issuing it to the corresponding device; wherein the forward propagation, backward propagation and gradient issuing in the server are performed in parallel in a pipeline manner.

[0092] The updating module 630, for example, performs operation S3 for each device; for each device: the updating module is configured to update the gradient using the set weight after the device receives the gradient, the set weight being inversely proportional to the update frequency of the device in the current round, and update the local bottom sub-model using the updated gradient; wherein the forward propagation, data uploading and model updating in the device are performed in parallel in a pipeline manner.

[0093] The split federated learning device based on adaptive flow parallelism 600 is used to implement the split federated learning method based on adaptive flow parallelism described in any one of the embodiments.

[0094] Corresponding to the method of any of the above embodiments based on the same inventive concept, the present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method of any of the above embodiments based on adaptive pipeline parallel split federated learning.

[0095] Figure 7 A more specific hardware structure of an electronic device is shown, which can include a processor 710, a memory 720, an input / output interface 730, a communication interface 740, and a bus 750. The processor 710, the memory 720, the input / output interface 730, and the communication interface 740 are connected to each other through the bus 750 for internal communication.

[0096] The processor 710 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, etc., for executing related programs to implement the technical solutions provided by the embodiments of the present application.

[0097] The memory 720 can be implemented by a ROM (Read Only Memory), a RAM (Random Access Memory), a static storage device, a dynamic storage device, etc. The memory 720 can store an operating system and other application programs, and when the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 720 and executed by the processor 710.

[0098] The input / output interface 730 is used to connect input / output modules to realize information input and output. The input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. The input device can include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device can include a display, a speaker, a vibrator, an indicator light, etc.

[0099] The communication interface 740 is used to connect the communication module (not shown in the figure) to realize the communication interaction between the device and other devices. The communication module can realize communication through wired means (such as USB, network cable, etc.) or through wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0100] Bus 750 includes a path for transferring information between the various components (e.g., processor 710, memory 720, input / output interface 730, and communication interface 740) of the device.

[0101] It should be noted that although the above device only shows the processor 710, the memory 720, the input / output interface 730, the communication interface 740 and the bus 750, in the specific implementation process, the device can also include other components necessary for normal operation. In addition, those skilled in the art can understand that the above device can also only contain the components necessary to implement the embodiments of the present application, and does not have to contain all the components shown in the figure.

[0102] The electronic device of the above embodiment is used to implement the corresponding adaptive flow parallel based split federated learning method in any of the preceding embodiments, and has the beneficial effects of the corresponding method embodiments, which are not repeated here.

[0103] Based on the same inventive concept, corresponding to the method of any of the above embodiments, the present application also provides a non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the adaptive flow parallel based split federated learning method according to any of the above embodiments.

[0104] The computer readable medium of the present embodiment includes permanent and non-permanent, removable and non-removable media, which can be implemented by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape magnetic disk storage or other magnetic storage device, or any other non-transmission medium that can be used to store information accessible by a computing device.

[0105] The computer instructions stored in the storage medium of the above embodiment are used to cause the computer to perform the adaptive flow parallel based split federated learning method according to any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which are not repeated here.

[0106] Those skilled in the art should understand that the above discussion of any embodiment is merely exemplary, and is not intended to be limiting of the scope of the present application (including claims) to these examples; many variations are possible under the teachings of the present application, as would be apparent to those skilled in the art. Steps recited in any of the examples herein can be executed in any order and are not necessarily performed in the order recited; multiple steps can be performed at once, or multiple steps can be performed in an iterative manner; different steps can be performed at the same time or at different times; and steps can be performed in any order, unless otherwise specified or implied by the context of the description.

[0107] In addition, to simplify the description and discussion, and so as not to make the embodiments of the present application difficult to understand, the well-known power / ground connections to integrated circuit (IC) chips and other components can or can not be shown in the provided drawings. Further, devices can be shown in block diagram form in order to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that details regarding implementation of these block diagram devices are highly dependent on the platform in which the embodiments of the present application are to be implemented (i.e., these details should be well within the understanding of one of ordinary skill in the art). Where specific details (e.g., circuitry) are set forth in order to describe an illustrative embodiment of the present application, it should be apparent to those skilled in the art that the present application can be practiced without or with variation of these specific details. Therefore, these descriptions should not be construed as limiting, but merely as descriptive of illustrative embodiments of the present application.

[0108] Although the present application has been described in connection with certain specific embodiments thereof, many modifications, changes, alterations and variations will become apparent to those skilled in the art once the foregoing description is read. For example, other memory architectures (e.g., dynamic RAM (DRAM)) can use the embodiments discussed.

[0109] It should be understood that the above-described embodiments are merely exemplary of the application of the principles of the present application, and that those skilled in the art will be able to design many alternative embodiments without departing from the principles of the present application. It is the intention, therefore, to be limited only as indicated by the scope of the claims appended hereto.

Claims

1. A partitioned federated learning method based on adaptive pipelined parallelism, characterized in that, The method is applied to a federated learning framework, which includes a server and multiple devices, and the method includes: For each of the aforementioned devices: the device divides the local bottom sub-model into multiple segmented models of a set size, performs forward propagation on each of the segmented models in sequence, and uploads the intermediate activation data corresponding to the segmented model after forward propagation to the server; After receiving the intermediate activation data, the server uses its top sub-model to perform forward and backward propagation to obtain the gradient of the corresponding segmentation model and sends it down to the corresponding device; wherein, the forward propagation, backward propagation and gradient sending in the server are performed in parallel in a pipeline manner. For each device: after receiving the gradient, the device updates the gradient using its set weights, where the set weights are inversely proportional to the update frequency of the device in the current round, and updates its local bottom sub-model using the updated gradient; wherein, the forward propagation, data upload, and model update in the device are performed in parallel in a pipeline manner; the updated bottom sub-model is: ； in, , These are the bottom sub-models of device i before and after the update in the kth round of training; The first learning rate; Set weights for device i; This represents the gradient received by device i for local computation; , The number of devices in the federated learning framework.

2. The partitioned federated learning method based on adaptive pipelined parallelism according to claim 1, characterized in that, The set size is the smallest integer that satisfies the following constraints: For each segmentation model of each device: the latency required for the device to upload the intermediate activation data corresponding to the segmentation model is not less than [amount missing]. ,in, The computational overhead for the device to perform forward propagation of the segmentation model. This represents the computational overhead of the server for the forward and backward propagation of this segmentation model.

3. The partitioned federated learning method based on adaptive pipelined parallelism according to claim 1, characterized in that, The set weight is: ； in, Assign weights to device i. ; Let j be the update frequency of device j in the current round. ; The number of devices in the federated learning framework.

4. The partitioned federated learning method based on adaptive pipelined parallelism according to any one of claims 1-3, characterized in that, The method further includes: The server calculates the current weighted cumulative label distribution based on the set weights of each device and the number of samples and label distribution contained in all the intermediate activation data it currently receives; The server calculates the KL divergence between the current weighted cumulative label distribution and the preset ideal distribution; When the KL divergence is lower than a first preset threshold or the cumulative number of samples received by the server reaches a second preset threshold, the server updates its top sub-model.

5. The partitioned federated learning method based on adaptive pipelined parallelism according to claim 4, characterized in that, The current weighted cumulative label distribution is as follows: ； in, For the current weighted cumulative label distribution, , These represent the number of samples and the label distribution contained in the intermediate activation data uploaded by device i, respectively. Assign weights to device i. , The number of devices in the federated learning framework.

6. The partitioned federated learning method based on adaptive pipelined parallelism according to claim 4, characterized in that, The updated top sub-model is as follows: ； in, , These are the top sub-models of the server before and after the update during the k-th round of training. As the second learning rate, The number of samples included in the intermediate activation data uploaded by device i. Assign weights to device i. This represents the gradient that the server uses to calculate locally after receiving the intermediate activation data uploaded by device i. , The number of devices in the federated learning framework.

7. A partitioned federated learning device based on adaptive pipelined parallelism, characterized in that, The apparatus is used in a federated learning framework, which includes a server and multiple devices. The apparatus includes: A segmentation and processing module is used for each of the aforementioned devices; for each of the aforementioned devices: the segmentation and processing module is used to segment the local bottom sub-model of the device into multiple segmentation models of a set size, perform forward propagation on each of the segmentation models in sequence, and upload the intermediate activation data corresponding to the segmentation model after forward propagation to the server; The processing and distribution module is used to perform forward and backward propagation using the top sub-model after the server receives the intermediate activation data, obtain the gradient of the corresponding segmentation model, and distribute it to the corresponding device; wherein, the forward propagation, backward propagation, and gradient distribution in the server are performed in parallel in a pipeline manner; An update module is provided for each of the aforementioned devices. For each device: the update module updates the gradient using a set weight after the device receives the gradient, the set weight being inversely proportional to the update frequency of the device in the current round, and updates its local bottom sub-model using the updated gradient; wherein, the forward propagation, data upload, and model update in the device are performed in parallel in a pipelined manner; the updated bottom sub-model is: ； in, , These are the bottom sub-models of device i before and after the update in the kth round of training; The first learning rate; Set weights for device i; This represents the gradient received by device i for local computation; , The number of devices in the federated learning framework; The apparatus is used to implement the partitioned federated learning method based on adaptive pipeline parallelism as described in any one of claims 1-6.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the partitioned federated learning method based on adaptive pipeline parallelism as described in any one of claims 1-6.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to execute the partitioned federated learning method based on adaptive pipeline parallelism as described in any one of claims 1-6.

Images