A federated learning training method and system based on model segmentation and resource allocation

Abstract

The present invention discloses a federated learning training method and system based on model segmentation and resource allocation. The method is aimed at the federated learning training tasks performed in resource-constrained and heterogeneous edge environments, adaptively divides the model to be trained into two parts, unloads the training tasks of one part of the model from the working node to the master node, and combines the dynamic adjustment of bandwidth resources and master node computing resources, greatly reduces the communication overhead in the federated learning training process and effectively balances the completion time of heterogeneous working nodes, thereby significantly improving the training speed of federated learning.

Description

Technical Field

[0001] The present invention belongs to the field of federated learning and edge computing, and specifically relates to a federated learning training method and system based on model segmentation and resource allocation. Background Art

[0002] The federated learning framework for collaborative training of deep neural networks aims to make full use of the rich computing and communication resources in the edge environment and collaboratively train high-precision machine learning models through multiple smart terminals. In the federated learning framework for edge computing environments, smart terminals update the current global model based on the calculation of local training data, and the edge server acts as a collaborative node to aggregate the intermediate results and synchronize the model parameters, and achieve the expected accuracy of the model through multiple iterative training. Federated learning has outstanding advantages such as easy distributed collaboration, local processing of edge data, and user privacy protection. It perfectly fits the core concept of edge computing and has received widespread attention from academia and industry in recent years.

[0003] However, as edge intelligence application scenarios become increasingly complex, the scale of parameters involved in deep neural network models in federated learning is rapidly increasing. For example, in typical application scenarios such as high-precision image recognition and natural language processing, the scale of parameters reaches millions or even hundreds of millions. In addition, the network environment of terminal devices is dynamic and changeable, and the communication performance is limited. It is difficult to meet the communication requirements of federated learning intermediate result aggregation and model parameter synchronization, resulting in increased time overhead and becoming a key factor restricting the training speed of federated learning in edge environments. In addition, in edge environments with heterogeneous resources, the computing power and communication capabilities of the terminal devices responsible for model training are different, and the time spent on completing tasks with the same amount of computing and communication is also significantly different. Under the model synchronization method based on batch synchronization and parallelism, terminal devices with good resource performance need to wait for terminal devices with poor resource performance to complete model aggregation. Therefore, the time it takes for federated learning to complete a global model update depends on the worst performing participating node, which not only leads to serious resource underutilization, but also further increases the time overhead of model training.

[0004] None of the existing solutions can deal with the above two problems at the same time. Although the data compression-based method can reduce the communication volume, the bandwidth-constrained and highly dynamic terminal devices will still face huge communication overhead, and compression cannot remove the limitation of resource heterogeneity on the parallel capability of federated learning. The model synchronization mechanism based on asynchronous parallelism can alleviate the impact of terminal device resource heterogeneity on the efficiency of federated learning training to a certain extent, but the staleness of local updates is still common in the training process, resulting in a decrease in the training convergence speed. In addition, the above two methods will have a certain impact on the accuracy of the trained model.

[0005] Therefore, it is necessary to propose a federated learning training acceleration method that can not only reduce communication overhead, but also effectively improve the parallel collaboration capability of heterogeneous devices during training without affecting the model accuracy. Summary of the invention

[0006] In view of the above-mentioned deficiencies in the prior art, the object of the present invention is to provide a federated learning training method based on model segmentation and resource allocation, which can effectively reduce the communication overhead of federated learning while balancing the training speed between heterogeneous edge devices, thereby greatly improving the training speed of federated learning without affecting the accuracy of the model.

[0007] The present invention also provides a corresponding federated learning training system based on model segmentation and resource allocation.

[0008] In order to achieve the above-mentioned object of the invention, the present invention adopts the following technical solutions:

[0009] In a first aspect, a federated learning training method based on model segmentation and resource allocation comprises the following steps:

[0010] The master node collects resource information of the working nodes, including computing power and wireless signal transmission power;

[0011] The master node calculates the model parameters, the output data volume of the model's intermediate layer, and the amount of computation required for training based on the configuration of the model to be trained;

[0012] The master node determines the model segmentation strategy and resource allocation strategy, including: initializing the model segmentation strategy and resource allocation strategy, estimating the global completion time of a round of model training according to the model segmentation strategy and resource allocation strategy, selecting the optimal model segmentation strategy according to the current state of system resources, selecting the optimal resource allocation strategy according to the current model segmentation strategy, repeating the above operations until the accuracy of the global completion time reduction is less than the preset accuracy, and completing the determination of the model segmentation strategy and resource allocation strategy, wherein the model segmentation strategy specifies the computing amount that the working node and the master node are responsible for respectively, and the resource allocation strategy specifies the bandwidth allocated to the working node and the master node computing resources allocated to the working node;

[0013] The master node divides the model of each working node according to the model division strategy, and adjusts the bandwidth allocated to each working node and the master node computing resources according to the resource allocation strategy;

[0014] The worker nodes and the master node work together to train the segmented model.

[0015] Furthermore, the calculation of the parameters of the model, the output data volume of the middle layer of the model and the calculation volume required for training according to the configuration of the model to be trained includes:

[0016] Calculate the number of parameters for each layer of the model based on the model structure and type;

[0017] Calculate the input and output size of each layer of the model based on the size of the training data and the structure of the model;

[0018] The number of floating-point operations required for each layer of the model to complete a training run is calculated based on the input and output sizes of each layer and the type of the layer. The amount of calculation required for a complete training run is the sum of the floating-point operations required for all layers.

[0019] Furthermore, the types include convolutional layers, pooling layers, and fully connected layers.

[0020] Furthermore, the master node initialization model segmentation strategy and resource allocation strategy include:

[0021] Allocate wireless bandwidth resources evenly to all working nodes;

[0022] Evenly distribute the computing resources available on the master node to all worker nodes;

[0023] The model split point of all working nodes is set at the last layer, that is, all training tasks are completed by the working nodes at the beginning.

[0024] Furthermore, the model segmentation strategy divides the model into two parts, the first part of the model is from the first layer to the segmentation layer, including the segmentation layer; the second part of the model is from the segmentation layer to the last layer, excluding the segmentation layer; the computing amount of the working node is the computing amount required for the first part of the model to complete forward propagation and backward propagation, and the computing amount of the master node is the computing amount required for the second part of the model to complete forward propagation and backward propagation.

[0025] Furthermore, estimating the global completion time of a round of model training according to the model segmentation strategy and the resource allocation strategy includes:

[0026] The communication volume between the working node and the master node is determined according to the segmentation strategy. The data transmission operation between the working node and the master node during the training process includes two transmissions of the first part of the model, the output result of the segmentation layer transmitted from the working node to the master node, and the gradient of the segmentation layer transmitted from the master node to the working node. The sum of the above transmission data is the communication volume between the working node and the master node;

[0027] The data transmission rate between each working node and the master node is calculated by bandwidth and wireless transmission power;

[0028] Estimate the computing time of the working nodes based on their computing workload and computing power;

[0029] Estimate the master node computing time based on the master node's computing workload and the master node computing resources allocated to the worker nodes;

[0030] Estimate the communication time based on the communication volume and data transmission rate between the working node and the master node;

[0031] The estimated completion time of the working node is obtained by adding the working node calculation time, the master node calculation time, and the communication time;

[0032] Estimate the completion time of all working nodes, and the longest completion time is the estimated global completion time.

[0033] Furthermore, the selecting the optimal model segmentation strategy according to the current system resource status includes:

[0034] Without changing the resource allocation strategy, for a working node, each layer of models is selected as the split layer starting from the first layer model, and the completion time of the working node is estimated at this time;

[0035] Determine the partitioning layer that minimizes the completion time as the optimal partitioning layer for the work node;

[0036] The above operations are performed on all working nodes, and the final set of optimal segmentation layers is the optimal model segmentation strategy.

[0037] Furthermore, the selecting the optimal allocation strategy according to the current model segmentation strategy includes:

[0038] Without changing the model segmentation strategy, the master node constructs a linear programming problem, where the variable to be optimized is the global completion time, and the decision variables are the bandwidth allocation strategy and the master node computing resource allocation strategy;

[0039] Solving the linear programming problem obtains the optimal bandwidth allocation strategy and master node computing resource allocation strategy.

[0040] Furthermore, the working node and the master node cooperate to train the segmented model, including:

[0041] The master node sends the first part of the model to the worker node;

[0042] The working node uses local data as model input to complete the forward propagation calculation of the first part of the model;

[0043] The worker node sends the output of the first part of the model to the master node;

[0044] The master node uses the received output as input to complete the forward propagation calculation of the second part of the model;

[0045] The master node completes the back propagation calculation of the second part of the model;

[0046] The master node sends the gradient of the second part of the model to the worker node;

[0047] The worker node takes the received gradient as input and completes the back propagation calculation of the first part of the model;

[0048] The worker node sends the first part of the updated model to the master node;

[0049] The master node concatenates the two parts of the model and aggregates them with other model updates.

[0050] In the second aspect, a federated learning training system based on model segmentation and resource allocation is provided, wherein the working cluster consists of a master node and multiple working nodes, and the computing task is to collaboratively train a deep learning model, characterized in that the master node is responsible for model segmentation, distribution, and aggregation, training of part of the model, and allocation of bandwidth resources to the working nodes and computing resources of the master node, the master node is configured to execute the operations completed by the master node according to the federated learning training method described in the first aspect, the working node is responsible for training part of the model and transmitting the training results back to the master node, and the working node is configured to execute the operations completed by the working node according to the federated learning training method described in the first aspect.

[0051] Beneficial effects: The present invention is aimed at federated learning training tasks performed in resource-constrained and heterogeneous edge environments. The model to be trained is adaptively divided into two parts, and the training tasks of one part of the model are offloaded from the working node to the master node. At the same time, combined with the dynamic adjustment of bandwidth resources and the computing resources of the master node, the communication overhead in the federated learning training process is greatly reduced and the completion time of heterogeneous working nodes is effectively balanced, thereby significantly improving the training speed of federated learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is an overall flow chart of the federated learning training method according to the present invention;

[0053] Figure 2 are three network structures of the deep learning model according to the present invention;

[0054] Figure 3 is the computational process of post-segmentation training of the model according to the present invention;

[0055] Figure 4 This is the overall process of the master node and multiple working nodes collaboratively training the model after model segmentation according to the present invention. DETAILED DESCRIPTION

[0056] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0057] like Figure 1As shown, the present invention proposes a federated learning training acceleration method based on model segmentation and resource allocation, comprising the following steps:

[0058] Step S1, the master node collects resource information of the working nodes, including computing power and wireless signal transmission power;

[0059] Step S2, the master node calculates the model parameters, the output data volume of the model middle layer and the calculation volume required for training according to the configuration of the model to be trained;

[0060] Step S3, the master node initializes the model segmentation strategy and resource allocation strategy;

[0061] Step S4, the master node estimates the global completion time of a round of model training according to the model segmentation strategy and resource allocation strategy;

[0062] Step S5, the master node selects the optimal model segmentation strategy according to the current state of system resources;

[0063] Step S6, the master node selects the optimal resource allocation strategy according to the current model segmentation strategy;

[0064] Step S7, the master node repeats the above three steps S4-S6 until the accuracy of the global completion time reduction is less than the preset accuracy σ;

[0065] Step S8: The master node divides the model of each working node into two parts according to the model segmentation strategy and assigns the training task of the second part of the model to the master node for execution, and adjusts the allocated bandwidth to each working node and the master node computing resources according to the resource allocation strategy;

[0066] Step S9: The working node and the master node cooperate to train the segmented model.

[0067] Figure 2 The following shows examples of the parameter structure, input data structure, and output data structure of three common network layers (convolutional layer, pooling layer, and fully connected layer) in deep learning models. In summary, in step S2, the master node calculates the model parameters, the output data volume of the model's intermediate layer, and the amount of computation required for training based on the configuration of the model to be trained. The calculations include: calculating the parameters of each layer of the model based on the model structure and type; calculating the input and output size of each layer of the model based on the size of the training data and the structure of the model; calculating the number of floating-point operations required for each layer of the model to complete a training session based on the input and output size of each layer of the model and the type of the layer. The amount of computation required for a complete training session is the sum of the floating-point operations required for all layers. Figure 2 A specific description is given of how to calculate the model parameters, the output data volume of the model's intermediate layers, and the amount of computation required for training based on this information.

[0068] like Figure 2 -(a) shows that the input of the convolution layer is a three-dimensional matrix, where w, h, and c represent the width, height, and number of input channels, respectively. A convolution layer consists of n three-dimensional matrices, which are called convolution kernels, where k, k, and c are the width, height, and number of channels of the convolution kernel, respectively. Here, the height and width of the convolution kernel are equal. The convolution kernel convolves the input matrix from left to right and from top to bottom, and calculates the dot product between the convolution kernel and the input. The kernel moves on the input matrix like a sliding window with a step size of s. In some cases, in order to control the size of the output, 0 is padded around the input, and the padding size is represented by p. Then the number of parameters of the convolution layer is The calculation formula is expressed as follows:

[0069]

[0070] The amount of data output by the convolutional layer The calculation formula is expressed as follows:

[0071]

[0072] Where w', h', c' are the width, height and number of channels of the output data respectively. The number of floating point operations required by the convolutional layer during forward propagation and backward propagation and The calculation formula is expressed as follows:

[0073]

[0074]

[0075] like Figure 2 -(b) shows that the pooling layer takes a three-dimensional matrix as input and outputs a smaller three-dimensional matrix. The pooling layer is controlled by only a few hyperparameters, so its number of parameters can be ignored. The output data size of the pooling layer is S o The calculation formula is as follows:

[0076]

[0077] Where w', h', c' are the width, height and number of channels of the output data respectively. The number of floating point operations required by the pooling layer during forward propagation and backward propagation and The calculation formula is expressed as follows:

[0078]

[0079] like Figure 2-(c) As shown, the input data of the fully connected layer is a vector of size 1*c, and the output data is a vector of size 1*c'. The weight size of the fully connected layer The calculation formula is expressed as follows:

[0080]

[0081] Output data size c', the number of floating point operations required during forward propagation and backward propagation and The calculation formula is expressed as follows:

[0082]

[0083]

[0084] In step S3, the master node initializes the model segmentation strategy and resource allocation strategy. According to an embodiment of the present invention, the initialization specifically includes: evenly distributing wireless bandwidth resources to all working nodes; evenly distributing the computing resources available to the master node to all working nodes; setting the model segmentation points of all working nodes to the last layer, that is, initially all training tasks are completed by the working nodes. It should be understood that this is only an example of initialization and is not a limitation of the present invention. In other embodiments, the master node bandwidth resources, computing resources, and segmentation points can be allocated according to other configurations, for example, allocating resources by weight and / or designating other layers as initial segmentation points. The present invention is not limited to this.

[0085] Steps S4-S7 constitute a loop body for determining the model segmentation strategy and resource allocation strategy. In step S4, the master node first estimates the global completion time of a round of model training. According to an embodiment of the present invention, the estimation operation includes:

[0086] (1) Determine the amount of computation that the working node and the master node are responsible for respectively according to the segmentation strategy. The amount of computation of the working node is the amount of computation required for the first part of the model to complete forward propagation and backward propagation, and the amount of computation of the master node is the amount of computation required for the second part of the model to complete forward propagation and backward propagation. The first part of the model is from the first layer to the segmentation layer, including the segmentation layer; the second part of the model is from the segmentation layer to the last layer, excluding the segmentation layer;

[0087] (2) Determine the communication volume between the working node and the master node according to the segmentation strategy. The data transmission operation between the working node and the master node during the training process includes two transmissions of the first part of the model (the master node distributes the model, and the working node returns the trained model), the working node transmits the output result of the segmentation layer to the master node, and the master node transmits the gradient of the segmentation layer to the working node. The sum of the above transmission data is the communication volume between the working node and the master node;

[0088] (3) Calculate the data transmission rate between each working node and the master node by using the bandwidth and wireless transmission power, where the wireless transmission power is collected in step S1;

[0089] (4) estimating the computing time of the working node according to the computing amount and computing power of the working node, wherein the computing power of the working node is collected and obtained in step S1;

[0090] (5) Estimate the master node computing time based on the master node's computing workload and the master node computing resources allocated to the worker nodes;

[0091] (6) Estimate the communication time based on the communication volume and data transmission rate between the working node and the master node;

[0092] (7) Add the working node calculation time, the master node calculation time, and the communication time to obtain the estimated completion time of the working node;

[0093] (8) Estimate the completion time of all work nodes, and the longest completion time is the estimated global completion time.

[0094] Steps S5 and S6 are used to achieve the optimal selection of two strategies respectively, and they are not necessarily executed in the order shown in the figure. For example, the optimal resource allocation strategy can be selected according to the model segmentation strategy in step S5, and the optimal model segmentation strategy can be selected according to the state of resources in step S6. The master node selects the optimal model segmentation strategy according to the current state of system resources, including: without changing the resource allocation strategy, for a working node, starting from the first layer of the model, each layer of the model is selected as the segmentation layer in turn, and the completion time of the working node at this time is estimated; the segmentation layer with the smallest completion time is determined as the optimal segmentation layer of the working node; the above operations are performed on all working nodes, and the set of optimal segmentation layers finally obtained is the optimal model segmentation strategy.

[0095] The master node selects the optimal allocation strategy according to the current model segmentation strategy, including: without changing the model segmentation strategy, the master node constructs a convex optimization problem, in which the variable to be optimized is the global completion time, and the decision variable is the bandwidth allocation strategy, that is, the set of bandwidths allocated by the master node to all working nodes, and the computing resource allocation strategy, that is, the set of computing resources allocated by the master node to all working nodes; solving the convex optimization problem to obtain the optimal bandwidth allocation strategy and master node computing resource allocation strategy.

[0096] Step S7 controls iteration according to the change of the completion time. When the decrease of the global completion time is lower than a preset threshold, the iteration is stopped and the process proceeds to step S8.

[0097] In step S8, the master node partitions the model according to the optimal model partitioning strategy, and allocates resources according to the optimal resource allocation strategy.

[0098] In step S9, the working node and the master node cooperate to train the segmented model. According to an embodiment of the present invention, the training process includes: the master node sends the first part of the model to the working node; the working node uses local data as model input to complete the forward propagation calculation of the first part of the model; the working node sends the output of the first part of the model to the master node; the master node uses the received output as input to complete the forward propagation calculation of the second part of the model; the master node completes the back propagation calculation of the second part of the model; the master node sends the gradient of the second part of the model to the working node; the working node uses the received gradient as input to complete the back propagation calculation of the first part of the model; the working node sends the updated first part of the model to the master node; the master node splices the two parts of the model and aggregates them with other model updates.

[0099] Figure 3This is an example of the computational flow of model training after segmentation. In the figure, a deep learning model consisting of three layers of networks (represented by W1, W2, and W3 respectively) is divided into two parts. W1 is trained by the working node, and W2W3 are trained by the master node. The training computational process starts from the working node. The training data x is sent to the model as input, followed by forward propagation (marked by solid arrows) and backward propagation (marked by dotted arrows). The forward propagation is carried out from front to back. Each layer of the network receives the output of the previous layer as the input of this layer, and after operating with the parameters of this layer, the result is output. In the figure, after W1 completes the forward propagation, the working node passes the output result z1 to the master node. The master node uses z1 as the input of W2 to continue the subsequent forward propagation. z2 and z3 represent the outputs of W1 and W2 respectively. At the end of the forward propagation, the loss function L is used to calculate the difference J between the model output and the actual label y. Back propagation is performed from back to front. Each layer receives the gradient of the previous layer and calculates the gradient of the weight of this layer relative to the loss function, and updates the value of the weight of this layer according to the gradient direction. ▽z and ▽W in the figure represent the output of the network of this layer and the gradient of the weight relative to the loss function respectively. γ represents the learning rate, which is used to control the step size of the weight update. After the master node completes the back propagation of W3W2, it passes the gradient ▽z1 to the working node, and the working node uses the gradient to complete the back propagation of W1. At this point, the weights of all layers are updated and a round of training is completed.

[0100] Figure 4 The figure shows the overall process of a master node and multiple worker nodes cooperating to train a three-layer neural network model (w1w2w3) after model segmentation. The main steps include:

[0101] The master node sends the first part of the split model to each worker node. In this example, the master node sends w1 to worker node 1 (1-1) and w1w2 to worker node 2 (2-1).

[0102] The worker node uses local data as model input to complete the forward propagation calculation of the first part of the model. Worker node 1 completes the forward propagation of w1 (1-2), and worker node 2 completes the forward propagation of w1w2 (2-2).

[0103] Worker 1 sends the output of w1 to the master node (1-3), and worker 2 sends the output of w2 to the master node (2-3);

[0104] The master node takes the received output as input and performs the forward propagation of w2w3 (1-4) and the forward propagation of w3 (2-4) in parallel;

[0105] The master node performs the reverse propagation of w2w3 (1-5) and the reverse propagation of w3 (2-5) in parallel;

[0106] The master node sends the gradient of w1 to worker node 1 (1-6), and sends the gradient of w2 to worker node 2 (2-6);

[0107] The worker node uses the received gradient as input to complete the back propagation calculation of the first part of the model. Worker node 1 completes the back propagation of w1 (1-7), and worker node 2 completes the back propagation of w1w2 (2-7).

[0108] The worker node sends the updated first part of the model to the master node. Worker node 1 sends w1' to the master node (1-8), and worker node 2 sends w1"w2" to the master node (2-8).

[0109] The master node aggregates the two update models (9).

[0110] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A federated learning training method based on model segmentation and resource allocation, characterized in that: The method comprises the following steps: The master node collects resource information of the working nodes, including computing power and wireless signal transmission power; The master node calculates the model parameters, the output data volume of the model's intermediate layer, and the amount of computation required for training based on the configuration of the model to be trained; The master node determines the model segmentation strategy and resource allocation strategy, including: initializing the model segmentation strategy and resource allocation strategy, estimating the global completion time of a round of model training according to the model segmentation strategy and resource allocation strategy, selecting the optimal model segmentation strategy according to the current state of system resources, selecting the optimal resource allocation strategy according to the current model segmentation strategy, repeating the above operations until the accuracy of the global completion time reduction is less than the preset accuracy, and completing the determination of the model segmentation strategy and resource allocation strategy, wherein the model segmentation strategy specifies the computing amount that the working node and the master node are responsible for respectively, and the resource allocation strategy specifies the bandwidth allocated to the working node and the master node computing resources allocated to the working node; The master node divides the model of each working node according to the model division strategy, and adjusts the bandwidth allocated to each working node and the master node computing resources according to the resource allocation strategy; the model division strategy divides the model into two parts, the first part of the model is from the first layer to the division layer, including the division layer; the second part of the model is from the division layer to the last layer, excluding the division layer; the computing amount of the working node is the computing amount required for the first part of the model to complete forward propagation and backward propagation, and the computing amount of the master node is the computing amount required for the second part of the model to complete forward propagation and backward propagation; The worker nodes and the master node work together to train the segmented model, including: The master node sends the first part of the model to the worker node; The working node uses local data as model input to complete the forward propagation calculation of the first part of the model; The worker node sends the output of the first part of the model to the master node; The master node uses the received output as input to complete the forward propagation calculation of the second part of the model; The master node completes the back propagation calculation of the second part of the model; The master node sends the gradient of the second part of the model to the worker node; The worker node takes the received gradient as input and completes the back propagation calculation of the first part of the model; The worker node sends the first part of the updated model to the master node; The master node concatenates the two parts of the model and aggregates them with other model updates.

2. The method according to claim 1, characterized in that The calculation of the parameters of the model, the output data volume of the middle layer of the model and the calculation volume required for training according to the configuration of the model to be trained includes: Calculate the number of parameters for each layer of the model based on the model structure and type; Calculate the input and output size of each layer of the model based on the size of the training data and the structure of the model; The number of floating-point operations required for each layer of the model to complete a training run is calculated based on the input and output sizes of each layer and the type of the layer. The amount of calculation required for a complete training run is the sum of the floating-point operations required for all layers.

3. The method according to claim 2, characterized in that The types include convolutional layers, pooling layers, and fully connected layers.

4. The method according to claim 1, characterized in that: The master node initialization model segmentation strategy and resource allocation strategy include: Allocate wireless bandwidth resources evenly to all working nodes; Evenly distribute the computing resources available on the master node to all worker nodes; The model split point of all working nodes is set at the last layer, that is, all training tasks are completed by the working nodes initially.

5. The method according to claim 1, characterized in that The global completion time of a round of model training estimated according to the model segmentation strategy and resource allocation strategy includes: The communication volume between the working node and the master node is determined according to the segmentation strategy. The data transmission operation between the working node and the master node during the training process includes two transmissions of the first part of the model, the output result of the segmentation layer transmitted from the working node to the master node, and the gradient of the segmentation layer transmitted from the master node to the working node. The sum of the above transmission data is the communication volume between the working node and the master node; The data transmission rate between each working node and the master node is calculated by bandwidth and wireless transmission power; Estimate the computing time of the working nodes based on their computing workload and computing power; Estimate the master node computing time based on the master node's computing workload and the master node computing resources allocated to the worker nodes; Estimate the communication time based on the communication volume and data transmission rate between the working node and the master node; The estimated completion time of the working node is obtained by adding the working node calculation time, the master node calculation time, and the communication time; Estimate the completion time of all working nodes, and the longest completion time is the estimated global completion time.

6. The method according to claim 1, characterized in that The step of selecting the optimal model segmentation strategy according to the current system resource status includes: Without changing the resource allocation strategy, for a working node, each layer of models is selected as the split layer starting from the first layer model, and the completion time of the working node is estimated at this time; Determine the partitioning layer that minimizes the completion time as the optimal partitioning layer for the work node; The above operations are performed on all working nodes, and the final set of optimal segmentation layers is the optimal model segmentation strategy.

7. The method according to claim 1, characterized in that The selecting the optimal allocation strategy according to the current model segmentation strategy includes: Without changing the model segmentation strategy, the master node constructs a linear programming problem, where the variable to be optimized is the global completion time, and the decision variables are the bandwidth allocation strategy and the master node computing resource allocation strategy; Solving the linear programming problem obtains the optimal bandwidth allocation strategy and master node computing resource allocation strategy.

8. A federated learning training system based on model segmentation and resource allocation, wherein the working cluster consists of a master node and multiple working nodes, and the computing task is to collaboratively train a deep learning model, characterized in that: The master node is responsible for the segmentation, distribution, and aggregation of the model, the training of some models, and the allocation of bandwidth resources to the working nodes and the computing resources of the master node. The master node is configured to perform the operations completed by the master node according to the federated learning training method described in any one of claims 1-7. The working node is responsible for the training of some models and transmits the training results back to the master node. The working node is configured to perform the operations completed by the working node according to the federated learning training method described in any one of claims 1-7.

Images