Self-adaptive model partitioning method and system applied to distributed training
Through the combination of deep reinforcement learning and Bayesian optimization, the intelligent partitioned deep learning model solves the problem of insufficient flexibility in model partitioning strategies in distributed training, and improves resource utilization and training efficiency.
Patent Information
- Application Number
- CN202411202794.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-29
- Publication Date
- 2024-11-22
AI Technical Summary
The existing model partitioning methods lack flexibility in distributed training and are difficult to adapt to variable task requirements and complex environments, resulting in inadequate resource utilization and training efficiency.
Deep reinforcement learning combined with Bayesian optimization algorithm is adopted to intelligently partition the deep learning model by building Q networks and reward functions, and optimize the model sharding strategy to improve resource utilization and training speed.
It realizes intelligent adaptive optimization of distributed model sharding, improves training speed, reduces resource overhead, and improves the intelligence and training efficiency of model partitioning.
Smart Images

Figure CN119004112A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of deep learning technology, and specifically to an adaptive model partitioning method and system for distributed training. Background Art
[0002] With the increasing complexity and scale of deep learning models, distributed training has become an important means to accelerate the training of large-scale models. At present, distributed training mainly includes: data parallelism and model parallelism strategies. Among them, the model parallelism strategy reduces the memory burden of a single node by allocating different parts of the model to different computing nodes, which is especially suitable for ultra-large-scale models. Compared with data parallelism, model parallelism can more effectively utilize computing resources and adapt to complex model structures. Therefore, studying model parallelism strategies is crucial to improving the efficiency of distributed training. The partitioning strategy of the model needs to be carefully designed to ensure that the load of each node is balanced. Otherwise, when there are too many computing tasks on some nodes, it will become a bottleneck and have a "barrel effect", thereby affecting the overall training speed. Therefore, how to effectively partition the model in a distributed environment to optimize resource utilization and training efficiency needs to be solved urgently. However, traditional model partitioning methods usually rely on fixed strategies or manual adjustments, lack flexibility, and are difficult to adapt to changing task requirements and complex environments.
[0003] With the continuous development of deep learning technology, intelligent and adaptive deep reinforcement learning has gradually become an important tool for solving complex decision-making problems due to its outstanding performance in adaptive systems. Deep reinforcement learning learns the optimal decision-making strategy through interactive training of intelligent agents in a simulated environment. In addition, the Bayesian optimization algorithm performs outstandingly in automated hyperparameter adjustment and can efficiently explore and optimize the model's hyperparameter configuration, thereby improving the training performance of distributed slicing models.
[0004] For distributed training, how to combine deep reinforcement learning and Bayesian optimization algorithm to achieve distributed training of models is a technical problem that needs to be solved. Summary of the invention
[0005] The technical task of the present invention is to address the above shortcomings and provide an adaptive model partitioning method and system for distributed training to solve the technical problems of distributed training and how to combine deep reinforcement learning and Bayesian optimization algorithm to realize distributed training of models.
[0006] In a first aspect, the present invention provides an adaptive model partitioning method for distributed training, which selects a sharding strategy for a distributed training model based on deep reinforcement learning, obtains multiple slice models based on the sharding strategy and applies them in parallel to distributed computing nodes for model training, and the method comprises the following steps:
[0007] Feature extraction: For a distributed training model with K layers, define feature types and encode various types of features using one-hot encoding to obtain feature vectors for each layer.
[0008] Build a deep reinforcement adaptive model: Build a Q network based on a deep neural network, define state information, actions, and reward functions. The state information includes the feature vector of the slicing model and the training time, resource utilization, and inter-node communication overhead when the slicing model is distributedly trained through distributed computing nodes. The action is the slicing strategy taken by the agent under the current state information. The reward function R is used to evaluate the effect of the current slicing strategy. The Q network takes the state information as input and predicts the Q value of each action that the output agent may take.
[0009] Model and parameter initialization: Initialize the parameters and experience playback pool of the Q network, and initialize the sharding strategy X0. Apply the slice model obtained based on the sharding strategy X0 to the distributed computing nodes in parallel. Each computing node performs distributed model training on the assigned slice model based on the local training set, and obtains the training time t0, resource utilization e0, and inter-node communication overhead c0 corresponding to the initial distributed model training, and obtains the initial state information s0;
[0010] Iterative training calculation: The deep reinforcement adaptive model is trained iteratively multiple times to obtain the final sharding strategy and the parameters of the Q network. During iterative training, the Q value of each possible action is output through the Q network according to the current state information. The intelligent agent selects the action corresponding to the maximum Q value based on the ∈-greedy strategy to execute, and obtains the next state and the reward value of the current action. The current state information, the action corresponding to the current state, the reward value of the action, and the action at the next moment are returned to the experience pool as prior knowledge. Prior knowledge is extracted from the experience pool for training and updating the parameters of the Q network until the intelligent agent converges to the optimal sharding strategy.
[0011] As a preference, the feature definition of each layer of the distributed training model has four categories, namely, operation type, input data volume, calculation volume, and output data volume;
[0012] Among them, the operation types include the basic components in the distributed training model, including convolution, pooling, full connection and optimizer;
[0013] The data volume is the amount of data input to the current layer, the computational complexity is the computational complexity of the current layer, and the output data volume is the amount of data output by the current layer;
[0014] The features F of each layer k It is expressed as:
[0015] F k =[Operator k ,InputDatak ,Compute k ,Outputdata k ],
[0016] Operator k Indicates the operation type characteristics, InputData k Indicates the data volume characteristics, Compute k Indicates the computational characteristics, Outputdata k Indicates the output data volume characteristics.
[0017] Preferably, the training time represents the total training completion time, which is calculated by the computing nodes performing the distributed training;
[0018] Resource utilization e0 includes GPU utilization CPU Utilization Memory Utilization and storage I / O utilization The calculation formula is as follows:
[0019]
[0020] The inter-node communication cost c0 includes the communication time Bandwidth Utilization and delay The calculation formula is as follows:
[0021]
[0022] As a preferred method, the Q value calculation formula is as follows:
[0023] Q(s,a)=Q(s,a)+α[r+γmaxQ(s′,a′)-Q(s,a)]Q(s,a),
[0024] Among them, s represents the current state information, a represents the action taken in state s, r represents the reward value for the action, α represents the learning rate, which determines the speed of Q value update, γ is the discount factor, which is used to balance the weight of current reward and future reward, s′ represents the new state after executing action a, maxQ(s′,a′) represents the maximum Q value of all possible actions a′ in the new state s′, and represents the future cumulative reward starting from the new state s′ under the optimal strategy;
[0025] The calculation formula of the reward value is expressed as:
[0026] r(s,a,s′)=ε(-t)+μe+η(-c),
[0027] The reward function is determined based on the training time t, resource utilization e, and inter-node communication overhead c, and ε, μ, and η are all weight factors;
[0028] For the first multiple iterations of the deep reinforcement adaptive model training, the corresponding Q value Q0 calculation formula is as follows:
[0029]
[0030] Preferably, for the Q network, when the deep reinforcement adaptive model is trained for multiple iterations, the loss function L(θ) of the Q network is minimized by the Adam gradient descent algorithm, and the parameter θ of the Q network is adjusted based on the Bayesian optimization method;
[0031] The loss function of the Q network is expressed as:
[0032]
[0033] Among them, N is the number of experience samples in a small batch, y i is the target Q value, Q(s i ,a i |θ) is the Q value estimated by the Q network, and the target Q value is calculated as follows:
[0034] y t =r t +γ·max a′ Q′(s t+1 ,a′|θ′),
[0035] Among them, s t+1 represents the state information at time t+1, and a′ represents s t+1 The action taken next, r t Represents the reward value for the action, max a′ Q′(s t+1 ,a′|θ′) represents s t+1 The maximum Q value among all possible actions a′;
[0036] When adjusting the parameters θ of the Q network based on the Bayesian optimization method, set θ∈{α,γ}. Bayesian optimization finds the parameter combination θ that makes the training most efficient by sampling and updating the posterior distribution. * , the optimization goal is to minimize the training time T, the formula is as follows:
[0037] θ * =argθminT(θ),
[0038] T(θ) represents the training time under the hyperparameter θ.
[0039] In a second aspect, the present invention provides an adaptive model partitioning system for distributed training, which is used to select a sharding strategy for a distributed training model by using an adaptive model partitioning method for distributed training as described in any one of the first aspects, including a feature extraction module, a deep reinforcement adaptive model construction module, a model and parameter initialization module, and an iterative training calculation module;
[0040] The feature extraction module is used to perform the following: for a distributed training model with K layers, define feature types and encode various types of features by one-hot encoding to obtain feature vectors for each layer;
[0041] The deep reinforcement adaptive model building module is used to perform the following: build a Q network based on a deep neural network, define state information, actions, and reward functions. The state information includes the feature vector of the slicing model and the training time, resource utilization, and inter-node communication overhead when the slicing model is distributedly trained through distributed computing nodes. The action is the slicing strategy taken by the agent under the current state information. The reward function R is used to evaluate the effect of the current slicing strategy. The Q network takes the state information as input and predicts the Q value of each action that the output agent may take.
[0042] The model and parameter initialization module is used to perform the following: initialize the parameters and experience playback pool of the Q network, and initialize the sharding strategy X0, apply the slice model obtained based on the sharding strategy X0 in parallel to the distributed computing nodes, each computing node performs distributed model training on the assigned slice model based on the local training set, and obtains the training time t0, resource utilization e0 and inter-node communication overhead c0 corresponding to the initial distributed model training, and obtains the initial state information s0;
[0043] The iterative training calculation module is used to perform the following: perform multiple iterative training on the deep reinforcement adaptive model to obtain the final sharding strategy and the parameters of the Q network. During iterative training, according to the current state information, the Q value of each possible action is output through the Q network. The intelligent agent selects the action corresponding to the maximum Q value based on the ∈-greedy strategy to execute, and obtains the next state and the reward value of the current action. The current state information, the action corresponding to the current state, the reward value of the action, and the action at the next moment are returned to the experience pool as prior knowledge, and the prior knowledge is extracted from the experience pool for training and updating the parameters of the Q network until the intelligent agent converges to the optimal sharding strategy.
[0044] As a preference, the feature definition of each layer of the distributed training model has four categories, namely, operation type, input data volume, calculation volume, and output data volume;
[0045] Among them, the operation types include the basic components in the distributed training model, including convolution, pooling, full connection and optimizer;
[0046] The data volume is the amount of data input to the current layer, the computational complexity is the computational complexity of the current layer, and the output data volume is the amount of data output by the current layer;
[0047] The features F of each layer k It is expressed as:
[0048] F k =[Operator k ,InputData k ,Compute k ,Outputdata k ],
[0049] Operator k Indicates the operation type characteristics, InputData k Indicates the data volume characteristics, Compute k Indicates the computational characteristics, Outputdata k Indicates the output data volume characteristics.
[0050] Preferably, the training time represents the total training completion time, which is calculated by the computing nodes performing the distributed training;
[0051] Resource utilization e0 includes GPU utilization CPU Utilization Memory Utilization and storage I / O utilization The calculation formula is as follows:
[0052]
[0053] The inter-node communication cost c0 includes the communication time Bandwidth Utilization and delay The calculation formula is as follows:
[0054]
[0055] As a preferred method, the Q value calculation formula is as follows:
[0056] Q(s,a)=Q(s,a)+α[r+γmaxQ(s′,a′)-Q(s,a)]Q(s,a),
[0057] Among them, s represents the current state information, a represents the action taken in state s, r represents the reward value for the action, α represents the learning rate, which determines the speed of Q value update, γ is the discount factor, which is used to balance the weight of current reward and future reward, s′ represents the new state after executing action a, maxQ(s′,a′) represents the maximum Q value of all possible actions a′ in the new state s′, and represents the future cumulative reward starting from the new state s′ under the optimal strategy;
[0058] The calculation formula of the reward value is expressed as:
[0059] r(s,a,s′)=ε(-t)+μe+η(-c),
[0060] The reward function is determined based on the training time t, resource utilization e, and inter-node communication overhead c, and ε, μ, and η are all weight factors;
[0061] For the first multiple iterations of the deep reinforcement adaptive model training, the corresponding Q value Q0 calculation formula is as follows:
[0062]
[0063] Preferably, for the Q network, when the deep reinforcement adaptive model is trained for multiple iterations, the loss function L(θ) of the Q network is minimized by the Adam gradient descent algorithm, and the parameter θ of the Q network is adjusted based on the Bayesian optimization method;
[0064] The loss function of the Q network is expressed as:
[0065]
[0066] Among them, N is the number of experience samples in a small batch, y i is the target Q value, Q(s i ,a i |θ) is the Q value estimated by the Q network, and the target Q value is calculated as follows:
[0067] y t =r t +γ·max a′ Q′(s t+1 ,a′|θ′),
[0068] Among them, s t+1 represents the state information at time t+1, and a′ represents s t+1 The action taken next, r t Represents the reward value for the action, max a′ Q′(s t+1 ,a′|θ′) represents s t+1 The maximum Q value among all possible actions a′;
[0069] When adjusting the parameters θ of the Q network based on the Bayesian optimization method, set θ∈{α,γ}. Bayesian optimization finds the parameter combination θ that makes the training most efficient by sampling and updating the posterior distribution. * , the optimization goal is to minimize the training time T, the formula is as follows:
[0070] θ * =argθminT(θ),
[0071] T(θ) represents the training time under the hyperparameter θ.
[0072] The adaptive model partitioning method and system for distributed training of the present invention have the following advantages:
[0073] 1. This algorithm can be used to achieve intelligent adaptive optimization of distributed model sharding, improve training speed, and reduce resource overhead
[0074] 2. Strengthen deep learning to expand the scope of model optimization exploration and improve the intelligence of model partitioning;
[0075] 3. The reward function adopts a multi-objective reward strategy, which strengthens the adaptive search target of the intelligent agent;
[0076] 4. The Bayesian optimization algorithm improves the learning speed of the intelligent agent and speeds up the overall distributed optimization training. BRIEF DESCRIPTION OF THE DRAWINGS
[0077] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0078] The present invention is further described below in conjunction with the accompanying drawings.
[0079] Figure 1 This is a flowchart of an adaptive model partitioning method applied to distributed training in Example 1. DETAILED DESCRIPTION
[0080] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments may be combined with each other.
[0081] The embodiments of the present invention provide an adaptive model partitioning method and system for distributed training, which are used to solve the technical problems of distributed training and how to combine deep reinforcement learning and Bayesian optimization algorithm to realize distributed training of models.
[0082] Embodiment 1:
[0083] The present invention discloses an adaptive model partitioning method for distributed training. The method selects a sharding strategy for a distributed training model based on deep reinforcement learning, obtains multiple slice models based on the sharding strategy and applies them in parallel to distributed computing nodes for model training. The method includes four steps: feature extraction, construction of a deep reinforcement adaptive model, model and parameter initialization, and iterative training calculation.
[0084] Step S100 feature extraction: For a distributed training model with K layers, define feature types, and encode various types of features through one-hot encoding to obtain feature vectors for each layer.
[0085] In step S100 of this embodiment, the feature definition of each layer of the distributed training model has four categories, namely, operation type, input data volume, calculation volume and output data volume. Among them, the operation type includes the basic components in the distributed training model, including convolution, pooling, full connection and optimizer; the data volume is the amount of data input to the current layer, the calculation volume is the calculation complexity of the current layer, and the output data volume is the amount of data output by the current layer; the feature F of each layer k It is expressed as:
[0086] F k =[Operator k ,InputData k ,Compute k ,Outputdata k ],
[0087] Operator k Indicates the operation type characteristics, InputData k Indicates the data volume characteristics, Compute k Indicates the computational characteristics, Outputdata k Indicates the output data volume characteristics.
[0088] As a specific implementation of feature vector encoding, each feature type is encoded as follows.
[0089] Operation Type k ): There are n types of operations, each type corresponds to an independent one-hot encoding vector. If the operation type of the current layer is the jth type, the jth position in the one-hot vector is 1, and the rest are 0.
[0090] Convolution: 1, 0, 0, 0;
[0091] Pooling: 0, 1, 0, 0;
[0092] Full connection: 0,0,1,0;
[0093] Optimizer: 0,0,0,1.
[0094] InputData k ) and output data volume (Outputdata k ): Assume that the data volume is divided into four intervals: 0-1MB, 1-10MB, 10-100MB, and more than 100MB. Since the amount of task training data received in distributed training is generally large, in order to test the model slicing effect, select some data and divide the data volume into 0-100MB.
[0095] InputData k ):
[0096] 0-1MB:1,0,0,0;
[0097] 1-10MB: 0,1,0,0;
[0098] 10-100MB: 0,0,1,0;
[0099] Above 100MB: 0,0,0,1.
[0100] Output data volume k ):
[0101] 0-1MB:1,0,0,0;
[0102] 1-10MB: 0,1,0,0;
[0103] 10-100MB: 0,0,1,0;
[0104] Above 100MB: 0,0,0,1.
[0105] Compute k ): Divide the model's computational volume into low, medium, relatively high, and high. First, use static analysis tools such as Intel's OpenVINO to measure the model's total computational volume. M Then calculate the base value base, which is calculated as follows:
[0106]
[0107] The formula shows that the benchmark value is the total amount of calculation divided by half of the total number of layers. Because the range of model slices is [1, K]. Therefore, the average number of slices is K / 2. The total amount of calculation divided by the average number of slices is the average amount of calculation of the slices, which is used as the benchmark value to divide the amount of calculation of each layer.
[0108] According to the benchmark value, the calculation amount is divided into:
[0109] Low: <0.5*base;
[0110] Medium: [0.5*base,1.5*base);
[0111] High: [1.5*base,3*base);
[0112] Higher: >= 3*base.
[0113] Similar to the data volume encoding method, this method divides the amount of calculation into several intervals, and each interval is represented by One-Hot encoding.
[0114] Low:1,0,0,0;
[0115] Middle: 0,1,0,0;
[0116] Higher: 0,0,1,0;
[0117] High:0,0,0,1.
[0118] Step S200 builds a deep reinforcement adaptive model: a Q network is built based on a deep neural network, and state information, actions, and reward functions are defined. The state information includes the feature vector of the slicing model and the training time, resource utilization, and inter-node communication overhead when the slicing model is distributedly trained through distributed computing nodes. The action is the slicing strategy taken by the agent under the current state information. The reward function R is used to evaluate the effect of the current slicing strategy. The Q network takes the state information as input and predicts the Q value of each action that the output agent may take.
[0119] In this embodiment, the state information s represents the state and performance of the current distributed training model segmentation.
[0120] The training time t0 represents the total training completion time, which is measured by a computer.
[0121] Resource utilization e0 includes GPU utilization CPU Utilization Memory Utilization and storage I / O utilization Measured by nvidia-smi, top, and iostat tools, the calculation formula is as follows:
[0122]
[0123] The inter-node communication cost c0 includes the communication time Bandwidth Utilization and delay It is mainly measured by the PyTorch Profiler or TensorFlow Profiler tool built into the distributed training framework. The calculation formula is as follows:
[0124]
[0125] Action a is defined as [X1,X2,...,X k ],X k Indicates whether the kth layer is merged with the next layer. For the action [1,0,1,0]: it means that the 1st layer is merged with the 2nd layer, and the 3rd layer is merged with the 4th layer.
[0126] The reward function R is used to evaluate the effect of the current segmentation strategy. The design goal is to improve load balancing, reduce communication overhead, and speed up overall training time. The reward function is:
[0127] r(s,a,s′)=ε(-t)+μe+η(-c);
[0128] The reward function is determined based on the training time t, resource utilization e, and inter-node communication overhead c, and ε, μ, and η are all weight factors.
[0129] For the Q network, the network structure includes an input layer, a hidden layer, and an output layer.
[0130] The input layer receives state information, the hidden layer uses 3 fully connected layers, the dimension is set to 128, the activation function can be ReLU, and the output of the Q network is a length of 2 K-1 A vector represents the Q value of each action combination; K represents the number of layers of the segmented deep learning model.
[0131] For example, there is a 4-layer neural network, the current state s t The slicing model characteristics, computation time, resource utilization, and communication overhead are described. After training, the Q network may output the following Q values:
[0132] Q(s,[1,0,1]|θ)=10
[0133] Q(s,[1,1,0]|θ)=7
[0134] Q(s,[0,1,0]|θ)=8
[0135] The optimal strategy is to choose the action combination with the largest Q value, that is, [1,0,1], which means that the first and second layers are merged, and the third and fourth layers are merged.
[0136] By continuously updating the parameters of the Q network, the DQN model can learn how to perform optimal segmentation in different states. The change in the final Q value gradually decreases and tends to be stable, indicating that the agent's strategy is close to the optimal. The output action a at this time is the optimal segmentation strategy.
[0137] For the first multiple iterations of the deep reinforcement adaptive model, the corresponding Q value Q0 is calculated as follows:
[0138]
[0139] Step S300 model and parameter initialization: Initialize the parameters and experience replay pool of the Q network, and initialize the sharding strategy X0. Apply the slice model obtained based on the sharding strategy X0 in parallel to the distributed computing nodes. Each computing node performs distributed model training on the assigned slice model based on the local training set, and obtains the training time t0, resource utilization e0, and inter-node communication overhead c0 corresponding to the initial distributed model training, and obtains the initial state information s0.
[0140] As a specific implementation of step S300, the initialization sharding strategy is: split the distributed training model layer by layer to obtain the initial distributed training model slice set Y0 = [0,0,...,0], obtain K slice models, and apply the obtained slice models in parallel to the distributed computing nodes. Each computing node performs model distributed training on the received slice models based on the local training set. Each computing node calculates the training time, resource utilization and inter-node communication overhead of executing the current distributed model training, thereby obtaining the initial state information.
[0141] Step S400 iterative training calculation: perform multiple iterative training on the deep reinforcement adaptive model to obtain the final sharding strategy and the parameters of the Q network. During iterative training, according to the current state information, the Q value of each possible action is output through the Q network. The intelligent agent selects the action corresponding to the maximum Q value based on the ∈-greedy strategy to execute, and obtains the next moment state and the reward value of the current action. The current state information, the action corresponding to the current state, the reward value of the action, and the action at the next moment are returned to the experience pool as prior knowledge, and the prior knowledge is extracted from the experience pool for training and updating the parameters of the Q network until the intelligent agent converges to the optimal sharding strategy.
[0142] In step S400 of this embodiment, according to the current state s t , the agent uses the Q network to select a split action a t , the Q network will be based on the current state s tOutput the Q values of all possible actions, and use the ∈-greedy strategy to select the action a corresponding to the maximum Q value t .
[0143] ∈-Greedy Strategy: It is one of the strategies used in reinforcement learning to balance exploration and exploitation. The main idea is: the agent selects the action that is currently considered the best, that is, the action with the largest Q value; exploration: a small part of the time, the agent randomly selects actions to explore new strategies or possible better actions. In this method, the value of ∈ is set to 0.01, and a probability p is randomly generated during execution. If p < ∈, an exploration operation is performed and an action is randomly selected. Otherwise, the action with the largest Q value in the current Q network is selected.
[0144] In this embodiment, by continuously updating the parameters of the Q network, the DQN model can learn how to perform optimal segmentation in different states. The change in the final Q value gradually decreases and tends to be stable, indicating that the strategy of the intelligent agent is close to the optimal. Output the action a at this time t , is the optimal segmentation strategy.
[0145] The Q network update strategy is to use the Adam gradient descent algorithm to minimize the loss function L(θ) and update the parameters θ of the Q network. During the training process, the Bayesian optimization method is used to automatically adjust the hyperparameter configuration.
[0146] The loss function L(θ) of the Q network is expressed as:
[0147]
[0148] Among them, N is the number of experience samples in a small batch, y i is the target Q value, Q(s i ,a i |θ) is the Q value estimated by the Q network, and the target Q value is calculated as follows:
[0149] y t =r t +γ·max a′ Q′(s t+1 ,a′|θ′),
[0150] Among them, s t+1 represents the state information at time t+1, and a′ represents s t+1 The action taken next, r t Represents the reward value for the action, max a′ Q′(s t+1 ,a′|θ′) represents s t+1 The maximum Q value among all possible actions a′.
[0151] The Adam optimizer updates the network parameters based on the first-order moment estimation and the second-order moment estimation. The specific steps are as follows:
[0152] Gradient calculation: For each parameter θ t Calculate the gradient g of the loss function L(θ) with respect to the parameter t , the calculation formula is as follows:
[0153]
[0154] Update the first-order momentum estimate:
[0155] m t =β1·m t-1 +(1-β1)·g t ,
[0156] Update the second-order momentum estimate:
[0157] v t =β2·v t-1 +(1-β2)·g t 2 ,
[0158] Bias correction: Since the momentum estimates m0 and v0 are initialized to zero, these estimates will be biased in the initial stage. Therefore, bias correction is required, and the formula is as follows:
[0159]
[0160] Parameter update: Use the corrected momentum estimate to update the parameters, the formula is as follows:
[0161]
[0162] Among them, α is the learning rate, which is set to 0.001 in this method, β1 is the first-order momentum decay rate, which is set to 0.9 in this method, β2 is the second-order momentum decay rate, which is set to 0.999 in this method, and ∈ is a small constant to prevent division by zero, which is set to 10 in this method. -8 .
[0163] As a specific implementation, the parameter θ of the Q network is adjusted based on the Bayesian optimization method, and θ∈{α,γ} is set. The Bayesian optimization finds the parameter combination θ that makes the training efficiency the highest by sampling and updating the posterior distribution. * , the optimization goal is to minimize the training time T, the formula is as follows:
[0164] θ * =argθmint(θ),
[0165] T(θ) represents the training time under the hyperparameter θ.
[0166] In this embodiment, the optimal hyperparameter combination is predicted by analyzing prior knowledge and current training data, thereby further optimizing the training efficiency and performance of DQN.
[0167] The method of this embodiment outputs the optimal model sharding strategy, divides the distributed training model based on the model sharding strategy, and distributes each slice model to each computing node for distributed training. The intelligent and adaptive model partitioning method of this method can effectively cope with the distributed training challenges of large-scale deep learning models and improve the overall training efficiency.
[0168] The distributed training model of this embodiment is a model built based on a neural network, such as a distributed network model built based on a CNN network. The network model can be used to take a facial image as input and perform expression recognition on the facial image.
[0169] Embodiment 2:
[0170] The present invention discloses an adaptive model partitioning system for distributed training, comprising a feature extraction module, a deep reinforcement adaptive model construction module, a model and parameter initialization module, and an iterative training calculation module.
[0171] The feature extraction module is used to perform the following: for a distributed training model with K layers, define feature types, and encode various types of features through one-hot encoding to obtain feature vectors for each layer.
[0172] As a specific implementation, the feature definition of each layer of the distributed training model has four categories, namely operation type, input data volume, calculation volume and output data volume. Among them, the operation type includes the basic components in the distributed training model, including convolution, pooling, full connection and optimizer; the data volume is the amount of data input to the current layer, the calculation volume is the calculation complexity of the current layer, and the output data volume is the amount of data output by the current layer; the feature F of each layer k It is expressed as:
[0173] F k =[Operator k ,InputData k ,Compute k ,Outputdata k ],
[0174] Operator k Indicates the operation type characteristics, InputData k Indicates the data volume characteristics, Compute k Indicates the computational characteristics, Outputdata k Indicates the output data volume characteristics.
[0175] As a specific implementation of feature vector encoding, each feature type is encoded as follows.
[0176] Operation Type k ): There are n types of operations, each type corresponds to an independent one-hot encoding vector. If the operation type of the current layer is the jth type, the jth position in the one-hot vector is 1, and the rest are 0.
[0177] Convolution: 1, 0, 0, 0;
[0178] Pooling: 0, 1, 0, 0;
[0179] Full connection: 0,0,1,0;
[0180] Optimizer: 0,0,0,1.
[0181] InputData k ) and output data volume (Outputdata k ): Assume that the data volume is divided into four intervals: 0-1MB, 1-10MB, 10-100MB, and more than 100MB. Since the amount of task training data received in distributed training is generally large, in order to test the model slicing effect, select some data and divide the data volume into 0-100MB.
[0182] InputData k ):
[0183] 0-1MB:1,0,0,0;
[0184] 1-10MB: 0,1,0,0;
[0185] 10-100MB: 0,0,1,0;
[0186] Above 100MB: 0,0,0,1.
[0187] Output data volume k ):
[0188] 0-1MB:1,0,0,0;
[0189] 1-10MB: 0,1,0,0;
[0190] 10-100MB: 0,0,1,0;
[0191] Above 100MB: 0,0,0,1.
[0192] Compute k): Divide the model's computational volume into low, medium, relatively high, and high. First, use static analysis tools such as Intel's OpenVINO to measure the model's total computational volume. M Then calculate the base value base, which is calculated as follows:
[0193]
[0194] The formula shows that the benchmark value is the total amount of calculation divided by half of the total number of layers. Because the range of model slices is [1, K]. Therefore, the average number of slices is K / 2. The total amount of calculation divided by the average number of slices is the average amount of calculation of the slices, which is used as the benchmark value to divide the amount of calculation of each layer.
[0195] According to the benchmark value, the calculation amount is divided into:
[0196] Low: <0.5*base;
[0197] Medium: [0.5*base,1.5*base);
[0198] High: [1.5*base,3*base);
[0199] Higher: >= 3*base.
[0200] Similar to the data volume encoding method, this method divides the amount of calculation into several intervals, and each interval is represented by One-Hot encoding.
[0201] Low:1,0,0,0;
[0202] Middle: 0,1,0,0;
[0203] Higher: 0,0,1,0;
[0204] High:0,0,0,1.
[0205] The deep reinforcement adaptive model building module is used to perform the following: build a Q network based on a deep neural network, define state information, actions and reward functions. The state information includes the feature vector of the slicing model and the training time, resource utilization and inter-node communication overhead when the slicing model is distributed trained through distributed computing nodes. The action is the slicing strategy taken by the agent under the current state information. The reward function R is used to evaluate the effect of the current slicing strategy. The Q network takes the state information as input and predicts the Q value of each action that the output agent may take.
[0206] In this embodiment, the state information s represents the state and performance of the current distributed training model segmentation.
[0207] The training time t0 represents the total training completion time, which is measured by a computer.
[0208] Resource utilization e0 includes GPU utilization CPU Utilization Memory Utilization and storage I / O utilization Measured by nvidia-smi, top, and iostat tools, the calculation formula is as follows:
[0209]
[0210] The inter-node communication cost c0 includes the communication time Bandwidth Utilization and delay It is mainly measured by the PyTorch Profiler or TensorFlow Profiler tool built into the distributed training framework. The calculation formula is as follows:
[0211]
[0212] Action a is defined as [X1,X2,...,X k ],X k Indicates whether the kth layer is merged with the next layer. For the action [1,0,1,0]: it means that the 1st layer is merged with the 2nd layer, and the 3rd layer is merged with the 4th layer.
[0213] The reward function R is used to evaluate the effect of the current segmentation strategy. The design goal is to improve load balancing, reduce communication overhead, and speed up overall training time. The reward function is:
[0214] r(s,a,s′)=ε(-t)+μe+η(-c);
[0215] The reward function is determined based on the training time t, resource utilization e, and inter-node communication overhead c, and ε, μ, and η are all weight factors.
[0216] For the Q network, the network structure includes an input layer, a hidden layer, and an output layer.
[0217] The input layer receives state information, the hidden layer uses 3 fully connected layers, the dimension is set to 128, the activation function can be ReLU, and the output of the Q network is a length of 2 K-1 A vector represents the Q value of each action combination; K represents the number of layers of the segmented deep learning model.
[0218] For example, there is a 4-layer neural network, the current state s t The slicing model characteristics, computation time, resource utilization, and communication overhead are described. After training, the Q network may output the following Q values:
[0219] Q(s,[1,0,1]|θ)=10
[0220] Q(s,[1,1,0]|θ)=7
[0221] Q(s,[0,1,0]|θ)=8
[0222] The optimal strategy is to choose the action combination with the largest Q value, that is, [1,0,1], which means that the first and second layers are merged, and the third and fourth layers are merged.
[0223] By continuously updating the parameters of the Q network, the DQN model can learn how to perform optimal segmentation in different states. The change in the final Q value gradually decreases and tends to be stable, indicating that the agent's strategy is close to the optimal. The output action a at this time is the optimal segmentation strategy.
[0224] For the first multiple iterations of the deep reinforcement adaptive model, the corresponding Q value Q0 is calculated as follows:
[0225]
[0226] The model and parameter initialization module is used to perform the following: initialize the parameters and experience replay pool of the Q network, and initialize the sharding strategy X0, apply the slice model obtained based on the sharding strategy X0 in parallel to the distributed computing nodes, and each computing node performs distributed model training on the assigned slice model based on the local training set, and obtains the training time t0, resource utilization e0 and inter-node communication overhead c0 corresponding to the initial distributed model training, and obtains the initial state information s0.
[0227] As a specific implementation, the model and parameter initialization module is used to execute the following to implement the initialization sharding strategy: split the distributed training model layer by layer to obtain the initial distributed training model slice set Y0 = [0,0,...,0], obtain K slice models, and apply the obtained slice models in parallel to the distributed computing nodes. Each computing node performs model distributed training on the received slice model based on the local training set. Each computing node calculates the training time, resource utilization and inter-node communication overhead of executing the current distributed model training, thereby obtaining the initial state information.
[0228] The iterative training calculation module is used to perform the following: perform multiple iterative training on the deep reinforcement adaptive model to obtain the final sharding strategy and the parameters of the Q network. During iterative training, according to the current state information, the Q value of each possible action is output through the Q network. The intelligent agent selects the action corresponding to the maximum Q value based on the ∈-greedy strategy to execute, and obtains the next state and the reward value of the current action. The current state information, the action corresponding to the current state, the reward value of the action, and the action at the next moment are returned to the experience pool as prior knowledge, and the prior knowledge is extracted from the experience pool for training and updating the parameters of the Q network until the intelligent agent converges to the optimal sharding strategy.
[0229] In this embodiment, according to the current state s t , the agent uses the Q network to select a split action a t , the Q network will be based on the current state s t Output the Q values of all possible actions, and use the ∈-greedy strategy to select the action a corresponding to the maximum Q value t .
[0230] ∈-Greedy Strategy: It is one of the strategies used in reinforcement learning to balance exploration and exploitation. The main idea is: the agent selects the action that is currently considered the best, that is, the action with the largest Q value; exploration: a small part of the time, the agent randomly selects actions to explore new strategies or possible better actions. In this method, the value of ∈ is set to 0.01, and a probability p is randomly generated during execution. If p < ∈, an exploration operation is performed and an action is randomly selected. Otherwise, the action with the largest Q value in the current Q network is selected.
[0231] In this embodiment, by continuously updating the parameters of the Q network, the DQN model can learn how to perform optimal segmentation in different states. The change in the final Q value gradually decreases and tends to be stable, indicating that the strategy of the intelligent agent is close to the optimal. Output the action a at this time t , is the optimal segmentation strategy.
[0232] The Q network update strategy is to use the Adam gradient descent algorithm to minimize the loss function L(θ) and update the parameters θ of the Q network. During the training process, the Bayesian optimization method is used to automatically adjust the hyperparameter configuration.
[0233] The loss function L(θ) of the Q network is expressed as:
[0234]
[0235] Among them, N is the number of experience samples in a small batch, y i is the target Q value, Q(s i ,a i |θ) is the Q value estimated by the Q network, and the target Q value is calculated as follows:
[0236] y t =r t +γ·max a′ Q′(s t+1 ,a′|θ′),
[0237] Among them, s t+1 represents the state information at time t+1, and a′ represents s t+1 The action taken next, r t Represents the reward value for the action, max a′ Q′(s t+1 ,a′|θ′) represents s t+1 The maximum Q value among all possible actions a′.
[0238] The Adam optimizer updates the network parameters based on the first-order moment estimation and the second-order moment estimation. The specific steps are as follows:
[0239] Gradient calculation: For each parameter θ t Calculate the gradient g of the loss function L(θ) with respect to the parameter t , the calculation formula is as follows:
[0240]
[0241] Update the first-order momentum estimate:
[0242] m t =β1·m t-1 +(1-β1)·g t ,
[0243] Update the second-order momentum estimate:
[0244] v t =β2·v t-1 +(1-β2)·g t 2 ,
[0245] Bias correction: Since the momentum estimates m0 and v0 are initialized to zero, these estimates will be biased in the initial stage. Therefore, bias correction is required, and the formula is as follows:
[0246]
[0247] Parameter update: Use the corrected momentum estimate to update the parameters, the formula is as follows:
[0248]
[0249] Among them, α is the learning rate, which is set to 0.001 in this method, β1 is the first-order momentum decay rate, which is set to 0.9 in this method, β2 is the second-order momentum decay rate, which is set to 0.999 in this method, and ∈ is a small constant to prevent division by zero, which is set to 10 in this method. -8 .
[0250] As a specific implementation, the parameter θ of the Q network is adjusted based on the Bayesian optimization method, and θ∈{α,γ} is set. The Bayesian optimization finds the parameter combination θ that makes the training efficiency the highest by sampling and updating the posterior distribution. * , the optimization goal is to minimize the training time T, the formula is as follows:
[0251] θ * =argθmint(θ),
[0252] T(θ) represents the training time under the hyperparameter θ.
[0253] In this embodiment, the optimal hyperparameter combination is predicted by analyzing prior knowledge and current training data, thereby further optimizing the training efficiency and performance of DQN.
[0254] The system of this embodiment can execute the method disclosed in Example 1 to implement the sharding strategy of the distributed training model.
[0255] The present invention has been shown and described in detail above through the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above multiple embodiments, those skilled in the art can know that the means in the above different embodiments can be combined to obtain more embodiments of the present invention, and these embodiments are also within the protection scope of the present invention.
Claims
1. An adaptive model partitioning method for distributed training, characterized in that: A sharding strategy for a distributed training model is selected based on deep reinforcement learning, and multiple slice models obtained based on the sharding strategy are applied in parallel to distributed computing nodes for model training. The method comprises the following steps: Feature extraction: For a distributed training model with K layers, define feature types and encode various types of features using one-hot encoding to obtain feature vectors for each layer. Build a deep reinforcement adaptive model: Build a Q network based on a deep neural network, define state information, actions, and reward functions. The state information includes the feature vector of the slicing model and the training time, resource utilization, and inter-node communication overhead when the slicing model is distributedly trained through distributed computing nodes. The action is the slicing strategy taken by the agent under the current state information. The reward function R is used to evaluate the effect of the current slicing strategy. The Q network takes the state information as input and predicts the Q value of each action that the output agent may take. Model and parameter initialization: Initialize the parameters and experience playback pool of the Q network, and initialize the sharding strategy X0. Apply the slice model obtained based on the sharding strategy X0 to the distributed computing nodes in parallel. Each computing node performs distributed model training on the assigned slice model based on the local training set, and obtains the training time t0, resource utilization e0, and inter-node communication overhead c0 corresponding to the initial distributed model training, and obtains the initial state information s0; Iterative training calculation: The deep reinforcement adaptive model is trained iteratively multiple times to obtain the final sharding strategy and the parameters of the Q network. During iterative training, the Q value of each possible action is output through the Q network according to the current state information. The intelligent agent selects the action corresponding to the maximum Q value based on the ∈-greedy strategy to execute, and obtains the next state and the reward value of the current action. The current state information, the action corresponding to the current state, the reward value of the action, and the action at the next moment are returned to the experience pool as prior knowledge. Prior knowledge is extracted from the experience pool for training and updating the parameters of the Q network until the intelligent agent converges to the optimal sharding strategy.
2. The adaptive model partitioning method for distributed training according to claim 1, characterized in that: There are four types of feature definitions for each layer of the distributed training model, namely, operation type, input data volume, computation volume, and output data volume; Among them, the operation types include the basic components in the distributed training model, including convolution, pooling, full connection and optimizer; The data volume is the amount of data input to the current layer, the computational complexity is the computational complexity of the current layer, and the output data volume is the amount of data output by the current layer; The features F of each layer k It is expressed as: F k =[Operator k ,InputData k ,Compute k ,Outputdata k ], Operator k Indicates the operation type characteristics, InputData k Indicates the data volume characteristics, Compute k Indicates the computational characteristics, Outputdata k Indicates the output data volume characteristics.
3. The adaptive model partitioning method for distributed training according to claim 1, characterized in that: Training time indicates the total training completion time, calculated by the computing nodes that perform distributed training; Resource utilization e0 includes GPU utilization CPU Utilization Memory Utilization and storage I / O utilization The calculation formula is as follows: The inter-node communication cost c0 includes the communication time Bandwidth Utilization and delay The calculation formula is as follows:
4. The adaptive model partitioning method for distributed training according to claim 1, characterized in that: The Q value calculation formula is as follows: Q(s,a)=Q(s,a)+α[r+γmaxQ(s ′ ,a ′ )-Q(s,a)]Q(s,a), Among them, s represents the current state information, a represents the action taken in state s, r represents the reward value for the action, α represents the learning rate, which determines the speed of Q value update, γ is the discount factor, which is used to balance the weight of current reward and future reward, s ′ represents the new state after executing action a, maxQ(s ′ ,a ′ ) indicates that in the new state s ′ All possible actions a ′ The maximum Q value in represents the optimal strategy from the new state s ′ The future accumulated rewards that have begun; The calculation formula of the reward value is expressed as: r(s,a,s ′ )=ε(-t)+μe+η(-c), The reward function is determined based on the training time t, resource utilization e, and inter-node communication overhead c, and ε, μ, and η are all weight factors; For the first multiple iterations of the deep reinforcement adaptive model training, the corresponding Q value Q0 calculation formula is as follows:
5. The adaptive model partitioning method for distributed training according to claim 1, characterized in that: For the Q network, when the deep reinforcement adaptive model is trained for multiple iterations, the loss function L(θ) of the Q network is minimized by the Adam gradient descent algorithm, and the parameter θ of the Q network is adjusted based on the Bayesian optimization method; The loss function of the Q network is expressed as: Among them, N is the number of experience samples in a small batch, y i is the target Q value, Q(s i ,a i |θ) is the Q value estimated by the Q network, and the target Q value is calculated as follows: y t =r t +γ·max a′ Q ′ (s t+1 ,a ′ |θ ′ ), Among them, s t+1 Indicates the status information at time t+1, a ′ Indicates t+1 The action taken next, r t Represents the reward value for the action, max a′ Q ′ (s t+1 ,a ′ |θ ′ ) means s t+1 All possible actions a ′ The maximum Q value in ; When adjusting the parameters θ of the Q network based on the Bayesian optimization method, set θ∈{α,γ}. Bayesian optimization finds the parameter combination θ that makes the training most efficient by sampling and updating the posterior distribution. * , the optimization goal is to minimize the training time T, the formula is as follows: i * =argθminT(θ), T(θ) represents the training time under the hyperparameter θ.
6. An adaptive model partitioning system for distributed training, characterized in that: A sharding strategy for a distributed training model is selected by an adaptive model partitioning method for distributed training as described in any one of claims 1 to 5, comprising a feature extraction module, a deep reinforcement adaptive model construction module, a model and parameter initialization module, and an iterative training calculation module; The feature extraction module is used to perform the following: for a distributed training model with K layers, define feature types and encode various types of features by one-hot encoding to obtain feature vectors for each layer; The deep reinforcement adaptive model building module is used to perform the following: build a Q network based on a deep neural network, define state information, actions, and reward functions. The state information includes the feature vector of the slicing model and the training time, resource utilization, and inter-node communication overhead when the slicing model is distributedly trained through distributed computing nodes. The action is the slicing strategy taken by the agent under the current state information. The reward function R is used to evaluate the effect of the current slicing strategy. The Q network takes the state information as input and predicts the Q value of each action that the output agent may take. The model and parameter initialization module is used to perform the following: initialize the parameters and experience playback pool of the Q network, and initialize the sharding strategy X0, apply the slice model obtained based on the sharding strategy X0 in parallel to the distributed computing nodes, each computing node performs distributed model training on the assigned slice model based on the local training set, and obtains the training time t0, resource utilization e0 and inter-node communication overhead c0 corresponding to the initial distributed model training, and obtains the initial state information s0; The iterative training calculation module is used to perform the following: perform multiple iterative training on the deep reinforcement adaptive model to obtain the final sharding strategy and the parameters of the Q network. During iterative training, according to the current state information, the Q value of each possible action is output through the Q network. The intelligent agent selects the action corresponding to the maximum Q value based on the ∈-greedy strategy to execute, and obtains the next state and the reward value of the current action. The current state information, the action corresponding to the current state, the reward value of the action, and the action at the next moment are returned to the experience pool as prior knowledge, and the prior knowledge is extracted from the experience pool for training and updating the parameters of the Q network until the intelligent agent converges to the optimal sharding strategy.
7. The adaptive model partitioning system for distributed training according to claim 6, characterized in that: There are four types of feature definitions for each layer of the distributed training model, namely, operation type, input data volume, computation volume, and output data volume; Among them, the operation types include the basic components in the distributed training model, including convolution, pooling, full connection and optimizer; The data volume is the amount of data input to the current layer, the computational complexity is the computational complexity of the current layer, and the output data volume is the amount of data output by the current layer; The features F of each layer k It is expressed as: F k =[Operator k ,InputData k ,Compute k ,Outputdata k ], Operator k Indicates the operation type characteristics, InputData k Indicates the data volume characteristics, Compute k Indicates the computational characteristics, Outputdata k Indicates the output data volume characteristics.
8. The adaptive model partitioning system for distributed training according to claim 6, characterized in that: Training time indicates the total training completion time, calculated by the computing nodes that perform distributed training; Resource utilization e0 includes GPU utilization CPU Utilization Memory Utilization and storage I / O utilization The calculation formula is as follows: The inter-node communication cost c0 includes the communication time Bandwidth Utilization and delay The calculation formula is as follows:
9. The adaptive model partitioning system for distributed training according to claim 6, characterized in that: The Q value calculation formula is as follows: Q(s,a)=Q(s,a)+α[r+γmaxQ(s ′ ,a ′ )-Q(s,a)]Q(s,a), Among them, s represents the current state information, a represents the action taken in state s, r represents the reward value for the action, α represents the learning rate, which determines the speed of Q value update, γ is the discount factor, which is used to balance the weight of current reward and future reward, s ′ represents the new state after executing action a, maxQ(s ′ ,a ′ ) indicates that in the new state s ′ All possible actions a ′ The maximum Q value in represents the optimal strategy from the new state s ′ The future accumulated rewards that have begun; The calculation formula of the reward value is expressed as: r(s,a,s ′ )=ε(-t)+μe+η(-c), The reward function is determined based on the training time t, resource utilization e, and inter-node communication overhead c, and ε, μ, and η are all weight factors; For the first multiple iterations of the deep reinforcement adaptive model training, the corresponding Q value Q0 calculation formula is as follows:
10. The adaptive model partitioning system for distributed training according to claim 6, characterized in that: For the Q network, when the deep reinforcement adaptive model is trained for multiple iterations, the loss function L(θ) of the Q network is minimized by the Adam gradient descent algorithm, and the parameter θ of the Q network is adjusted based on the Bayesian optimization method; The loss function of the Q network is expressed as: Among them, N is the number of experience samples in a small batch, y i is the target Q value, Q(s i ,a i |θ) is the Q value estimated by the Q network, and the target Q value is calculated as follows: y t =r t +γ·max a′ Q′(s t+1 ,a′|θ′), Among them, s t+1 represents the state information at time t+1, and a′ represents s t+1 The action taken next, r t Represents the reward value for the action, max a′ Q′(s t+1 ,a′|θ′) represents s t+1 The maximum Q value among all possible actions a′; When adjusting the parameters θ of the Q network based on the Bayesian optimization method, set θ∈{α,γ}. Bayesian optimization finds the parameter combination θ that makes the training most efficient by sampling and updating the posterior distribution. * , the optimization goal is to minimize the training time T, the formula is as follows: i * =argθminT(θ), T(θ) represents the training time under the hyperparameter θ.
Citation Information
Cited By
Self-adaptive collaborative optimization electromagnetic signal intelligent model distributed learning training method
CN120256142A
Large model distributed training fault processing method based on dynamic check point strategy
CN120317318A
Adaptive model partitioning method and system applied to distributed training
WO2026045419A1
