Adaptive model partitioning method and system applied to distributed training

Abstract

An adaptive model partitioning method and system applied to distributed training, which method and system belong to the technical field of deep learning, and aim at solving the technical problem of how to implement, in respect of distributed training, distributed model training by means of combining deep reinforcement learning and a Bayesian optimization algorithm. The method comprises the following steps: constructing a Q network on the basis of a deep neural network, and defining state information, actions and a reward function, wherein the state information comprises feature vectors of partitioned models, and training times, resource utilization rates and inter-node communication overheads of when the partitioned models are subjected to distributed training by means of distributed computing nodes, each action is a partitioning strategy used by an agent under the current state information, the reward function R is used for evaluating the effect of the current partitioning strategy, and the Q network uses the state information as input to predict and output a Q value of each action that the agent may take; and performing multiple iterative training on a deep reinforcement adaptive model, so as to obtain a final partitioning strategy and parameters of the Q network.

Description

Adaptive Model Partitioning Method and System for Distributed Training

[0001] This application claims priority to Chinese Patent Application No. 202411202794.X, filed on August 29, 2024, entitled "Adaptive Model Partitioning Method and System for Distributed Training", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This invention relates to the field of deep learning technology, specifically to an adaptive model partitioning method and system for distributed training. Background Technology

[0003] With the increasing complexity and scale of deep learning models, distributed training has become a crucial means to accelerate the training of large-scale models. Currently, distributed training mainly includes data parallelism and model parallelism strategies. Model parallelism, by distributing different parts of the model across different computing nodes, reduces the memory burden on a single node, making it particularly suitable for ultra-large-scale models. Compared to data parallelism, model parallelism can utilize computing resources more effectively and adapt to complex model structures. Therefore, researching model parallelism strategies is essential for improving the efficiency of distributed training. A carefully designed model partitioning strategy is needed to ensure load balancing across each node. Otherwise, if some nodes have too many computational tasks, they will become bottlenecks, exhibiting a "barrel effect," thus affecting the overall training speed. Therefore, how to effectively partition models in a distributed environment to optimize resource utilization and training efficiency is a pressing issue. Traditional model partitioning methods typically rely on fixed strategies or manual adjustments, lacking flexibility and struggling to adapt to changing task requirements and complex environments.

[0004] With the continuous development of deep learning technology, intelligent adaptive deep reinforcement learning has gradually become an important tool for solving complex decision-making problems due to its outstanding performance in adaptive systems. Deep reinforcement learning learns the optimal decision-making strategy through interactive training of intelligent agents in a simulated environment. In addition, Bayesian optimization algorithms excel in automated hyperparameter tuning, efficiently exploring and optimizing the hyperparameter configuration of the model, thereby improving the training performance of distributed slice models.

[0005] Distributed training, and how to combine deep reinforcement learning and Bayesian optimization algorithms to achieve distributed training of models, are technical problems that need to be solved. Summary of the Invention

[0006] The technical objective of this invention is to address the above-mentioned shortcomings by providing an adaptive model partitioning method and system for distributed training, thereby solving the technical problems of how to combine deep reinforcement learning and Bayesian optimization algorithms to achieve distributed training of models.

[0007] In a first aspect, the present invention provides an adaptive model partitioning method for distributed training, which selects a partitioning strategy for the distributed training model based on deep reinforcement learning, and applies multiple slice models obtained based on the partitioning strategy to distributed computing nodes in parallel for model training. The method includes the following steps:

[0008] Feature extraction: For a distributed training model with K layers, define feature types and encode each type of feature using one-hot encoding to obtain the feature vector for each layer;

[0009] Constructing a deep reinforcement adaptive model: Based on a deep neural network, a Q-network is constructed, defining state information, actions, and a reward function. The state information includes the feature vector of the slice model and the training time, resource utilization, and inter-node communication overhead when the slice model is trained in a distributed manner through distributed computing nodes. The action is the slice strategy adopted by the agent under the current state information. The reward function R is used to evaluate the effect of the current slice strategy. The Q-network takes the state information as input and predicts the Q value of each possible action taken by the agent.

[0010] Model and parameter initialization: Initialize the parameters and experience replay pool of the Q network, and initialize the sharding strategy X0. Apply the slice model obtained based on the sharding strategy X0 to the distributed computing nodes in parallel. Each computing node performs distributed model training on the allocated slice model based on the local training set, and obtains the training time t0, resource utilization e0 and inter-node communication overhead c0 corresponding to the first distributed model training, and obtains the initial state information s0.

[0011] Iterative training computation: The deep reinforcement adaptive model is trained iteratively multiple times to obtain the final partitioning policy and the parameters of the Q network. During iterative training, based on the current state information, the Q value of each possible action is output through the Q network. The agent selects the action corresponding to the maximum Q value based on the ∈-greedy policy to execute, and obtains the reward value of the next state and the current action. The current state information, the action corresponding to the current state, the reward value of the action, and the action of the next state are returned to the experience pool as prior knowledge. Prior knowledge is extracted from the experience pool for training and updating the parameters of the Q network until the agent converges to the optimal partitioning policy.

[0012] As a preferred approach, the feature definitions for each layer of the distributed training model fall into four categories: operation type, input data volume, computational cost, and output data volume.

[0013] The operation types include the basic components of a distributed training model, including convolution, pooling, fully connected, and optimizers;

[0014] The input data volume is the amount of data input to the current layer, the computational volume is the computational complexity of the current layer, and the output data volume is the amount of data output by the current layer.

[0015] Features F of each layer k Represented as: F k =[Operator k InputData k Compute k Outputdata k ],

[0016] Operator k Indicates the operation type characteristic, InputData k Indicating data volume characteristics, Compute k Output data represents computational complexity characteristics. k This indicates the characteristics of the output data volume.

[0017] Preferably, the training time represents the total training completion time, which is calculated by the computing nodes performing distributed training;

[0018] Resource utilization e0 includes GPU utilization. CPU utilization Memory utilization and storage I / O utilization The calculation formula is as follows:

[0019] The inter-node communication overhead c0 includes communication time. Bandwidth utilization and delay The calculation formula is as follows:

[0020] As a preferred option, the formula for calculating the Q value is as follows: Q(s,a)=Q(s,a)+α[r+γmaxQ(s′,a′)-Q(s,a)]Q(s,a),

[0021] Where s represents the current state information, a represents the action taken in state s, r represents the reward value for the action, α represents the learning rate, which determines the speed of Q value update, γ is the discount factor, used to balance the weight of current reward and future reward, s′ represents the new state after executing action a, maxQ(s′,a′) represents the maximum Q value among all possible actions a′ in the new state s′, and represents the future cumulative reward starting from the new state s′ under the optimal policy;

[0022] The formula for calculating the reward value is: r(s,a,s′)=ε(-t)+μe+η(-c),

[0023] The reward function is determined based on training time t, resource utilization e, and inter-node communication overhead c, where ε, μ, and η are all weighting factors.

[0024] The formula for calculating the Q-value Q0 during the initial multiple iterations of training of the deep reinforcement adaptive model is as follows:

[0025] As a preferred approach, for the Q-network, when training the deep reinforcement adaptive model multiple times, the loss function L(θ) of the Q-network is minimized by the Adam gradient descent algorithm, and the parameters θ of the Q-network are adjusted based on the Bayesian optimization method.

[0026] The loss function of the Q network is expressed as:

[0027] Where N is the number of empirical samples in a mini-batch, y i Let Q(s) be the target Q value. i ,a i |θ) is the Q-value estimated by the Q-network, and the formula for calculating the target Q-value is as follows:

[0028] Among them, s t+1 This represents the state information at time t+1, and a′ represents s. t+1 The action to be taken, r t This represents the reward value for the action. s t+1 The maximum Q value among all possible actions a′;

[0029] When adjusting the parameters θ of a Q-network using Bayesian optimization, assuming θ∈{α,γ}, Bayesian optimization finds the parameter combination θ that maximizes training efficiency by sampling and updating the posterior distribution. * The optimization objective is to minimize the training time T, as shown in the following formula: θ * =argθminT(θ),

[0030] T(θ) represents the training time under hyperparameter θ.

[0031] In a second aspect, the present invention provides an adaptive model partitioning system for distributed training, used to select a partitioning strategy for a distributed training model through an adaptive model partitioning method for distributed training as described in any of the first aspects, comprising a feature extraction module, a deep reinforcement adaptive model construction module, a model and parameter initialization module, and an iterative training calculation module.

[0032] The feature extraction module performs the following: For a distributed training model with K layers, it defines feature types and encodes each type of feature using one-hot encoding to obtain the feature vector for each layer;

[0033] The deep reinforcement adaptive model building module is used to perform the following: build a Q network based on a deep neural network, define state information, actions, and reward functions. The state information includes the feature vector of the slice model and the training time, resource utilization, and inter-node communication overhead when the slice model is trained in a distributed manner through distributed computing nodes. The action is the slice strategy adopted by the agent under the current state information. The reward function R is used to evaluate the effect of the current slice strategy. The Q network takes the state information as input and predicts the Q value of each possible action taken by the agent.

[0034] The model and parameter initialization module is used to perform the following: initialize the parameters and experience replay pool of the Q network, initialize the sharding strategy X0, apply the slice model obtained based on the sharding strategy X0 in parallel to the distributed computing nodes, each computing node performs distributed model training on the allocated slice model based on the local training set, and obtains the training time t0, resource utilization e0 and inter-node communication overhead c0 corresponding to the first distributed model training, and obtains the initial state information s0.

[0035] The iterative training computation module is used to perform the following: iterative training of the deep reinforcement adaptive model is performed multiple times to obtain the final partitioning policy and the parameters of the Q network. During iterative training, based on the current state information, the Q value of each possible action is output through the Q network. The agent selects the action corresponding to the maximum Q value based on the ε-greedy policy to execute, and obtains the reward value of the next state and the current action. The current state information, the action corresponding to the current state, the reward value of the action, and the action of the next state are returned to the experience pool as prior knowledge. Prior knowledge is extracted from the experience pool for training and updating the parameters of the Q network until the agent converges to the optimal partitioning policy.

[0036] As a preferred approach, the feature definitions for each layer of the distributed training model fall into four categories: operation type, input data volume, computational cost, and output data volume.

[0037] The operation types include the basic components of a distributed training model, including convolution, pooling, fully connected, and optimizers;

[0038] The input data volume is the amount of data input to the current layer, the computational volume is the computational complexity of the current layer, and the output data volume is the amount of data output by the current layer.

[0039] Features F of each layer k Represented as: F k=[Operator k InputData k Compute k Outputdata k ],

[0040] Operator k Indicates the operation type characteristic, InputData k Indicating data volume characteristics, Compute k Output data represents computational complexity characteristics. k This indicates the characteristics of the output data volume.

[0041] Preferably, the training time represents the total training completion time, which is calculated by the computing nodes performing distributed training;

[0042] Resource utilization e0 includes GPU utilization. CPU utilization Memory utilization and storage I / O utilization The calculation formula is as follows:

[0043] The inter-node communication overhead c0 includes communication time. Bandwidth utilization and delay The calculation formula is as follows:

[0044] As a preferred option, the formula for calculating the Q value is as follows: Q(s,a)=Q(s,a)+α[r+γmaxQ(s′,a′)-Q(s,a)]Q(s,a),

[0045] Where s represents the current state information, a represents the action taken in state s, r represents the reward value for the action, α represents the learning rate, which determines the speed of Q value update, γ is the discount factor, used to balance the weight of current reward and future reward, s′ represents the new state after executing action a, maxQ(s′,a′) represents the maximum Q value among all possible actions a′ in the new state s′, and represents the future cumulative reward starting from the new state s′ under the optimal policy;

[0046] The formula for calculating the reward value is: r(s,a,s′)=ε(-t)+μe+η(-c),

[0047] The reward function is determined based on training time t, resource utilization e, and inter-node communication overhead c, where ε, μ, and η are all weighting factors.

[0048] The formula for calculating the Q-value Q0 during the initial multiple iterations of training of the deep reinforcement adaptive model is as follows:

[0049] As a preferred approach, for the Q-network, when training the deep reinforcement adaptive model multiple times, the loss function L(θ) of the Q-network is minimized by the Adam gradient descent algorithm, and the parameters θ of the Q-network are adjusted based on the Bayesian optimization method.

[0050] The loss function of the Q network is expressed as:

[0051] Where N is the number of empirical samples in a mini-batch, y i Let Q(s) be the target Q value. i ,a i |θ) is the Q-value estimated by the Q-network, and the formula for calculating the target Q-value is as follows:

[0052] Among them, s t+1 This represents the state information at time t+1, and a′ represents s. t+1 The action to be taken, r t This represents the reward value for the action. s t+1 The maximum Q value among all possible actions a′;

[0053] When adjusting the parameters θ of a Q-network using Bayesian optimization, assuming θ∈{α,γ}, Bayesian optimization finds the parameter combination θ that maximizes training efficiency by sampling and updating the posterior distribution. * The optimization objective is to minimize the training time T, as shown in the following formula: θ * =argθminT(θ),

[0054] T(θ) represents the training time under hyperparameter θ.

[0055] The adaptive model partitioning method and system for distributed training of the present invention have the following advantages:

[0056] 1. This algorithm can be used to achieve intelligent adaptive optimization of distributed model sharding, thereby improving training speed and reducing resource consumption;

[0057] 2. Enhanced deep learning expands the scope of model optimization and improves the intelligence of model partitioning;

[0058] 3. The reward function adopts a multi-objective reward strategy, which enhances the agent's adaptive search target.

[0059] 4. Bayesian optimization algorithms improve the learning speed of agents and accelerate the overall distributed optimization training speed. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0061] The invention will be further described below with reference to the accompanying drawings.

[0062] Figure 1 is a flowchart of an adaptive model partitioning method applied to distributed training in Example 1. Detailed Implementation

[0063] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments are not intended to limit the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0064] This invention provides an adaptive model partitioning method and system for distributed training, which addresses the technical problem of how to combine deep reinforcement learning and Bayesian optimization algorithms to achieve distributed training of models.

[0065] Example 1:

[0066] This invention provides an adaptive model partitioning method for distributed training. Based on deep reinforcement learning, it selects a partitioning strategy for the distributed training model and applies multiple slice models obtained based on the partitioning strategy to distributed computing nodes in parallel for model training. The method includes four steps: feature extraction, construction of a deep reinforcement adaptive model, model and parameter initialization, and iterative training computation.

[0067] Step S100 Feature Extraction: For a distributed training model with K layers, define the feature types and encode each type of feature using one-hot encoding to obtain the feature vector for each layer.

[0068] In step S100 of this embodiment, the feature definition of each layer of the distributed training model has four categories: operation type, input data volume, computational complexity, and output data volume. The operation type includes the basic components of the distributed training model, including convolution, pooling, fully connected layers, and optimizers; the input data volume is the amount of data input to the current layer; the computational complexity is the computational complexity of the current layer; and the output data volume is the amount of data output by the current layer. The feature F of each layer... k Represented as: F k =[Operator kInputData k Compute k Outputdata k ],

[0069] Operator k Indicates the operation type characteristic, InputData k Indicating data volume characteristics, Compute k Output data represents computational complexity characteristics. k This indicates the characteristics of the output data volume.

[0070] As a specific implementation of feature vector encoding, the encoding method for each feature type is as follows.

[0071] Operation type (Operator) k Let there be n operation types, each corresponding to an independent one-hot encoded vector. If the operation type of the current layer is the j-th type, then the j-th position in the one-hot vector is 1, and the rest are 0.

[0072] Convolution: 1,0,0,0;

[0073] Pooling: 0,1,0,0;

[0074] Fully connected: 0,0,1,0;

[0075] Optimizer: 0,0,0,1.

[0076] Input data volume k ) and output data volume (Outputdata) k Assume the data volume is divided into four intervals: 0-1MB, 1-10MB, 10-100MB, and over 100MB. Since the amount of training data received in distributed training is generally large, a portion of the data is selected and divided into 0-100MB intervals to test the model slicing effect.

[0077] Input data volume k ):

[0078] 0-1MB: 1,0,0,0;

[0079] 1-10MB: 0,1,0,0;

[0080] 10-100MB: 0,0,1,0;

[0081] Above 100MB: 0,0,0,1.

[0082] Output data volumek ):

[0083] 0-1MB: 1,0,0,0;

[0084] 1-10MB: 0,1,0,0;

[0085] 10-100MB: 0,0,1,0;

[0086] Above 100MB: 0,0,0,1.

[0087] Computational complexity k The computational load of the model is categorized into low, medium, relatively high, and high. First, the total computational load (Compute) of the model is measured using static analysis tools such as Intel's OpenVINO. M Then calculate the baseline value as follows:

[0088] This formula means that the baseline value is the total computational cost divided by half the total number of layers. Since the model slice range is [1, K], the average number of slices is K / 2. The total computational cost divided by the average number of slices gives the average computational cost per slice, which is used as the baseline value to divide the computational cost of each layer.

[0089] Based on the baseline value, the computational workload is divided as follows:

[0090] Low: <0.5*base;

[0091] Medium: [0.5*base,1.5*base);

[0092] High: [1.5*base,3*base);

[0093] Higher: >= 3*base.

[0094] Similar to data volume encoding methods, this method divides the computational volume into several intervals, and each interval is represented using One-Hot encoding.

[0095] Low: 1,0,0,0;

[0096] Middle: 0,1,0,0;

[0097] Higher: 0,0,1,0;

[0098] High: 0,0,0,1.

[0099] Step S200: Construct a deep reinforcement adaptive model: Construct a Q-network based on a deep neural network, defining state information, actions, and a reward function. The state information includes the feature vector of the slice model and the training time, resource utilization, and inter-node communication overhead when the slice model is trained in a distributed manner through distributed computing nodes. The action is the slice strategy adopted by the agent under the current state information. The reward function R is used to evaluate the effect of the current slice strategy. The Q-network takes the state information as input and predicts the Q value of each action that the agent may take.

[0100] In this embodiment, the state information s represents the current state and performance of the distributed training model segmentation.

[0101] Training time t0 represents the total training completion time, which is measured by the computer.

[0102] Resource utilization e0 includes GPU utilization. CPU utilization Memory utilization and storage I / O utilization Measured by the nvidia-smi, top, and iostat tools, the calculation formula is as follows:

[0103] The inter-node communication overhead c0 includes communication time. Bandwidth utilization and delay The data is primarily measured using the built-in PyTorch Profiler or TensorFlow Profiler tools in distributed training frameworks, and the calculation formula is as follows:

[0104] Action a is defined as [X1, X2, ..., X... k ],X k This indicates whether the k-th layer merges with the next layer. For the action [1,0,1,0], it means that the 1st and 2nd layers merge, and the 3rd and 4th layers merge.

[0105] The reward function R is used to evaluate the effectiveness of the current segmentation strategy. The design goal is to improve load balancing, reduce communication overhead, and speed up the overall training time. The reward function is: r(s,a,s′)=ε(-t)+μe+η(-c);

[0106] The reward function is determined based on training time t, resource utilization e, and inter-node communication overhead c, with ε, μ, and η being weighting factors.

[0107] For Q-networks, the network structure includes an input layer, hidden layers, and an output layer.

[0108] The input layer receives state information, the hidden layers use three fully connected layers with a dimension of 128, the activation function can be ReLU, and the output of the Q network is a 2^32-bit microcontroller. K-1 The vector represents the Q-value of each action combination; K represents the number of layers in the segmented deep learning model.

[0109] For example, consider a 4-layer neural network with the current state s. t The characteristics of the slice model, computation time, resource utilization, and communication overhead are described. After training, the Q-network may output the following Q-values:

[0110] Q(s,[1,0,1]|θ)=10;

[0111] Q(s,[1,1,0]|θ)=7;

[0112] Q(s,[0,1,0]|θ)=8.

[0113] The optimal strategy is to select the action combination with the largest Q value, i.e., [1,0,1], which means merging the 1st and 2nd layers and merging the 3rd and 4th layers.

[0114] By continuously updating the parameters of the Q-network, the DQN model can learn how to perform optimal segmentation under different states. Eventually, the change in the Q-value gradually decreases and tends to stabilize, indicating that the agent's policy has approached its optimum. The output action 'a' at this point represents the optimal segmentation policy.

[0115] For the first multiple iterations of training of a deep reinforcement adaptive model, the formula for calculating the corresponding Q-value Q0 is as follows:

[0116] Step S300 Model and parameter initialization: Initialize the parameters and experience replay pool of the Q network, and initialize the sharding strategy X0. Apply the slice model obtained based on the sharding strategy X0 in parallel to the distributed computing nodes. Each computing node performs distributed model training on the allocated slice model based on the local training set, and obtains the training time t0, resource utilization e0 and inter-node communication overhead c0 corresponding to the first distributed model training, and obtains the initial state information s0.

[0117] As a specific implementation of step S300, the initial sharding strategy is as follows: the distributed training model is sharded layer by layer to obtain an initial set of shards Y0 = [0,0,...,0] of the distributed training model, resulting in K shard models. The obtained shard models are applied in parallel to the distributed computing nodes. Each computing node performs distributed training on the received shard models based on its local training set. Each computing node calculates the training time, resource utilization, and inter-node communication overhead for executing the current distributed model training, thereby obtaining the initial state information.

[0118] Step S400 Iterative Training Calculation: The deep reinforcement adaptive model is trained iteratively multiple times to obtain the final partitioning policy and the parameters of the Q-network. During iterative training, based on the current state information, the Q-value of each possible action is output through the Q-network. The agent selects the action corresponding to the maximum Q-value based on the ε-greedy policy to execute, and obtains the reward value of the next state and the current action. The current state information, the action corresponding to the current state, the reward value of the action, and the action of the next state are returned to the experience pool as prior knowledge. Prior knowledge is extracted from the experience pool for training and updating the parameters of the Q-network until the agent converges to the optimal partitioning policy.

[0119] In step S400 of this embodiment, based on the current state s t The agent uses a Q-network to select a segmentation action a t The Q network will adjust according to the current state s t Output the Q-values ​​of all possible actions, and use an ε-greedy strategy to select the action 'a' corresponding to the maximum Q-value. t .

[0120] Greedy strategy: This is one of the strategies used in reinforcement learning to balance exploration and exploitation. The main idea is that the agent chooses what it currently considers the optimal action, i.e., the action with the largest Q-value; exploration: for a small portion of the time, the agent randomly selects actions to explore new strategies or potentially better actions. In this method, the value of ε is set to 0.01. During execution, a probability p is randomly generated. If p < ε, an exploration operation is performed, and an action is randomly selected; otherwise, the action with the largest Q-value in the current Q-network is selected.

[0121] In this embodiment, by continuously updating the parameters of the Q-network, the DQN model can learn how to perform optimal segmentation under different states. Eventually, the change in the Q-value gradually decreases and tends to stabilize, indicating that the agent's strategy has approached its optimality. The action 'a' at this point is then output. t , is the optimal segmentation strategy.

[0122] The Q-network update strategy is as follows: The Adam gradient descent algorithm is used to minimize the loss function L(θ), and the parameters θ of the Q-network are updated accordingly. During training, Bayesian optimization is used to automatically adjust the hyperparameter configuration.

[0123] The loss function L(θ) of the Q network is expressed as:

[0124] Where N is the number of empirical samples in a mini-batch, y i Let Q(s) be the target Q value. i ,a i |θ) is the Q-value estimated by the Q-network, and the formula for calculating the target Q-value is as follows:

[0125] Among them, s t+1 This represents the state information at time t+1, and a′ represents s. t+1 The action to be taken, r t This represents the reward value for the action. s t+1 Find the maximum Q value among all possible actions a′.

[0126] The Adam optimizer updates network parameters based on first-order and second-order moment estimates. The specific steps are as follows:

[0127] Gradient calculation: for each parameter θ t Calculate the gradient g of the loss function L(θ) with respect to this parameter. t The calculation formula is as follows:

[0128] Update first-order momentum estimate: m t =β1·m t-1 +(1-β1)·g t ,

[0129] Update the second-order momentum estimate: v t =β2·v t-1 +(1-β2)·g t 2 ,

[0130] Bias Correction: Since the momentum estimates m0 and v0 are initialized to zero initially, these estimates will be biased in the initial stage. Therefore, bias correction is required, and the formula is as follows:

[0131] Parameter update: The parameters are updated using the corrected momentum estimate, as shown in the following formula:

[0132] Where α is the learning rate, set to 0.001 in this method; β1 is the first-order momentum decay rate, set to 0.9 in this method; β2 is the second-order momentum decay rate, set to 0.999 in this method; and ∈ is a small constant to prevent division by zero, set to 10 in this method. -8 .

[0133] In a specific implementation, the parameters θ of the Q-network are adjusted based on the Bayesian optimization method. Let θ∈{α,γ}, Bayesian optimization finds the parameter combination θ that maximizes training efficiency by sampling and updating the posterior distribution. * The optimization objective is to minimize the training time T, as shown in the following formula: θ * =argθmint(θ),

[0134] T(θ) represents the training time under hyperparameter θ.

[0135] In this embodiment, the optimal combination of hyperparameters is predicted by analyzing prior knowledge and current training data, thereby further optimizing the training efficiency and performance of DQN.

[0136] The method in this embodiment outputs an optimal model sharding strategy, which is used to partition the distributed training model and distribute each slice to various computing nodes for distributed training. This intelligent and adaptive model partitioning method effectively addresses the challenges of distributed training of large-scale deep learning models, improving overall training efficiency.

[0137] The distributed training model in this embodiment is a model built on a neural network, such as a distributed network model built on a CNN network. This network model can be used to perform expression recognition on facial images as input.

[0138] Example 2:

[0139] The present invention provides an adaptive model partitioning system for distributed training, comprising a feature extraction module, a deep reinforcement adaptive model construction module, a model and parameter initialization module, and an iterative training computation module.

[0140] The feature extraction module performs the following: For a distributed training model with K layers, it defines feature types and encodes each type of feature using one-hot encoding to obtain the feature vector for each layer.

[0141] In practice, the features of each layer in the distributed training model are defined in four categories: operation type, input data volume, computational complexity, and output data volume. The operation type includes the basic components of the distributed training model, including convolution, pooling, fully connected layers, and optimizers; the input data volume is the amount of data input to the current layer; the computational complexity is the computational complexity of the current layer; and the output data volume is the amount of data output by the current layer. The features F of each layer... k Represented as: F k =[Operator k InputData k Compute k Outputdata k ],

[0142] Operator k Indicates the operation type characteristic, InputData k Indicating data volume characteristics, Compute k Output data represents computational complexity characteristics. k This indicates the characteristics of the output data volume.

[0143] As a specific implementation of feature vector encoding, the encoding method for each feature type is as follows.

[0144] Operation type (Operator) k Let there be n operation types, each corresponding to an independent one-hot encoded vector. If the operation type of the current layer is the j-th type, then the j-th position in the one-hot vector is 1, and the rest are 0.

[0145] Convolution: 1,0,0,0;

[0146] Pooling: 0,1,0,0;

[0147] Fully connected: 0,0,1,0;

[0148] Optimizer: 0,0,0,1.

[0149] Input data volume k ) and output data volume (Outputdata) k Assume the data volume is divided into four intervals: 0-1MB, 1-10MB, 10-100MB, and over 100MB. Since the amount of training data received in distributed training is generally large, a portion of the data is selected and divided into 0-100MB intervals to test the model slicing effect.

[0150] Input data volume k ):

[0151] 0-1MB: 1,0,0,0;

[0152] 1-10MB: 0,1,0,0;

[0153] 10-100MB: 0,0,1,0;

[0154] Above 100MB: 0,0,0,1.

[0155] Output data volume k ):

[0156] 0-1MB: 1,0,0,0;

[0157] 1-10MB: 0,1,0,0;

[0158] 10-100MB: 0,0,1,0;

[0159] Above 100MB: 0,0,0,1.

[0160] Computational complexityk The computational load of the model is categorized into low, medium, relatively high, and high. First, the total computational load (Compute) of the model is measured using static analysis tools such as Intel's OpenVINO. M Then calculate the baseline value as follows:

[0161] This formula means that the baseline value is the total computational cost divided by half the total number of layers. Since the model slice range is [1, K], the average number of slices is K / 2. The total computational cost divided by the average number of slices gives the average computational cost per slice, which is used as the baseline value to divide the computational cost of each layer.

[0162] Based on the baseline value, the computational workload is divided as follows:

[0163] Low: <0.5*base;

[0164] Medium: [0.5*base,1.5*base);

[0165] High: [1.5*base,3*base);

[0166] Higher: >= 3*base.

[0167] Similar to data volume encoding methods, this method divides the computational volume into several intervals, and each interval is represented using One-Hot encoding.

[0168] Low: 1,0,0,0;

[0169] Middle: 0,1,0,0;

[0170] Higher: 0,0,1,0;

[0171] High: 0,0,0,1.

[0172] The deep reinforcement adaptive model building module is used to perform the following: build a Q-network based on a deep neural network, define state information, actions, and reward functions. The state information includes the feature vector of the slice model and the training time, resource utilization, and inter-node communication overhead when the slice model is trained in a distributed manner through distributed computing nodes. The action is the slice strategy adopted by the agent under the current state information. The reward function R is used to evaluate the effect of the current slice strategy. The Q-network takes the state information as input and predicts the Q value of each possible action taken by the agent.

[0173] In this embodiment, the state information s represents the current state and performance of the distributed training model segmentation.

[0174] Training time t0 represents the total training completion time, which is measured by the computer.

[0175] Resource utilization e0 includes GPU utilization. CPU utilization Memory utilization and storage I / O utilization Measured by the nvidia-smi, top, and iostat tools, the calculation formula is as follows:

[0176] The inter-node communication overhead c0 includes communication time. Bandwidth utilization and delay The data is primarily measured using the built-in PyTorch Profiler or TensorFlow Profiler tools in distributed training frameworks, and the calculation formula is as follows:

[0177] Action a is defined as [X1, X2, ..., X... k ],X k This indicates whether the k-th layer merges with the next layer. For the action [1,0,1,0], it means that the 1st and 2nd layers merge, and the 3rd and 4th layers merge.

[0178] The reward function R is used to evaluate the effectiveness of the current segmentation strategy. The design goal is to improve load balancing, reduce communication overhead, and speed up the overall training time. The reward function is: r(s,a,s′)=ε(-t)+μe+η(-c);

[0179] The reward function is determined based on training time t, resource utilization e, and inter-node communication overhead c, with ε, μ, and η being weighting factors.

[0180] For Q-networks, the network structure includes an input layer, hidden layers, and an output layer.

[0181] The input layer receives state information, the hidden layers use three fully connected layers with a dimension of 128, the activation function can be ReLU, and the output of the Q network is a 2^32-bit microcontroller. K-1 The vector represents the Q-value of each action combination; K represents the number of layers in the segmented deep learning model.

[0182] For example, consider a 4-layer neural network with the current state s. t The characteristics of the slice model, computation time, resource utilization, and communication overhead are described. After training, the Q-network may output the following Q-values:

[0183] Q(s,[1,0,1]|θ)=10;

[0184] Q(s,[1,1,0]|θ)=7;

[0185] Q(s,[0,1,0]|θ)=8.

[0186] The optimal strategy is to select the action combination with the largest Q value, i.e., [1,0,1], which means merging the 1st and 2nd layers and merging the 3rd and 4th layers.

[0187] By continuously updating the parameters of the Q-network, the DQN model can learn how to perform optimal segmentation under different states. Eventually, the change in the Q-value gradually decreases and tends to stabilize, indicating that the agent's policy has approached its optimum. The output action 'a' at this point represents the optimal segmentation policy.

[0188] For the first multiple iterations of training of a deep reinforcement adaptive model, the formula for calculating the corresponding Q-value Q0 is as follows:

[0189] The model and parameter initialization module is used to perform the following: initialize the parameters and experience replay pool of the Q network, initialize the sharding strategy X0, apply the slice model obtained based on the sharding strategy X0 in parallel to the distributed computing nodes, each computing node performs distributed model training on the allocated slice model based on the local training set, and obtains the training time t0, resource utilization e0 and inter-node communication overhead c0 corresponding to the first distributed model training, and obtains the initial state information s0.

[0190] In a specific implementation, the model and parameter initialization module executes the following to implement the initial sharding strategy: the distributed training model is sharded layer by layer to obtain an initial set of shards Y0 = [0,0,...,0] of the distributed training model, resulting in K shard models. The obtained shard models are then applied in parallel to the distributed computing nodes. Each computing node performs distributed training on the received shard models based on its local training set. Each computing node calculates the training time, resource utilization, and inter-node communication overhead for executing the current distributed model training, thereby obtaining the initial state information.

[0191] The iterative training computation module is used to perform the following: iterative training of the deep reinforcement adaptive model is performed multiple times to obtain the final partitioning policy and the parameters of the Q network. During iterative training, based on the current state information, the Q value of each possible action is output through the Q network. The agent selects the action corresponding to the maximum Q value based on the ∈-greedy policy to execute, and obtains the reward value of the next state and the current action. The current state information, the action corresponding to the current state, the reward value of the action, and the action of the next state are returned to the experience pool as prior knowledge. Prior knowledge is extracted from the experience pool for training and updating the parameters of the Q network until the agent converges to the optimal partitioning policy.

[0192] In this embodiment, based on the current state s tThe agent uses a Q-network to select a segmentation action a t The Q network will adjust according to the current state s t Output the Q-values ​​of all possible actions, and use an ε-greedy strategy to select the action 'a' corresponding to the maximum Q-value. t .

[0193] The ε-greedy strategy is one of the strategies used in reinforcement learning to balance exploration and exploitation. The main idea is that the agent chooses what it currently considers the optimal action, i.e., the action with the largest Q-value. Exploration: For a small portion of the time, the agent randomly selects actions to explore new strategies or potentially better actions. In this method, the value of ε is set to 0.01. During execution, a probability p is randomly generated. If p < ε, an exploration operation is performed, and an action is randomly selected; otherwise, the action with the largest Q-value in the current Q-network is selected.

[0194] In this embodiment, by continuously updating the parameters of the Q-network, the DQN model can learn how to perform optimal segmentation under different states. Eventually, the change in the Q-value gradually decreases and tends to stabilize, indicating that the agent's strategy has approached its optimality. The action 'a' at this point is then output. t , is the optimal segmentation strategy.

[0195] The Q-network update strategy is as follows: The Adam gradient descent algorithm is used to minimize the loss function L(θ), and the parameters θ of the Q-network are updated accordingly. During training, Bayesian optimization is used to automatically adjust the hyperparameter configuration.

[0196] The loss function L(θ) of the Q network is expressed as:

[0197] Where N is the number of empirical samples in a mini-batch, y i Let Q(s) be the target Q value. i ,a i |θ) is the Q-value estimated by the Q-network, and the formula for calculating the target Q-value is as follows:

[0198] Among them, s t+1 This represents the state information at time t+1, and a′ represents s. t+1 The action to be taken, r t This represents the reward value for the action. s t+1 Find the maximum Q value among all possible actions a′.

[0199] The Adam optimizer updates network parameters based on first-order and second-order moment estimates. The specific steps are as follows:

[0200] Gradient calculation: for each parameter θ t Calculate the gradient g of the loss function L(θ) with respect to this parameter.t The calculation formula is as follows:

[0201] Update first-order momentum estimate: m t =β1·m t-1 +(1-β1)·g t ,

[0202] Update the second-order momentum estimate: v t =β2·v t-1 +(1-β2)·g t 2 ,

[0203] Bias Correction: Since the momentum estimates m0 and v0 are initialized to zero initially, these estimates will be biased in the initial stage. Therefore, bias correction is required, and the formula is as follows:

[0204] Parameter update: The parameters are updated using the corrected momentum estimate, as shown in the following formula:

[0205] Where α is the learning rate, set to 0.001 in this method; β1 is the first-order momentum decay rate, set to 0.9 in this method; β2 is the second-order momentum decay rate, set to 0.999 in this method; and ∈ is a small constant to prevent division by zero, set to 10 in this method. -8 .

[0206] In a specific implementation, the parameters θ of the Q-network are adjusted based on the Bayesian optimization method. Let θ∈{α,γ}, Bayesian optimization finds the parameter combination θ that maximizes training efficiency by sampling and updating the posterior distribution. * The optimization objective is to minimize the training time T, as shown in the following formula: θ * =argθmint(θ),

[0207] T(θ) represents the training time under hyperparameter θ.

[0208] In this embodiment, the optimal combination of hyperparameters is predicted by analyzing prior knowledge and current training data, thereby further optimizing the training efficiency and performance of DQN.

[0209] The system in this embodiment can execute the method disclosed in Embodiment 1 to implement the sharding strategy of the distributed training model.

[0210] The present invention has been shown and described in detail above with reference to the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above embodiments, those skilled in the art will know that more embodiments of the present invention can be obtained by combining the means in the different embodiments described above, and these embodiments are also within the protection scope of the present invention.

Claims

1. An adaptive model partitioning method applied to distributed training, characterized in that, The method for selecting a sharding strategy for a distributed training model based on deep reinforcement learning, and applying multiple slice models obtained based on the sharding strategy in parallel to distributed computing nodes for model training, includes the following steps: Feature extraction: For a distributed training model with K layers, define feature types and encode each type of feature using one-hot encoding to obtain the feature vector for each layer; Constructing a deep reinforcement adaptive model: Based on a deep neural network, a Q-network is constructed, defining state information, actions, and a reward function. The state information includes the feature vector of the slice model and the training time, resource utilization, and inter-node communication overhead when the slice model is trained in a distributed manner through distributed computing nodes. The action is the slice strategy adopted by the agent under the current state information. The reward function R is used to evaluate the effect of the current slice strategy. The Q-network takes the state information as input and predicts the Q value of each possible action taken by the agent. Model and parameter initialization: Initialize the parameters and experience replay pool of the Q network, and initialize the sharding strategy X0. Apply the slice model obtained based on the sharding strategy X0 to the distributed computing nodes in parallel. Each computing node performs distributed model training on the allocated slice model based on the local training set, and obtains the training time t0, resource utilization e0 and inter-node communication overhead c0 corresponding to the first distributed model training, and obtains the initial state information s0. Iterative training computation: The deep reinforcement adaptive model is trained iteratively multiple times to obtain the final partitioning policy and the parameters of the Q network. During iterative training, based on the current state information, the Q value of each possible action is output through the Q network. The agent selects the action corresponding to the maximum Q value based on the ε-greedy policy to execute, and obtains the reward value of the next state and the current action. The current state information, the action corresponding to the current state, the reward value of the action, and the action of the next state are returned to the experience pool as prior knowledge. Prior knowledge is extracted from the experience pool for training and updating the parameters of the Q network until the agent converges to the optimal partitioning policy.

2. The adaptive model partitioning method for distributed training according to claim 1, characterized in that, The features of each layer in the distributed training model are defined in four categories: operation type, input data volume, computational cost, and output data volume. The operation types include the basic components of a distributed training model, including convolution, pooling, fully connected, and optimizers; The input data volume is the amount of data input to the current layer, the computational volume is the computational complexity of the current layer, and the output data volume is the amount of data output by the current layer. Features F of each layer k Represented as: F k =[Operator k ,InputData k ,Compute k ,Outputdata k ], Operator k Indicates the operation type characteristic, InputData k Indicating data volume characteristics, Compute k Output data represents computational complexity characteristics. k This indicates the characteristics of the output data volume.

3. The adaptive model partitioning method for distributed training according to claim 1, characterized in that, Training time represents the total training completion time, calculated by the computing nodes performing distributed training; Resource utilization e0 includes GPU utilization. CPU utilization Memory utilization and storage I / O utilization The calculation formula is as follows: The inter-node communication overhead c0 includes communication time. Bandwidth utilization and delay The calculation formula is as follows:

4. The adaptive model partitioning method for distributed training according to claim 1, characterized in that, The formula for calculating the Q value is as follows: Q(s,a)=Q(s,a)+α[r+γmaxQ(s′,a′)-Q(s,a)]Q(s,a), Where s represents the current state information, a represents the action taken in state s, r represents the reward value for the action, α represents the learning rate, which determines the speed of Q value update, γ is the discount factor, used to balance the weight of current reward and future reward, s′ represents the new state after executing action a, maxQ(s′,a′) represents the maximum Q value among all possible actions a′ in the new state s′, and represents the future cumulative reward starting from the new state s′ under the optimal policy; The formula for calculating the reward value is as follows: r(s,a,s′)=ε(-t)+μe+η(-c), The reward function is determined based on training time t, resource utilization e, and inter-node communication overhead c, with ε, μ, and η being weighting factors. The formula for calculating the Q-value Q0 during the initial multiple iterations of training of the deep reinforcement adaptive model is as follows:

5. The adaptive model partitioning method for distributed training according to claim 1, characterized in that, For the Q network, when training the deep reinforcement adaptive model multiple times, the loss function L(θ) of the Q network is minimized by the Adam gradient descent algorithm, and the parameters θ of the Q network are adjusted based on the Bayesian optimization method. The loss function of the Q network is expressed as: Where N is the number of empirical samples in a mini-batch, y i Let Q(s) be the target Q value. i ,a i |θ) is the Q-value estimated by the Q-network, and the formula for calculating the target Q-value is as follows: Among them, s t+1 This represents the state information at time t+1, and a′ represents s. t+1 The action to be taken, r t This represents the reward value for the action. s t+1 The maximum Q value among all possible actions a′; When adjusting the parameters θ of a Q-network using Bayesian optimization, assuming θ∈{α,γ}, Bayesian optimization finds the parameter combination θ that maximizes training efficiency by sampling and updating the posterior distribution. * The optimization objective is to minimize the training time T, as shown in the following formula: i * =argθminT(θ), T(θ) represents the training time under hyperparameter θ.

6. An adaptive model partitioning system for distributed training, characterized in that, The sharding strategy for selecting a distributed training model using an adaptive model partitioning method for distributed training as described in any one of claims 1-5 includes a feature extraction module, a deep reinforcement adaptive model construction module, a model and parameter initialization module, and an iterative training computation module. The feature extraction module performs the following: For a distributed training model with K layers, it defines feature types and encodes each type of feature using one-hot encoding to obtain the feature vector for each layer; The deep reinforcement adaptive model building module is used to perform the following: build a Q network based on a deep neural network, define state information, actions, and reward functions. The state information includes the feature vector of the slice model and the training time, resource utilization, and inter-node communication overhead when the slice model is trained in a distributed manner through distributed computing nodes. The action is the slice strategy adopted by the agent under the current state information. The reward function R is used to evaluate the effect of the current slice strategy. The Q network takes the state information as input and predicts the Q value of each possible action taken by the agent. The model and parameter initialization module is used to perform the following: initialize the parameters and experience replay pool of the Q network, initialize the sharding strategy X0, apply the slice model obtained based on the sharding strategy X0 in parallel to the distributed computing nodes, each computing node performs distributed model training on the allocated slice model based on the local training set, and obtains the training time t0, resource utilization e0 and inter-node communication overhead c0 corresponding to the first distributed model training, and obtains the initial state information s0. The iterative training computation module is used to perform the following: iterative training of the deep reinforcement adaptive model is performed multiple times to obtain the final partitioning policy and the parameters of the Q network. During iterative training, based on the current state information, the Q value of each possible action is output through the Q network. The agent selects the action corresponding to the maximum Q value based on the ε-greedy policy to execute, and obtains the reward value of the next state and the current action. The current state information, the action corresponding to the current state, the reward value of the action, and the action of the next state are returned to the experience pool as prior knowledge. Prior knowledge is extracted from the experience pool for training and updating the parameters of the Q network until the agent converges to the optimal partitioning policy.

7. The adaptive model partitioning system for distributed training according to claim 6, characterized in that, The features of each layer in the distributed training model are defined in four categories: operation type, input data volume, computational cost, and output data volume. The operation types include the basic components of a distributed training model, including convolution, pooling, fully connected, and optimizers; The input data volume is the amount of data input to the current layer, the computational volume is the computational complexity of the current layer, and the output data volume is the amount of data output by the current layer. Features F of each layer k Represented as: F k =[Operator k ,InputData k ,Compute k ,Outputdata k ], Operator k Indicates the operation type characteristic, InputData k Indicating data volume characteristics, Compute k Output data represents computational complexity characteristics. k This indicates the characteristics of the output data volume.

8. The adaptive model partitioning system for distributed training according to claim 6, characterized in that, Training time represents the total training completion time, calculated by the computing nodes performing distributed training; Resource utilization e0 includes GPU utilization. CPU utilization Memory utilization and storage I / O utilization The calculation formula is as follows: The inter-node communication overhead c0 includes communication time. Bandwidth utilization and delay The calculation formula is as follows:

9. The adaptive model partitioning system for distributed training according to claim 6, characterized in that, The formula for calculating the Q value is as follows: Q(s,a)=Q(s,a)+α[r+γmaxQ(s′,a′)-Q(s,a)]Q(s,a), Where s represents the current state information, a represents the action taken in state s, r represents the reward value for the action, α represents the learning rate, which determines the speed of Q value update, γ is the discount factor, used to balance the weight of current reward and future reward, s′ represents the new state after executing action a, maxQ(s′,a′) represents the maximum Q value among all possible actions a′ in the new state s′, and represents the future cumulative reward starting from the new state s′ under the optimal policy; The formula for calculating the reward value is as follows: r(s,a,s′)=ε(-t)+μe+η(-c), The reward function is determined based on training time t, resource utilization e, and inter-node communication overhead c, with ε, μ, and η being weighting factors. The formula for calculating the Q-value Q0 during the initial multiple iterations of training of the deep reinforcement adaptive model is as follows:

10. The adaptive model partitioning system for distributed training according to claim 6, characterized in that, For the Q network, when training the deep reinforcement adaptive model multiple times, the loss function L(θ) of the Q network is minimized by the Adam gradient descent algorithm, and the parameters θ of the Q network are adjusted based on the Bayesian optimization method. The loss function of the Q network is expressed as: Where N is the number of empirical samples in a mini-batch, y i Let Q(s) be the target Q value. i ,a i |θ) is the Q-value estimated by the Q-network, and the formula for calculating the target Q-value is as follows: Among them, s t+1 This represents the state information at time t+1, and a′ represents s. t+1 The action to be taken, r t This represents the reward value for the action. s t+1 The maximum Q value among all possible actions a′; When adjusting the parameters θ of a Q-network using Bayesian optimization, assuming θ∈{α,γ}, Bayesian optimization finds the parameter combination θ that maximizes training efficiency by sampling and updating the posterior distribution. * The optimization objective is to minimize the training time T, as shown in the following formula: i * =argθminT(θ), T(θ) represents the training time under hyperparameter θ.

Images