A neural network adaptive distributed parallel training method based on reinforcement learning
Through the adaptive distributed parallel training method of neural networks based on reinforcement learning, a multi-dimensional performance evaluation model and a two-layer policy network are built, which solves the problem of relying on expert experience and communication overhead in the existing technology, and realizes efficient parallelism Strategy search and resource scheduling improve the performance and stability of large-scale neural network training.
Patent Information
- Application Number
- CN202110406478.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-04-15
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2041-04-15
AI Technical Summary
The existing technology model parallel training method relies on expert experience in large-scale neural network training, has low cluster utilization rate, large communication overhead, and is difficult to adapt to dynamic environments and different network types, so it cannot obtain universal promotion.
Adaptive distributed parallel training method for neural networks based on reinforcement learning is adopted, and by building a multi-dimensional performance evaluation model and a two-layer policy network, operator adaptive grouping and resource scheduling are realized, search space is reduced, and the comprehensive performance of parallel strategies is improved.
The comprehensive performance of parallel strategies is improved, and the complete end-to-end search of model parallel strategies is realized without human intervention, which speeds up strategy execution and performance evaluation, and improves algorithm stationarity and convergence rate.
Smart Images

Figure CN113128702B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a neural network adaptive distributed parallel training method based on reinforcement learning, which provides an optimal model parallel training scheme for large-scale complex neural networks. Background Art
[0002] In recent years, benefiting from the development of AI algorithms, hardware computing power and data sets, deep neural network technology has been widely used in natural language processing, computer vision and search recommendation. As these fields continue to iterate and develop larger and more complex neural networks, "Moore's Law" has been unable to match computing needs, and a single device can no longer support large-scale deep network training. Therefore, studying the segmentation of neural network calculation graphs and scheduling the segmented networks to clusters with multiple CPUs and GPUs to achieve model parallel training has become a common method to solve large-scale neural network training.
[0003] The industry usually divides neural network computational graphs horizontally by layers, vertically across layers, or randomly and schedules them to different devices for execution. However, these methods rely heavily on expert experience, have unreasonable segmentation methods, low cluster utilization, and usually have large communication overhead.
[0004] The heuristic method based on "graph search" is one of the methods to guide model parallel training. F. Pellegrini et al. proposed the Scotch static graph segmentation method, which guides the static graph segmentation by calculating the cost of each node and the amount of data flowing through each edge, balancing the workload to minimize the communication cost, but this method cannot be directly applied to dynamic environments. Jia et al. proposed the FlexFlow framework, which uses a directed graph to describe all DNN calculations and states, and automatically searches for effective strategies in the parallel space of four dimensions: operators, samples, parameters, and features; Zhang et al. proposed AutoSync for the problem of data synchronization and parallelism, randomly searching the synchronization parallel space to obtain the best synchronization and parallel strategy; Peng et al. proposed the Optimus framework to establish a performance model for the PS framework, accurately estimate the speed of machine learning training, and minimize the job training time by building an online resource performance model. Although these methods have good effects in specific scenarios, they are limited by network types (such as FlexFlow is only applicable to DNN) and scenarios (such as AutoSync is only applicable to data synchronization and parallelism, and only for parameter server architecture), and it is difficult to obtain universal promotion.
[0005] Another method is to use machine learning to guide model segmentation and resource scheduling. Parallex developed by Kim et al. adjusts the variable size by learning linear models to achieve adaptive tensor partitioning; Peter et al. use Bayesian optimization to determine the credit size of resource scheduling; Uber team develops Horovod to use Bayesian to optimize the adjustable part. However, the parallel dimension of the above methods is relatively single, the main tuning model is simple, and these methods focus on optimizing the running time of distributed operation strategies, and cannot guarantee other performance of the strategy except running time.
[0006] Therefore, the present invention studies the above defects, designs and implements a neural network adaptive distributed parallel training method based on reinforcement learning. Summary of the invention
[0007] The present invention designs and implements a neural network adaptive distributed parallel training method based on reinforcement learning. By constructing a multidimensional performance evaluation model that can fully reflect the distributed training performance, the comprehensive performance of the parallel strategy is improved; the operator adaptive grouping is realized according to the operator attribute characteristics, and the parallelism is determined, so as to reduce the search space and achieve the purpose of adaptive training; the proximal policy gradient method is used to iteratively optimize the reinforcement learning model, expand the offline learning ability of the policy network, and improve the algorithm stability, convergence rate and policy search performance.
[0008] A neural network adaptive distributed parallel training method based on reinforcement learning, the steps are as follows:
[0009] Step 1: Construct a multidimensional performance evaluation model R(π g ,π s ), measure the comprehensive performance of distributed parallel strategies. First, analyze the factors that affect the execution performance of neural networks, including neural network model structure, computing properties, and cluster topology; then extract the computing cost E i , communication cost C i and memory usage M i Finally, a multi-dimensional performance evaluation model that can fully reflect the distributed training performance is constructed through the following two steps:
[0010] 1) By calculating the cost E i and the communication cost C i The complex linear relationship f predicts the single iteration running time r of the distributed parallel strategy t =f(E i ,C i );
[0011] 2) Establish a multidimensional performance evaluation model R = -log(αr t +βr c +γr m);where r t , r c and r m They represent the single iteration running time, peak communication cost and peak memory occupancy respectively. α, β and γ represent weight hyperparameters respectively. The weight hyperparameters are obtained by manual tuning based on expert experience to obtain the best weight combination.
[0012] Step 2: Construct a distributed parallel strategy search space, determine the grouping and parallelism, and generate a complete model parallel strategy.
[0013] First, according to the structural characteristics of the neural network model and the cluster topology, the computational graph is abstracted. and cluster device resource groups Construct a distributed parallel strategy search space. Computational graph In the figure, vertex O represents a neural network operator, and E represents a directed edge connecting vertex O.
[0014] Secondly, extract the model and operator attributes and generate the operator feature vector. Extract the relevant attributes of all operators, including operator type, input and output, and adjacent node index, perform feature encoding on the operators based on the above attributes, and generate operator feature vectors.
[0015] Again, build a segmentation strategy network Group operators, determine parallelism, and generate segmentation strategy π g , and then generate group feature codes according to the operator grouping. The segmentation strategy network uses a feedforward neural network with SoftMax to combine operator feature vectors and cluster device resource groups. As a segmentation strategy network The coarse-grained grouping of operators is realized by inputting , and the parallelism is determined. Among them, operators within the same group are not suitable for parallel execution, while operators between groups are suitable for parallel execution. After completing the grouping, the feature vectors of operators within the group are selected to generate the group feature coding sequence.
[0016] Finally, build the scheduling strategy network Determine the scheduling strategy π for different groups on the device s The scheduling strategy network is a Seq2Seq double recurrent neural network with an attention mechanism and long short-term memory, which uses an encoder and a decoder to process input and output variable-length sequences respectively. Split the strategy network The obtained group feature code is used as an input sequence to generate a device sequence. The generated device sequence and the input group feature code sequence are in a one-to-one correspondence order, that is, all operators in the first group will be scheduled to execute on the first device in the output device sequence, and so on. This correspondence is the scheduling strategy π s. Splitting strategy π g and scheduling strategy π s Together they form a complete model parallel strategy (π g ,π s ).like Figure 1 As shown on the left, the reinforcement learning agent Agent contains the above segmentation strategy network and scheduling strategy network Two-layer policy network.
[0017] Step 3: Based on importance sampling, the proximal policy gradient is used to iteratively optimize the reinforcement learning model to search for the optimal segmentation and scheduling strategy.
[0018] First of all, the core idea of importance sampling is to separate the sampling and optimization model objects, and completely copy the segmentation strategy network and the scheduling strategy network. Instances Instances Responsible for continuous sampling and segmentation scheduling strategies, Responsible for the continuous iteration and update of model parameters, and using KL divergence to capture the difference between the two models. When the difference between the two models is too large, synchronize the replica parameters once
[0019] Secondly, the complete model parallel strategy obtained in step 2 is input into the execution simulator ( Figure 1 The execution simulator on the right shows the multi-dimensional collection strategy execution performance and calculates the cost E i and the communication cost C i The complex linear relationship f predicts the single running time r of the distributed parallel strategy t =f(E i ,C i ), communication overhead, memory usage and computing load. On this basis, the multi-dimensional performance evaluation model R(π g ,π s ) is used to guide the calculation of the reinforcement learning reward value R.
[0020] Finally, the reward-guided splitting strategy network and the scheduling strategy network are used Iterative optimization. According to the reward value R, the proximal policy gradient method is used to continuously iterate and collaboratively optimize the segmentation strategy network and the scheduling strategy network. Parameters, incentives The policy network generates a model parallel strategy with better overall performance in the next round of training. Finally, the optimal parallel strategy is generated and executed in a real distributed environment, such as Figure 1 Shown on the right.
[0021] The beneficial effects of the present invention are as follows: the establishment of a multidimensional performance evaluation model improves the comprehensive performance of the searched parallel strategy; the establishment of a split and scheduled two-layer strategy network realizes a complete end-to-end search of the model parallel strategy without human intervention; the use of an execution simulator to take over the real distributed execution environment accelerates the strategy execution and performance evaluation speed and improves sampling efficiency; the use of a distributed proximal strategy optimization algorithm based on importance sampling improves the original policy gradient algorithm, expands the offline learning ability of the strategy network, and improves the algorithm stability and convergence rate. At the same time, the multidimensional performance evaluation model and execution simulator in the present invention can be extended to other adaptive search scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 This is a schematic diagram of searching for the optimal parallel strategy based on reinforcement learning;
[0023] Figure 2 This is a joint architecture diagram of the segmentation network and the policy network. DETAILED DESCRIPTION
[0024] The present invention will be further described below with reference to the accompanying drawings and specific implementation steps:
[0025] like Figure 1 As shown, a neural network adaptive distributed parallel training method based on reinforcement learning includes the following steps:
[0026] Step 1: Construct a multidimensional performance evaluation model R(π g ,π s ), measure the comprehensive performance of the strategy. First, analyze the factors that affect the execution performance of the neural network, including the neural network model structure, computing properties, and cluster topology; then extract the computational cost E i , communication cost C i and memory usage M i Equal performance factors, where the computational cost E i , communication cost C i and memory usage M i The definition is as follows:
[0027] The computational cost is expressed by dividing the tensor precision involved in the operation by the device computation density. It can measure the computational load of the device during training, denoted by T j is the j-th dimension depth of the tensor, is the tensor precision calculation, c i Calculate the density for the i-th device, and the calculation cost is defined as follows:
[0028]
[0029] The communication cost is expressed by dividing the communication volume by the communication bandwidth, which can measure the communication load of the device training process. Indicates the size of the tensor, b i represents the communication bandwidth between devices, and the communication cost is expressed as follows:
[0030]
[0031] Memory usage, memory usage M d Use the physical storage size of the model parameters in the current device i memory and the total device memory m i The ratio of memory usage is expressed as follows:
[0032]
[0033] Through the above definition, the present invention will model the scheduling strategy performance evaluation from three dimensions: running time, peak memory and peak communication cost. A multi-dimensional performance evaluation model based on reinforcement learning is established, and R is used as the reinforcement learning reward value for continuous iteration and improvement. The comprehensive performance of the partitioning and scheduling strategy network is used to adaptively find the optimal distributed parallel partitioning and scheduling strategy (π g ,π s ). The multidimensional performance evaluation model is modeled as follows:
[0034] R(π g ,π s )=-log(αr t +βr c +γr m ) (4)
[0035] The larger the reward value R, the better the searched strategy. It is worth noting that the natural logarithm operation in the above formula can also be replaced by the square root operation, where α, β and γ represent the weight hyper parameters, and the optimal weight combination is obtained by manual tuning. t , r c and r m They represent three performance factors: running time, peak communication load, and peak memory usage. The definition of each performance factor is as follows:
[0036]
[0037] where r t It represents the running time of a single iteration of the strategy in a simulated or real environment, and uses the simulator to indirectly calculate the cost E i and the communication cost C i The nonlinear relationship between i ,C i ) prediction, R cThe peak communication cost of multiple devices is represented by max{C1,C2,...,C k}Calculated, where C i The communication cost corresponding to hardware device i, Indicates the maximum amount of communication allowed for a user. If the maximum amount of communication is exceeded, a corresponding penalty will be imposed. m Indicates the peak memory usage of multiple devices, through max{M1,M2,...,M k} calculated, where M i is the memory usage, Indicates the maximum memory usage that the user can tolerate.
[0038] Step 2: If Figure 2 As shown, a distributed parallel strategy search space is constructed, grouping and parallelism are determined, and a model parallel strategy is generated.
[0039] First, according to the structural characteristics of the neural network model and the cluster topology, the computational graph is abstracted. and cluster device resource groups Constructing a distributed parallel strategy search space. Abstract computational graph and cluster device resource groups The vertex O represents the neural network operator, denoted as O = {o1, o2, ..., o N O is a directed edge connecting vertex P, indicating the communication data dependency between computing nodes, denoted as E = {e 11 ,e 12 ,…,e ij}. For cluster device resource groups Abstract the CPU and GPU devices in the cluster into a series of resource groups Now assume that the cluster has M available resources, then the available resource group Can be abstracted into
[0040] Secondly, extract the model and operator attributes and generate the operator feature vector. All operator attributes in the graph, including operator type (type), input and output (output shape) and adjacent node index (adj), feature encode the operator attributes and generate an operator feature vector. The encoding method is as follows: extract the three attributes of the operator in the calculation graph as the operator attribute feature vector, (1) Operator type (type): for example, operator type definitions such as convolution (Conv2D), maximum pooling (MaxPool) or matrix multiplication (MalMul). The present invention adopts a natural language processing method to collect 200 commonly used operator vocabularies in the TensorFlow API, establish a vocabulary, use the vocabulary as a natural language processing task, and obtain a word vector of size 20 as the embedded feature of the operator type part; (2) Operator output tensor size (output shape): Assume that the current operator is o i , get the computation graph vertex edges {e i1 ,e i2 ,…,e ij} as the list of output tensors of the operator. In order to unify the number of output tensors, the present invention unifies the number of output tensors to 6, and specifies the maximum dimension of the output tensor to be 4. If it is insufficient, the vector is filled with -1. For example, the existing convolution operator outputs a four-dimensional tensor with a shape of (2,2,1,64), then its size is: 256=2×2×1×64, and the remaining output tensor positions are filled with -1; (3) Adjacent operator index (adj): Traverse all operators according to topological sorting to obtain the operator index, limit the in-degree and out-degree size of each operator to 6, and fill the index set with the parent node operator and child node operator of the operator, and still use -1 to fill the in-degree and out-degree less than 6.
[0041] Again, build a segmentation strategy network (like Figure 2 The first layer architecture is shown in Figure 1) to group operators, determine the degree of parallelism, and generate a segmentation strategy π g , and then generate group feature codes according to the operator grouping. It is a feed-forward neural network with two hidden layers of size 64 and 128 respectively, and a 30% dropout layer is introduced between the hidden layers to prevent overfitting. A feedforward neural network with SoftMax is used to coarsely group the operators in the computation graph. As the input of the segmentation strategy network, coarse-grained grouping is generated and the degree of parallelism is determined. Operators within the same group are not suitable for parallel execution, while operators between groups are suitable for parallel execution. The operator feature codes in the group are combined into a group feature code output, which includes three parts: (1) the average of all operator type codes in the group is used as the first part of the group code; (2) the average of all operator output tensor sizes in the group is used as the second part of the group code. Note that there is no -1 value in this part of the code; (3) the one-hot encoding of the group, which indicates the connection between groups. The length of this part of the vector is the number of groups (for example, if the operator is divided into 256 groups, the vector length is 256). If an operator in the group is connected to an operator in the i-th group, the i-th position of the vector is set to 1, otherwise it is set to 0.
[0042] Finally, build the scheduling strategy network (like Figure 2 The second layer architecture is shown in Figure 2), which determines the scheduling strategy π for devices in different groups. s The scheduling network is a Seq2Seq double recurrent neural network with an attention mechanism and long short-term memory. It uses an encoder and a decoder to process the input and output variable-length sequences respectively. (1) The group feature encoding obtained in step 2 is used as the scheduling strategy network The encoder is used to process the variable length sequence of input packets, reading one packet g at a time. i The group feature encoding vector is generated and k hidden states are generated, where k is a hyperparameter equal to the number of groups. (2) The decoder is used to process the variable length sequence of the output device, and each prediction obtains a scheduling device d j , the resulting device sequence and input group sequence have a one-to-one correspondence order, that is, all operators in the first group will be scheduled to the first device output by the decoder, and so on. It is worth noting that each device has its own trainable feature encoding, and the feature encoding of the previous device will be input to the next decoder. Scheduling Strategy Network π s The attention mechanism is used to pay attention to the encoder state. During the training process, the decoder will sample the device d from the SoftMax layer at step t. t In order to make the scheduling strategy network activation function u t To make it flatter, the present invention introduces SoftMax temperature and logit clipping, and uses temperature T and tanh constant C to express the scheduling strategy network activation function u t , that is, sampling is done in the following way:
[0043] d t ~softmax(Ctanh(u t / T)) (6)
[0044] The device sequence output by the decoder is the scheduling strategy π corresponding to the input group s, segmentation strategy π g and scheduling strategy π s Parallel strategies that together form a complete model (π g ,π s ).
[0045] Step 3: Based on importance sampling, the proximal policy gradient is used to iteratively optimize the reinforcement learning model to search for the optimal segmentation and scheduling strategy.
[0046] First of all, the core idea of importance sampling is to separate the sampling and optimization model objects, and completely copy the segmentation strategy network and the scheduling strategy network. Instances Instances Responsible for continuously sampling the segmentation scheduling strategy and generating segmentation scheduling strategy samples for subsequent iterative optimization of the reinforcement learning algorithm; It is only responsible for the continuous iteration and update of model parameters. The proximal policy gradient method uses KL divergence to capture the difference between the two models. If the model difference is too large, the replica parameters will be synchronized once. Otherwise, the old model copy will continue to be used to complete the sampling.
[0047] Secondly, the complete model parallel strategy obtained in step 2 is input into the execution simulator to collect the execution performance of various aspects of the strategy and calculate the cost E i and the communication cost C i The complex linear relationship f predicts the single-step running time r of the distributed parallel strategy t =f(E i ,C i On this basis, the multi-dimensional performance evaluation model R(π g ,π s ) is used to guide the calculation of the reinforcement learning reward value R.
[0048] The execution simulator is designed as follows: the first execution of the strategy will be performed in a real distributed environment to collect the running performance of a given model on all devices. After that, the execution simulator will take over the real distributed environment and model the computational cost as E i , model the communication cost as C i ,The present invention assumes that the communication cost is linearly related to the size of the ,communication data flow between devices.,Each device uses two threads to maintain the following two ,FIFO queues. Indicates the operator running queue on device d, recording the order in which operators are about to run; Represents the tensor queue that will be transferred from device d to other devices, recording the set of tensors that will be communicated to other devices.
[0049] The execution simulator follows the following three principles during the design process: (1) Operation o can only start running on device d when all dependent parent nodes have been executed and all dependent tensors have been obtained; (2) The computation and communication processes can overlap with each other, that is, it has a pipeline execution mechanism; (3) Operators on the same device are executed in serial mode. The simulator in this invention uses the following three trigger mechanisms to generate a time pipeline:
[0050] Trigger mechanism 1: trigger execution. Here, the current operator o i Installing k Taking the above execution as an example, the complete trigger execution process will complete the following process: 1) Complete the current operator o i In device d k Execute the above process to obtain all child nodes o that depend on this node j 2) Get o j Device where the node is located l , if d j ≠d i , then the operation o i Output tensor t i Enqueue If d j =d i , check o j Does it comply with the operating principles? If so, add it to the team. 3) If Idle, it will trigger the device d k Idle state; if not idle, select the next queue The operator execution is triggered.
[0051]
[0052] Trigger mechanism 2: trigger communication. Here, tensor t i From device d k Transfer to d j For example, the complete trigger communication process will execute the following process: 1) Complete the current tensor t i From device d k Transfer to d j 2) Check the dependence on t i Tensor operator o k Does it comply with the operating principles? If so, add it to the team. 3) Judgment Is it empty? If it is empty, the device d k The communication trigger status is idle. If not, select The next transfer tensor triggers the communication process.
[0053]
[0054]
[0055] Trigger mechanism three: trigger state. First determine and Is it empty? If it is empty, it triggers the execution idle or communication idle state. If it is not empty, it dequeues to realize the next operator trigger execution or tensor trigger transmission.
[0056]
[0057] Finally, after completing the execution simulation and obtaining the reward value R, the segmentation strategy network and the scheduling strategy network are guided according to the reward value R. Iterate the optimization and output the optimal parallel strategy to execute in a real distributed environment. In the selection of optimization algorithm, the present invention adopts the proximal strategy optimization method to coordinately optimize the segmentation strategy network and the scheduling strategy network. In essence, we hope to maximize the optimization of the following cost function
[0058]
[0059]
[0060] Let θ g and θ s are the parameters of the segmentation strategy network and the scheduling strategy network respectively. is the model copy parameter, and θ g and θ s The joint is denoted as θ, and The joint is denoted as θ k p(g,s;θ) represents the segmentation and scheduling strategy network The joint probability distribution of the strategies, q(g,s;θ k ) indicates a model copy The joint probability distribution obtained by sampling, R is the reward value, and b is the average moving baseline.
[0061] The splitting and policy network parameters of the above formula can also be written as follows:
[0062]
[0063] in
[0064]
[0065]
[0066] where p(g;θ g ) represents the segmentation strategy network The probability distribution of the generated strategy, p(s|g; η s ) represents the scheduling strategy network Generate conditional probability distribution of scheduling policies; This means that the split strategy network copy The probability distribution of the sampling and segmentation strategy, Represents the scheduling strategy network copy Conditional probability distribution of the sampling scheduling strategy; η is the KL divergence weight hyperparameter. b is the exponential moving average reward baseline. The introduction of the baseline can effectively improve the sampling efficiency. If N is the hyperparameter representing the period, then the exponential moving average reward baseline EMA N (b n ) is calculated as follows:
[0067]
[0068] Cost Function The derivation process is as follows:
[0069]
[0070] The present invention uses Adam to complete the gradient descent process, and the proximal strategy optimization method introduces KL divergence to avoid θ and θ k The large gap leads to low sampling efficiency.
Claims
1. A neural network adaptive distributed parallel training method based on reinforcement learning, It is characterized in that The method comprises the following steps: (1) Analyze the impact of neural network model structure and computing properties on neural network execution performance, extract performance factors including computing cost, communication cost, and memory usage, and build a multidimensional performance evaluation model that can fully reflect the performance of distributed training; The computational cost refers to the ratio of the tensor precision involved in the operation to the device computation density, the communication cost refers to the ratio of the communication volume to the communication bandwidth, and the memory usage refers to the ratio of the current device model parameter physical storage size to the device total memory; and the performance factor also includes video memory and throughput; (2) Based on the structural characteristics of the neural network model and the cluster topology, a distributed parallel strategy search space is constructed; the model operator attributes are extracted, including operator type, input and output, and adjacent node index, and all operators of the model are adaptively grouped using a feedforward network according to the operator attributes to determine the parallelism of the deep network model. Operators in the same operator group are not suitable for parallel execution, while operators in different operator groups are suitable for parallel execution. (3) Based on importance sampling, we collect reinforcement learning training samples, iteratively optimize the reinforcement learning model through the proximal policy gradient method, and find the optimal distributed parallel strategy; The multidimensional performance evaluation model in step (1) is established through the following two steps: (1) By calculating the cost E i and the communication cost C i The complex linear relationship f predicts the single-step running time r of the distributed parallel strategy t =f(E i ,C i ); (2) Establish a multidimensional performance evaluation model R = -log(ar t +βr c +γr m );where; r t , r c and r m They represent the single-step running time, peak communication cost and peak memory usage respectively. α, β and γ represent weight hyperparameters respectively. The best weight combination is obtained based on experience adjustment. Determining the parallelism of the deep network model in step (2) refers to: feature encoding the operator attributes, calculating the reward value based on the multidimensional performance evaluation model, using reinforcement learning to guide the feedforward neural network containing SoftMax to complete the operator grouping, and grouping the operators that are interdependent and whose communication cost exceeds the set value into one group, thereby determining the parallelism of the deep network model.
2. According to claim 1, a neural network adaptive distributed parallel training method based on reinforcement learning, Features: The proximal policy gradient method described in step (3) refers to: separating the sampling and optimization model objects, using KL divergence to capture the difference between the two models, realizing offline iterative updates of the reinforcement learning algorithm, and introducing an exponential moving average reward baseline b to reduce the convergence variance.
3. According to claim 1, a neural network adaptive distributed parallel training method based on reinforcement learning, Features: The complex linear relationship f is constructed in the following way: each device is maintained and executed by two threads and communication The execution process of the two FIFO queue construction strategy follows the following principles: (1) Operator o can only be executed on the current device when all its dependent parent nodes have been executed and all its dependent tensors have been obtained; (2) the computation and communication processes can overlap with each other; (3) operators on the same device are executed in serial mode.
Citation Information
Patent Citations
Autonomous underwater vehicle trajectory tracking control method based on deep reinforcement learning
CN108803321A
Intelligent task scheduling strategy training method based on strategy gradient reinforcement learning
CN109857534A
Learning-based low-delay task scheduling method in edge computing network
CN109976909A
Traffic signal control method based on deep reinforcement learning
CN112216124A
