Heterogeneous computing resource configuration optimization method for deep reinforcement learning model training

Abstract

The application discloses a heterogeneous computing resource configuration optimization method for deep reinforcement learning model training, constructs a heterogeneous graph to describe the characteristics and data flow relationship of the core components of the model-based Muzero series algorithm, maps the characteristics of nodes and edges to the same hidden space based on a multi-head attention node-edge encoder, predicts the algorithm training time under different configurations by using the summarized global characteristics, and then optimizes the configuration by using the simulated annealing algorithm according to the predicted training time output by the model. CPU resources, GPU resources and memory resources are allocated according to the optimized configuration to perform a training task. According to the training result, a resource balance factor is calculated to evaluate the resource configuration scheme and continuously monitor and optimize. The method realizes the best balance between performance and resource consumption in the model training process, significantly improves the efficiency and effect of model training, and reduces the technical difficulty of resource configuration optimization in a complex scene.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer science, relates to deep learning and reinforcement learning, and in particular to a heterogeneous computing resource configuration optimization method for deep reinforcement learning model training. BACKGROUND

[0002] Since the advent of AlphaGo, deep reinforcement learning technology has made significant progress in computational efficiency and resource optimization. The model-based Muzero series of algorithms has achieved an efficient training process with lower hardware requirements, significantly improving sample efficiency, representing an important development direction of the technology.

[0003] The core architecture of the model-based Muzero series of algorithms includes Replay Buffer, Self-Play Worker, Representation Network, Prediction Network, Dynamics Network, Training Loop, and multiple key components connected through complex graph structures to realize data and control flow interaction.

[0004] Among them, Replay Buffer is responsible for storing the data transitions generated during the training process, the key hyperparameter is batch_size, and the main data indicators are the number of transitions and the buffer length. Self-Play Worker executes the self-play task to generate new training data, the key hyperparameter is the number of Monte Carlo tree search parallel mcts_simulations, and the computational cost is the number of self-play runs. Representation Network is used to convert the current observation state into a hidden state representation, the key hyperparameters include network depth and hidden layer unit number, and the computational cost is related to the forward propagation calculation amount. Prediction Network predicts the strategy and value based on the hidden state, the key hyperparameters include the structure and size of the strategy head and value head, and the computational cost includes the calculation complexity of the strategy and value. Dynamics Network predicts the next hidden state and reward based on the current hidden state and action, the key hyperparameters are the network structure of state and reward prediction, and the computational cost is related to state transition and reward calculation. Training Loop uses the data in Replay Buffer for model training, involving parameter updates of representation network, prediction network and dynamics network, the key hyperparameters include training batch size train_batch_size and learning rate, and the computational cost includes the number of iterations of backpropagation and gradient descent.

[0005] Although the model-based MuZero series algorithm has made significant progress in computational efficiency and sample efficiency, it still faces challenges in managing computing resources. For example, the core components of the above-mentioned algorithm have extremely high demands for computing and memory resources. High-parallel Monte Carlo tree search requires a large amount of CPU resources, neural network computation requires a large amount of GPU resources, and the generation, storage and processing of large-scale training data increase the burden of memory and storage devices. These high-intensity computing and memory requirements directly affect the scalability and practical application range of the algorithm, and also increase the complexity and management difficulty of the system. In addition, the performance and resource utilization efficiency of the model-based MuZero series algorithm are highly dependent on the precise adjustment of a series of key hyperparameters. These parameters are not only closely related to the specific hardware configuration and problem characteristics, but also interact with each other. The complex interaction between multiple components causes the resource demand to change dynamically during the training process, further increasing the difficulty of optimization. Finding the optimal configuration usually requires a large number of experiments, which is itself a resource-intensive task, forming a paradox that needs to be weighed.

[0006] Therefore, there is an urgent need for a method that can improve resource utilization efficiency and energy efficiency while maintaining algorithm performance, to achieve unified optimization of computational efficiency, sample efficiency and resource efficiency. SUMMARY

[0007] In view of the deficiencies of the prior art, the present application proposes a heterogeneous computing resource configuration optimization method for deep reinforcement learning model training, which dynamically allocates GPU and CPU resources of the model-based Muzero series algorithm according to the needs of different training stages, improves resource utilization efficiency and energy efficiency while maintaining algorithm performance.

[0008] The heterogeneous computing resource configuration optimization method for deep reinforcement learning model training specifically includes the following steps:

[0009] Step 1, running time prediction

[0010] Step 1.1, construct a prediction model, use the components in the model-based Muzero series algorithm as nodes, and use the interaction relationship between components as edges, use a heterogeneous graph to describe the relationship between components, where, is the node set, ε is the edge set, denotes the node feature matrix, denotes the edge feature matrix, n is the total number of nodes, d n is the dimension of the node feature, m is the total number of edges, d eis the dimension of edge features. The types of the nodes include Replay Buffer, Self-Play Worker, Neural Network and Training Loop.

[0011] Step 1.2, map the features of nodes and edges to the same hidden space:

[0012]

[0013] where, and are the feature transformation functions of nodes and edges respectively, h is the dimension of the hidden layer. represents the feature vector of node i; represents the feature vector of edge j, 1≤i≤n, 1≤j≤m. h i and h j represent the transformed node and edge features respectively.

[0014] Update the node features using the multi-head attention mechanism:

[0015]

[0016] where, is the set of adjacent nodes of node i, h i ′ represents the updated node feature, h k represents the adjacent node feature of node i.

[0017] Step 1.3, globally aggregate the features of all nodes and edges to obtain the global node feature g N and the global edge feature g E :

[0018]

[0019] According to the global features, predict the running time of the model-based Muzero series algorithm

[0020]

[0021] where, W1 and W2 are weight matrices, b is a bias vector. Calculate the mean square error (MSE) according to the predicted running time and the actual running time, and optimize the parameters of the prediction model.

[0022] Step 2, calculate resource allocation optimization

[0023] Step 2.1: Set initial configuration parameters, including batch buffer size (batch_buffer_size), CPU behavior (cpu_actor), storage size (mcts_storage_size), and loop count (loop_count). Execute baseline tests and record the training time under the initial configuration.

[0024] Step 2.2: Input the initial configuration parameters into the prediction model trained in Step 1, and predict the training time under the current configuration. Use the historical data from the baseline test in Step 2.1 to validate and calibrate the prediction model.

[0025] Step 2.3: Optimize the resource allocation scheme of the model-based Muzero series algorithms using simulated annealing, specifically including the following steps:

[0026] Step 2.3.1, Parameter Initialization:

[0027] The state in the simulated annealing algorithm is set to the configuration parameters of the model-based Muzero series algorithm, using the initial configuration parameters in step 2.1 as the initial state S0. Temperature is set as an adjustment parameter for the probability of accepting suboptimal solutions, with the initial temperature set to T0. Energy is set as the prediction training time for the prediction model output.

[0028] Step 2.3.2, State Transition:

[0029] Within the neighborhood of the current state, randomly select a new state S according to either a uniform distribution or a normal distribution. k Record the state transition probability P k Use a predictive model to predict the energy E corresponding to the new state. k k is the number of iterations.

[0030] Step 2.3.3, Optimization Iteration:

[0031] Compare the energy E before and after the state update. k-1 E k The size of E k <E k-1 Then update the state to S. k Update the temperature to T k =T0·α k Then return to step 2.3.2 for the next iteration, where α is the cooling factor. Otherwise, compare the state transition probabilities P. k and The size of P k If the value is greater than P, then update the state to S. k Update the temperature to T kand return to step 2.3.2 for the next iteration, otherwise, keep the state S k-1 , update the temperature to T k and return to step 2.3.2 for the next iteration. When the preset temperature or the preset number of iterations is reached, stop the iteration and output the configuration parameters S current corresponding to the current state.

[0032] Based on the comparison result of the state transition probability and the random probability P, the state is updated, which can maintain the balance between exploration and utilization.

[0033] Step 2.4, based on the configuration parameters S current and historical running data H obtained in step 2.3, dynamically allocate CPU, GPU and memory resources, wherein the historical data H includes resource utilization and training time under past configurations:

[0034] CPU resource allocation:

[0035]

[0036] wherein, CPU allocated represents the amount of resources dynamically allocated to the CPU; CPU min is the minimum CPU resource limit allowed by the system to ensure basic running requirements; CPU max is the maximum CPU resource limit allowed by the system to prevent excessive resource allocation; f CPU () is a function to predict CPU demand, which is calculated according to the current configuration parameters S current , temperature T and historical data H. Through the constraint conditions max() and min(), it is ensured that the allocated CPU resources are between the minimum and maximum limits, thereby improving the running efficiency and performance of the system.

[0037] GPU resource allocation:

[0038]

[0039] wherein, GPU allocated represents the number of GPUs allocated, GPU min is the minimum required GPU number, which is determined by the algorithm complexity and data size; GPU max is the maximum available GPU number, representing the upper limit of resources to avoid resource waste. f GPU () is a GPU resource prediction function applied to the current configuration parameters S current , temperature T and historical running data H.

[0040] Memory allocation:

[0041]

[0042] where f Memory () is a function that predicts the memory requirement, Memory allocated represents the amount of memory resources allocated. Memory min represents the minimum limit of system available memory, ensuring that the system will not crash or performance degradation due to insufficient memory during operation. And Memory max is the maximum memory limit supported by the system, to avoid unstable operation due to exceeding system capacity.

[0043] Step 2.5, resource balancing and adjustment

[0044] According to the results of step 2.4, the resources are allocated to complete the training task. According to the training results, the resource balancing factor β is calculated:

[0045]

[0046] When β>1, it means that the resources are over-allocated, at this time, CPU and memory resources are preferentially reduced; when β<1, it means that the resources are insufficient, at this time, GPU resources are preferentially increased.

[0047] Compared with the prior art, the present application has the following beneficial effects:

[0048] 1. Accurate performance prediction: By capturing the complex interaction between components of the model-based Muzero series algorithm through a heterogeneous graph neural network, the running time under different configurations can be accurately predicted. This accurate prediction provides a reliable basis for decision-making in the simulated annealing process, significantly reducing invalid searches.

[0049] 2. Dynamic adaptability: Combining real-time prediction of the prediction model and the temperature regulation mechanism of simulated annealing, the configuration can be dynamically adjusted according to the performance changes during training. This adaptability enables the system to automatically adjust resource allocation at different training stages of the model-based Muzero series algorithm, such as increasing computing resources in the early exploration stage and appropriately reducing resources in the late convergence stage.

[0050] 3. Efficient exploration of global optimal solution: The randomness of the simulated annealing algorithm combined with the guidance of the prediction model enables the system to more effectively explore the global optimal solution in the high-dimensional configuration space of MuZero. Compared with traditional grid search or random search, it can converge to high-quality solutions faster to find optimal solutions comparable to exhaustive search. This feature not only improves search efficiency, but also discovers some non-intuitive but excellent performance configuration combinations.

[0051] 4、The method not only significantly improves the training efficiency and resource utilization of the model-based Muzero series algorithm, but also provides the possibility for the application of the algorithm in large-scale complex problems. By reducing the training time and optimizing resource usage, it helps to reduce the deployment cost of the model-based Muzero series algorithm and accelerate its landing process in practical applications. Moreover, the method has good generality and is easy to extend to other complex reinforcement learning algorithm configuration optimization. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 Flow chart of the heterogeneous computing resource configuration optimization method for deep reinforcement learning model training.

[0053] Figure 2 Algorithm training time prediction flow chart. DETAILED DESCRIPTION

[0054] The application will be further explained in conjunction with the accompanying drawings; taking the EfficientZero algorithm, one of the model-based Muzero series algorithms, as an example, the specific process of the heterogeneous computing resource configuration optimization method for deep reinforcement learning model training is introduced, and the EfficientZero algorithm realizes the interaction of data and control flow through the following key components:

[0055] Replay Buffer: responsible for storing data transitions generated during training. Key hyperparameters include batch_size, and main data indicators include number of transitions and buffer length.

[0056] Reanalyze Worker CPU / GPU: uses CPU or GPU to perform data reanalysis. The key hyperparameter is the number of worker threads, and the calculation cost is mainly related to the replay buffer full count.

[0057] Selfplay Worker: performs self-play tasks to generate new training data. Key hyperparameters include environment parallel number p_mcts_num, and calculation cost includes self-play acceleration count.

[0058] The heterogeneous computing resource configuration optimization method for deep reinforcement learning model training configures resources for the EfficientZero algorithm, as shown in Figure 1 The specific steps are as follows:

[0059] Step 1, running time prediction

[0060] Step 1.1. Construct a prediction model with components in the EfficientZero algorithm as nodes and interaction relationships between components as edges, using a heterogeneous graph to describe the relationship between components, where, is a set of nodes, and ε is a set of edges, represents a node feature matrix, represents an edge feature matrix, n is the total number of nodes, d n is the dimension of the node feature, m is the total number of edges, d e is the dimension of the edge feature.

[0061] The node features include node type h t , hyperparameters h p , data size h d , and computing cost h c . Among them, the node type h t is used to describe the functional category of the node; the hyperparameters h p are used to describe the specific configuration of each node operation, such as batch size and the number of workers, etc. The data size h d is used to describe the amount of data processed or generated by the node. The computing cost h c is used to describe the consumption of computing resources when the node performs its function.

[0062] The edge features include the type and data volume of the edge. The type of the edge defines the data flow between nodes, and the data volume is used to represent the data density and traffic between different nodes.

[0063] Step 1.2. In order to deeply understand the dependency between nodes, a node-edge encoder based on the multi-head attention mechanism is introduced, a node feature transformation function and an edge feature transformation function map the features of nodes and edges to the same hidden space:

[0064]

[0065] where h is the dimension of the hidden layer. represents the feature vector of node i; represents the feature vector of edge j, 1≤i≤n, 1≤j≤m. h i and h j represent the transformed node and edge features, respectively.

[0066] The multi-head attention mechanism MultiHead() is used to refine the attention distribution of node i to its adjacent nodes, and update the feature representation of the node:

[0067]

[0068] where, is the set of neighboring nodes of node i, h i denotes the updated node feature, h j denotes the neighboring node feature of node i.

[0069] Step 1.3, globally aggregate the features of all nodes and edges to obtain global node features g N and global edge features g E :

[0070]

[0071] Predict the running time of the model-based Muzero series algorithm according to the global features

[0072]

[0073] where W1 and W2 are weight matrices, and b is a bias vector.

[0074] Use the mean square error (MSE) of the predicted running time and the actual running time as the loss function Optimize the parameters of the prediction model:

[0075]

[0076] where y i denotes the actual running time of node i.

[0077] Step 2, computing resource configuration optimization

[0078] Based on the prediction results of the prediction model, combined with the simulated annealing algorithm, the configuration optimization of the EfficientZero algorithm is carried out, aiming to meet the performance requirements and minimize the consumption of computing resources. The specific steps are as follows:

[0079] Step 2.1, set the initial configuration parameters, including batch buffer size (batch_buffer_size), CPU behavior (cpu_actor), storage size (mcts_storage_size), and loop count (loop_count), perform baseline testing, and test the results in detail in the base.Log4j file, which is used for preliminary analysis of system performance bottlenecks and optimization starting point.

[0080] Step 2.2, input the initial configuration parameters to the prediction model trained in step 1 to predict the training time under the current configuration. Use the historical data of the baseline test in step 2.1 to verify and calibrate the prediction model.

[0081] Step 2.3. Optimize the resource configuration scheme of the EfficientZero algorithm using the simulated annealing algorithm, using the initial configuration parameters in step 2.1 as the initial state S0 of the simulated annealing algorithm. Set the temperature as an adjustment parameter for the probability of accepting suboptimal solutions, and set a higher initial temperature T0 = 1000 to ensure that the algorithm can explore a wider configuration space in the early exploration stage and avoid falling into a local optimal solution too early. Set the energy as the predicted training time output by the prediction model. When the preset temperature 1 or the preset number of iterations is reached, stop iteration and output the configuration parameters S corresponding to the current state current .

[0082] Step 2.4. Based on the configuration parameters S current and historical running data H, dynamically allocate CPU, GPU and memory resources.

[0083] Through the above optimization process, the resource configuration of the EfficientZero algorithm can be effectively adjusted and optimized, minimizing the consumption of computing resources while ensuring performance. This method combines the prediction ability of graph neural networks and the global search characteristics of the simulated annealing algorithm, providing an efficient configuration optimization strategy for complex deep learning models.

Claims

1. A method for optimizing the allocation of heterogeneous computing resources for training deep reinforcement learning models, characterized by: Specifically, the following steps are included: Step 1: Runtime Prediction Construct a prediction model using components from the model-based Muzero series algorithms as nodes and the interactions between components as edges, employing a heterogeneous graph. To describe the relationships between components, the features of nodes and edges are mapped to the same latent space. A multi-head attention mechanism is used to refine the attention distribution of nodes to their neighboring nodes and update node features. Finally, the features of all nodes and edges are globally summarized, and the running time of the model-based Muzero series algorithm is predicted based on the global features. Optimize the parameters of the prediction model based on the predicted runtime and the actual runtime; Step 2: Optimize computing resource allocation Set the configuration parameters for the model-based Muzero series algorithms, perform baseline tests, and record historical running data H; The simulated annealing algorithm is used to optimize the resource allocation scheme of the model-based Muzero series algorithms. The state in the simulated annealing algorithm is set as the configuration parameter of the model-based Muzero series algorithms; temperature is set as an adjustment parameter for the probability of accepting suboptimal solutions; energy is set as the prediction training time under the corresponding configuration of the prediction model output; when iterative optimization reaches a preset temperature or a preset number of iterations, the configuration parameter S corresponding to the current state is output. current ; Based on configuration parameter S current Based on historical running data H, CPU, GPU, and memory resources are dynamically allocated to complete the training of the model-based Muzero series algorithms.

2. The heterogeneous computing resource allocation optimization method for deep reinforcement learning model training as described in claim 1, characterized in that: The heterogeneous graph in, ε is the set of nodes, and ε is the set of edges. Represents the node feature matrix, This represents the edge feature matrix, where n is the total number of nodes, and d n It is the dimension of node features, m is the total number of edges, and d is the dimension of node features. e It is the dimension of the edge feature.

3. The heterogeneous computing resource allocation optimization method for deep reinforcement learning model training as described in claim 1 or 2, characterized in that: The types of nodes include Replay Buffer, Self-Play Worker, Neural Network, and TrainingLoop.

4. The heterogeneous computing resource allocation optimization method for deep reinforcement learning model training as described in claim 2, characterized in that: The node characteristics include node type h. t Hyperparameter h p Data size h d and calculation cost h c Among them, node type h t Used to describe the functional category of a node; hyperparameter h p Used to describe the specific configuration for each node's operation; data size h d Used to describe the amount of data processed or generated by a node; computational cost h c Used to describe the computational resource consumption of a node when it performs its functions; The edge features include the edge type and the amount of data; the edge type defines the data flow direction between nodes, and the amount of data is used to characterize the data density and flow of interaction between different nodes.

5. The heterogeneous computing resource allocation optimization method for deep reinforcement learning model training as described in claim 1, characterized in that: The updated node features are represented as follows: in, h is the set of adjacent nodes of node i. i ′ represents the updated node feature, h j Let i represent the characteristics of the adjacent nodes of node i, where 1 ≤ i ≤ n, and n is the total number of nodes.

6. The heterogeneous computing resource allocation optimization method for deep reinforcement learning model training as described in claim 1, characterized in that: The mean squared error between the predicted and actual running times is used as the loss function. Optimize the parameters of the prediction model: Among them, y i This represents the actual running time of node i.

7. The heterogeneous computing resource allocation optimization method for deep reinforcement learning model training as described in claim 1, characterized in that: The configuration parameters for the model-based Muzero series algorithms include batch buffer size (batch_buffer_size), CPU behavior (cpu_actor), storage size (mcts_storage_size), and loop count (loop_count).

8. The heterogeneous computing resource allocation optimization method for deep reinforcement learning model training as described in claim 1, characterized in that: The optimization of resource allocation schemes for model-based Muzero series algorithms using simulated annealing includes the following steps: Step 2.1, Parameter Initialization: Set the state in the simulated annealing algorithm to the configuration parameters of the model-based Muzero series algorithm; set the temperature to the adjustment parameter of the probability of accepting suboptimal solutions, and set the initial temperature to T0; set the energy to the prediction training time of the prediction model output. Step 2.2, State Transition: Within the neighborhood of the current state, randomly select a new state S according to either a uniform distribution or a normal distribution. k Record the state transition probability P k Use a predictive model to predict the energy E corresponding to the new state. k k is the number of iterations; Step 2.3, Optimization and Iteration: Compare the energy E before and after the state update. k-1 E k The size of E k <E k-1 Then update the state to S. k Update the temperature to T k =T0·α k Then return to step 2.2 for the next iteration, where α is the cooling factor; otherwise, compare the state transition probabilities P. k and The size of P k If the value is greater than P, then update the state to S. k Update the temperature to T k If the iteration continues, return to step 2.2 for the next iteration; otherwise, retain state S. k-1 Update temperature to T k Then return to step 2.2 for the next iteration; stop iterating when the preset temperature or preset number of iterations is reached.

9. The heterogeneous computing resource allocation optimization method for deep reinforcement learning model training as described in claim 1, characterized in that: Based on configuration parameter S current Based on temperature T and historical operating data H, CPU, GPU, and memory resources are dynamically allocated. The specific method is as follows: CPU resource allocation: Among them, CPU allocated This represents the amount of resources dynamically allocated to the CPU; CPU min This is the minimum CPU resource limit allowed by the system; CPU max This is the maximum CPU resource limit allowed by the system; f CPU () is a function that predicts CPU demand; max() and min() represent the maximum and minimum constraints, respectively; GPU resource allocation: Among them, GPU allocated Indicates the number of GPUs allocated. min Minimum number of GPUs required; GPU max f is the maximum number of available GPUs. GPU () is the GPU resource prediction function; Memory allocation: Among them, f Memory () is a function that predicts memory requirements. allocated Represents the amount of memory resources allocated; Memory min Indicates the minimum amount of memory available in the system. max This is the maximum memory limit supported by the system.

10. The method for optimizing the allocation of heterogeneous computing resources for training deep reinforcement learning models as described in claim 1 or 9, characterized in that: Based on the training results of the model-based Muzero series algorithms, the resource balance factor β is calculated: When β>1, it indicates over-allocation of resources, so reduce the allocation of CPU and memory resources; when β<1, it indicates insufficient resources, so increase the allocation of GPU resources.

Images