Heterogeneous computing power resource allocation optimization method for deep reinforcement learning model training

Abstract

The invention discloses a heterogeneous computing power resource configuration optimization method for deep reinforcement learning model training, and the method comprises the steps: constructing a heterogeneous graph for describing the features of a module and a data flow relation for a core module of a model-based Muzero series algorithm, mapping the features of nodes and edges to a same hidden space through a node-edge encoder based on multi-head attention, and carrying out the optimization of the heterogeneous computing power resource configuration. And predicting algorithm training time under different configurations by using the summarized global features. And performing configuration optimization according to the prediction training time output by the model by using a simulated annealing algorithm. And allocating CPU resources, GPU resources and memory resources according to the optimized configuration, and carrying out a training task. And calculating a resource equalization factor according to a training result, evaluating a resource configuration scheme, and continuously monitoring and optimizing. According to the method, the optimal balance between the performance and the resource consumption in the model training process is realized, the efficiency and the effect of model training are remarkably improved, and the technical difficulty of resource configuration optimization in a complex scene is reduced.

Description

Technical Field

[0001] The present invention belongs to the field of computer science and technology, and relates to deep learning and reinforcement learning, and in particular to a heterogeneous computing resource configuration optimization method for deep reinforcement learning model training. Background Art

[0002] Since the advent of AlphaGo, deep reinforcement learning technology has made significant progress in computing efficiency and resource optimization. The model-based Muzero series of algorithms achieves an efficient training process with lower hardware requirements, significantly improves sample efficiency, and represents an important development direction of the technology.

[0003] The core architecture of the model-based Muzero series of algorithms includes multiple key components such as Replay Buffer, Self-PlayWorker, Representation Network, Prediction Network, Dynamics Network, TrainingLoop, etc. These components are connected through a complex graph structure to realize the interaction of data and control flow.

[0004] Among them, Replay Buffer is responsible for storing data transitions generated during training. The key hyperparameter is batch_size, and the main data indicators are the number of transitions and buffer length. Self-Play Worker performs self-play tasks to generate new training data. The key hyperparameter is the number of parallel Monte Carlo tree searches mcts_simulations, and the computational cost is the number of self-play runs. RepresentationNetwork is used to convert the current observed state into a hidden state representation. The key hyperparameters include network depth and the number of hidden layer units. The computational cost is related to the amount of forward propagation calculations. Prediction Network predicts strategies and values ​​based on hidden states. The key hyperparameters include the structure and size of the strategy head and value head. The computational cost includes the computational complexity of strategies and values. Dynamics Network predicts the next hidden state and reward based on the current hidden state and action. The key hyperparameter is the network structure for state and reward prediction. The computational cost is related to state transfer and reward calculation. Training Loop uses the data in the Replay Buffer for model training, which involves updating the parameters of the representation network, prediction network, and power network. Key hyperparameters include the training batch size train_batch_size and the learning rate. The computational cost includes the number of iterations of backpropagation and gradient descent.

[0005] Although the model-based MuZero series of algorithms have made significant progress in computational efficiency and sample efficiency, they still face challenges in computing resource management. For example, the core components of the above algorithms all have extremely high requirements for computing and memory resources. Highly parallel Monte Carlo tree search requires a lot of CPU resources, the computational process of neural networks requires a lot of GPU resources, and the generation, storage and processing of large-scale training data increases the burden on memory and storage devices. These high-intensity computing and memory requirements directly affect the scalability and practical application scope of the algorithm, and also increase the complexity and management difficulty of the system. In addition, the performance and resource utilization efficiency of the model-based MuZero series of algorithms are highly dependent on the precise adjustment of a series of key hyperparameters. These parameters are not only closely related to the specific hardware configuration and problem characteristics, but also have an impact on each other. The complex interaction between multiple components causes the resource requirements to change dynamically during the training process, further increasing the difficulty of optimization. Finding the optimal configuration usually requires a lot of experiments, which is itself a resource-intensive task, forming a paradox that requires trade-offs.

[0006] Therefore, there is an urgent need for a method that can improve resource utilization efficiency and energy efficiency while maintaining algorithm performance, so as to achieve unified optimization of computational efficiency, sample efficiency and resource efficiency. Summary of the invention

[0007] In view of the shortcomings of the existing technology, the present invention proposes a heterogeneous computing resource configuration optimization method for deep reinforcement learning model training, which is used to dynamically allocate GPU and CPU resources of the model-based Muzero series algorithms according to the needs of different training stages, while maintaining algorithm performance, and improving resource utilization efficiency and energy efficiency.

[0008] The heterogeneous computing resource configuration optimization method for deep reinforcement learning model training specifically includes the following steps: Step 1: Run time prediction Step 1.1: Build a prediction model using the components in the model-based Muzero series algorithm as nodes and the interactions between components as edges, using a heterogeneous graph To describe the relationship between components, is a collection of nodes, is the edge set, represents the node feature matrix, represents the edge feature matrix, is the total number of nodes, is the dimension of node features, is the total number of edges, is the dimension of edge features. The types of nodes include Replay Buffer, Self-Play Worker, Neural Network and Training Loop.

[0009] Step 1.2: Map the features of nodes and edges to the same latent space: in, and are the feature conversion functions of nodes and edges respectively, is the dimension of the hidden layer. Representation Node The eigenvector of Represents edge The characteristic vector of , . and Represent the converted node and edge features respectively.

[0010] Update node features using a multi-head attention mechanism: in, Is a node The set of adjacent nodes of represents the updated node features, Representation Node Adjacent node features.

[0011] Step 1.3: Globally summarize the features of all nodes and edges to obtain the global node features representing the entire graph structure and global edge features : Predicting the running time of the model-based Muzero series algorithm based on global features : in, and is the weight matrix, is the bias vector. The mean square error (MSE) is calculated based on the predicted running time and the actual running time, and the parameters of the prediction model are optimized.

[0012] Step 2: Optimize computing resource configuration Step 2.1. Set the initial configuration parameters, including batch buffer size (batch_buffer_size), CPU behavior (cpu_actor), storage size (mcts_storage_size), and loop count (loop_count). Perform a baseline test and record the training time under the initial configuration.

[0013] Step 2.2: Input the initial configuration parameters to the prediction model trained in step 1 and predict the training time under the current configuration. Use the historical data from the baseline test in step 2.1 to verify and calibrate the prediction model.

[0014] Step 2.3: Use the simulated annealing algorithm to optimize the resource allocation scheme of the model-based Muzero series algorithm, which specifically includes the following steps: Step 2.3.1, parameter initialization: Set the state in the simulated annealing algorithm to the configuration parameters of the model-based Muzero series algorithm, using the initial configuration parameters in step 2.1 as the initial state Set the temperature as a tuning parameter for the probability of accepting a suboptimal solution, and set the initial temperature to T0. Set the energy to the predicted training time of the prediction model output.

[0015] Step 2.3.2, state transfer: Randomly select a new state in the neighborhood of the current state according to a uniform or normal distribution , record the state transition probability , use the prediction model to predict the energy corresponding to the new state , is the number of iterations.

[0016] Step 2.3.3, optimization iteration: Compare the energy before and after the state update , The size of < , then update the status to , update the temperature to , and return to step 2.3.2 for the next iteration, where is the cooling factor. Otherwise, compare the state transition probability and The size of > , then update the status to , update the temperature to , and return to step 2.3.2 for the next iteration, otherwise, keep the state , update temperature to Then return to step 2.3.2 for the next iteration. When the preset temperature or the preset number of iterations is reached, the iteration stops and the configuration parameters corresponding to the current state are output. .

[0017] The state is updated based on the comparison result between the state transition probability and the random probability P, which can maintain the balance between exploration and utilization.

[0018] Step 2.4: Configuration parameters based on step 2.3 Dynamically allocate CPU, GPU and memory resources based on historical operation data H, where the historical data H includes resource usage and training time under past configurations: CPU resource allocation: in, Indicates the amount of resources dynamically allocated to the CPU; It is the minimum CPU resource limit allowed by the system to ensure basic operating requirements; It is the maximum CPU resource limit allowed by the system to prevent over-allocation of resources; () is a function that predicts CPU demand based on current configuration parameters ,temperature and historical data Calculate. By constraints and , ensuring that the allocated CPU resources are between the minimum and maximum limits, thereby improving the system's operating efficiency and performance.

[0019] GPU resource allocation: in, Indicates the number of GPUs allocated, is the minimum number of GPUs required, which is determined by the algorithm complexity and data size; The maximum number of available GPUs represents the upper limit of resources to avoid resource waste. () is applied to the current configuration parameters ,temperature And historical operation data GPU resource prediction function.

[0020] Memory allocation: in, () is a function that predicts memory requirements, Represents the amount of memory resources allocated. Indicates the minimum amount of available memory in the system, ensuring that the system will not crash or experience performance degradation due to insufficient memory during operation. This is the maximum memory limit supported by the system, to avoid unstable operation caused by exceeding system capabilities.

[0021] Step 2.5: Resource balancing and adjustment Allocate resources based on the results of step 2.4 to complete the training task. Calculate the resource balancing factor based on the training results :

[0022] when When , it indicates that resources are over-allocated, and priority should be given to reducing CPU and memory resources; when , it indicates that resources are insufficient, and priority should be given to increasing GPU resources.

[0023] Compared with the prior art, the present invention has the following beneficial effects: 1. Accurate performance prediction: The complex interactions between the components of the model-based Muzero series of algorithms are captured through heterogeneous graph neural networks, which can accurately predict the running time under different configurations. This accurate prediction provides a reliable basis for decision-making in the simulated annealing process and significantly reduces invalid searches.

[0024] 2. Dynamic adaptability: Combining the real-time prediction of the prediction model and the temperature adjustment mechanism of simulated annealing, the configuration can be dynamically adjusted according to the performance changes during the training process. This adaptability enables the system to automatically adjust resource allocation at different training stages of the model-based Muzero series algorithm, such as increasing computing resources in the early exploration stage and appropriately reducing resources in the later convergence stage.

[0025] 3. Efficient exploration of global optimal solutions: The randomness of the simulated annealing algorithm combined with the guidance of the prediction model enables the system to more effectively explore the global optimal solution in MuZero's high-dimensional configuration space. Compared with traditional grid search or random search, it can converge to high-quality solutions faster to find the optimal solution equivalent to exhaustive search. This feature not only improves search efficiency, but also discovers some non-intuitive but high-performance configuration combinations.

[0026] 4. This method not only significantly improves the training efficiency and resource utilization of the model-based Muzero series of algorithms, but also makes it possible to apply the algorithm to large-scale complex problems. By reducing training time and optimizing resource usage, it helps to reduce the deployment cost of the model-based Muzero series of algorithms and accelerate their implementation in practical applications. In addition, this method has good generalization and can be easily extended to other complex reinforcement learning algorithm configuration optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Flowchart of the optimization method for heterogeneous computing resource configuration for deep reinforcement learning model training.

[0028] Figure 2 Flowchart for algorithm training time prediction. DETAILED DESCRIPTION

[0029] The present invention is further explained below in conjunction with the accompanying drawings; taking the EfficientZero algorithm, one of the model-based Muzero series algorithms, as an example, the specific process of the heterogeneous computing resource configuration optimization method for deep reinforcement learning model training is introduced. The EfficientZero algorithm realizes the interaction of data and control flow through the following key components: Replay Buffer: Responsible for storing data transitions generated during training. Key hyperparameters include batch_size, and the main data indicators are the number of transitions and buffer length.

[0030] Reanalyze Worker CPU / GPU: Uses CPU or GPU to perform reanalysis of data. The key hyperparameter is the number of worker threads, and the computational cost is mainly related to the replay buffer full count.

[0031] Selfplay Worker: performs self-play tasks to generate new training data. Key hyperparameters include the number of parallel environments p_mcts_num, and the computational cost includes the count of self-play speedup.

[0032] The heterogeneous computing resource allocation optimization method for deep reinforcement learning model training performs resource allocation on the EfficientZero algorithm, such as Figure 1 As shown, the specific steps are as follows: Step 1: Run time prediction Step 1.1: Build a prediction model using the components in the EfficientZero algorithm as nodes and the interactions between components as edges, using a heterogeneous graph To describe the relationship between components, is a collection of nodes, is the edge set, represents the node feature matrix, represents the edge feature matrix, is the total number of nodes, is the dimension of node features, is the total number of edges, is the dimension of the edge feature.

[0033] The node characteristics include node type , hyperparameters , Data size and calculate the cost Among them, the node type Functional categories used to describe nodes; hyperparameters Used to describe the specific configuration of each node operation, such as batch size and number of workers. Data size Used to describe the amount of data processed or generated by a node. Computational cost Used to describe the computing resources consumed by a node when performing its functions.

[0034] The edge features include edge type and data volume, where the edge type defines the data flow direction between nodes, and the data volume is used to characterize the data density and flow of the interaction between different nodes.

[0035] Step 1.2: In order to deeply understand the dependencies between nodes, a node-edge encoder based on a multi-head attention mechanism is introduced, and the node feature conversion function And edge feature conversion function , mapping the features of nodes and edges to the same latent space: in, is the dimension of the hidden layer. Representation Node The eigenvector of Represents edge The characteristic vector of , . and Represent the converted node and edge features respectively.

[0036] Use multi-head attention mechanism MultiHead() to refine nodes Attention distribution of its adjacent nodes, update the feature representation of the node: in, Is a node The set of adjacent nodes of represents the updated node features, Representation Node Adjacent node features.

[0037] Step 1.3: Globally summarize the features of all nodes and edges to obtain the global node features representing the entire graph structure and global edge features : Predicting the running time of the model-based Muzero series algorithm based on global features : in, and is the weight matrix, is the bias vector.

[0038] The loss function is the mean square error (MSE) between the predicted runtime and the actual runtime. , optimize the parameters of the prediction model: in, Representation Node The actual running time.

[0039] Step 2: Optimize computing resource configuration Based on the prediction results of the prediction model, the configuration of the EfficientZero algorithm is optimized in combination with the simulated annealing algorithm, aiming to simultaneously meet performance requirements and minimize computing resource consumption. The specific steps are as follows: Step 2.1. Set the initial configuration parameters, including batch buffer size (batch_buffer_size), CPU behavior (cpu_actor), storage size (mcts_storage_size), and loop count (loop_count), perform baseline testing, and record the test results in the base.Log4j file for preliminary analysis of system performance bottlenecks and optimization starting points.

[0040] Step 2.2: Input the initial configuration parameters to the prediction model trained in step 1 and predict the training time under the current configuration. Use the historical data from the baseline test in step 2.1 to verify and calibrate the prediction model.

[0041] Step 2.3: Use the simulated annealing algorithm to optimize the resource allocation scheme of the EfficientZero algorithm, and use the initial configuration parameters in step 2.1 as the initial state of the simulated annealing algorithm. The temperature is set as a tuning parameter for the probability of accepting a suboptimal solution. In order to ensure that the algorithm can explore a wider configuration space in the early exploration stage and avoid falling into a local optimal solution too early, a higher initial temperature is set. . Set the energy to the predicted training time output by the prediction model. When the preset temperature 1 or the preset number of iterations is reached, stop the iteration and output the configuration parameters corresponding to the current state .

[0042] Step 2.4: Configuration parameters based on step 2.3 and historical running data H, dynamically allocate CPU, GPU, and memory resources.

[0043] Through the above optimization process, the resource configuration of the EfficientZero algorithm can be effectively adjusted and optimized, minimizing the consumption of computing resources while ensuring performance. This method combines the predictive ability of graph neural networks and the global search characteristics of simulated annealing algorithms to provide an efficient configuration optimization strategy for complex deep learning models.

Claims

1. A heterogeneous computing resource configuration optimization method for deep reinforcement learning model training, characterized by: The specific steps include: Step 1: Run time prediction Build a prediction model using the components in the model-based Muzero series algorithm as nodes and the interactions between components as edges, using a heterogeneous graph To describe the relationship between components, map the features of nodes and edges to the same hidden space, and use the multi-head attention mechanism to refine the attention distribution of nodes to their adjacent nodes and update node features; finally, globally summarize the features of all nodes and edges, and predict the running time of the model-based Muzero series algorithm based on the global features. ;Optimize the parameters of the prediction model based on the predicted running time and the actual running time; Step 2: Optimize computing resource configuration Set the configuration parameters of the model-based Muzero series algorithm, perform baseline testing, and record historical operation data H; Use the simulated annealing algorithm to optimize the resource allocation scheme of the model-based Muzero series algorithm, set the state in the simulated annealing algorithm as the configuration parameter of the model-based Muzero series algorithm; set the temperature as the adjustment parameter for the probability of accepting suboptimal solutions; set the energy as the predicted training time under the corresponding configuration of the prediction model output; when iterative optimization reaches the preset temperature or the preset number of iterations, output the configuration parameters corresponding to the current state ; Based on configuration parameters And historical running data H, dynamically allocate CPU, GPU and memory resources to complete the training of model-based Muzero series algorithms.

2. The method for optimizing heterogeneous computing resource configuration for deep reinforcement learning model training according to claim 1, characterized in that: The heterogeneous graph ,in, is a collection of nodes, is the edge set, represents the node feature matrix, represents the edge feature matrix, is the total number of nodes, is the dimension of node features, is the total number of edges, is the dimension of the edge feature.

3. The method for optimizing heterogeneous computing resource configuration for deep reinforcement learning model training according to claim 1 or 2, characterized in that: The types of nodes include Replay Buffer, Self-Play Worker, Neural Network and TrainingLoop.

4. The method for optimizing heterogeneous computing resource configuration for deep reinforcement learning model training according to claim 2, characterized in that: The node characteristics include node type , hyperparameters , Data size and calculate the cost ; Among them, the node type Functional categories used to describe nodes; hyperparameters Used to describe the specific configuration of each node operation; data size Used to describe the amount of data processed or generated by a node; calculation cost Used to describe the computing resource consumption of a node when performing its function; The edge features include edge type and data volume; the edge type defines the data flow direction between nodes, and the data volume is used to characterize the data density and flow of the interaction between different nodes.

5. The method for optimizing heterogeneous computing resource configuration for deep reinforcement learning model training according to claim 1, characterized in that: The updated node feature is expressed as: in, Is a node The set of adjacent nodes of represents the updated node features, Representation Node The adjacent node features of , is the total number of nodes.

6. The method for optimizing heterogeneous computing resource configuration for deep reinforcement learning model training according to claim 1, characterized in that: The loss function is the mean square error between the predicted runtime and the actual runtime. , optimize the parameters of the prediction model: in, Representation Node The actual running time.

7. The method for optimizing heterogeneous computing resource configuration for deep reinforcement learning model training according to claim 1, characterized in that: The configuration parameters of the model-based Muzero series algorithm include batch buffer size batch_buffer_size, CPU behavior cpu_actor, storage size mcts_storage_size, and loop count loop_count.

8. The method for optimizing heterogeneous computing resource configuration for deep reinforcement learning model training according to claim 1, characterized in that: The simulated annealing algorithm is used to optimize the resource allocation scheme of the model-based Muzero series algorithm, which includes the following steps: Step 2.1, parameter initialization: Set the state in the simulated annealing algorithm as the configuration parameter of the model-based Muzero series algorithm; set the temperature as the adjustment parameter for the probability of accepting suboptimal solutions, and set the initial temperature to T0; set the energy to the prediction training time of the prediction model output; Step 2.2, state transfer: Randomly select a new state in the neighborhood of the current state according to a uniform or normal distribution , record the state transition probability , use the prediction model to predict the energy corresponding to the new state , is the number of iterations; Step 2.3, optimization iteration: Compare the energy before and after the state update , The size of < , then update the status to , update the temperature to , and return to step 2.2 for the next iteration, where is the cooling factor; otherwise, compare the state transition probability and The size of > , then update the status to , update the temperature to , and return to step 2.2 for the next iteration, otherwise, keep the state , update the temperature to Then return to step 2.2 for the next iteration; when the preset temperature or the preset number of iterations is reached, the iteration is stopped.

9. The method for optimizing heterogeneous computing resource configuration for deep reinforcement learning model training according to claim 1, characterized in that: Based on configuration parameters and historical running data H, dynamically allocate CPU, GPU and memory resources. The specific method is: CPU resource allocation: in, Indicates the amount of resources dynamically allocated to the CPU; It is the minimum CPU resource limit allowed by the system; It is the maximum CPU resource limit allowed by the system; () is a function that predicts CPU demand; and Represent the maximum and minimum constraints respectively; GPU resource allocation: in, Indicates the number of GPUs allocated, is the minimum number of GPUs required; is the maximum number of GPUs available; () is the GPU resource prediction function; Memory allocation: in, () is a function that predicts memory requirements, Represents the amount of memory resources allocated; Indicates the minimum amount of available memory in the system. The maximum memory limit supported by the system.

10. The method for optimizing heterogeneous computing resource configuration for deep reinforcement learning model training according to claim 1 or 9, characterized in that: Calculate resource balancing factors based on the training results of the model-based Muzero series algorithm : when When , it indicates that resources are over-allocated, reduce the amount of CPU and memory resources allocated; when , it indicates insufficient resources and increases GPU resource allocation.

Images