A reinforcement learning training framework and method for GPU

Abstract

The present application provides a reinforcement learning training framework implemented on a single GPU, including a configurator, an executor, a learner, a model pool and a buffer. The executor and the learner run on the GPU, the model pool and the buffer are set in the video memory of the GPU, and the configurator runs on the CPU; the configurator determines the operating parameters of the executor and the operating parameters of the learner, and sends the operating parameters of the executor and the operating parameters of the learner to the GPU. The executor extracts the model from the model pool and stores the empirical data obtained using the model in the buffer; the learner extracts the empirical data from the buffer, and generates a model based on the empirical data, and places the generated model in an empty position in the model pool according to the operating parameters of the learner. The present invention utilizes a multi-stream mechanism to efficiently utilize a single GPU for reinforcement learning training. By controlling the execution of different training modules, resource utilization is improved, and the performance of reinforcement learning training on the GPU is improved.

Description

Technical Field

[0001] One or more embodiments of this specification relate to the field of computer technology, and more particularly, to a reinforcement learning training framework and method for GPUs. Background Art

[0002] Reinforcement learning consists of steps such as organizing the operating environment, interpreting model inference, optimizing model teaching, and managing the dynamics of the replay buffer. Different steps have different computational loads and resource requirements. Environmental simulations in scientific applications are often very complex and require a large amount of computing resources. Currently, distributed simulators with multiple GPUs or multiple nodes are used to accelerate the generation and updating of environments. However, these simulators require the coordination of heterogeneous computing resources, resulting in cross-device computing performance losses and inability to fully utilize the computing resources of a single node.

[0003] Application Contents

[0004] This application describes a reinforcement learning training framework and method for GPU, which can solve the above technical problems.

[0005] According to a first aspect, a reinforcement learning training framework implemented on a single GPU is provided, the reinforcement learning training framework comprising a configurator, at least one executor, a learner, a model pool, and a buffer, wherein the executor and the learner run on the GPU, the model pool and the buffer are set in the GPU's video memory, and the configurator runs on the CPU;

[0006] the configurator is configured to determine operating parameters of the executor and the learner, and send the operating parameters of the executor and the learner to the GPU, so that the GPU starts a corresponding number of GPU threads for the executor according to the operating parameters of the executor, wherein the operating parameters of the executor include the number of executors;

[0007] At least one of the executors occupies at least one GPU thread, and is used to extract a model from the model pool, perform reasoning using the model to obtain an action, and store the experience data obtained after executing the action in a buffer;

[0008] The learner occupies at least one GPU thread, which is used to extract experience data from the buffer, use the experience data to perform gradient backpropagation and parameter update on the model, and place the updated model into an empty position in the model pool based on the operating parameters of the learner. The operating parameters of the learner include the interval at which the learner places the model into the model pool.

[0009] Based on the above embodiment, the configurator is specifically used to obtain multiple sets of operating parameters of the actuator and the operating parameters of the learner according to the configuration parameters, wherein the operating parameters of the actuator and the operating parameters of the learner are regarded as a set of operating parameters;

[0010] The operating parameters of each set of the executor and the operating parameters of the learner are sent to the GPU for pre-execution to obtain the operating parameters of the executor and the operating parameters of the learner that require the shortest time to complete the reinforcement learning training.

[0011] Based on the above further embodiment, the configurator is specifically used to calculate the number of experience data updated by the actuator in one step according to the experience income and expenditure ratio k, the number of experience data updated by the actuator in one step, the number of experience data batch_size used by the learner for a single model iteration, and the ratio formula

[0012]

[0013] Get the relationship between the number of executors n_actors and the interval update_interva at which the learner puts the model into the model pool;

[0014] By using the association relationship, the number n_actors of the plurality of groups of executors and the interval at which the learner puts the model into the model pool are obtained.

[0015] Based on the above embodiment, there are multiple positions in the model pool, and the model at each position is taken away by one of the actuators;

[0016] The actuator is configured to check whether there is a model at a target position in the model pool after generating a first preset amount of experience data;

[0017] If so, take away the model;

[0018] Otherwise, the actuator enters a waiting state until a model is placed on the extraction target position.

[0019] Based on the above embodiment, the learner is specifically configured to place the models into the empty positions in the model pool in order after completing the second preset number of models;

[0020] If a model is stored in one of the positions, the model pool is polled until an empty position is found to put the model in.

[0021] Based on the above embodiment, the buffer is used to store the experience data collected by the executor, and an independent buffer space is allocated to each of the executors.

[0022] According to a second aspect, a reinforcement learning training method implemented on a single GPU is provided. The reinforcement learning training framework includes a configurator, at least one executor, a learner, a model pool, and a buffer. The executor and the learner run on the GPU, the model pool and the buffer are set in the GPU's video memory, and the configurator runs on the CPU.

[0023] The method comprises:

[0024] The configurator determines operating parameters of the executor and the learner, and sends the operating parameters of the executor and the learner to the GPU, so that the GPU starts a corresponding number of GPU threads for the executor according to the operating parameters of the executor, wherein the operating parameters of the executor include the number of executors;

[0025] At least one of the executors occupies at least one GPU thread, extracts a model from the model pool, uses the model to perform reasoning to obtain an action, and stores experience data obtained after executing the action in a buffer;

[0026] The learner occupies at least one GPU thread, extracts experience data from the buffer, uses the experience data to perform gradient backpropagation and parameter update on the model, and places the updated model into an empty position in the model pool based on the operating parameters of the learner. The operating parameters of the learner include the interval at which the learner places the model into the model pool.

[0027] Based on the above embodiment, the configurator determines the operating parameters of the actuator and the operating parameters of the learner, specifically including:

[0028] The configurator obtains multiple sets of operating parameters of the actuator and the operating parameters of the learner according to the configuration parameters, wherein the operating parameters of the actuator and the operating parameters of the learner are taken as a set of operating parameters;

[0029] The operating parameters of each set of the executor and the operating parameters of the learner are sent to the GPU for pre-execution to obtain the operating parameters of the executor and the operating parameters of the learner that require the shortest time to complete the reinforcement learning training.

[0030] Based on the above embodiment, the configurator calculates the ratio of experience revenue and expenditure k, the number of experience data updatenum updated by the actuator in one step, the number of experience data batch_size used by the learner for a single model iteration, and the ratio formula

[0031]

[0032] Get the relationship between the number of executors n_actors and the interval update_interva at which the learner puts the model into the model pool;

[0033] By using the association relationship, the number n_actors of the plurality of executors and the interval at which the learner puts the model into the model pool are obtained.

[0034] Based on the above embodiment, there are multiple positions in the model pool, and the model at each position is taken away by one of the actuators;

[0035] After generating a first preset amount of experience data, the actuator checks whether there is a model at a target position in the model pool;

[0036] If so, take away the model;

[0037] Otherwise, the actuator enters a waiting state until a model is placed on the extraction target position.

[0038] Based on the above embodiment, after the learner completes the second preset number of models, the models are placed into the empty positions in the model pool in order;

[0039] If a model is stored in one of the positions, the model pool is polled until an empty position is found to put the model in.

[0040] Based on the above embodiment, the buffer stores the experience data collected by the executor, and an independent buffer space is allocated to each of the executors.

[0041] According to a third aspect, an electronic device is provided, comprising a memory, a processor, and a computer program stored on the memory and running on the processor, wherein when the processor executes the program, the reinforcement learning training method implemented on a single GPU as described in the above technical solution is implemented.

[0042] According to a fourth aspect, an electronic device is provided, comprising a memory, a processor, and a computer program stored on the memory and running on the processor, wherein when the processor executes the program, the reinforcement learning training method implemented on a single GPU as described in the above technical solution is implemented.

[0043] In the above-mentioned system and method provided in the embodiments of this specification, it is proposed to use a multi-stream mechanism to efficiently utilize a single GPU for reinforcement learning training, improve resource utilization by controlling the execution of different training modules, and at the same time, optimize the training modules by utilizing the GPU hardware characteristics, thereby improving the performance of reinforcement learning training on the GPU. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0045] Figure 1 This is a schematic diagram of the reinforcement learning training framework structure implemented on a single GPU provided by the present invention;

[0046] Figure 2 This is a schematic diagram of the operation process of a reinforcement learning framework implemented on a single GPU provided by the present invention;

[0047] Figure 3 This is a flow chart of a reinforcement learning training method implemented on a single GPU provided by the present invention. DETAILED DESCRIPTION

[0048] The solution provided in this specification is described below in conjunction with the accompanying drawings.

[0049] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings.

[0050] Reinforcement learning consists of steps such as organizing the operating environment, explaining model inference, optimizing model teaching, and managing the dynamics of the replay buffer. Different steps have different computational loads and resource requirements. Environmental simulations in scientific applications are often very complex and require a large amount of computing resources. Consider using multi-GPU or multi-node distributed simulators to accelerate environment generation and updates. Such training architectures require heterogeneous computing resources to cooperate with each other, resulting in cross-device computing performance losses. Current deep learning frameworks such as Haiku, Flax, PyTorch, and TensorFlow support distributed training, but these deep learning frameworks do not fully utilize the computing resources of a single node.

[0051] The execution process of reinforcement learning involves numerous unavoidable conditional branches, and these logical decisions are typically executed on the CPU. Neural network parameter updates, being computationally intensive, can be significantly accelerated on GPUs. However, the policy network used in reinforcement learning tasks has a relatively simple neural network structure and a small number of parameters, which prevents full utilization of GPU computing power. Data exchange and synchronization between different devices can also lead to performance losses on GPUs.

[0052] Multi-streaming is a technology implemented in modern GPU architectures that optimizes the allocation and utilization of computing resources by allowing multiple independent execution streams to run simultaneously on the same GPU. This mechanism enhances computational overlap, significantly improving processing speed and resource efficiency, particularly when managing large-scale parallel computing tasks. It effectively reduces overall task execution time and enhances the decoupling of computation and data transfer through the parallel operation of independent streams, thereby increasing the throughput of computing tasks. However, optimizing GPU computing power based on multiple training modules in reinforcement learning using multi-streaming is a technical challenge that needs to be addressed.

[0053] This paper proposes a reinforcement learning training framework on modern GPUs. It uses a multi-stream mechanism to efficiently utilize a single GPU for reinforcement learning training. By controlling the execution of different training modules, it improves resource utilization. At the same time, by utilizing the GPU hardware characteristics and optimizing the training modules, it further improves the performance of reinforcement learning training on the GPU.

[0054] like Figure 1 The reinforcement learning training framework structure implemented on a single GPU shows that the reinforcement learning training framework includes five modules: executor actor, learner learner, model pool, parallel buffer and configurator. Among them, the executor generates experience data, and the learner uses the experience data generated by the executor to train the agent model. Unlike the existing CPU-based reinforcement learning training framework, the GPU-based reinforcement learning training framework adds three modules: parallel buffer, model pool and configurator. Among them, the parallel buffer and model pool assist the executor-learner cycle, and the configurator is used to adjust the operating parameters to fully utilize computing resources while reducing training time.

[0055] Reinforcement learning training consists of two main steps: First, the agent interacts with the environment, exploring and acquiring a large amount of experience data, which is stored in a parallel buffer. Second, the learner extracts a certain amount of experience data from the parallel buffer and uses this experience data to calculate gradients and update the neural network parameters.

[0056] The reinforcement learning training framework uses a JSON file to configure the training process. This JSON file is divided into a training parameter section and a network parameter section. The network parameter section is used to set the necessary values ​​and control parameters required to implement different neural networks. The training parameter section includes settings related to reinforcement learning training, such as the buffer size, the amount of experience collected, the model update frequency, the environment exploration rate, and the learning rate.

[0057] The reinforcement learning training framework consists of five parts, including five modules: actor, learner, model pool, parallel buffer and configurator.

[0058] The executor actor is responsible for collecting experience data. In the process of collecting each piece of experience data, the executor uses the model obtained by training the learner and the environment state data to perform reasoning and obtain actions. The executor inputs the obtained actions into the environment, makes the environment step forward, and obtains the next environment state as well as information such as rewards and termination status. In the present invention, each environment is assigned one or more GPU threads for calculation according to the complexity of the environment. At the same time, the present invention supports multiple GPU threads to start multiple executors to collect experience data together. In the face of an environment with fewer resources, an actor in the present invention contains two main parts, the auxiliary function part auv and the environment simulation part env. The configuration information provided by the configurator can allow an auxiliary function part auv to cooperate with the simulation of multiple environment simulation parts env at the same time, making full use of GPU resources, so that the executor can obtain more experience data in one experience collection process, and cooperate with multiple executors started by multiple GPU threads to improve the GPU utilization efficiency and experience acquisition efficiency.

[0059] The parallel experience buffer is used to store the large amount of experience data collected by the executors. As more experience data is acquired, the experience pool is continuously flushed, eliminating old experience and replacing it with new experience. Each executor is allocated independent space, allowing them to store experience without interfering with each other.

[0060] The learner is responsible for updating the neural network. It randomly extracts multiple experiences from the buffer and uses these experiences to calculate the loss, perform gradient backpropagation, and update parameters.

[0061] The model pool is a lock-free circular queue with multiple positions, each corresponding to an executor to extract the model. When the learner completes a certain number of models, it places the models into the empty positions in the queue in order. If it finds that the model previously stored in a certain position has not been removed by the executor, it polls the queue until it finds an empty position to put the model in. After generating a certain amount of experience, the executor will check the corresponding empty position and remove the model if there is a model in the empty position. If it finds that there is no new model in the empty position, it means that the rate at which the executor actor generates experience is greater than the rate at which the learner learns, and the executor enters a waiting state until it can obtain the newly generated model. Using the lock-free model pool, the executor actor and the learner can achieve asynchronous execution, fully utilizing the computing power of the GPU.

[0062] The configurator is used to evaluate the workload of the learner and executor actors. Before actual operation, the configurator uses tools provided by Nvidia to evaluate the GPU resource usage of the learner and executor actors and recommends the configuration with the highest resource utilization. Based on this configuration, the configurator determines the interval at which the learner parameters are placed in the model pool and the number of executor actors according to the following proportional formula:

[0063]

[0064] In the above formula, the experience balance ratio k, the number of experiences updated during a joint step by the executor actors updatenum, and the number of experiences used by the learner for each model iteration batch_size are all user-configured. update_interval represents the interval at which the learner adds a model to the model pool, and n_actors represents the number of executor actors.

[0065] The numerator of the formula represents the number of experiences the learner uses each time it places the model, and the denominator represents the number of experiences an executor generates each time it removes the model. k should be greater than or equal to 1, ensuring that the expected number of experiences generated by an executor actor being used by the learner is at least 1.

[0066] Under the constraints of the above formula, different update_interval values ​​(the interval at which the learner adds models to the model pool) are calculated for different numbers of executor actors. The configurator configures multiple numbers of executor actors, n_actors, with small training steps and conducts trial runs to find the number of executor actors, n_actors, and the update_interval interval at which the learner adds models to the model pool that minimizes reinforcement learning completion time. This approach provides recommendations for the number of executor actors and the interval at which the learner adds models to the model pool based on three criteria: optimal resource utilization, minimal computational waste, and the fastest computational speed.

[0067] In addition, if Figure 2 As shown in the operational flow diagram of the reinforcement learning training framework implemented on a single GPU, the reinforcement learning training process is a continuous iterative optimization cycle.

[0068] The operation process of the reinforcement learning training framework is as follows:

[0069] Step 1: The configurator evaluates a single run of the learner and executor actors and recommends parameter settings, including the number of executor actors and the interval at which the learner puts models into the model pool.

[0070] Step 2: All executor actors perform random actions together to fill the parallel experience buffer.

[0071] Step 3: The learner updates the model and places the new model into the empty space in the model pool after a certain number of iterations. At the same time, the executor actor starts reasoning and generating experience, placing the experience into the parallel experience buffer. After generating a certain amount of experience, a new model is taken from the model pool.

[0072] Step 4: Repeat step 3 until the pre-set number of learner iterations is reached and execute the downstream task.

[0073] like Figure 1 As shown in the schematic diagram of the reinforcement learning training framework implemented on a single GPU, the reinforcement learning training framework includes a configurator, at least one executor, a learner, a model pool, and a buffer. The executor and learner run on the GPU, the model pool and buffer are set in the GPU's video memory, and the configurator runs on the CPU.

[0074] A configurator, configured to determine the operating parameters of the executor and the learner, and to send the operating parameters of the executor and the learner to the GPU so that the GPU starts a corresponding number of GPU threads for the executor based on the executor's operating parameters, including the number of executors.

[0075] At least one executor occupies at least one GPU thread and is used to extract a model from the model pool, use the model to perform inference to obtain an action, and store the experience data obtained after executing the action in a buffer;

[0076] The learner occupies at least one GPU thread to extract experience data from the buffer, use the experience data to perform gradient backpropagation and parameter update on the model, and place the updated model into an empty position in the model pool according to the learner's operating parameters. The learner's operating parameters include the interval at which the learner places the model into the model pool.

[0077] Based on the above embodiment, the configurator is specifically configured to obtain multiple sets of operating parameters of the actuator and the operating parameters of the learner according to the configuration parameters, wherein the operating parameters of the actuator and the operating parameters of the learner are regarded as a set of operating parameters;

[0078] The operating parameters of each set of executors and learners are sent to the GPU for pre-execution to obtain the operating parameters of the executors and learners that require the shortest time to complete the reinforcement learning training.

[0079] Based on the above further embodiment, the configurator is specifically used to calculate the number of experience data updated by the actuator in one step according to the experience income and expenditure ratio k, the number of experience data updated by the actuator in one step, the number of experience data batch_size used by the learner for a single model iteration, and the ratio formula

[0080]

[0081] Get the relationship between the number of executors n_actors and the interval update_interva at which the learner puts the model into the model pool;

[0082] Using the association relationship, we can get the number of multiple groups of executors n_actors and the interval at which the learner puts the model into the model pool.

[0083] Based on the above embodiment, there are multiple positions in the model pool, and the model at each position is taken away by an actuator;

[0084] An executor, configured to check whether there is a model at a target position in a model pool after generating a first preset amount of experience data;

[0085] If so, take away the model;

[0086] Otherwise, the actuator enters a waiting state until a model is placed at the extraction target position.

[0087] Based on the above embodiment, the learner is specifically configured to place the models into empty positions in the model pool in order after completing the second preset number of models;

[0088] If a model is stored in one of the locations, the model pool is polled until an empty location is found to put the model in.

[0089] Based on the above embodiment, the buffer is used to store the experience data collected by the executor, and an independent buffer space is allocated to each executor.

[0090] In the above embodiment, it is proposed to use a multi-stream mechanism to efficiently utilize a single GPU for reinforcement learning training, improve resource utilization by controlling the execution of different training modules, and at the same time, optimize the training modules by utilizing the GPU hardware characteristics, thereby improving the performance of reinforcement learning training on the GPU.

[0091] like Figure 3The operation flow chart of the reinforcement learning training framework implemented on a single GPU shows a reinforcement learning training method implemented on a single GPU. The reinforcement learning training framework includes a configurator, an executor, a learner, a model pool, and a buffer. The executor and learner run on the GPU, the model pool and buffer are set in the GPU's video memory, and the configurator runs on the CPU.

[0092] The following steps are involved:

[0093] 110. The configurator determines operating parameters of the executor and the learner, and sends the operating parameters of the executor and the learner to the GPU, so that the GPU starts a corresponding number of GPU threads for the executor according to the operating parameters of the executor, where the operating parameters of the executor include the number of executors.

[0094] 120. At least one executor occupies at least one GPU thread, extracts a model from the model pool, uses the model to perform reasoning to obtain an action, and stores the experience data obtained after executing the action in a buffer;

[0095] 130. The learner occupies at least one GPU thread, extracts experience data from the buffer, uses the experience data to perform gradient backpropagation and parameter update on the model, and places the updated model into an empty position in the model pool according to the operating parameters of the learner. The operating parameters of the learner include the interval at which the learner places the model into the model pool.

[0096] Based on the above embodiment, step 110 specifically includes:

[0097] The configurator obtains multiple sets of operating parameters of the actuators and the operating parameters of the learner according to the configuration parameters, wherein the operating parameters of the actuators and the operating parameters of the learner are regarded as a set of operating parameters;

[0098] The operating parameters of each set of executors and the operating parameters of the learner are sent to the GPU for pre-execution to obtain the operating parameters of the executors and the operating parameters of the learner that require the shortest time to complete the reinforcement learning training.

[0099] Based on the above embodiment, step 110 specifically includes:

[0100] The configurator uses the experience income and expenditure ratio k, the number of experience data updatenum that the executor updates in one step, the number of experience data batch_size used by the learner for a single model iteration, and the ratio formula

[0101]

[0102] Get the relationship between the number of executors n_actors and the interval update_interva at which the learner puts the model into the model pool;

[0103] Using the association relationship, we can get the number of multiple groups of executors n_actors and the interval at which the learner puts the model into the model pool.

[0104] Based on the above embodiment, there are multiple positions in the model pool, and the model at each position is taken away by one of the actuators;

[0105] After generating a first preset amount of experience data, the actuator checks whether there is a model at the target position in the model pool;

[0106] If so, take away the model;

[0107] Otherwise, the actuator enters a waiting state until a model is placed at the extraction target position.

[0108] Based on the above embodiment, after the learner completes the second preset number of models, the models are placed into the empty positions in the model pool in order;

[0109] If a model is stored in one of the locations, the model pool is polled until an empty location is found to put the model in.

[0110] Based on the above embodiment, the buffer stores the experience data collected by the executor, and an independent buffer space is allocated to each executor.

[0111] In the above embodiment, it is proposed to use a multi-stream mechanism to efficiently utilize a single GPU for reinforcement learning training, improve resource utilization by controlling the execution of different training modules, and at the same time, optimize the training modules by utilizing the GPU hardware characteristics, thereby improving the performance of reinforcement learning training on the GPU.

[0112] According to another embodiment, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the program, the reinforcement learning training method implemented on a single GPU as in the above technical solution is implemented.

[0113] According to another embodiment, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute a reinforcement learning training method implemented on a single GPU.

[0114] Those skilled in the art will appreciate that, in one or more of the above examples, the functions described herein may be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0115] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of this application. It should be understood that the above description is only the specific implementation methods of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solutions of this application should be included in the scope of protection of this application.

Claims

1. A reinforcement learning training framework implemented on a single GPU, characterized by: The reinforcement learning training framework includes a configurator, at least one executor, a learner, a model pool, and a buffer, wherein the executor and the learner run on the same GPU, the model pool and the buffer are set in the GPU memory, and the configurator runs on the CPU; the configurator is configured to determine operating parameters of the executor and the learner, and send the operating parameters of the executor and the learner to the GPU, so that the GPU starts a corresponding number of GPU threads for the executor according to the operating parameters of the executor, wherein the operating parameters of the executor include the number of executors; The model pool is a lock-free circular queue, and the executors and the learner are executed asynchronously using the model pool; at least one of the executors occupies at least one GPU thread, and is used to extract a model from the model pool, perform inference using the model to obtain an action, and store the experience data obtained after executing the action in a buffer; The learner occupies at least one GPU thread, which is used to extract experience data from the buffer, use the experience data to perform gradient backpropagation and parameter update on the model, and place the updated model into an empty position in the model pool based on the operating parameters of the learner. The operating parameters of the learner include the interval at which the learner places the model into the model pool.

2. The reinforcement learning training framework according to claim 1, characterized in that The configurator is specifically configured to obtain multiple sets of operating parameters of the actuator and the operating parameters of the learner according to the configuration parameters, wherein the operating parameters of the actuator and the operating parameters of the learner are regarded as a set of operating parameters; The operating parameters of each set of the executor and the operating parameters of the learner are sent to the GPU for pre-execution to obtain the operating parameters of the executor and the operating parameters of the learner that require the shortest time to complete the reinforcement learning training.

3. The reinforcement learning training framework according to claim 2, characterized in that The configurator is specifically used to calculate the ratio of experience revenue and expenditure k, the number of experience data updatenum that the executor steps together to update once, the number of experience data batch_size used by the learner for a single model iteration, and the ratio formula. Get the relationship between the number of executors n_actors and the interval update_interva at which the learner puts the model into the model pool; By using the association relationship, the number n_actors of the plurality of executors and the interval at which the learner puts the model into the model pool are obtained.

4. The reinforcement learning training framework according to claim 1, characterized in that There are multiple positions in the model pool, and the model at each position is taken away by one of the actuators; The actuator is configured to check whether there is a model at a target position in the model pool after generating a first preset amount of experience data; If so, take away the model; Otherwise, the actuator enters a waiting state until a model is placed on the extraction target position.

5. The reinforcement learning training framework according to claim 4, characterized in that The learner is specifically configured to place the models into the empty positions in the model pool in order after completing a second preset number of models; If a model is stored in one of the positions, the model pool is polled until an empty position is found to put the model in.

6. The reinforcement learning training framework according to claim 1, characterized in that The buffer is used to store the experience data collected by the executor, and an independent buffer space is allocated to each of the executors.

7. A reinforcement learning training method implemented on a single GPU, characterized in that: The reinforcement learning training framework includes a configurator, at least one executor, a learner, a model pool, and a buffer, wherein the executor and the learner run on the same GPU, the model pool and the buffer are set in the GPU memory, and the configurator runs on the CPU; The method comprises: The configurator determines operating parameters of the executor and the learner, and sends the operating parameters of the executor and the learner to the GPU, so that the GPU starts a corresponding number of GPU threads for the executor according to the operating parameters of the executor, wherein the operating parameters of the executor include the number of executors; The model pool is a lock-free circular queue, and the executor and the learner are executed asynchronously using the model pool; At least one of the executors occupies at least one GPU thread, extracts a model from the model pool, uses the model to perform reasoning to obtain an action, and stores experience data obtained after executing the action in a buffer; The learner occupies at least one GPU thread, extracts experience data from the buffer, uses the experience data to perform gradient backpropagation and parameter update on the model, and places the updated model into an empty position in the model pool based on the operating parameters of the learner. The operating parameters of the learner include the interval at which the learner places the model into the model pool.

8. The reinforcement learning training method according to claim 7, characterized in that: The configurator determines the operating parameters of the actuator and the operating parameters of the learner, specifically including: The configurator obtains multiple sets of operating parameters of the actuator and the operating parameters of the learner according to the configuration parameters, wherein the operating parameters of the actuator and the operating parameters of the learner are taken as a set of operating parameters; The operating parameters of each set of the executor and the operating parameters of the learner are sent to the GPU for pre-execution to obtain the operating parameters of the executor and the operating parameters of the learner that require the shortest time to complete the reinforcement learning training.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the reinforcement learning training method implemented on a single GPU as described in any one of claims 7-8 is implemented.

Images