Reinforcement learning training framework and method for GPU

Abstract

The invention provides a reinforcement learning training framework implemented on a single GPU, the reinforcement learning training framework comprises a configurator, an actuator, a learning device, a model pool and a buffer area, the actuator and the learning device run on the GPU, the model pool and the buffer area are arranged on a video memory of the GPU, and the configurator runs on a CPU; the configurator determines operation parameters of the executor and operation parameters of the learner, the operation parameters of the executor and the operation parameters of the learner are issued to the GPU, the executor extracts a model from the model pool, and empirical data obtained through the model are stored in a buffer area; the learner extracts empirical data from the buffer area, a model is generated according to the empirical data, and the generated model is placed at an empty position in the model pool according to operation parameters of the learner. According to the method, the multi-stream mechanism is utilized, the single GPU is efficiently utilized to carry out reinforcement learning training, and the execution of different training modules is controlled, so that the resource utilization rate is improved, and the performance of reinforcement learning training on the GPU is improved.

Description

Technical Field

[0001] One or more embodiments of this specification relate to the field of computer technology, and more particularly, to a reinforcement learning training framework and method for GPU. Background Art

[0002] Reinforcement learning consists of steps such as organizing the operating environment, explaining model inference, optimizing model teaching, and managing the dynamics of the playback buffer. Different steps have different computing loads and resource requirements. Environmental simulations in scientific applications are usually very complex and require a lot of computing resources. Currently, multi-GPU or multi-node distributed simulators are used to accelerate the generation and update of the environment. Heterogeneous computing resources need to cooperate with each other, resulting in cross-device computing performance loss and inability to fully utilize the computing resources of a single node.

[0003] Application Contents

[0004] This application describes a reinforcement learning training framework and method for GPU, which can solve the above technical problems.

[0005] According to a first aspect, a reinforcement learning training framework implemented on a single GPU is provided, the reinforcement learning training framework comprising a configurator, at least one executor, a learner, a model pool and a buffer, wherein the executor and the learner run on the GPU, the model pool and the buffer are set in the video memory of the GPU, and the configurator runs on the CPU;

[0006] The configurator is used to determine the operating parameters of the executor and the operating parameters of the learner, and send the operating parameters of the executor and the operating parameters of the learner to the GPU, so that the GPU starts a corresponding number of GPU threads for the executor according to the operating parameters of the executor, and the operating parameters of the executor include the number of executors;

[0007] At least one of the executors occupies at least one GPU thread, and is used to extract a model from the model pool, perform reasoning using the model to obtain an action, and store the experience data obtained after executing the action in a buffer;

[0008] The learner occupies at least one GPU thread, which is used to extract experience data from the buffer, use the experience data to perform gradient backpropagation and parameter update on the model, and place the updated model into an empty position in the model pool according to the operating parameters of the learner. The operating parameters of the learner include the interval at which the learner places the model into the model pool.

[0009] Based on the above embodiment, the configurator is specifically used to obtain multiple sets of operating parameters of the actuator and the operating parameters of the learner according to the configuration parameters, wherein the operating parameters of the actuator and the operating parameters of the learner are used as a set of operating parameters;

[0010] Each group of operating parameters of the executor and the operating parameters of the learner are sent to the GPU for pre-execution to obtain the operating parameters of the executor and the operating parameters of the learner that require the shortest time to complete the reinforcement learning training.

[0011] Based on the above further embodiment, the configurator is specifically used to calculate the ratio of experience revenue and expenditure k, the number of experience data updatenum updated by the actuator in one step, the number of experience data batch_size used by the learner for a single model iteration, and the ratio formula

[0012]

[0013] Get the relationship between the number of executors n_actors and the interval update_interva at which the learner puts the model into the model pool;

[0014] By using the association relationship, the number n_actors of the plurality of groups of the actuators and the interval at which the learner puts the model into the model pool are obtained.

[0015] Based on the above embodiment, there are multiple positions in the model pool, and the model at each position is taken away by one of the actuators;

[0016] The actuator is used to check whether there is a model at a target position in the model pool after generating a first preset amount of experience data;

[0017] If so, take away the model;

[0018] Otherwise, the actuator enters a waiting state until a model is placed at the extraction target position.

[0019] Based on the above embodiment, the learner is specifically used to put the models into the empty positions in the model pool in order after completing the second preset number of models;

[0020] If a model is stored in one of the positions, the model pool is polled until an empty position is found to put the model.

[0021] Based on the above embodiment, the buffer is used to store the experience data collected by the actuator, and an independent buffer space is allocated to each actuator.

[0022] According to a second aspect, a reinforcement learning training method implemented on a single GPU is provided, wherein the reinforcement learning training framework includes a configurator, at least one executor, a learner, a model pool, and a buffer, wherein the executor and the learner run on the GPU, the model pool and the buffer are set in the video memory of the GPU, and the configurator runs on the CPU;

[0023] The method comprises:

[0024] The configurator determines the operating parameters of the executor and the operating parameters of the learner, and sends the operating parameters of the executor and the operating parameters of the learner to the GPU, so that the GPU starts a corresponding number of GPU threads for the executor according to the operating parameters of the executor, wherein the operating parameters of the executor include the number of executors;

[0025] At least one of the executors occupies at least one GPU thread, extracts a model from the model pool, uses the model to perform reasoning to obtain an action, and stores the experience data obtained after executing the action in a buffer;

[0026] The learner occupies at least one GPU thread, extracts experience data from the buffer, uses the experience data to perform gradient backpropagation and parameter update on the model, and places the updated model into an empty position in the model pool according to the operating parameters of the learner. The operating parameters of the learner include the interval at which the learner places the model into the model pool.

[0027] Based on the above embodiment, the configurator determines the operating parameters of the actuator and the operating parameters of the learner, specifically including:

[0028] The configurator obtains multiple sets of operating parameters of the actuator and the operating parameters of the learner according to the configuration parameters, wherein the operating parameters of the actuator and the operating parameters of the learner are taken as a set of operating parameters;

[0029] Each group of operating parameters of the executor and the operating parameters of the learner are sent to the GPU for pre-execution to obtain the operating parameters of the executor and the operating parameters of the learner that require the shortest time to complete the reinforcement learning training.

[0030] Based on the above embodiment, the configurator calculates the ratio k of the experience income and expenditure, the number of experience data updatenum updated by the actuator in one step, the number of experience data batch_size used by the learner for a single model iteration, and the ratio formula

[0031]

[0032] Get the relationship between the number of executors n_actors and the interval update_interva at which the learner puts the model into the model pool;

[0033] By using the association relationship, the number n_actors of the plurality of groups of the executors and the interval at which the learner puts the model into the model pool are obtained.

[0034] Based on the above embodiment, there are multiple positions in the model pool, and the model at each position is taken away by one of the actuators;

[0035] After generating a first preset amount of experience data, the actuator checks whether there is a model at a target position in the model pool;

[0036] If so, take away the model;

[0037] Otherwise, the actuator enters a waiting state until a model is placed at the extraction target position.

[0038] Based on the above embodiment, after the learner completes the second preset number of models, the models are placed into the empty positions in the model pool in order;

[0039] If a model is stored in one of the positions, the model pool is polled until an empty position is found to put the model.

[0040] Based on the above embodiment, the buffer stores the experience data collected by the executor, and an independent buffer space is allocated to each of the executors.

[0041] According to a third aspect, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the program, the reinforcement learning training method implemented on a single GPU as described in the above technical solution is implemented.

[0042] According to a fourth aspect, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the program, the reinforcement learning training method implemented on a single GPU as described in the above technical solution is implemented.

[0043] In the above-mentioned system and method provided in the embodiments of this specification, it is proposed to utilize a multi-stream mechanism to efficiently utilize a single GPU for reinforcement learning training, improve resource utilization by controlling the execution of different training modules, and at the same time, optimize the training module by utilizing the GPU hardware characteristics, thereby improving the performance of reinforcement learning training on the GPU. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 It is a schematic diagram of a reinforcement learning training framework structure implemented on a single GPU provided by the present invention;

[0046] Figure 2 It is a schematic diagram of the operation flow of a reinforcement learning framework implemented on a single GPU provided by the present invention;

[0047] Figure 3 It is a flowchart of a reinforcement learning training method implemented on a single GPU provided by the present invention. DETAILED DESCRIPTION

[0048] The solution provided in this specification is described below in conjunction with the accompanying drawings.

[0049] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0050] Reinforcement learning consists of steps such as organizing the operating environment, explaining model inference, optimizing model teaching, and managing the dynamics of the playback buffer. Different steps have different computing loads and resource requirements. Environmental simulations in scientific applications are usually very complex and require a lot of computing resources. You can consider using multi-GPU or multi-node distributed simulators to accelerate the generation and update of the environment. Such a training architecture requires heterogeneous computing resources to cooperate with each other, resulting in cross-device computing performance loss. Currently, deep learning frameworks such as Haiku, Flax, PyTorch, and TensorFlow support distributed training, but these deep learning frameworks that support distributed training do not fully utilize the computing resources of a single node.

[0051] There are many unavoidable conditional branches in the execution process of reinforcement learning, and these logical judgments are usually run on the CPU. The parameter update of the neural network, as an intensive calculation, can achieve a good acceleration effect on the GPU. However, the neural network structure of the policy network in the reinforcement learning task is relatively simple, with a small number of parameters, and cannot fully utilize the computing performance of the GPU. Data exchange and synchronization between different devices will also cause performance loss of the GPU.

[0052] The multi-stream mechanism is a technology implemented in modern GPU architectures that optimizes the allocation and utilization of computing resources by allowing multiple independent execution streams to run simultaneously on the same GPU. This mechanism enhances the overlap of computations, especially when managing large-scale parallel computing tasks, significantly improving processing speed and resource efficiency. It effectively reduces the overall task execution time and strengthens the decoupling of computing and data transmission through parallel operations of independent streams, thereby improving the throughput of computing tasks. However, under the multi-stream mechanism, optimizing the computing power of the GPU based on multiple training modules in reinforcement learning is a technical problem that needs to be solved urgently.

[0053] This paper proposes a reinforcement learning training framework on a modern GPU. It uses a multi-stream mechanism to efficiently use a single GPU for reinforcement learning training. It improves resource utilization by controlling the execution of different training modules. At the same time, it optimizes the training modules by utilizing the GPU hardware characteristics, thereby further improving the performance of reinforcement learning training on the GPU.

[0054] like Figure 1 The structure of the reinforcement learning training framework implemented on a single GPU shows that the reinforcement learning training framework includes five modules, executor actor, learner learner, model pool, parallel buffer and configurator. Among them, the executor generates experience data, and the learner uses the experience data generated by the executor to train the agent model. Different from the existing CPU-based reinforcement learning training framework, the GPU-based reinforcement learning training framework adds three modules, namely parallel buffer, model pool and configurator. Among them, the parallel buffer and model pool assist the executor-learner cycle, and the configurator is used to adjust the operating parameters to fully utilize computing resources while reducing training time.

[0055] Reinforcement learning training mainly consists of the following two steps: First, the agent interacts with the environment, explores and obtains a large amount of experience data, which is stored in a parallel buffer. Second, the learner extracts a certain amount of experience data from the parallel buffer and uses this experience data for gradient calculation and updating the neural network parameters.

[0056] The reinforcement learning training framework uses JSON files to configure the training process. The JSON file is divided into a training parameter section and a network parameter section. The network parameter section is used to set the necessary values ​​and control parameters required to implement different neural networks. The training parameter section includes settings related to reinforcement learning training, such as buffer size, number of experience collections, model update frequency, environment exploration rate, and learning rate.

[0057] The reinforcement learning training framework consists of five parts, including five modules: actor, learner, model pool, parallel buffer and configurator.

[0058] The executor actor is responsible for collecting experience data. In the process of collecting each piece of experience data, the executor uses the model obtained by the learner training and the environmental state data to perform reasoning and obtain actions. The executor inputs the obtained actions into the environment, so that the environment steps forward and obtains the next environmental state as well as information such as rewards and termination conditions. In the present invention, one or more GPU threads are assigned to each environment for calculation according to the complexity of the environment. At the same time, the present invention supports multiple GPU threads to start multiple executors to collect experience data together. In the face of an environment with less resources, an actor in the present invention includes two main parts, the auxiliary function part auv and the environment simulation part env. The configuration information provided by the configurator can allow an auxiliary function part auv to cooperate with the simulation of multiple environment simulation parts env at the same time, making full use of GPU resources, so that the executor can obtain more experience data in one experience collection process, and cooperate with multiple executors started by multiple GPU threads to improve the GPU utilization efficiency and experience acquisition efficiency.

[0059] The parallel experience buffer is used to store a large amount of experience data collected by the executor. As a large amount of experience data is obtained, the experience pool is continuously flushed, old experience is eliminated and replaced with new experience. Each executor is allocated independent space so that experience can be stored without interfering with each other.

[0060] The learner is responsible for updating the neural network. The learner randomly extracts multiple experiences from the buffer, uses these experiences to calculate the loss, perform gradient backpropagation and parameter update.

[0061] The model pool is a lock-free circular queue with multiple positions in the queue, and each position corresponds to an executor to extract the model. When the learner learner completes a certain number of models, the models are placed in the empty positions in the queue in order. If it is found that the model previously stored in a certain position has not been taken away by the executor, the queue is polled until an empty position is found to put the model. After generating a certain amount of experience, the executor will check the corresponding empty position, and if there is a model in the empty position, it will be taken away; if it is found that there is no new model in the empty position, it means that the rate at which the executor actor generates experience is greater than the rate at which the learner learns, and the executor enters a waiting state until it can obtain the newly generated model. Using the lock-free model pool, the executor actor and the learner learner can achieve asynchronous execution and make full use of the computing power of the GPU.

[0062] The configurator is used to evaluate the workload of the learner and the executor actor. Before the actual operation, the configurator uses the tools provided by Nvidia to evaluate the GPU resource usage of the executor learner and the learner actor, and gives the recommended configuration with the highest resource utilization. Based on this configuration, the configurator determines the interval at which the learner places parameters in the model pool and the number of executor actors according to the following proportional formula:

[0063]

[0064] In the above formula, the experience income and expenditure ratio k, the number of experiences updated when the executor actor steps together once updatenum, and the number of experiences used by the learner learner for each model iteration batch_size are all configured by the user. update_interval indicates the interval at which the learner learner puts the model into the model pool, and n_actors indicates the number of executors actors.

[0065] The numerator of the formula represents the amount of experience used by the learner each time the model is placed, and the denominator represents the amount of experience generated by an executor each time the model is taken away. k should be greater than or equal to 1, that is, the expectation that each experience generated by the executor actor is used by the learner must be at least 1.

[0066] Under the constraints of the above formula, different update_intervals, i.e., the intervals at which the learner puts the model into the model pool, are calculated for different numbers of executor actors. The configurator configures a number of different executor actors n_actors with small training steps and runs them in a trial run to find the number of executor actors n_actors and the update_interval interval at which the learner puts the model into the model pool with the shortest reinforcement learning completion time. The system provides recommendations on the number of executor actors and the interval at which the learner puts the model into the model pool for the user's parameter configuration from three dimensions: full resource utilization, no waste of computing resources, and fastest computing speed.

[0067] In addition, if Figure 2 As shown in the operating flow diagram of the reinforcement learning training framework implemented on a single GPU, the reinforcement learning training process is a continuous iterative optimization cycle.

[0068] The operation process of the reinforcement learning training framework is as follows:

[0069] Step 1: The configurator evaluates a single run of the learner and executor actors and recommends parameter settings, including the number of executor actors and the interval at which the learner puts the model into the model pool.

[0070] Step 2: All executor actors perform random actions together to fill the parallel experience buffer.

[0071] Step 3: The learner updates the model and puts the new model into the empty space in the model pool every certain number of iterations. At the same time, the executor actor starts reasoning and experience generation, putting the experience into the parallel experience buffer. After generating a certain amount of experience, a new model is taken from the model pool.

[0072] Step 4: Repeat step 3 until the preset number of learner iterations is reached and execute the downstream task.

[0073] like Figure 1 As shown in the schematic diagram of the reinforcement learning training framework implemented on a single GPU, the reinforcement learning training framework includes a configurator, at least one executor, a learner, a model pool and a buffer, wherein the executor and the learner run on the GPU, the model pool and the buffer are set in the video memory of the GPU, and the configurator runs on the CPU;

[0074] A configurator, used to determine the operating parameters of the executor and the operating parameters of the learner, and send the operating parameters of the executor and the operating parameters of the learner to the GPU, so that the GPU starts a corresponding number of GPU threads for the executor according to the operating parameters of the executor, and the operating parameters of the executor include the number of executors;

[0075] At least one executor occupies at least one GPU thread, and is used to extract a model from the model pool, perform reasoning using the model to obtain an action, and store the experience data obtained after executing the action in a buffer;

[0076] The learner occupies at least one GPU thread to extract experience data from the buffer, use the experience data to perform gradient backpropagation and parameter update on the model, and place the updated model into an empty position in the model pool according to the operating parameters of the learner. The operating parameters of the learner include the interval at which the learner places the model into the model pool.

[0077] Based on the above embodiment, the configurator is specifically used to obtain multiple sets of operating parameters of the actuator and the operating parameters of the learner according to the configuration parameters, wherein the operating parameters of the actuator and the operating parameters of the learner are taken as a set of operating parameters;

[0078] The operating parameters of each group of executors and the operating parameters of the learner are sent to the GPU for pre-execution, so as to obtain the operating parameters of the executors and the operating parameters of the learner that require the shortest time to complete the reinforcement learning training.

[0079] Based on the above further embodiment, the configurator is specifically used to calculate the ratio of experience revenue and expenditure k, the number of experience data updatenum updated by the actuator in one step, the number of experience data batch_size used by the learner for a single model iteration, and the ratio formula

[0080]

[0081] Get the relationship between the number of executors n_actors and the interval update_interva at which the learner puts the model into the model pool;

[0082] Using the association relationship, we get the number of multiple groups of executors n_actors and the interval at which the learner puts the model into the model pool.

[0083] Based on the above embodiment, there are multiple positions in the model pool, and the model at each position is taken away by an actuator;

[0084] An executor, configured to check whether there is a model at a target position in the model pool after generating a first preset amount of experience data;

[0085] If so, take away the model;

[0086] Otherwise, the actuator enters a waiting state until a model is placed at the extraction target position.

[0087] Based on the above embodiment, the learner is specifically used to put the models into the empty positions in the model pool in order after completing the second preset number of models;

[0088] If a model is stored in one of the positions, the model pool is polled until an empty position is found to put the model in.

[0089] Based on the above embodiment, the buffer is used to store the experience data collected by the actuator, and an independent buffer space is allocated to each actuator.

[0090] In the above embodiment, it is proposed to use a multi-stream mechanism to efficiently use a single GPU for reinforcement learning training, improve resource utilization by controlling the execution of different training modules, and at the same time, optimize the training modules by utilizing the GPU hardware characteristics, thereby improving the performance of reinforcement learning training on the GPU.

[0091] like Figure 3As shown in the operation flow chart of the reinforcement learning training framework implemented on a single GPU, a reinforcement learning training method implemented on a single GPU, the reinforcement learning training framework includes a configurator, an executor, a learner, a model pool and a buffer, wherein the executor and the learner run on the GPU, the model pool and the buffer are set in the video memory of the GPU, and the configurator runs on the CPU;

[0092] The following steps are involved:

[0093] 110. The configurator determines the operating parameters of the executor and the operating parameters of the learner, and sends the operating parameters of the executor and the operating parameters of the learner to the GPU, so that the GPU starts a corresponding number of GPU threads for the executor according to the operating parameters of the executor, where the operating parameters of the executor include the number of executors.

[0094] 120. At least one executor occupies at least one GPU thread, extracts a model from the model pool, uses the model to perform reasoning to obtain an action, and stores the experience data obtained after executing the action in a buffer;

[0095] 130. The learner occupies at least one GPU thread, extracts experience data from the buffer, uses the experience data to perform gradient backpropagation and parameter update on the model, and places the updated model into an empty position in the model pool according to the operating parameters of the learner. The operating parameters of the learner include the interval at which the learner places the model into the model pool.

[0096] Based on the above embodiment, step 110 specifically includes:

[0097] The configurator obtains multiple sets of operating parameters of the actuators and the operating parameters of the learner according to the configuration parameters, wherein the operating parameters of the actuators and the operating parameters of the learner are taken as a set of operating parameters;

[0098] The operating parameters of each group of executors and the operating parameters of the learner are sent to the GPU for pre-execution, so as to obtain the operating parameters of the executors and the operating parameters of the learner that require the shortest time to complete the reinforcement learning training.

[0099] Based on the above embodiment, step 110 specifically includes:

[0100] The configurator uses the experience income and expenditure ratio k, the number of experience data updatenum that the actuator updates in one step, the number of experience data batch_size used by the learner for a single model iteration, and the ratio formula

[0101]

[0102] Get the relationship between the number of executors n_actors and the interval update_interva at which the learner puts the model into the model pool;

[0103] Using the association relationship, we get the number of multiple groups of executors n_actors and the interval at which the learner puts the model into the model pool.

[0104] Based on the above embodiment, there are multiple positions in the model pool, and the model at each position is taken away by one of the actuators;

[0105] After generating a first preset amount of experience data, the actuator checks whether there is a model at the target position in the model pool;

[0106] If so, take away the model;

[0107] Otherwise, the actuator enters a waiting state until a model is placed at the extraction target position.

[0108] Based on the above embodiment, after the learner completes the second preset number of models, the models are placed into the empty positions in the model pool in order;

[0109] If a model is stored in one of the positions, the model pool is polled until an empty position is found to put the model in.

[0110] Based on the above embodiment, the buffer stores the experience data collected by the executor, and an independent buffer space is allocated to each executor.

[0111] In the above embodiment, it is proposed to use a multi-stream mechanism to efficiently use a single GPU for reinforcement learning training, improve resource utilization by controlling the execution of different training modules, and at the same time, optimize the training modules by utilizing the GPU hardware characteristics, thereby improving the performance of reinforcement learning training on the GPU.

[0112] According to an embodiment of another aspect, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the program, the reinforcement learning training method implemented on a single GPU as in the above technical solution is implemented.

[0113] According to another aspect of the embodiment, there is also provided a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute a reinforcement learning training method implemented on a single GPU.

[0114] Those skilled in the art should be aware that in one or more of the above examples, the functions described in this application can be implemented by hardware, software, firmware, or any combination thereof. When implemented by software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0115] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only the specific implementation methods of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the present application should be included in the scope of protection of the present application.

Claims

1. A reinforcement learning training framework implemented on a single GPU, characterized in that: The reinforcement learning training framework includes a configurator, at least one executor, a learner, a model pool and a buffer, wherein the executor and the learner run on a GPU, the model pool and the buffer are set in a video memory of the GPU, and the configurator runs on a CPU; The configurator is used to determine the operating parameters of the executor and the operating parameters of the learner, and send the operating parameters of the executor and the operating parameters of the learner to the GPU, so that the GPU starts a corresponding number of GPU threads for the executor according to the operating parameters of the executor, and the operating parameters of the executor include the number of executors; At least one of the executors occupies at least one GPU thread, and is used to extract a model from the model pool, perform reasoning using the model to obtain an action, and store the experience data obtained after executing the action in a buffer; The learner occupies at least one GPU thread, which is used to extract experience data from the buffer, use the experience data to perform gradient backpropagation and parameter update on the model, and place the updated model into an empty position in the model pool according to the operating parameters of the learner. The operating parameters of the learner include the interval at which the learner places the model into the model pool.

2. The reinforcement learning training framework according to claim 1, characterized in that: The configurator is specifically used to obtain multiple sets of operating parameters of the actuator and the operating parameters of the learner according to the configuration parameters, wherein the operating parameters of the actuator and the operating parameters of the learner are taken as a set of operating parameters; Each group of operating parameters of the executor and the operating parameters of the learner are sent to the GPU for pre-execution to obtain the operating parameters of the executor and the operating parameters of the learner that require the shortest time to complete the reinforcement learning training.

3. The reinforcement learning training framework according to claim 2, characterized in that: The configurator is specifically used to calculate the ratio of experience revenue and expenditure k, the number of experience data updatenum updated by the actuator in one step, the number of experience data batch size used by the learner for a single model iteration, and the ratio formula Get the relationship between the number of executors n_actors and the interval update_interva at which the learner puts the model into the model pool; By using the association relationship, the number n_actors of the plurality of groups of the executors and the interval at which the learner puts the model into the model pool are obtained.

4. The reinforcement learning training framework according to claim 1, characterized in that: There are multiple positions in the model pool, and the model at each position is taken away by one of the actuators; The actuator is used to check whether there is a model at a target position in the model pool after generating a first preset amount of experience data; If so, take away the model; Otherwise, the actuator enters a waiting state until a model is placed at the extraction target position.

5. The reinforcement learning training framework according to claim 4, characterized in that: The learner is specifically used to place the models into empty positions in the model pool in order after completing the second preset number of models; If a model is stored in one of the positions, the model pool is polled until an empty position is found to put the model.

6. The reinforcement learning training framework according to claim 1, characterized in that: The buffer is used to store the experience data collected by the actuator, and an independent buffer space is allocated to each actuator.

7. A reinforcement learning training method implemented on a single GPU, characterized in that: The reinforcement learning training framework includes a configurator, at least one executor, a learner, a model pool and a buffer, wherein the executor and the learner run on a GPU, the model pool and the buffer are set in a video memory of the GPU, and the configurator runs on a CPU; The method comprises: The configurator determines the operating parameters of the executor and the operating parameters of the learner, and sends the operating parameters of the executor and the operating parameters of the learner to the GPU, so that the GPU starts a corresponding number of GPU threads for the executor according to the operating parameters of the executor, wherein the operating parameters of the executor include the number of executors; At least one of the executors occupies at least one GPU thread, extracts a model from the model pool, uses the model to perform reasoning to obtain an action, and stores the experience data obtained after executing the action in a buffer; The learner occupies at least one GPU thread, extracts experience data from the buffer, uses the experience data to perform gradient backpropagation and parameter update on the model, and places the updated model into an empty position in the model pool according to the operating parameters of the learner. The operating parameters of the learner include the interval at which the learner places the model into the model pool.

8. The reinforcement learning training method according to claim 7, characterized in that: The configurator determines the operating parameters of the actuator and the operating parameters of the learner, specifically including: The configurator obtains multiple sets of operating parameters of the actuator and the operating parameters of the learner according to the configuration parameters, wherein the operating parameters of the actuator and the operating parameters of the learner are taken as a set of operating parameters; Each group of operating parameters of the executor and the operating parameters of the learner are sent to the GPU for pre-execution to obtain the operating parameters of the executor and the operating parameters of the learner that require the shortest time to complete the reinforcement learning training.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the reinforcement learning training method implemented on a single GPU as described in any one of claims 7-8 is implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the reinforcement learning training method implemented on a single GPU as described in any one of claims 7-8 is implemented.

Images