Distributed reinforcement learning system and data processing method and device

Abstract

The embodiment of the invention provides a distributed reinforcement learning system and a data processing method and device, and is suitable for processing a plurality of training tasks in parallel. A main node and working nodes in the distributed reinforcement learning system process any round of any training task in the following mode: any working node is used for collecting empirical data through a collecting process and sending the empirical data to the main node; the main node is used for calling the training task to calculate the empirical data collected by any working node in the GPU corresponding to the round after receiving the empirical data collected by any working node; after the to-be-updated parameters of the training task in the round are obtained through calculation, the GPU corresponding to the training task in the round is released, and the to-be-updated parameters of the training task in the round are sent to each working node; any working node is further used for updating the model parameters corresponding to the training tasks stored in the working node according to the to-be-updated parameters of the training tasks in the round.

Description

Technical Field

[0001] The present invention relates to the field of reinforcement learning technology, and in particular to a distributed reinforcement learning system, a data processing method and a device. Background Art

[0002] Distributed reinforcement learning training is a method in which the master node distributes the training process of the reinforcement learning algorithm to multiple worker nodes to accelerate training and improve model performance.

[0003] In existing distributed reinforcement learning training, if multiple training tasks are trained simultaneously, each training task needs to occupy a GPU exclusively for a long time, which will lead to low utilization of GPU computing resources and cause serious waste of GPU computing resources.

[0004] In summary, how to improve the utilization rate of GPU computing resources while training multiple training tasks at the same time is a technical problem that needs to be solved urgently. Summary of the invention

[0005] The embodiments of the present invention provide a distributed reinforcement learning system, a data processing method and a device, which are used to solve the problem in the prior art that the utilization rate of GPU computing resources is low under the premise of simultaneously training multiple training tasks.

[0006] In a first aspect, an embodiment of the present invention provides a distributed reinforcement learning system suitable for processing multiple training tasks in parallel; any training task is used to generate target parameters of a corresponding model after multiple rounds of iteration; the master node and each working node in the distributed reinforcement learning system process any round of any training task in the following manner: any working node is used to collect experience data through an acquisition process and send the experience data to the master node; the master node is used to, after receiving the experience data collected by any working node, call the GPU corresponding to the training task in this round to calculate the experience data collected by any working node; after obtaining the parameters to be updated of the training task in this round by calculation, release the GPU corresponding to the training task in this round and send the parameters to be updated of the training task in this round to each working node; any working node is also used to update the model parameters corresponding to the training task stored in itself according to the parameters to be updated of the training task in this round.

[0007] Optionally, any working node is also used to close the collection process after sending the experience data to the master node; the master node is also used to issue a preheating instruction to any working node according to the historical iteration time of the historical round corresponding to the training task; the preheating instruction is used to instruct any working node to start the next round of collection process according to the preheating time; the preheating time is the time for any working node to start the next round of collection process in advance.

[0008] Optionally, the master node is also used to determine whether the training task has a corresponding GPU in this round after receiving the experience data collected by any working node for the training task in this round; if not, call an idle GPU from the GPU resource pool as the GPU of the training task in this round; if yes, calculate the experience data collected by any working node through the GPU of the training task in this round.

[0009] In a second aspect, an embodiment of the present invention provides a data processing method, which is applicable to a distributed reinforcement learning system that processes multiple training tasks in parallel, wherein any training task is used to generate target parameters of a corresponding model after multiple rounds of iterations, including:

[0010] For any round of any training task, after receiving the experience data collected by any working node, the master node in the distributed reinforcement learning system calls the GPU corresponding to the training task in this round to calculate the experience data collected by any working node; after obtaining the parameters to be updated of the training task in this round through calculation, releases the GPU corresponding to the training task in this round and sends the parameters to be updated of the training task in this round to each working node; the parameters to be updated of the training task in this round are used to update the model parameters corresponding to the training task stored in each working node itself.

[0011] Optionally, after receiving the experience data collected by any working node, the master node in the distributed reinforcement learning system calls the GPU corresponding to the training task in this round to calculate the experience data collected by any working node, including: after receiving the experience data collected by any working node for the training task in this round, the master node in the distributed reinforcement learning system determines whether the training task has a corresponding GPU in this round; if not, calls an idle GPU from the GPU resource pool as the GPU of the training task in this round; if yes, calculates the experience data collected by any working node through the GPU of the training task in this round.

[0012] Optionally, after receiving the experience data collected by any working node, the master node in the distributed reinforcement learning system calls the training task to calculate the experience data collected by any working node on the GPU corresponding to this round, including: after receiving the experience data collected by the i-th working node, the master node in the distributed reinforcement learning system determines whether the gradient calculation of the experience data collected by the j-th working node is completed; if it has not yet ended, wait until the gradient calculation is completed, and call the training task to perform gradient calculation on the experience data collected by the i-th working node on the GPU corresponding to this round; the time when the j-th working node sends the collected experience data to the master node is earlier than the time when the i-th working node sends the collected experience data to the master node; after the experience data collected by each working node completes their own gradient calculation, the parameters to be updated of the training task in this round are determined according to the gradient calculation results of each working node.

[0013] Optionally, before the master node in the distributed reinforcement learning system receives the experience data collected by any working node, it also includes: the master node in the distributed reinforcement learning system issues a preheating instruction to any working node according to the historical iteration time of the historical round corresponding to the training task; the preheating instruction is used to instruct any working node to start the next round of collection process according to the preheating time; the preheating time is the time for any working node to start the next round of collection process in advance.

[0014] Optionally, the preheating time can be determined in the following manner: the master node selects a target historical iteration time that meets the stability condition from the historical iteration time of multiple historical rounds corresponding to the training task; determines the start time of the next round of each working node based on the target historical iteration time; determines the preheating time of each working node based on the start time of the next round of each working node.

[0015] In the third aspect, an embodiment of the present invention also provides a data processing device, which is suitable for a distributed reinforcement learning system that processes multiple training tasks in parallel, wherein any training task is used to generate target parameters of a corresponding model after multiple rounds of iterations, including: an acquisition unit is used to, for any round of any training task, after receiving the experience data collected by any working node, call the GPU corresponding to the training task in this round to calculate the experience data collected by any working node; a processing unit is used to release the GPU corresponding to the training task in this round and send the parameters to be updated of the training task in this round to each working node after obtaining the parameters to be updated of the training task in this round through calculation; the parameters to be updated of the training task in this round are used to update the model parameters corresponding to the training task stored in each working node itself.

[0016] Optionally, the acquisition unit is specifically used for: after receiving the experience data collected by any working node for the training task in this round, the master node in the distributed reinforcement learning system determines whether the training task has a corresponding GPU in this round; if not, calling an idle GPU from the GPU resource pool as the GPU of the training task in this round; if yes, calculating the experience data collected by any working node through the GPU of the training task in this round.

[0017] Optionally, the acquisition unit is specifically used for: after receiving the experience data collected by the i-th working node, the master node in the distributed reinforcement learning system determines whether the gradient calculation of the experience data collected by the j-th working node is completed; if it has not yet ended, wait until the gradient calculation is completed, and call the GPU corresponding to the training task in this round to perform gradient calculation on the experience data collected by the i-th working node; the time when the j-th working node sends the collected experience data to the master node is earlier than the time when the i-th working node sends the collected experience data to the master node; after the experience data collected by each working node have completed their own gradient calculation, determine the parameters to be updated of the training task in this round according to the gradient calculation results of each working node.

[0018] Optionally, the acquisition unit is also used for: the master node in the distributed reinforcement learning system issues a preheating instruction to any working node according to the historical iteration time of the historical round corresponding to the training task; the preheating instruction is used to instruct any working node to start the next round of acquisition process according to the preheating time; the preheating time is the time for any working node to start the next round of acquisition process in advance.

[0019] Optionally, the preheating time can be determined in the following manner: the master node selects a target historical iteration time that meets the stability condition from the historical iteration time of multiple historical rounds corresponding to the training task; determines the start time of the next round of each working node based on the target historical iteration time; determines the preheating time of each working node based on the start time of the next round of each working node.

[0020] In a fourth aspect, an embodiment of the present application further provides a computer device, comprising: a memory for storing programs; a processor for calling the programs stored in the memory, and executing a data processing method as in the second aspect according to the obtained programs.

[0021] In a fifth aspect, an embodiment of the present application further provides a computer-readable non-volatile storage medium, comprising a computer-readable program. When a computer reads and executes the computer-readable program, the computer executes a data processing method as in the second aspect.

[0022] In a sixth aspect, an embodiment of the present application provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer device, the computer device executes the steps of a data processing method in the second aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0024] Figure 1 A schematic diagram of data transmission between a master node and each working node provided in an embodiment of the present invention;

[0025] Figure 2 A schematic diagram of the structure of a distributed reinforcement learning system provided by an embodiment of the present invention;

[0026] Figure 3 A schematic diagram of the GPU ratio in each time period provided by an embodiment of the present invention;

[0027] Figure 4 A schematic diagram of GPU ratios of multiple training tasks provided in an embodiment of the present invention;

[0028] Figure 5 A schematic diagram of the proportion of the startup overhead time of the acquisition process to the entire training process in an existing distributed reinforcement learning training provided by an embodiment of the present invention;

[0029] Figure 6 A schematic diagram of period distribution of distributed reinforcement learning provided by an embodiment of the present invention;

[0030] Figure 7 A schematic diagram of the time overhead ratio of a training task with a preheating mechanism and without a preheating mechanism provided in an embodiment of the present invention;

[0031] Figure 8 A schematic diagram of data transmission between a master node and each working node provided in an embodiment of the present invention;

[0032] Fig. 9 A schematic diagram of the proportion of data transmission and training time for distributed reinforcement learning provided by an embodiment of the present invention;

[0033] Fig.10 A flow chart of a data processing method provided by an embodiment of the present invention;

[0034] Fig.11 A schematic diagram of the structure of a data processing device provided by an embodiment of the present invention;

[0035] Fig.12 A schematic diagram of the structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] Reinforcement Learning (RL) is an important branch of machine learning that focuses on how to enable intelligent agents to learn optimal strategies through interaction with the environment to maximize long-term rewards. Reinforcement learning has a wide range of applications in many fields, such as game artificial intelligence (such as chess, Go, Atari games), robot control, autonomous driving, recommendation systems, etc. It provides a general learning framework that enables intelligent agents to continuously improve their decision-making ability through interaction with the environment.

[0038] Distributed reinforcement learning training is a method to accelerate training and improve model performance by distributing the training process of reinforcement learning algorithms to multiple computing nodes. It takes advantage of distributed computing to process large-scale training data and explore the environment in parallel, thereby speeding up learning and obtaining better strategies.

[0039] The following introduces an existing distributed reinforcement learning system training process. For ease of understanding, the following takes a training task as an example to describe the existing distributed reinforcement learning system training process:

[0040] The first step is to initialize parameters on the master node and synchronize the parameters to each working node. Then, multiple collection processes are created on each working node. Each collection process creates an independent environment instance for interacting with the environment.

[0041] In the second step, the collection process on each working node interacts with its own environment instance independently, collects experience data from the environment instance, and uploads the experience data to the master node. Since each working node completes experience data collection at different times, the master node needs to wait for the slowest working node to complete experience data collection before proceeding to the third step. Figure 1 .

[0042] In the third step, the master node calculates the parameters to be updated based on the empirical data collected from each working node, and sends the parameters to be updated to each working node.

[0043] In the fourth step, each working node restarts multiple collection processes and replaces the parameters in the collection process with the parameters to be updated.

[0044] Step 5: Repeat steps 2 to 5 until the preset number of iterations is reached or the stopping condition is met.

[0045] Among them, for the second step, since the time points at which each working node completes the collection of experience data are inconsistent, the master node needs to wait for the slowest working node to complete data collection before proceeding to the third step, which is very time-consuming and inefficient.

[0046] In the second step, after each working node sends the experience data to the master node, the working node will close its own collection process. Then, in the fourth step, when the master node sends the parameters to be updated to the working node, the working node will restart the collection process. Restarting the collection process will take a long time, resulting in a relatively large time overhead, which reduces the efficiency of computing resources.

[0047] Since the above is distributed reinforcement learning training for one training task, in actual application, multiple training tasks will be trained simultaneously in distributed reinforcement learning to obtain the target parameters of the corresponding model. Each training task needs to occupy a GPU for a long time, which will result in low utilization of GPU computing resources.

[0048] In summary, the embodiments of the present invention provide a distributed reinforcement learning system to solve the problems in the prior art of large time overhead and low efficiency in the second step, low utilization rate of computing resources in the fourth step, and low utilization rate of GPU computing resources when occupying the GPU.

[0049] like Figure 2As shown, a distributed reinforcement learning system provided by an embodiment of the present invention is suitable for processing multiple training tasks in parallel; any training task is used to generate target parameters of the corresponding model after multiple rounds of iteration; the master node 201 and each working node 202 in the distributed reinforcement learning system 200 process any round of any training task in the following manner: any working node 202 is used to collect experience data through the collection process and send the experience data to the master node 201; the master node 201 is used to call the GPU corresponding to the training task in this round to calculate the experience data collected by any working node 202 after receiving the experience data collected by any working node 202; after obtaining the parameters to be updated of the training task in this round through calculation, the GPU corresponding to the training task in this round is released and the parameters to be updated of the training task in this round are sent to each working node 202; any working node 202 is also used to update the model parameters corresponding to the training task stored in itself according to the parameters to be updated of the training task in this round.

[0050] In the embodiment of the present invention, since the training task needs to be iterated for multiple rounds, in the existing technology, any training task always occupies a GPU during the training process, and the GPU computing resources will not be released until the training task is completed. However, in actual use, except for periodic usage peaks, GPU resources are idle most of the time. Figure 3 , Figure 3 Used to represent the real-time GPU utilization for training tasks with different configurations. Lines of different colors represent training tasks with different configurations. It can be observed from the figure that regardless of the training task, the GPU utilization shows a periodic peak, indicating that for traditional RL training, only a small part of the time is used for GPU training, resulting in a large amount of waste of GPU resources during troughs. Therefore, in any round of this application, when the master node receives the empirical data collected by any working node, it will call the training task on the GPU corresponding to that round to calculate the empirical data. After the calculation is completed and the parameters to be updated are obtained, the GPU corresponding to the training task in that round will be released for use by other training tasks, thereby achieving a significant improvement in the utilization of the GPU.

[0051] For example, to execute four identical 16 acquisition processes in parallel, with a step size of 512, using three GPUs, see Figure 4 , Figure 4The training performance of multiple training tasks sharing the GPU is shown. Compared with a single training load (FaaS-Steps512-N16) that runs the same task alone, the training load of multiple tasks sharing GPU resources (FaaS-Quadruple-A ~ FaaS-Quadruple-D) has only about 10% more clock synchronization overhead, but compared with the existing solution Ray (Ray-Steps 512-N16), the performance difference is less than 5%. Since Ray monopolizes the GPU for a long time during training, at least 25% of GPU resources can be saved. Under the premise of more reasonable utilization of fragmented GPU idle time, theoretically GPU resources can be saved up to 50%.

[0052] Optionally, any working node is also used to close the collection process after sending the experience data to the master node; the master node is also used to issue a preheating instruction to any working node according to the historical iteration time of the historical rounds corresponding to the training task; the preheating instruction is used to instruct any working node to start the next round of collection process according to the preheating time; the preheating time is the time for any working node to start the next round of collection process in advance.

[0053] Optionally, the warm-up time can be determined in the following manner: the master node selects a target historical iteration time that meets the stability condition from the historical iteration time of multiple historical rounds corresponding to the training task; determines the next round start time of each working node based on the target historical iteration time; determines the warm-up time of each working node based on the next round start time of each working node.

[0054] In the embodiment of the present invention, after the working node sends the experience data to the master node, it will close the collection process. Then, after receiving the parameters to be updated sent by the master node, the working node needs to restart the collection process so that the next round of experience data can be collected through the collection process. When restarting the collection process, there will be a relatively large time overhead, which will reduce the efficiency of computing resources. Figure 5 , the embodiment of the present invention provides a conventional distributed reinforcement learning training in which the acquisition process startup overhead time accounts for the proportion of the entire training process time. Figure 5 The vertical axis is the proportion of startup overhead to the entire round of time overhead. Figure 5 It can be seen that the startup overhead of the acquisition process can even take up more than 20% of the training time. Since any training task requires multiple rounds of iteration, this repetitive overhead will greatly reduce the training speed and reduce the efficiency of computing resource utilization.

[0055] Therefore, this application takes into account that distributed reinforcement learning has significant periodicity, that is, any round of iteration process consists of four parts: restarting the collection process, collecting experience data, uploading experience data, and calculating the parameters to be updated. Figure 6 . Since the running time of each part of distributed reinforcement learning is relatively stable, combined with this periodicity, the present invention designs a preheating mechanism to solve the problem of large time overhead. Specifically, the master node can issue a preheating instruction to any working node according to the historical iteration time of the historical rounds corresponding to the training task, wherein the preheating instruction is used to instruct any working node to start the next round of acquisition process according to the preheating time. In this way, the working node starts the next round of acquisition process in advance, and when the master node sends the parameters to be updated to the working node, the working node can directly update its own parameters according to the parameters to be updated, and collect through the acquisition process, avoiding the problem of large time overhead caused by restarting the acquisition process after the working node obtains the parameters to be updated.

[0056] For example, reinforcement learning training workloads with a fixed step size of 512 steps and multiple batch sizes of 8, 16 and dynamically adjusted number of worker processes were selected, and each workload was run continuously for 200 iterations. Figure 7 The figure shows the time taken for each iteration without the preheating mechanism (left) and with the preheating mechanism (right). Figure 7 The different colored lines in the figure respectively show the time taken by restarting the collection process, uploading the experience data, collecting the experience data, and calculating the parameters to be updated. This is the time taken by each part in each round.

[0057] The preheating mechanism of this application significantly reduces the time overhead of starting a work process. Thanks to lightweight analysis and adjustment of dynamic preheating indicators, it is possible to maintain extremely low overhead, and dynamically adjust the preheating indicators in the scenario of dynamically adjusting the number of work processes, so that the startup overhead is stably maintained at a low level. Compared with the baseline performance, it can reduce the startup time overhead by 10%.

[0058] Optionally, the master node is also used to determine whether the training task has a corresponding GPU in this round after receiving the experience data collected by any working node for the training task in this round; if not, call an idle GPU from the GPU resource pool as the GPU for the training task in this round; if yes, calculate the experience data collected by any working node through the GPU of the training task in this round.

[0059] In the embodiment of the present invention, in order to improve the utilization efficiency of the GPU, after the master node calculates the parameters to be updated in each round of the training task, it will release the GPU corresponding to the training task in that round to facilitate use by other training tasks.

[0060] For example, training task A corresponds to a master node and multiple worker nodes, namely worker node 1, worker node 2, and worker node 3. In the third round of training task A, the master node first receives the experience data collected by worker node 2, and needs to determine whether there is a corresponding GPU in the third round of training task A. Since training task A is in the third round, the master node receives the experience data collected by the worker node for the first time. Therefore, training task A does not have a corresponding GPU in the third round, and it is necessary to call an idle GPU from the GPU resource pool as the GPU of training task A in the third round to calculate the experience data sent by worker node 2. Then the master node receives the experience data collected by working node 1, and needs to determine whether there is a corresponding GPU in the third round of training task A, where there is a corresponding GPU in the third round of training task A. Therefore, the master node calculates the experience data collected by working node 1 through the GPU corresponding to the third round of training task A. Similarly, when the master node receives the experience data collected by working node 3, it needs to determine whether there is a corresponding GPU in the third round of training task A. Where there is a corresponding GPU in the third round of training task A, therefore, the master node calculates the experience data collected by working node 3 through the GPU corresponding to the third round of training task A.

[0061] Optionally, the master node is also used to determine whether the gradient calculation of the experience data collected by the jth work node is completed after receiving the experience data collected by the i-th work node; if it is not completed, wait until the gradient calculation is completed, and call the training task to perform gradient calculation on the experience data collected by the i-th work node on the GPU corresponding to this round; after the experience data collected by each work node completes their own gradient calculation, determine the parameters to be updated of the training task in this round according to the gradient calculation results of each work node. The time when the jth work node sends the collected experience data to the master node is earlier than the time when the i-th work node sends the collected experience data to the master node.

[0062] In the embodiment of the present invention, since the calculation of the experience data collected by the master node for each working node can be divided into two parts, the first part is to perform gradient calculation on the experience data collected by each working node respectively, and the second part is to take the average value according to the gradient calculation of each working node after the experience data collected by each working node has completed their own gradient calculation to obtain the parameters to be updated of the training task in this round. Therefore, the present application takes into account the characteristics of these two parts, and the master node does not need to wait until all working nodes have completed collecting experience data before starting the calculation. After receiving the experience data collected by any working node, the master node can perform gradient calculation on the experience data of the working node. For details, see Figure 8 .

[0063] For example, training task A corresponds to worker node 1 and worker node 2, where worker node 1 sends experience data to the master node earlier than worker node 2 sends experience data to the master node. Therefore, the master node first receives the experience data collected by worker node 1, and then calls training task A to perform gradient calculation on the experience data collected by worker node 1 on the GPU corresponding to this round. Then the master node receives the experience data collected by worker node 2, and the master node needs to determine whether the gradient calculation of the experience data collected by worker node 1 is completed. If it has not yet ended, it is necessary to wait until the gradient calculation of the experience data collected by worker node 1 is completed, and then call training task A to perform gradient calculation on the experience data collected by worker node 2 on the GPU corresponding to this round. After the master node completes the gradient calculation of worker nodes 1 and 2, the average value of the gradient calculation of worker nodes 1 and 2 can be used to obtain the parameters to be updated for training task A in this round.

[0064] For example, we selected 16 parallel worker processes and different sampling steps (128, 256, 512). We calculated the data transfer and training time under different batch sizes. Fig. 9 The pipeline optimization performance compared with the existing solution is shown. It can be seen that the entire process of data transmission and policy network training benefits from pipeline parallel optimization, and the data transmission performance is improved. In addition, since part of the data transmission and policy network training can be parallelized through the pipeline, the required policy network training time is also reduced after the data transmission is completed. The effect of pipeline parallel optimization becomes more obvious as the batch size increases, and the maximum performance optimization can reach 10% under the experimental setting.

[0065] Optionally, after receiving the experience data collected by the pending work node set n composed of multiple work nodes, the master node determines whether the gradient calculation of the experience data collected by the pending work node set k is completed. If it is not completed, it waits until the gradient calculation is completed, and calls the training task in the round corresponding to the GPU to perform gradient calculation on the experience data collected by the pending work node set k. After the experience data collected by each work node completes their own gradient calculation, the training task in the round is determined according to the gradient calculation results of each work node. The time when the pending work node set k sends the collected experience data to the master node is earlier than the time when the pending work node set n sends the collected experience data.

[0066] like Fig.10 FIG. 1 is a flow chart of a data processing method provided by an embodiment of the present invention, and the method comprises the following steps:

[0067] Step 1001, for any round of any training task, after receiving the experience data collected by any working node, the master node in the distributed reinforcement learning system calls the GPU corresponding to the training task in this round to calculate the experience data collected by any working node.

[0068] Step 1002, after obtaining the parameters to be updated of the training task in this round through calculation, release the GPU corresponding to the training task in this round and send the parameters to be updated of the training task in this round to each working node.

[0069] It can be seen from the above steps 1001 to 1002 that any training task needs to go through multiple rounds of iterations. For any round, when the master node receives the experience data collected by any working node, it will call the GPU corresponding to the training task in this round to calculate the experience data. After the calculation is completed and the parameters to be updated are obtained, the GPU corresponding to the training task in this round will be released for use by other training tasks, thereby greatly improving the utilization rate of the GPU.

[0070] Based on the same technical concept, the present application embodiment provides a data processing device, which can be seen in Fig.11 The device 1100 is suitable for a distributed reinforcement learning system that processes multiple training tasks in parallel, and any training task is used to generate target parameters of a corresponding model after multiple rounds of iterations, including: an acquisition unit 1101 is used for, for any round of any training task, after receiving the experience data collected by any working node, calling the GPU corresponding to the training task in this round to calculate the experience data collected by any working node; a processing unit 1102 is used for, after obtaining the parameters to be updated of the training task in this round by calculation, releasing the GPU corresponding to the training task in this round and sending the parameters to be updated of the training task in this round to each working node; the parameters to be updated of the training task in this round are used to update the model parameters corresponding to the training task stored in each working node.

[0071] Optionally, the acquisition unit 1101 is specifically used for: after receiving the experience data collected by any working node for the training task in this round, the master node in the distributed reinforcement learning system determines whether the training task has a corresponding GPU in this round; if not, calling an idle GPU from the GPU resource pool as the GPU of the training task in this round; if yes, calculating the experience data collected by any working node through the GPU of the training task in this round.

[0072] Optionally, the acquisition unit 1101 is specifically used for: after receiving the experience data collected by the i-th working node, the master node in the distributed reinforcement learning system determines whether the gradient calculation of the experience data collected by the j-th working node is completed; if it has not yet ended, wait until the gradient calculation is completed, and call the GPU corresponding to the training task in this round to perform gradient calculation on the experience data collected by the i-th working node; the time when the j-th working node sends the collected experience data to the master node is earlier than the time when the i-th working node sends the collected experience data to the master node; after the experience data collected by each working node have completed their own gradient calculation, determine the parameters to be updated of the training task in this round according to the gradient calculation results of each working node.

[0073] Optionally, the acquisition unit 1101 is also used for: the master node in the distributed reinforcement learning system issues a preheating instruction to any working node according to the historical iteration time of the historical round corresponding to the training task; the preheating instruction is used to instruct any working node to start the next round of acquisition process according to the preheating time; the preheating time is the time for any working node to start the next round of acquisition process in advance.

[0074] Optionally, the preheating time can be determined in the following manner: the master node selects a target historical iteration time that meets the stability condition from the historical iteration time of multiple historical rounds corresponding to the training task; determines the start time of the next round of each working node based on the target historical iteration time; determines the preheating time of each working node based on the start time of the next round of each working node.

[0075] Based on the same technical concept, the embodiment of the present application provides a computer device 1200, such as Fig.12 As shown, it includes at least one processor 1201 and a memory 1202 connected to the at least one processor. The specific connection medium between the processor 1201 and the memory 1202 is not limited in the embodiment of the present application. Fig.12 For example, the processor 1201 and the memory 1202 are connected via a bus. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0076] In the embodiment of the present application, the memory 1202 stores instructions that can be executed by at least one processor 1201, and the at least one processor 1201 can perform the steps of the above-mentioned data processing method by executing the instructions stored in the memory 1202.

[0077] Among them, the processor 1201 is the control center of the computing device, and can use various interfaces and lines to connect various parts of the computing device, and realize the processing of the data processing method by running or executing instructions stored in the memory 1202 and calling data stored in the memory 1202. Optionally, the processor 1201 may include one or more processing units, and the processor 1201 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor can also be integrated into the processor 1201. In some embodiments, the processor 1201 and the memory 1202 can be implemented on the same chip, and in some embodiments, they can also be implemented separately on independent chips.

[0078] Processor 1201 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware processor for execution, or can be executed by a combination of hardware and software modules in the processor.

[0079] Memory 1202, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules. Memory 1202 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (Random Access Memory, RAM), static random access memory (Static Random Access Memory, SRAM), programmable read-only memory (Programmable Read Only Memory, PROM), read-only memory (Read Only Memory, ROM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), magnetic memory, disk, optical disk, etc. Memory 1202 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer device, but is not limited thereto. The memory 1202 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0080] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program executable by a computer device. When the program runs on the computer device, the computer device executes the steps of the above-mentioned data processing method.

[0081] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0082] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0083] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0084] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0085] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A distributed reinforcement learning system, characterized in that: Suitable for processing multiple training tasks in parallel; any training task is used to generate the target parameters of the corresponding model after multiple rounds of iteration; The master node and each worker node in the distributed reinforcement learning system process any round of any training task in the following manner: Any working node is used to collect experience data through the collection process and send the experience data to the master node; The master node is used to, after receiving the experience data collected by any working node, call the GPU corresponding to the training task in this round to calculate the experience data collected by any working node; after obtaining the parameters to be updated of the training task in this round through calculation, release the GPU corresponding to the training task in this round and send the parameters to be updated of the training task in this round to each working node; Any working node is also used to update the model parameters corresponding to the training task stored in itself according to the parameters to be updated of the training task in this round.

2. The system according to claim 1, characterized in that Any working node is further used to close the collection process after sending the experience data to the master node; The master node is also used to issue a preheating instruction to any working node according to the historical iteration time of the historical round corresponding to the training task; the preheating instruction is used to instruct any working node to start the next round of collection process according to the preheating time; the preheating time is the time for any working node to start the next round of collection process in advance.

3. The system according to claim 1, characterized in that The master node is also used to determine whether the training task has a corresponding GPU in this round after receiving the experience data collected by any working node for the training task in this round; if not, call an idle GPU from the GPU resource pool as the GPU of the training task in this round; if yes, calculate the experience data collected by any working node through the GPU of the training task in this round.

4. A data processing method, characterized in that: A distributed reinforcement learning system suitable for processing multiple training tasks in parallel. Any training task is used to generate the target parameters of the corresponding model after multiple rounds of iteration, including: For any round of any training task, after receiving the experience data collected by any working node, the master node in the distributed reinforcement learning system calls the GPU corresponding to the training task in this round to calculate the experience data collected by any working node; after obtaining the parameters to be updated of the training task in this round through calculation, releases the GPU corresponding to the training task in this round and sends the parameters to be updated of the training task in this round to each working node; the parameters to be updated of the training task in this round are used to update the model parameters corresponding to the training task stored in each working node itself.

5. The method according to claim 4, characterized in that After receiving the experience data collected by any working node, the master node in the distributed reinforcement learning system calls the training task to calculate the experience data collected by any working node on the GPU corresponding to the round, including: After receiving the experience data collected by any working node for the training task in this round, the master node in the distributed reinforcement learning system determines whether the training task has a corresponding GPU in this round; if not, an idle GPU is called from the GPU resource pool as the GPU of the training task in this round; if yes, the experience data collected by any working node is calculated through the GPU of the training task in this round.

6. The method according to claim 4, characterized in that After receiving the experience data collected by any working node, the master node in the distributed reinforcement learning system calls the training task to calculate the experience data collected by any working node on the GPU corresponding to the round, including: After receiving the experience data collected by the i-th working node, the master node in the distributed reinforcement learning system determines whether the gradient calculation of the experience data collected by the j-th working node is completed; if it is not completed, wait until the gradient calculation is completed, and call the training task in the round corresponding to the GPU to perform gradient calculation on the experience data collected by the i-th working node; the time when the j-th working node sends the collected experience data to the master node is earlier than the time when the i-th working node sends the collected experience data to the master node; After the experience data collected by each working node have completed their respective gradient calculations, the parameters to be updated for the training task in this round are determined according to the gradient calculation results of each working node.

7. The method according to any one of claims 4 to 6, characterized in that Before the master node in the distributed reinforcement learning system receives the experience data collected by any working node, the method further includes: The master node in the distributed reinforcement learning system issues a preheating instruction to any working node according to the historical iteration time of the historical round corresponding to the training task; the preheating instruction is used to instruct any working node to start the next round of acquisition process according to the preheating time; the preheating time is the time for any working node to start the next round of acquisition process in advance.

8. The method according to claim 7, characterized in that The warm-up time can be determined in the following ways: The master node selects a target historical iteration time that meets the stability condition from the historical iteration times of multiple historical rounds corresponding to the training task; Determine the next round of start time of each working node according to the target historical iteration time; The preheating time of each working node is determined according to the next round of startup time of each working node.

9. A data processing device, characterized in that: A distributed reinforcement learning system suitable for processing multiple training tasks in parallel. Any training task is used to generate the target parameters of the corresponding model after multiple rounds of iteration, including: The acquisition unit is used for, for any round of any training task, after receiving the experience data collected by any working node, calling the GPU corresponding to the training task in this round to calculate the experience data collected by any working node; The processing unit is used to release the GPU corresponding to the training task in this round and send the parameters to be updated of the training task in this round to each working node after obtaining the parameters to be updated of the training task in this round through calculation; the parameters to be updated of the training task in this round are used to update the model parameters corresponding to the training task stored in each working node itself.

10. A computer device, characterized in that: include: Memory for storing computer programs; A processor, configured to call a computer program stored in the memory, and execute the steps of the method according to any one of claims 4 to 8 according to the obtained program.

Images