An efficient reinforcement learning training acceleration system and method

Abstract

The application discloses a high-efficiency reinforcement learning training acceleration system and method, and belongs to the acceleration field of reinforcement learning. The reinforcement learning training acceleration system comprises a software system on a CPU and a hardware system on an FPGA. The reinforcement learning training acceleration method implemented on the system comprises the following steps: CPU-side data preprocessing, CPU-side data sending to the FPGA, FPGA-side data caching, FPGA-side parallel environment calculation and calculation result caching, CPU-side reading of the calculation result data from the FPGA, and CPU-side unpacking of the result data for training model updating. The method uses local storage technology, pipeline technology and time division multiplexing technology to improve the operation speed, realizes hardware acceleration of reinforcement learning environment updating, has high universality due to modularization and parameterization design, and finally accelerates the reinforcement learning training process by accelerating the reinforcement learning environment updating.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of accelerating reinforcement learning, in particular to accelerating environment update of reinforcement learning. BACKGROUND

[0002] Reinforcement Learning (RL) is a method of machine learning, unlike traditional machine learning methods that rely on labeled training data for model training, reinforcement learning does not require pre-labeled example data, it learns by interacting between agents and environments to obtain the data needed for training models. The agent improves its action strategy by obtaining rewards or punishments from the interaction with the environment. Important elements include: Figure 1 State of the environment: State is a way to describe the environment, which contains information about the current situation of the environment. The state can be discrete (for example, the position of the chess pieces on the chessboard) or continuous (for example, the sensor data of the robot).

[0003] Action of the agent: Action is the behavior or decision that the agent can take in a certain state. The action set is usually discrete or continuous, depending on the specific problem.

[0004] Reward obtained by the action of the agent: Reward is the feedback signal provided by the environment to the agent to evaluate the behavior of the agent. The goal of the agent is to maximize the cumulative reward. The reward can be positive (reward), negative (penalty) or zero, indicating the degree of good or bad of the agent's action in a certain state.

[0005] Policy of the agent: Policy is the decision rule of the agent, which maps the state to the action. The policy defines how the agent selects actions in different states. The ultimate goal of reinforcement learning is to learn a good policy, which can be deterministic (each state has a determined action, generally the action with the highest value is selected) or random (each state has a certain probability to select different actions). Deterministic policy selection has higher computational efficiency, but it is easy to fall into local optimal solution and has low spatial exploration; completely random policy selection has low computational efficiency, but it can increase the spatial exploration.

[0006] In reinforcement learning, the process of continuous interaction between the environment and the agent is as follows

[0007] Figure 2 ​As shown, the agent makes a choice of the next action according to the current state of the environment and the existing strategy; the environment provides the observation value of the agent to the environment according to the influence of the action of the agent on the environment, the reward corresponding to the action of the agent, and whether the task is cut off.

[0008] Therefore, reinforcement learning faces the key challenge of balancing exploration (randomly selecting actions) and utilization (selecting actions according to existing data), and in order to improve the efficiency of data collection, a commonly used method is to use multiple agents and environments to interact with each other simultaneously, and quickly accumulate a large amount of experience, such as Figure 3 A diagram of multiple agents and multiple environments interacting. However, updating a large number of environments in parallel requires a large amount of computing resources.

[0009] Because reinforcement learning involves a variety of types of environments and long logical chains of interaction, current reinforcement learning environment updates are mostly performed on CPUs that are more flexible in computing, but the hardware architecture of the CPU itself is not suitable for large-scale parallel computing; for some environments, people have also proposed a scheme for parallel environments on GPUs, but different types of environments may involve complex update logic, and it is difficult to efficiently deploy the parallel environment on the GPU.

[0010] FPGA (Field Programmable Gate Arrays) is a semi-custom circuit that can be reprogrammed multiple times, and it has flexibility and reprogrammability, which can easily instantiate multiple identical computing circuits, and is very suitable for reinforcement learning environment parallel computing. Therefore, it is of great significance to study how to implement reinforcement learning environment parallel computing on FPGA. SUMMARY

[0011] To solve the above problems of the prior art, the present application provides an efficient reinforcement learning training acceleration system and method, which uses data compression, pipeline design, time division multiplexing and other techniques to achieve efficient reinforcement learning environment update during training. Because reinforcement learning involves many types of environment tasks, the application uses modular and parameterized design techniques to easily deploy different types of reinforcement learning tasks. The application uses FPGA to instantiate multiple environment computing circuits, with a single environment computation as a minimum parallel computing unit, to efficiently update the environment during reinforcement learning training.

[0012] The technical solutions of the present application are as follows:

[0013] The application discloses a high-efficiency reinforcement learning training acceleration system which is composed of a software part and a hardware part, wherein the software part is realized on a CPU and comprises a data packaging module, a data transmission module, a data control module and a data unpacking module, and is used for connecting neural network forward propagation, model training and environment updating in a reinforcement learning training process; the hardware part is realized on an FPGA and comprises a data transmission module, a parallel computing control module, a data buffer module and an environment computing module, and is used for realizing parallel computing of environment updating on the hardware.

[0014] The software part, the data packaging module is used for compressing and packaging initial state of the environment in the reinforcement learning training process and action information of an agent according to a current strategy into a byte form, so as to contain more environment information with less bytes; the data transmission module is used for data transmission between the CPU end and the FPGA end, and is used for sending the packaged byte data from the CPU end to the FPGA end and reading the byte data from the FPGA end on the CPU end; the data control module controls the data transmission module to read data from the FPGA end after the environment computing on the FPGA end is completed; and the data unpacking module unpacks the byte data read from the FPGA end into a data format required in the training process.

[0015] The hardware part, the data transmission module of the CPU end software part is connected with the data transmission module of the FPGA end, is used for data transmission between the CPU end and the FPGA end, and is used for receiving data sent by the CPU end; the data buffer module comprises register and memory BRAM two storage types of storage media, and is used for storing received data, intermediate data in a computing process and computed data; the parallel computing control module is used for controlling data flow between the data buffer module and the environment computing module on the FPGA end; and the environment computing module is composed of a plurality of single-environment computing modules, each single-environment computing module corresponds to a piece of hardware resource capable of computing a single environment, and takes current state of the environment and action of an agent corresponding to the environment as input, and takes next-stage state of the environment, observation value of the agent to the environment, reward obtained by the agent and whether a current task is cut off as output of the environment computing module.

[0016] Further, the modules of the software part are realized through software programming codes.

[0017] Further, the environment computing module of the hardware part is realized through hardware programming codes Verilog codes.

[0018] Further, the parallel computing control module is realized through hardware programming codes Verilog codes.

[0019] Further, the data transmission module of the hardware part is realized by calling the pre-designed and reusable hardware module Xilinx IP core XDMA.

[0020] The application further provides a high-efficiency reinforcement learning training acceleration method realized on the system, wherein in a reinforcement learning training task, the calculation of m parallel reinforcement learning environments is performed to obtain m sets of training data for updating a training model, so as to accelerate the reinforcement learning training; during the training process, the software part CPU end sends m actions to the hardware part FPGA end at each calculation step step, the FPGA calculates the calculation results of the m environments at the calculation step step, the CPU reads the calculation results of the m environments after the calculation is completed, and the calculation results are used for training the model; after repeating a plurality of steps, the trained model reaches an expected training effect, which indicates that the reinforcement learning training task is completed; wherein the calculation of the m environments at the calculation step step includes the following steps:

[0021] 1) CPU end data preprocessing, when the calculation step step is the first step of the parallel environment, the initialization of the environment is required at this step, the data packaging module of the software part splices the state information of the m parallel environments and the action information of the m agents corresponding to the m parallel environments in the training process, and finally adds a signal bit: "1"; when the calculation step step is a standard step of a non-first step of the parallel environment, the state information of the parallel environment has been stored in the FPGA, and the data packaging module of the software part only needs to splice the action information of the agents corresponding to the m environments in the training process, and finally adds a signal bit: "2"; the data is compressed, spliced and packaged into byte form;

[0022] 2) Data transmission, the data transmission module of the software part sends the data processed in step 1) from the CPU end to the data transmission module of the FPGA end;

[0023] 3) Buffering the data received by the FPGA end, the data buffering module of the hardware part saves the received data in the BRAM, and reads the data into the register when the data is used, and the specific working process is as follows:

[0024] When the FPGA receives a signal "1", it indicates that this is the first step of the round, and the parallel computing control module reads the initial state of the environment and the action information of the agent in the received data, reads the initial state information of the environment into the register, and when the register is full, the parallel computing control module reads the information in the full register into the memory BRAM; when all the initial state information of the environment is written, the parallel computing control module reads and stores all the action information of the agent into the register; by storing the state information of the environment locally in the FPGA, the data transfer between the CPU and the FPGA is reduced;

[0025] When the FPGA receives a signal "2", it indicates that this step is a standard step, i.e. the parallel computing control module only reads the action information of the agent in the received data; reads and stores all the action information of the agent into the register;

[0026] 4) The FPGA performs parallel computing of the environment and caching of the computing results, and the parallel computing control module controls the data flow between the data buffer module and the environment computing module in the FPGA; the environment computing module is composed of n single environment computing modules, and each single environment computing module corresponds to a piece of hardware resource capable of computing a single environment, i.e. corresponds to a number of computing cores; the specific number of computing cores n is determined by the number of clock cycles a required for computing a single environment and the number of clock cycles b required for writing all the computed data back to the data buffer, n = a / b;

[0027] In the computing phase of the environment and the caching phase of the computing results, the self-termination technology, the pipeline technology and the time division multiplexing technology are used to divide all the m environments to be parallel computed into m / n batches, and after each batch is computed, the computing result of the batch starts to be cached while the next batch of environment starts to be computed, until the update of all the environments is completed; the computing phase of the environment is the first stage of the pipeline, and the caching phase of the computing result is the second stage of the pipeline; specifically including the following steps:

[0028] 4-1) The environment computing module reads the current state, the initial state, the cutoff signal of the previous state of the environment and the action signal of the corresponding agent of n corresponding environments from the data buffer module storing the current state of the m environments, the initial state of the environment, the cutoff signal of the previous state of the environment and the action signal of the corresponding agent;

[0029] 4-2) When the environment corresponding to the stop signal is a digital signal "0", the FPGA takes the value of the current state register of the environment as the state input of the environment in step 4-3) environment calculation module, and when the environment corresponding to the stop signal is a digital signal "1", the FPGA takes the value of the initial state register of the environment as the state input of the environment in step 4-3) environment calculation module. The state of the environment used in step 4-3) is obtained after this step. This design enables the corresponding environment to be automatically reset to the initial state if the round of the environment in the n parallel environments ends, without the need for manual reset;

[0030] 4-3) n cores start parallel computing environments, the input of the calculation is the state of the n environments and the action of the corresponding n agents, and the output is the next state of the n environments, the observation value of the agent to the environment, the reward of the action of the agent in this step, and whether the round is terminated. After a clock cycle, the calculation of the n environments is completed, and the calculation results of the n environments are cached in the buffer area at the same time, and the calculation of the second batch of n environments also starts. The next state information of the environment and the information whether the round is terminated are stored in the BRAM specially used to store the current state of the environment in the FPGA, and are used as input when the new action of the agent is transmitted. The observation value of the agent to the environment, the reward of the action of the agent in this step, and whether the round is terminated are cached in the buffer area waiting to be read by the CPU;

[0031] When the update calculation and the calculation result caching of all m environments are completed, step 5) is entered.

[0032] 5) After waiting for the completion of the calculation of all the environments, the data control module controls the CPU side data transmission module to read data from the FPGA side;

[0033] 6) Data unpacking, the data unpacking module unpacks the observation value of the agent to the environment, the reward of the action, and the information whether the round is terminated from the byte form to the data type required for training;

[0034] The above steps 1) to 6) complete the calculation of one calculation step in the parallel m-environment reinforcement learning training process. The CPU uses the m sets of data generated in this step to update the training model, and the updated model generates m sets of actions according to the state information of the m sets of environments sent back, and then repeats steps 1) to 6), until the updated model can achieve the expected training effect, that is, one reinforcement learning training task is completed.

[0035] The technical effects of the present application are as follows:

[0036] This invention provides a highly efficient reinforcement learning training acceleration system and method, achieving hardware acceleration for reinforcement learning environment updates. It enables parallel environment updates within the hardware. For the specific task of reinforcement learning environment updates, it utilizes local storage, pipelined technology, and time-division multiplexing to fully utilize computing and storage resources, thereby improving computational speed. This invention employs a modular and parameterized design, making it applicable to various types of reinforcement learning environments and algorithms, demonstrating strong versatility. By accelerating the environment update process, this invention ultimately accelerates the reinforcement learning training process. Attached Figure Description

[0037] Figure 1 The difference between traditional machine learning (left) and reinforcement learning (right);

[0038] Figure 2 To enhance the interaction graph between the environment and the agent during the learning and training process;

[0039] Figure 3 This is a schematic diagram of a parallel reinforcement learning environment;

[0040] Figure 4 This is a schematic diagram of the hardware and software components of the system of the present invention;

[0041] Figure 5 This is a flowchart of the method of the present invention;

[0042] Figure 6 This is a schematic diagram of the specific data flow during the calculation phase of this invention;

[0043] Figure 7 This is a schematic diagram of the reinforcement learning task of "balance bar on the car" in an embodiment of the present invention. Detailed Implementation

[0044] The present invention will be further clearly and completely described below with reference to the accompanying drawings and specific embodiments.

[0045] This invention proposes a highly efficient reinforcement learning training acceleration system, consisting of software and hardware components, such as... Figure 4 As shown, the data packaging module, data transmission module, data control module, and data unpacking module in the software are all implemented using Python code (a type of software programming code). This invention adopts a parameterized design, which can adapt to different types of environmental tasks and different numbers of parallel environments. The hardware is implemented on an FPGA and includes a data transmission module, a parallel computing control module, a data buffer module, and an environment computing module, used to implement parallel computing for environment updates on the hardware.

[0046] The software part, the data packaging module is used for data compression and packaging into byte form according to the initial state of the environment in the reinforcement learning training process and the action information of the agent according to the current policy, so that more environment information is contained with less byte number; the data transmission module is used for data transmission between the CPU end and the FPGA end, and the packaged byte data is sent from the CPU end to the FPGA end, and the byte data is also read from the FPGA end by the CPU end; the data control module controls the data transmission module to read data from the FPGA end after the calculation of the environment on the FPGA end is completed; and the data unpacking module unpacks the byte data read from the FPGA end into a data format required in the training process.

[0047] The hardware part, the data transmission module and the data transmission module of the CPU end software part are connected, used for data transmission between the CPU end and the FPGA end, receiving data sent by the CPU end; the data buffer module includes two storage types of storage media, registers and memories BRAM, used for storing received data, intermediate data in the calculation process and calculated data; the parallel computing control module is used for controlling the data flow between the data buffer module and the environment calculation module of the FPGA end; the environment calculation module is composed of a plurality of single environment calculation modules, each single environment calculation module corresponds to a piece of hardware resource capable of calculating a single environment, and for different environment tasks, the current state of the environment and the agent corresponding to the environment are taken as inputs, and the next stage state of the environment, the observation value of the agent to the environment, the reward obtained by the agent and whether the current task is cut off are taken as the outputs of the environment calculation module.

[0048] In the embodiment, the data transmission module of the hardware part is realized by calling the XDMA IP core of Xilinx (the IP core is a pre-designed and verified hardware module, which can accelerate and simplify the FPGA design process, improve the system performance and development efficiency, the XDMA is a PCIe IP core, and the PCIe is a high-speed data transmission mode between different hardware).

[0049] The environment calculation module and the parallel computing control module are realized by Verilog code (a kind of hardware programming code), and the modular and parameterized design is helpful to adapt to different environment tasks and different parallel degrees.

[0050] Figure 5 The flowchart of the method of the application is as follows: the efficient reinforcement learning training acceleration method realized on the system described in the application is introduced below.

[0051] The balance bar on the car is a classic reinforcement learning problem, and the task is as follows: Figure 7As shown, a wooden pole is vertically placed on the dolly, and the dolly is controlled to move left and right to keep the wooden pole upright. The dolly falling down marks the end of a round. In this task, the environment has 4 states, which are the position x of the wooden pole, the angle θ of the wooden pole, the velocity of the wooden pole and the angular velocity of the wooden pole The agent, i.e., the dolly, has 2 actions, which are moving the dolly to the left and moving the dolly to the right. If the position x of the wooden pole is between -4.8 and +4.8, and the angle θ of the wooden pole is between ±12°, it is considered that the wooden pole has not fallen down. In this embodiment, it takes 80 clock cycles to calculate a single dolly environment, and it takes 4 clock cycles to write the calculation result of a single environment into a buffer.

[0052] For this reinforcement learning training task, 160 such "pole balancing on the dolly" environments are designed in parallel to calculate, and 160 groups of training data are obtained to update the training model to accelerate the training of reinforcement learning. During the training process, the software part CPU end sends 160 actions to the hardware part FPGA end at each calculation step step, the FPGA calculates the calculation results of 160 environments at this calculation step step, and the CPU reads the calculation results of 160 environments after the calculation is completed to train the model. After repeating several steps, the trained model reaches the expected training effect, marking the completion of a reinforcement learning training task. The calculation of one calculation step step of 160 environments includes the following steps:

[0053] 1) CPU end data preprocessing. When the calculation step step is the first step of the parallel environment, this step needs to initialize the environment. The data packaging module of the software part splices the state information of 160 parallel environments and the action information of the corresponding 160 agents in the training process, and finally adds a signal bit: "1". When the calculation step step is a standard step of the parallel environment, the state information of the parallel environment has been stored in the FPGA. The data packaging module of the software part only needs to splice the action information of the corresponding 160 agents of the environment in the training process, and finally adds a signal bit: "2". After data compression and splicing, it is packaged into byte form;

[0054] 2) Data transmission. The data transmission module of the software part sends the data processed in step 1) from the CPU end to the data transmission module of the FPGA end;

[0055] 3) Buffering the data received by the FPGA end. The data buffering module of the hardware part saves the received data in the BRAM, and reads it into the register when needed. The specific working process is as follows:

[0056] When the FPGA receives the signal "1", it indicates that this is the first step of the round, and the parallel computing control module reads the initial state of the environment and the action information of the agent in the received data, reads the initial state information of the environment into the register, and when the register is full, the parallel computing control module reads the information in the register that has been written full into the memory BRAM; when all the initial state information of the environment is written, the parallel computing control module reads and stores all the action information of the agent into the register; by storing the state information of the environment locally in the FPGA, the data transfer between the CPU and the FPGA is reduced.

[0057] When the FPGA receives the signal "2", it indicates that this step is a standard step, i.e. the parallel computing control module only reads the action information of the agent in the received data; reads and stores all the action information of the agent into the register;

[0058] 4) The FPGA performs parallel computing of the environment and caching of the computing results, and the parallel computing control module controls the data flow between the data buffer module and the environment computing module in the FPGA; the environment computing module is composed of n single environment computing modules, and each single environment computing module corresponds to a piece of hardware resource capable of computing a single environment, i.e. each single environment computing module corresponds to a computing core number. The specific number of computing cores n is determined by the number of clock cycles a required for the training task to compute a single environment and the number of clock cycles b required for writing all the computed data back to the data buffer, n = a / b; in this embodiment, 80 clock cycles are required to compute a single car environment, and 4 clock cycles are required to write the computing results of a single environment into the buffer, so the number of computing cores required n = 80 / 4 = 20; in the environment computing stage and the computing result caching stage, the self-termination technology, the pipeline technology and the time division multiplexing technology are used to divide all the 160 environments to be parallel computed into 8 batches (160 / 20 = 8), and after each batch is computed, the computing of the next batch of environments starts at the same time as the caching of the computing results of the batch starts, until the update of all the environments is completed; the environment computing stage is the first stage of the pipeline, and the computing result caching stage is the second stage of the pipeline; including the following steps:

[0059] 4-1) The environment computing module reads the current state, the initial state of the environment, the cutoff signal of whether the previous state of the environment is cutoff and the action signal of the corresponding agent of 20 corresponding environments from the data buffer module storing the current state of 160 environments, the initial state of the environment, the cutoff signal of whether the previous state of the environment is cutoff and the action signal of the corresponding agent.

[0060] 4-2) Among the 20 environment information read, if the environment corresponding cutoff signal is digital signal "0", the FPGA takes the value of the current state register of the environment as the state input of the environment calculation module, if the environment corresponding cutoff signal is digital signal "1", the FPGA takes the value of the initial state register of the environment as the state input of the environment calculation module. This step can simultaneously obtain whether the state of the 20 environments used in step 4-3) is the current environment state or the initial environment state. This design makes the corresponding environment automatically reset to the initial state if the round of the environment in the n parallel environments ends, without the need for manual reset.

[0061] 4-3) 20 cores start parallel computing, the input is the state of 20 environments and the corresponding 20 agent actions, the output is the next step state of 20 environments (that is, the position, velocity, angular velocity and angular acceleration of the wooden rod after the car moves), the observation value of the agent to the environment (when the training task is a globally observable task, the agent obtains all the state information of the environment, and the observation value of the agent to the environment is the same as the state of the environment, when it is a partially observable task, the observation value of the agent to the environment is only part of the state of the environment, because this embodiment is a globally observable task, the agent can obtain all the state information of the environment, so in this task, the observation value of the agent to the environment is the same as the state of the environment), the reward of the agent's action at this step (whether the wooden rod falls after the car moves, if not, it will get a reward of +1) and whether the round is terminated (if the wooden rod falls, it is considered as round termination).

[0062] After 80 clock cycles, the calculation of 20 environments is completed, and the calculation results of the 20 environments are cached to the buffer area at the same time, and the calculation of the second batch of 20 environments also starts. Among them, the next step state information of the environment and the information whether the round is terminated are stored in the BRAM specially used to store the current environment state in the FPGA, waiting for the new action of the agent to be transmitted as input. The observation value of the agent to the environment, the reward of the agent's action at this step and whether the round is terminated will be cached in the buffer area waiting to be read by the CPU. Figure 6 The arrow in the figure shows the data flow of the calculation state. The current state of the environment and the corresponding agent action information are transmitted into the environment calculation module synchronously, and after the environment calculation module calculates, the next state of the environment is obtained and stored in the corresponding data buffer area; the observation information of the agent to the environment, the reward value corresponding to the action and the information whether the current round is cut off are cached, waiting to be read by the CPU.

[0063] After completing the update calculation and calculation result caching of all 160 environments, step 5) is entered.

[0064] 5) After all the calculations of the environment are completed, the data control module controls the CPU-side data transmission module to read data from the FPGA side.

[0065] 6) Data unpacking, the data unpacking module disassembles the observation value of the agent to the environment, the reward of the action and the information of whether the round is cut off from the byte form read by the CPU from the FPGA into the floating point type required for training.

[0066] The above steps 1) to step 6) complete the calculation of one calculation step in the reinforcement learning training process of 160 “balance rods on the car”. The CPU uses the 160 groups of data generated in this step to update the training model. The updated model generates 160 groups of actions according to the 160 groups of state information sent back, and then repeats steps 1) to 6), until the updated model can achieve the expected training effect, that is, a reinforcement learning training task is completed.

[0067] Through the above steps, the classic environment of 160 “balance rods on the car” can be calculated in parallel in the FPGA, and cooperated with other training codes. Compared with other general methods, the training time is reduced by 18% while reaching the same training effect after 12 million environment updates. Through the parameterization and modularization design, various types and parallel degrees of reinforcement learning environments can be easily deployed on the FPGA for parallel calculation, which has strong universality and low requirement for the hardware knowledge of users.

[0068] In the embodiments of the present application, the parallel update of the “balance rod on the car” task is taken as an example, but the present application is not limited to a certain reinforcement learning task, and is also applicable to the environment update of other reinforcement learning tasks; not limited to parallel environments, but also applicable to the environment update of a single reinforcement learning task. Although the present application emphasizes the training acceleration of reinforcement learning, it is also applicable to the inference process of reinforcement learning. It can be applied to application scenarios involving reinforcement learning environment update.

[0069] Finally, it should be noted that the purpose of publishing the embodiments is to help further understand the present application, so that those skilled in the art can understand that various substitutions and modifications are possible without departing from the spirit and scope of the present application and the appended claims. Therefore, the present application should not be limited to the disclosed content of the embodiments, and the scope of protection claimed by the present application is subject to the scope defined by the claims.

Claims

1. A system for efficient reinforcement learning training acceleration, the system comprising: The system comprises a software part and a hardware part, the software part is implemented on a CPU and comprises a data packaging module, a data transmission module, a data control module and a data unpacking module, and is used for connecting neural network forward propagation in a reinforcement learning training process, model training and environment updating performed in hardware; the hardware part is implemented on an FPGA and comprises a data transmission module, a parallel computing control module, a data buffer module and an environment computing module, and is used for implementing parallel computing of environment updating on hardware. The software part, the data packaging module is used for data compression and packaging in byte form according to initial state of the environment in the reinforcement learning training process and action information of the agent according to a current policy, so as to contain more environment information with fewer bytes. The data transmission module is used for data transmission between the CPU end and the FPGA end, and is used for sending the packaged byte data from the CPU end to the FPGA end and reading byte data from the FPGA end by the CPU end; the data control module controls the data transmission module to read data from the FPGA end after the environment computing of the FPGA end is completed; and the data unpacking module unpacks the byte data read from the FPGA end into a data format required in the training process. The hardware part, the data transmission module of the CPU end software part is connected, is used for data transmission between the CPU end and the FPGA end, and receives data sent by the CPU end; the data buffer module comprises register and memory BRAM two storage types of storage media, and is used for storing received data, intermediate data in a computing process and computed data; the parallel computing control module is used for controlling data flow between the data buffer module and the environment computing module of the FPGA end; and the environment computing module is composed of a plurality of single-environment computing modules, each single-environment computing module corresponds to a piece of hardware resource capable of computing a single environment, and inputs a current state of the environment and an action of the agent corresponding to the environment, and outputs a next stage state of the environment, an observation value of the agent to the environment, a reward obtained by the agent and whether a current task is cut off.

2. The system of claim 1, wherein the modules of the software part are implemented through software programming codes.

3. The system of claim 1, wherein the environment computing module of the hardware part is implemented through hardware programming codes Verilog codes.

4. The system of claim 1, wherein the parallel computing control module is implemented through hardware programming codes Verilog codes.

5. The system of claim 1, wherein the data transmission module of the hardware part is implemented by calling a pre-designed reusable hardware module Xilinx IP core XDMA. The system is characterized in that, in a reinforcement learning training task, parallel computing of m environments is performed to obtain m groups of training data for updating a training model, so as to accelerate reinforcement learning training.

6. A method of training acceleration for efficient reinforcement learning implemented on a system as in claim 1, ​ ​ In the training process, the software part CPU end sends m actions to the hardware part FPGA end at each calculation step, the FPGA calculates the calculation results of m environments at this calculation step, and the CPU reads the calculation results of m environments after the calculation is completed to train the model. After repeating several steps, the trained model achieves the expected training effect, indicating the completion of a reinforcement learning training task; wherein, the completion of the calculation of m environments at each calculation step includes the following steps: 1) CPU data preprocessing, when the calculation step is the first step of the parallel environment, this step needs to initialize the environment, the data packaging module of the software part splices the state information of m parallel environments and the action information of the corresponding m agents in the training process, and finally adds a signal bit: "1"; when the calculation step is a standard step of a parallel environment that is not the first step, the state information of the parallel environment has been stored in the FPGA, and the data packaging module of the software part only needs to splice the action information of the corresponding agent of m environments in the training process, and finally add a signal bit: "2"; compress the data, splice the compressed data, and pack it into byte form; 2) Data transmission, the data transmission module of the software part sends the data processed in step 1) from the CPU end to the data transmission module of the FPGA end; 3) Buffering the data received by the FPGA end, the data buffering module of the hardware part saves the received data in the BRAM, and reads it into the register when needed, the specific working process is as follows: When the FPGA receives the signal "1", it indicates that this is the first step of the round, and the parallel calculation control module reads the initial state of the environment and the action information of the agent in the received data, reads the initial state information of the environment into the register, and when the register is full, the parallel calculation control module reads the information in the full register into the memory BRAM; when the initial state information of all environments is written, the parallel calculation control module reads and stores the action information of all agents into the register; by storing the state information of the environment locally in the FPGA, the data transfer between the CPU and the FPGA is reduced; When the FPGA receives the signal "2", it indicates that this step is a standard step, that is, the parallel calculation control module only reads the action information of the agent in the received data; read and store the action information of all agents into the register; 4) FPGA parallel environment calculation and calculation result buffering, the parallel calculation control module controls the calculation of the FPGA internal ​ ​ ​ According to the data flow between the buffer module and the environment calculation module; the environment calculation module is composed of a single environment calculation module, one single environment calculation module corresponds to a piece of hardware resource capable of calculating a single environment, that is, corresponds to a number of calculation cores; the specific number of calculation cores is determined by the number of clock cycles required for calculating a single environment and the number of clock cycles b required for writing all the calculated data back to the data buffer, ; In the computing phase of the environment and the caching phase of the computing result, all m environments to be computed in parallel are divided into m / n batches by using self-termination technology, pipeline technology and time division multiplexing technology. After each batch is computed, the computing of the next batch of environment is started at the same time when the caching of the computing result of the batch is started, until the updating of all environments is completed. The computing phase of the environment is the first stage of the pipeline, and the caching phase of the computing result is the second stage of the pipeline. The specific steps include the following steps: 4-1) The environment calculation module reads the current state of the m-th environment, the initial state of the m-th environment, the cutoff signal of whether the previous state of the m-th environment is cutoff, and the action signal of the m-th agent from the data buffer module in which the current state of the m-th environment, the initial state of the m-th environment, the cutoff signal of whether the previous state of the m-th environment is cutoff, and the action signal of the m-th agent are stored. the current state of the m-th environment, the initial state of the m-th environment, the cutoff signal of whether the previous state of the m-th environment is cutoff, and the action signal of the m-th agent. 4-2) read When the corresponding cut-off signal of the environment is digital signal "0", the FPGA takes the value of the corresponding environment in the current state register of this environment as the state input of the environment in the environment calculation module in step 4-3), and when the corresponding cut-off signal of the environment is digital signal "1", the FPGA takes the value of the corresponding environment in the initial state register of this environment as the state input of the environment in the environment calculation module in step 4-3). After this step, the state of the environment used in the calculation of the environment in step 4-3) is obtained at the same time. This design makes it possible for the corresponding environment to automatically reset to the initial state if the round of the environment in one of the parallel environments ends, without the need for manual reset. ​ 4-3) A parallel computing environment is started with one core, and the input for the computation is... The state of an environment and its corresponding The action of an agent, the output is The next state of the environment, the agent's observations of the environment, the agent's reward for this action, and whether the round ends; after a clock cycles, The computation for this environment is complete, and this is cached. At the same time as the calculation results of the first environment are sent to the buffer, the second batch... The computation of the environment also begins; the state information of the environment in the next step and the information of whether the round has ended are stored in the BRAM inside the FPGA, which is specially used to store the current state of the environment, waiting for the new action of the agent to be passed in as input; the agent's observation of the environment, the reward of the agent's action in this step and whether the round has ended are cached in a buffer waiting to be read by the CPU. After the updating computation and the caching of the computing result of all m environments are completed, step 5) is entered. 5) After the computation of all environments is completed, the data control module controls the CPU-side data transmission module to read data from the FPGA side; 6) Data unpacking: the data unpacking module unpacks the observation value of the agent to the environment, the reward of the action and the information whether the round is cut off from the byte form read by the CPU from the FPGA into the data types required for training. The above steps 1) to 6) complete the computation of one computing step step in the reinforcement learning training process of parallel m environments. The CPU uses the m groups of data generated in this step to update the training model. The updated model generates m groups of actions according to the state information of the m groups of environments sent back. Then steps 1) to 6) are repeated until the updated model reaches the expected training effect, that is, one reinforcement learning training task is completed. ​ ​

Images