Efficient reinforcement learning training acceleration system and method
By performing reinforcement learning environment parallel computing on FPGA, using data compression and pipeline technology, the problem of large-scale environmental update computing resources in the existing technology is solved, and efficient environmental update and training acceleration is achieved.
Patent Information
- Application Number
- CN202411097118.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-12
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-12
AI Technical Summary
The existing reinforcement learning environment updates take up too much computing resources when they are in progress on the CPU, making it difficult to efficiently deploy on the GPU, especially under complex parallel logic, resulting in inefficient computing.
FPGA is used for parallel computing of reinforcement learning environments, and efficient parallel computing of environmental updates is achieved through data compression, pipeline design and time division multiplexing technology, combined with modular and parameterized design.
Parallel computing of environment updates is implemented on hardware, making full use of computing resources and storage resources, improving computing speed, and is suitable for a variety of types of reinforcement learning environments and algorithms, significantly accelerating the training process.
Smart Images

Figure CN119067185A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of acceleration of reinforcement learning, and in particular to acceleration of environment update of reinforcement learning. Background Art
[0002] Reinforcement Learning (RL) is a machine learning method. Figure 1 As shown in the figure, unlike traditional machine learning methods that rely on labeled training data for model training, reinforcement learning training does not require pre-labeled example data. It obtains the data needed for training models through the interaction between the agent and the environment. The agent obtains rewards or penalties for actions through interaction with the environment and continuously improves its action strategy. Its important elements include:
[0003] The state of the environment: A state is a way of describing the environment that contains information about the current situation of the environment. The state can be discrete (e.g., the position of a chess piece on a chessboard) or continuous (e.g., sensor data from a robot).
[0004] Action of the agent: An action is a behavior or decision that the agent can take in a certain state. The set of actions is usually discrete or continuous, depending on the specific problem.
[0005] Reward for the agent's actions: Reward is the feedback signal provided by the environment to the agent to evaluate the agent's behavior. The agent's goal is to maximize the cumulative reward. Reward can be positive (reward), negative (penalty), or zero, indicating how good or bad the agent is at taking an action in a certain state.
[0006] The strategy (Policy) for the agent to generate actions: The strategy is the decision-making rule of the agent, which maps the mapping relationship from state to action. The strategy defines how the agent chooses actions in different states. The ultimate goal of reinforcement learning is to learn a good strategy. The strategy can be deterministic (there is a certain action in each state, generally choosing the action with the highest value) or random (there is a certain probability of choosing different actions in each state). Although deterministic strategy selection has high computational efficiency, it is easy to fall into the situation of local optimal solution and has low exploration of space; completely random strategy selection has low computational efficiency, but it can increase the exploration of space.
[0007] In reinforcement learning, the process of continuous interaction between the environment and the agent is like Figure 2As shown, the agent chooses the next action based on the current state of the environment and the existing strategy; the environment chooses the next action based on the agent's observation of the environment, the reward corresponding to the agent's action, and whether the task is expired.
[0008] Therefore, reinforcement learning faces the key challenge of balancing exploration (randomly selecting actions) and utilization (selecting actions based on existing data). In order to improve the efficiency of data collection, a common method is to use a parallel environment, where multiple agents interact with the environment at the same time to quickly accumulate a large amount of experience, such as Figure 3 A graph of multiple agents interacting with multiple environments. However, updating a large number of environments in parallel requires a lot of computational resources.
[0009] Since reinforcement learning involves a variety of environment types and a long interactive logic chain, the current reinforcement learning environment update is mostly performed on the CPU, which has more flexible computing. However, the CPU's hardware architecture is not suitable for large-scale parallel computing. For some environments, people have also proposed parallel environments on GPUs. However, different types of environments may involve complex update logic, and it is difficult to efficiently deploy parallel environments on GPUs.
[0010] FPGA (Field Programmable Gate Arrays) is a semi-custom circuit that can be reprogrammed multiple times. It is flexible and reprogrammable. It can easily instantiate multiple copies of the same computing circuit, which is very suitable for environmental parallel computing of reinforcement learning. Therefore, it is of great significance to study how to implement environmental parallel computing of reinforcement learning on FPGA. Summary of the invention
[0011] In view of the problems existing in the above-mentioned prior art, the present invention proposes an efficient reinforcement learning training acceleration system and method. For the specific scenario of reinforcement learning environment update, data compression, pipeline design, time division multiplexing and other techniques are used to realize efficient reinforcement learning environment update during training. Since many types of environmental tasks may be encountered in reinforcement learning, the present invention uses modularization and parameterization and other design techniques to achieve the effect of easily deploying different types of reinforcement learning tasks; the present invention uses FPGA to instantiate computing circuits of multiple environments, and uses a single environment calculation as a minimum parallel computing unit to efficiently update the environment during reinforcement learning training.
[0012] The technical solution of the present invention is as follows:
[0013] An efficient reinforcement learning training acceleration system is composed of a software part and a hardware part. The software part is implemented on a CPU, including a data packaging module, a data transmission module, a data control module and a data unpacking module, and is used to connect the neural network forward propagation, model training and environment update performed in the hardware during the reinforcement learning training process; the hardware part is implemented on an FPGA, including a data transmission module, a parallel computing control module, a data buffer module and an environment computing module, and is used to implement parallel computing of environment updates on hardware;
[0014] In the software part, the data packaging module is used to compress and package the initial state of the environment in the reinforcement learning training process and the action information of the intelligent agent according to the current strategy into bytes, so that more environmental information can be contained in fewer bytes; the data transmission module is used for data transmission between the CPU end and the FPGA end, and sends the packaged byte data from the CPU end to the FPGA end, and also includes the CPU end reading byte data from the FPGA end; the data control module controls the data transmission module to read data from the FPGA end after the calculation of the FPGA end environment is completed; the data unpacking module unpacks the byte data read from the FPGA end into the data format required in the training process;
[0015] The hardware part, the data transmission module and the data transmission module of the CPU-side software part are connected, and are used for data transmission between the CPU-side and the FPGA-side, and for receiving data sent by the CPU-side; the data buffer module includes two types of storage media, namely registers and memory BRAM, which are used to store received data, intermediate data in the calculation process, and data completed by calculation; the parallel computing control module is used to control the data flow between the data buffer module and the environment computing module on the FPGA side; the environment computing module is composed of a number of single environment computing modules, and each single environment computing module corresponds to a hardware resource that can calculate a single environment. For different environment tasks, the current state of the environment and the action of the intelligent agent corresponding to the environment are used as input, and the state of the next stage of the environment, the observation value of the intelligent agent on the environment, the reward obtained by the intelligent agent, and whether the current task is terminated are used as outputs of the environment computing module.
[0016] Furthermore, the modules of the software part are implemented through software programming codes.
[0017] Furthermore, the environment calculation module of the hardware part is implemented by hardware programming code Verilog code.
[0018] Furthermore, the parallel computing control module is implemented by hardware programming code Verilog code.
[0019] Furthermore, the data transmission module of the hardware part is implemented by calling the IP core XDMA of the pre-designed and reusable hardware module Xilinx.
[0020] The present invention also proposes an efficient reinforcement learning training acceleration method implemented on the system. In a reinforcement learning training task, the calculation of m reinforcement learning environments is performed in parallel to obtain m sets of training data for updating the training model to accelerate the reinforcement learning training. During the training process, the software part CPU sends m actions to the hardware part FPGA at each calculation step. The FPGA calculates the calculation results of the m environments at this calculation step. After the calculation is completed, the CPU reads the calculation results of the m environments for training the model. After repeating several steps, the trained model achieves the expected training effect, indicating the completion of a reinforcement learning training task. Among them, completing the calculation of a calculation step of m environments includes the following steps:
[0021] 1) Data preprocessing on the CPU side: when the calculation step is the first step of the parallel environment, the environment needs to be initialized in this step. The data packaging module of the software part will splice the state information of the m parallel environments and the corresponding action information of the m intelligent agents in the training process, and add a signal bit at the end: "1"; when the calculation step is a standard step other than the first step of the parallel environment, the state information of the parallel environment has been stored inside the FPGA. The data packaging module of the software part only needs to splice the action information of the intelligent agents corresponding to the m environments in the training process, and add a signal bit at the end: "2"; compress it, splice the compressed data, and then pack it into byte form;
[0022] 2) Data transmission: the data transmission module of the software part sends the data processed in step 1) from the CPU end to the data transmission module of the FPGA end;
[0023] 3) Cache the data received on the FPGA side. The data buffer module of the hardware part saves the received data in BRAM and reads it into the register when it is used. The specific workflow is as follows:
[0024] When the FPGA receives the signal "1", it means that this is the first step in the round. The parallel computing control module reads the initial state of the environment and the action information of the agent in the received data, and reads the initial state information of the environment into the register. When the register is full, the parallel computing control module reads the information in the full register into the memory BRAM; when the initial state information of all environments is written, the parallel computing control module reads and stores the action information of all agents into the register; by storing the state information of the environment locally inside the FPGA, the data transfer between the CPU and the FPGA is reduced;
[0025] When the FPGA receives the signal "2", it means that this step is a standard step, that is, the parallel computing control module only reads the action information of the intelligent agent in the received data; the action information of all intelligent agents is read and stored in the register;
[0026] 4) The FPGA side performs parallel computing of the environment and caches the computing results. The parallel computing control module controls the data flow between the data buffer module and the environment computing module inside the FPGA. The environment computing module is composed of n single environment computing modules. A single environment computing module corresponds to a piece of hardware resources that can calculate a single environment, that is, corresponds to a number of computing cores. The specific number of computing cores n is determined by the number of clock cycles a required to calculate a single environment and the number of clock cycles b required to write all the calculated data back to the data buffer, n = a / b;
[0027] In the calculation stage of the environment and the caching stage of the calculation results, self-termination technology, pipeline technology and time division multiplexing technology are used to divide all m environments to be calculated in parallel into m / n batches. After each batch is calculated, when the calculation results of the batch begin to be cached, the calculation of the next batch of environments begins at the same time until the update of all environments is completed; the calculation stage of the environment is the first stage of the pipeline, and the caching stage of the calculation results is the second stage of the pipeline; specifically, the following steps are included:
[0028] 4-1) The environment calculation module reads the current state of n corresponding environments, the initial state of the environment, the cut-off signal indicating whether the previous state of the environment is cut-off, and the action signal of the corresponding intelligent agent from the data buffer module storing the current state of m environments, the initial state of the environment, the cut-off signal indicating whether the previous state of the environment is cut-off, and the action signal of the corresponding intelligent agent;
[0029] 4-2) Among the n environment information read, when the cutoff signal corresponding to the environment is a digital signal "0", the FPGA uses the value of the environment corresponding to the current state register of this environment as the state input of the environment calculation module environment in step 4-3), and when the cutoff signal corresponding to the environment is a digital signal "1", the FPGA uses the value of the environment corresponding to the initial state register of this environment as the state input of the environment calculation module environment in step 4-3). After this step, the states of the n environments to be used in the calculation of step 4-3) are obtained at the same time; this design ensures that if the round of an environment in the n parallel environments ends, the corresponding environment can be automatically reset to the initial state without manual reset;
[0030] 4-3) n cores start to calculate the environment in parallel. The input of the calculation is the state of the n environments and the corresponding actions of the n agents. The output is the state of the next step of the n environments, the agent's observation of the environment, the reward of the agent's action at this step, and whether the round is terminated. After a clock cycle, the calculation of the n environments is completed. While the calculation results of the n environments are cached in the buffer, the calculation of the second batch of n environments also starts. The next state information of the environment and whether the round is terminated are stored in the BRAM inside the FPGA that is specifically used to store the current environment state, waiting for the next action of the agent to be passed in as input. The agent's observation of the environment, the reward of the agent's action at this step, and whether the round is terminated will be cached in the buffer waiting to be read by the CPU.
[0031] After completing the update calculation and calculation result caching of all m environments, proceed to step 5);
[0032] 5) After waiting for all environmental calculations to be completed, the data control module controls the CPU-side data transmission module to read data from the FPGA side;
[0033] 6) Data unpacking: The data unpacking module decompresses the agent's observations of the environment, the reward of the action, and the information on whether the round is over, which are read from the FPGA by the CPU, from bytes into the data types required for training;
[0034] The above steps 1) to 6) complete the calculation of a calculation step in the reinforcement learning training process of m parallel environments. The CPU uses the m groups of data generated in this step to update the training model. The updated model generates m groups of actions based on the state information of the m groups of environments sent back. Steps 1) to 6) are repeated until the updated model can achieve the expected training effect, that is, a reinforcement learning training task is completed.
[0035] The technical effects of the present invention are as follows:
[0036] The present invention provides an efficient reinforcement learning training acceleration system and method, which realizes hardware acceleration of reinforcement learning environment update, can perform environment update in parallel in hardware, and for the specific task of reinforcement learning environment update, uses local storage technology, pipeline technology and time division multiplexing technology to make full use of computing resources and storage resources, thereby improving the operation speed. The present invention adopts a modular and parameterized design, can be applied to various types of reinforcement learning environments, can be applied to various types of reinforcement learning algorithms, and has strong versatility. The present invention ultimately accelerates the reinforcement learning training process by accelerating the environment update part of reinforcement learning. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 The difference between traditional machine learning (left) and reinforcement learning (right);
[0038] Figure 2 This is a diagram of the interaction between the environment and the agent during reinforcement learning training;
[0039] Figure 3 Schematic diagram of the parallel reinforcement learning environment;
[0040] Figure 4 It is a schematic diagram of the hardware and software parts of the system of the present invention;
[0041] Figure 5 is a flow chart of the method of the present invention;
[0042] Figure 6 This is a specific data flow diagram for the calculation stage of the present invention;
[0043] Figure 7 Schematic diagram of the reinforcement learning task of "balance bar on a cart" in an embodiment of the present invention. DETAILED DESCRIPTION
[0044] The present invention will be further clearly and completely described below through specific embodiments in conjunction with the accompanying drawings.
[0045] The present invention proposes an efficient reinforcement learning training acceleration system, which is composed of software and hardware parts. Figure 4 As shown, the data packaging module, data transmission module, data control module, and data unpacking module of the software part are all implemented by python code (a software programming code). The present invention adopts a parameterized design and can adapt to different types of environmental tasks and different numbers of environmental parallelism. The hardware part is implemented on FPGA, including a data transmission module, a parallel computing control module, a data buffer module, and an environmental computing module, which are used to implement parallel computing of environmental updates on hardware;
[0046] In the software part, the data packaging module is used to compress and package the initial state of the environment in the reinforcement learning training process and the action information of the intelligent agent according to the current strategy into bytes, so that more environmental information can be contained in fewer bytes; the data transmission module is used for data transmission between the CPU end and the FPGA end, and sends the packaged byte data from the CPU end to the FPGA end, and also includes the CPU end reading byte data from the FPGA end; the data control module controls the data transmission module to read data from the FPGA end after the calculation of the FPGA end environment is completed; the data unpacking module unpacks the byte data read from the FPGA end into the data format required in the training process;
[0047] The hardware part, the data transmission module and the data transmission module of the CPU-side software part are connected, and are used for data transmission between the CPU-side and the FPGA-side, and for receiving data sent by the CPU-side; the data buffer module includes two types of storage media, namely registers and memory BRAM, which are used to store received data, intermediate data in the calculation process, and data completed by calculation; the parallel computing control module is used to control the data flow between the data buffer module and the environment computing module on the FPGA side; the environment computing module is composed of a number of single environment computing modules, and each single environment computing module corresponds to a hardware resource that can calculate a single environment. For different environment tasks, the current state of the environment and the action of the intelligent agent corresponding to the environment are used as input, and the state of the next stage of the environment, the observation value of the intelligent agent on the environment, the reward obtained by the intelligent agent, and whether the current task is terminated are used as outputs of the environment computing module.
[0048] In this embodiment, the data transmission module of the hardware part is implemented by calling Xilinx's XDMA IP core (IP core is a pre-designed and verified hardware module that can accelerate and simplify the FPGA design process, improve system performance and development efficiency. XDMA is a PCIe IP core, and PCIe is a high-speed method for data transmission between different hardware).
[0049] The environmental computing module and the parallel computing control module are both implemented through Verilog code (a hardware programming code). The modular and parameterized design helps to adapt to different environmental tasks and different degrees of parallelism.
[0050] Figure 5 Flowchart of the method of the present invention. The following uses a “balance bar on a cart” to introduce an efficient reinforcement learning training acceleration method implemented on the system of the present invention.
[0051] "Balancing a pole on a cart" is a classic reinforcement learning problem, where tasks such as Figure 7As shown in the figure, a wooden pole with a fixed bottom is placed vertically on the cart. The wooden pole is kept upright by controlling the left and right movement of the cart. The fall of the cart marks the end of a round. In this task, the environment has four states, namely the position x of the wooden pole, the angle θ of the wooden pole, and the speed of the wooden pole. and the angular velocity of the pole The intelligent agent, i.e. the car, has two actions, namely, the car moves left and the car moves right. If the position x of the wooden pole is between -4.8 and +4.8, and the angle θ of the wooden pole is between ±12°, it is considered that it has not fallen. In this embodiment, it takes 80 clock cycles to calculate a single car environment, and it takes 4 clock cycles to write the calculation results of a single environment into the buffer.
[0052] For this reinforcement learning training task, the calculation of 160 such "balance bar on a small car" environments is designed to be performed in parallel, and 160 sets of training data are obtained to update the training model to accelerate the reinforcement learning training; during the training process, the software part CPU sends 160 actions to the hardware part FPGA at each calculation step, and the FPGA calculates the calculation results of the 160 environments at this calculation step. After the calculation is completed, the CPU reads the calculation results of the 160 environments to train the model; after repeating several steps, the trained model achieves the expected training effect, marking the completion of a reinforcement learning training task; Among them, completing the calculation of a calculation step of 160 environments includes the following steps:
[0053] 1) Data preprocessing on the CPU side: When the calculation step is the first step of the parallel environment, the environment needs to be initialized. The data packaging module of the software part will splice the state information of the 160 parallel environments and the corresponding action information of the 160 intelligent agents during the training process, and add a signal bit at the end: "1"; when the calculation step is a standard step of the parallel environment, the state information of the parallel environment has been stored inside the FPGA. The data packaging module of the software part only needs to splice the action information of the intelligent agents corresponding to the 160 environments during the training process, and add a signal bit at the end: "2". After compressing and splicing the data, it is packaged into bytes;
[0054] 2) Data transmission: the data transmission module of the software part sends the data processed in step 1) from the CPU end to the data transmission module of the FPGA end;
[0055] 3) Cache the data received on the FPGA side. The data buffer module of the hardware part saves the received data in BRAM and reads it into the register when it is used. The specific workflow is as follows:
[0056] When the FPGA receives the signal "1", it means that this is the first step in the round. The parallel computing control module reads the initial state of the environment and the action information of the agent in the received data, and reads the initial state information of the environment into the register. When the register is full, the parallel computing control module reads the information in the full register into the memory BRAM; when the initial state information of all environments is written, the parallel computing control module reads and stores the action information of all agents into the register; by storing the state information of the environment locally inside the FPGA, the data transfer between the CPU and the FPGA is reduced.
[0057] When the FPGA receives the signal "2", it means that this step is a standard step, that is, the parallel computing control module only reads the action information of the intelligent agent in the received data; the action information of all intelligent agents is read and stored in the register;
[0058] 4) The FPGA side performs parallel computing of the environment and caches the computing results. The parallel computing control module controls the data flow between the data buffer module inside the FPGA and the environment computing module. The environment computing module is composed of n single environment computing modules. A single environment computing module corresponds to a piece of hardware resources that can calculate a single environment, that is, a single environment computing module corresponds to one computing core number. The specific number of computing cores n is determined by the number of clock cycles a required for the training task to calculate a single environment and the number of clock cycles b required to write all calculated data back to the data buffer, n=a / b; in this embodiment, it takes 80 clock cycles to calculate a single car environment, and it takes 4 clock cycles to write the calculation results of a single environment into the buffer, so the number of computing cores required is n=80 / 4=20, and each core corresponds to a hardware resource that can calculate a single environment; in the calculation stage of the environment and the caching stage of the calculation results, self-termination technology, pipeline technology and time division multiplexing technology are used to divide all 160 environments to be calculated in parallel into 8 (160 / 20=8) batches. After each batch is calculated, when the calculation results of the batch begin to be cached, the calculation of the next batch of environments is started at the same time until the update of all environments is completed; the calculation stage of the environment is the first stage of the pipeline, and the caching stage of the calculation results is the second stage of the pipeline; the following steps are included:
[0059] 4-1) The environment calculation module reads 20 corresponding current states of the environment, the initial state of the environment, the cut-off signal indicating whether the previous state of the environment was terminated, and the action signals of the corresponding intelligent agents from the data buffer module that stores 160 current states of the environment, the initial state of the environment, the cut-off signal indicating whether the previous state of the environment was terminated, and the action signals of the corresponding intelligent agents.
[0060] 4-2) Among the 20 environment information read, if the cutoff signal corresponding to the environment is a digital signal "0", the FPGA uses the value of the environment in the current state register corresponding to the environment as the state input of the environment calculation module environment. When the cutoff signal corresponding to the environment is a digital signal "1", the FPGA uses the value of the environment in the initial state register corresponding to the environment as the state input of the environment calculation module environment. After this step, the state of the environment to be used in the calculation of step 4-3) can be obtained at the same time for 20 environments, whether it is the current environment state or the initial environment state. This design ensures that if the round of an environment ends in n parallel environments, the corresponding environment can be automatically reset to the initial state without manual reset.
[0061] 4-3) 20 cores start parallel computing. The input of the calculation is the state of the 20 environments and the corresponding actions of the 20 agents. The output is the state of the 20 environments in the next step (that is, the position, speed, angular velocity and angular acceleration of the wooden pole after the car moves), the agent's observation of the environment (when the training task is a globally observable task, the agent obtains the state information of the entire environment, and the agent's observation of the environment is the same as the state of the environment. When it is a partially observable task, the agent's observation of the environment is only a partial state of the environment. Because this embodiment is a globally observable task, the agent can obtain the state information of the entire environment. Therefore, in this task, the agent's observation of the environment is the same as the state of the environment), the reward for the agent's action in this step (whether the wooden pole falls after the car moves, if not, it will get a +1 reward) and whether the round is terminated (if the wooden pole falls, it is considered that the round is terminated).
[0062] After 80 clock cycles, the calculation of 20 environments is completed. While the calculation results of these 20 environments are cached in the buffer, the calculation of the second batch of 20 environments also begins. The next state information of the environment and whether the round is terminated are stored in the BRAM inside the FPGA, which is specially used to store the current environment state, waiting for the next new action of the agent to be passed in as input. The observation value of the agent on the environment, the reward of the agent's action at this step, and whether the round is terminated will be cached in the buffer waiting to be read by the CPU. Figure 6 The arrows in the figure show the data flow of the calculation state. The current state of the environment and the corresponding action information of the agent are synchronously transmitted to the environment calculation module. After the environment calculation module calculates, the next state of the environment is stored in the corresponding data buffer; the observation information of the agent on the environment, the reward value corresponding to the action, and the information on whether the current round is over are cached and wait to be read by the CPU.
[0063] After completing the update calculation and calculation result caching of all 160 environments, proceed to step 5).
[0064] 5) After all calculations of the environment are completed, the data control module controls the CPU-side data transmission module to read data from the FPGA side.
[0065] 6) Data unpacking: The data unpacking module decompresses the agent's observations of the environment, action rewards, and round expiration information read from the FPGA by the CPU from bytes into floating-point data required for training.
[0066] The above steps 1) to 6) complete the calculation of a calculation step in the reinforcement learning training process of 160 parallel "balance bars on the cart". The CPU uses the 160 sets of data generated in this step to update the training model. The updated model generates 160 sets of actions based on the 160 sets of environment state information sent back, and then repeats steps 1) to 6) until the updated model can achieve the expected training effect, that is, a reinforcement learning training task is completed.
[0067] Through the above steps, 160 "balance bars on a small car" classic environments can be calculated in parallel in the FPGA and coordinated with other parts of the training code. Compared with other general methods, while the environment is updated 12 million times to achieve the same training effect, the training time is reduced by 18%. The whole process is parameterized and modularized, and various types of reinforcement learning environments with various degrees of parallelism can be easily deployed on the FPGA for parallel computing. It has strong versatility and low hardware knowledge requirements for users.
[0068] In the embodiments of the present invention, the task of updating the "balance bar on the cart" in parallel is taken as an example. However, the present invention is not limited to a certain reinforcement learning task, but is also applicable to the environment update of other reinforcement learning tasks; it is not limited to a parallel environment, but is also applicable to the environment update of a single reinforcement learning task. Although the present invention emphasizes the acceleration of reinforcement learning training, it is also applicable to the reasoning process of reinforcement learning. It can be applied to any application scenario involving the environment update of reinforcement learning.
[0069] Finally, it should be noted that the purpose of publishing the embodiments is to help further understand the present invention, so that those skilled in the art can understand that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the contents disclosed in the embodiments, and the scope of protection claimed by the present invention shall be subject to the scope defined in the claims.
Claims
1. An efficient reinforcement learning training acceleration system, characterized in that: It consists of a software part and a hardware part. The software part is implemented on the CPU, including a data packaging module, a data transmission module, a data control module and a data unpacking module, which is used to connect the neural network forward propagation, model training and environment update performed in the hardware during the reinforcement learning training process; the hardware part is implemented on the FPGA, including a data transmission module, a parallel computing control module, a data buffer module and an environment computing module, which is used to implement parallel computing of environment updates on the hardware; The software part, the data packaging module is used to compress the initial state of the environment in the reinforcement learning training process and the action information of the intelligent agent according to the current strategy, and package them into bytes, so that more environment information can be contained in fewer bytes; The data transmission module is used for data transmission between the CPU and FPGA. It sends the packaged byte data from the CPU to the FPGA, and also includes the CPU reading byte data from the FPGA. The data control module controls the data transmission module to read data from the FPGA after the calculation of the FPGA environment is completed. The data unpacking module unpacks the byte data read from the FPGA into the data format required in the training process. The hardware part, the data transmission module and the data transmission module of the CPU-side software part are connected, and are used for data transmission between the CPU-side and the FPGA-side, and for receiving data sent by the CPU-side; the data buffer module includes two types of storage media, namely registers and memory BRAM, which are used to store received data, intermediate data in the calculation process, and data completed by calculation; the parallel computing control module is used to control the data flow between the data buffer module and the environment computing module on the FPGA side; the environment computing module is composed of a number of single environment computing modules, and each single environment computing module corresponds to a hardware resource that can calculate a single environment. For different environment tasks, the current state of the environment and the action of the intelligent agent corresponding to the environment are used as input, and the state of the next stage of the environment, the observation value of the intelligent agent on the environment, the reward obtained by the intelligent agent, and whether the current task is terminated are used as outputs of the environment computing module.
2. The system as claimed in claim 1, wherein the modules of the software part are implemented by software programming codes.
3. The system as claimed in claim 1, wherein the environment computing module of the hardware part is implemented by hardware programming code Verilog code.
4. The system as described in claim 1, wherein the parallel computing control module is implemented by hardware programming code Verilog code.
5. The system as claimed in claim 1, wherein the data transmission module of the hardware part is implemented by calling the IP core XDMA of the pre-designed and reusable hardware module Xilinx.
6. An efficient reinforcement learning training acceleration method implemented on the system as claimed in claim 1, characterized in that: In a reinforcement learning training task, m reinforcement learning environments are calculated in parallel to obtain m sets of training data for updating the training model to accelerate reinforcement learning training; During the training process, the software CPU sends m actions to the hardware FPGA at each calculation step. The FPGA calculates the calculation results of the m environments at this calculation step. After the calculation is completed, the CPU reads the calculation results of the m environments to train the model. After repeating several steps, the trained model achieves the expected training effect, marking the completion of a reinforcement learning training task. Among them, completing the calculation of a calculation step of m environments includes the following steps: 1) Data preprocessing on the CPU side: when the calculation step is the first step of the parallel environment, the environment needs to be initialized in this step. The data packaging module of the software part will splice the state information of the m parallel environments and the corresponding action information of the m intelligent agents in the training process, and add a signal bit "1" at the end; when the calculation step is a standard step other than the first step of the parallel environment, the state information of the parallel environment has been stored inside the FPGA. The data packaging module of the software part only needs to splice the action information of the intelligent agents corresponding to the m environments in the training process, and add a signal bit "2" at the end; compress it, splice the compressed data, and then pack it into byte form; 2) Data transmission: the data transmission module of the software part sends the data processed in step 1) from the CPU end to the data transmission module of the FPGA end; 3) Cache the data received on the FPGA side. The data buffer module of the hardware part saves the received data in BRAM and reads it into the register when it is used. The specific workflow is as follows: When the FPGA receives the signal "1", it means that this is the first step in the round. The parallel computing control module reads the initial state of the environment and the action information of the agent in the received data, and reads the initial state information of the environment into the register. When the register is full, the parallel computing control module reads the information in the full register into the memory BRAM; when the initial state information of all environments is written, the parallel computing control module reads and stores the action information of all agents into the register; by storing the state information of the environment locally inside the FPGA, the data transfer between the CPU and the FPGA is reduced; When the FPGA receives the signal "2", it means that this step is a standard step, that is, the parallel computing control module only reads the action information of the agent in the received data; the action information of all agents is read and stored in the register; 4) The FPGA side performs parallel computing of the environment and caches the computing results. The parallel computing control module controls the data flow between the data buffer module and the environment computing module inside the FPGA. The environment computing module is composed of n single environment computing modules. A single environment computing module corresponds to a piece of hardware resources that can calculate a single environment, that is, corresponds to a number of computing cores. The specific number of computing cores n is determined by the number of clock cycles a required to calculate a single environment and the number of clock cycles b required to write all the calculated data back to the data buffer, n = a / b; In the calculation stage of the environment and the caching stage of the calculation results, self-termination technology, pipeline technology and time division multiplexing technology are used to divide all m environments to be calculated in parallel into m / n batches. After each batch is calculated, when the calculation results of the batch begin to be cached, the calculation of the next batch of environments begins at the same time until the update of all environments is completed; the calculation stage of the environment is the first stage of the pipeline, and the caching stage of the calculation results is the second stage of the pipeline; specifically, the following steps are included: 4-1) The environment calculation module reads the current state of n corresponding environments, the initial state of the environment, the cut-off signal indicating whether the previous state of the environment is cut-off, and the action signal of the corresponding intelligent agent from the data buffer module storing the current state of m environments, the initial state of the environment, the cut-off signal indicating whether the previous state of the environment is cut-off, and the action signal of the corresponding intelligent agent; In the n environment information read in step 4-2), when the cutoff signal corresponding to the environment is a digital signal "0", the FPGA uses the value of the environment corresponding to the current state register of the environment as the state input of the environment calculation module in step 4-3), and when the cutoff signal corresponding to the environment is a digital signal "1", the FPGA uses the value of the environment corresponding to the initial state register of the environment as the state input of the environment calculation module in step 4-3). After this step, the states of the n environments to be used in the calculation in step 4-3) are obtained at the same time; this design ensures that if the round of an environment in the n parallel environments ends, the corresponding environment can be automatically reset to the initial state without manual reset; 4-3) n cores start to calculate the environment in parallel. The input of the calculation is the state of n environments and the corresponding actions of n agents. The output is the state of the next step of m environments, the agent's observation of the environment, the reward of the agent's action at this step, and whether the round is terminated. After a clock cycle, the calculation of m environments is completed. While the calculation results of these m environments are cached in the buffer, the calculation of the second batch of m environments also starts. The state information of the next step of the environment and the information of whether the round is terminated are stored in the BRAM inside the FPGA that is specially used to store the current environment state, waiting for the next action of the agent to be passed in as input. The agent's observation of the environment, the reward of the agent's action at this step, and whether the round is terminated will be cached in the buffer waiting to be read by the CPU. After completing the update calculation and calculation result caching of all m environments, proceed to step 5); 5) After waiting for all environmental calculations to be completed, the data control module controls the CPU-side data transmission module to read data from the FPGA side; 6) Data unpacking: The data unpacking module decompresses the agent's observations of the environment, the reward of the action, and the information on whether the round is over, which are read from the FPGA by the CPU, from bytes into the data types required for training; The above steps 1) to 6) complete the calculation of a calculation step in the reinforcement learning training process of m parallel environments. The CPU uses the m groups of data generated in this step to update the training model. The updated model generates m groups of actions based on the state information of the m groups of environments sent back. Steps 1) to 6) are repeated until the updated model can achieve the expected training effect, that is, a reinforcement learning training task is completed.
Citation Information
Patent Citations
Training method for reinforcement learning network, device, training apparatus and storage medium
CN109242099A
Model training method based on reinforcement learning and related device
CN111985640A
Deep reinforcement learning acceleration method for multi-core processor-single graphics processor
CN112732436A
Reinforced learning method
CN112734048A
Data enhancement training for reinforcement learning software agents
CN114730306A
Cited By
Symbol guidance-oriented neural symbol reinforcement learning agent reasoning and training method and hardware architecture thereof
CN121981159A
A reasoning and training method and its hardware architecture for a symbol-guided neurosymbolic reinforcement learning agent.
CN121981159B
