A hardware implementation method of Bellman equation based on policy iteration

Abstract

The present invention discloses a hardware implementation method for the Bellman equation based on policy iteration. First, a reward value is input into a Bellman expectation equation circuit to calculate the policy value of the reward value. The obtained policy value is input into a Bellman optimal equation circuit for policy iteration to solve the optimal value. The obtained optimal value is mapped to a policy map composed of a memristor array to complete the optimal value solution for each state. The movement direction of each state is determined based on the magnitude of the optimal value, thereby achieving the purpose of using hardware to accelerate the solution of the optimal value of the Bellman equation. This method can hardwareize the Bellman equation using a memristor array multiplication and addition method, thereby significantly optimizing the performance of reinforcement learning hardware systems.

Description

Technical Field

[0001] The present invention relates to the technical field of memristors, and in particular to a hardware implementation method of a Bellman equation based on policy iteration. Background Art

[0002] Reinforcement learning, a branch of machine learning, aims to leverage the interactions between an agent and its environment to learn a strategy that maximizes reward. With the rapid development of artificial intelligence (AI), reinforcement learning has broad application prospects in as many as 12 fields. However, compared to unsupervised and supervised learning, reinforcement learning requires significantly more computing power, necessitating full hardware implementation and comprehensive optimization of hardware accelerators. To date, a number of memristor-based reinforcement learning acceleration systems have emerged. For example, one uses a memristor array to implement the classic reinforcement learning algorithm, DQN. However, this work only hardware-enables the neural network portion of the Deep-Q-Network (DQN) algorithm, while the reinforcement learning policy solution still occurs in the CPU. Another approach uses phase-change memory to implement the eligibility trace, but this system only generates the policy graph on a PCM array, while the main policy iteration calculations still occur in the computer.

[0003] In summary, the use of memristor arrays as reinforcement learning accelerators has improved computing efficiency to a certain extent, but the majority of the computational process still occurs in the CPU, especially for the Bellman expectation equation, the cornerstone of reinforcement learning. Research has shown that a corresponding hardware circuit has not yet been designed. During the trial-and-error exploration of reinforcement learning, policy iteration calculations consume the most resources. How to use the matrix dot product of the memristor array to implement policy iteration in hardware and solve the optimal policy is a problem that must be solved for a fully hardware-based reinforcement learning system. Summary of the Invention

[0004] The purpose of the present invention is to provide a hardware implementation method of the Bellman equation based on policy iteration. This method can hardwareize the Bellman equation using a memristor array multiplication and addition method, thereby greatly optimizing the performance of the reinforcement learning hardware system.

[0005] The purpose of the present invention is achieved through the following technical solutions:

[0006] A hardware implementation method of the Bellman equation based on policy iteration, the method comprising:

[0007] Step 1: Input the reward value into the Bellman expectation equation circuit and calculate the strategic value of the reward value;

[0008] Step 2: Input the strategy value obtained in step 1 into the Bellman optimal equation circuit to perform strategy iteration and solve the optimal value;

[0009] Step 3: Map the optimal value obtained in step 2 to the strategy map composed of the memristor array to complete the optimal value solution for each state, and determine the movement direction of each state based on the size of the optimal value, so as to achieve the purpose of using hardware to accelerate the solution of the Bellman equation for the optimal value.

[0010] It can be seen from the technical solution provided by the present invention that the above method can hardware the Bellman equation using the multiplication and addition method of the memristor array, thereby greatly optimizing the performance of the reinforcement learning hardware system. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0012] Figure 1 A schematic flow chart of a hardware implementation method of the Bellman equation based on policy iteration provided in an embodiment of the present invention;

[0013] Figure 2 Schematic diagram of the iterative solution transformation process according to an embodiment of the present invention;

[0014] Figure 3 This is a schematic diagram illustrating the implementation principle of the Bellman expectation equation circuit according to an embodiment of the present invention;

[0015] Figure 4 This is a schematic diagram illustrating the implementation principle of the Bellman optimal equation circuit according to an embodiment of the present invention;

[0016] Figure 5 This is a circuit diagram for the full hardware implementation of the Bellman equation described in an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments, and do not constitute a limitation of the present invention. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0018] like Figure 1 FIG. 1 is a flow chart of a hardware implementation method of the Bellman equation based on policy iteration according to an embodiment of the present invention. The method includes:

[0019] Step 1: Input the reward value into the Bellman expectation equation circuit and calculate the strategic value of the reward value;

[0020] In this step, the Bellman expectation equation, as an expected fixed point equation, does not have the properties of matrix multiplication and addition. The embodiment of the present invention optimizes and derives the structure of the Bellman equation, such as Figure 2 FIG. 1 is a schematic diagram showing the implementation principle of the Bellman expectation equation circuit according to an embodiment of the present invention, which uses the fixed point principle to convert the expected form of the Bellman expectation equation into an iterative solution in the form of matrix multiplication and addition, as shown in FIG. Figure 2 The figure is a schematic diagram of the process of iterative solution transformation according to an embodiment of the present invention:

[0021] The original Bellman equation is expressed as: V(s)=R(s)+γ∑ s′∈S P(s′|s)V(s′), where R(s) represents the input reward value, γ∑ s′∈S P(s′|s)V(s′) represents the sum of future discounted rewards, γ is the discount factor, and P(s′|s) is the transition matrix from the current state to the next state;

[0022] The Bellman equation indicates that the value function of the current state can be calculated by the value function of the next state. After writing it in matrix form and solving it, its iterative analytical solution is: V = (I-γP) -1 R;

[0023] Based on this iterative analytical solution, the two parts on the right side of the equation can be replaced by the matrix multiplication function of the memristor array itself, thereby realizing the Bellman equation in hardware.

[0024] The memristor array is constructed by stacking individual crossbar memristors in two-dimensional rows and columns. Each intersection in the array has a memristor, each with an adjustable conductance. When voltage is applied row by row to the memristor array, each memristor in the same row multiplies the input voltage by its conductance to obtain the current value at that point. When multiple rows of the array are simultaneously input, Kirchhoff's current law applies to each column, resulting in the sum of the current values of all memristors in that column, achieving matrix multiplication and addition.

[0025] In this embodiment, the memristor array used is a 1T1R structure, that is, one transistor and one memristor. The drain of the transistor is connected to the electrode on the memristor to integrate the memristor array. The transistor switch is controlled by gate pressure to suppress current crosstalk between different rows and columns, reduce read errors, and improve the recognition accuracy of the hardware system.

[0026] like Figure 3The figure shows the implementation principle diagram of the Bellman expectation equation circuit described in an embodiment of the present invention. The input voltage signal represents the reward value R(s) input to the system; the conductance value of each memristor in the memristor array represents the state transition probability P(s′|s). The memristor array is inferred and mapped by using a peripheral FPGA board circuit; the output current value is converted into a voltage signal through a constant resistance array, representing the output value, that is, the strategic value of the input reward value.

[0027] In a specific implementation, when a parallel level signal is input into the memristor array as a reward value, the peripheral circuit of the memristor array controls the first column to turn on. Then, the input level signal and the probability value written in the first column (0.1-1) are vector-multiplied and added by the non-volatile device according to Ohm's law and Kirchhoff's law, and the current value of the first column is output. The current value is then converted into a voltage signal, attenuated by 50%, and input into the array together with the reward level signal as the input of the first row under the control of the peripheral circuit. At the same time, the array is controlled to turn on the output current signal of the second column, completing a cycle operation.

[0028] After traversing all columns, observe whether the output current no longer changes. If so, use the output current value as the value obtained in the first epoch, and use this value to update the weights in the array, that is, the strategy value of the entire system.

[0029] Step 2: Input the strategy value obtained in step 1 into the Bellman optimal equation circuit to perform strategy iteration and solve the optimal value;

[0030] In this step, the Bellman optimal equation circuit is to repeatedly recursively operate the strategy value obtained by the Bellman expectation equation circuit, update the value probability matrix using the greedy algorithm, and repeatedly solve the strategy value until it is attributed to a certain value, that is, the optimal value, such as Figure 4 FIG. 1 is a schematic diagram illustrating the implementation principle of the Bellman optimal equation circuit according to an embodiment of the present invention;

[0031] Among them, the circuit input voltage signal is the strategy value matrix calculated by the Bellman expectation equation circuit; the array conductance value in the memristor array represents the value probability transfer matrix. After the matrix multiplication and addition operation of the memristor array, the corresponding voltage value is output and then the input value is returned again for repeated recursive operations. The value probability matrix is updated through the algorithm to output a new optimized value; it can be seen from the fixed point iteration method that after repeated iterations, the output value gradually stabilizes, and finally the maximum value is obtained by using the winner-takes-all circuit, which is the optimal value.

[0032] Step 3: Map the optimal value obtained in step 2 to the strategy map composed of the memristor array to complete the optimal value solution for each state, and determine the movement direction of each state based on the size of the optimal value, so as to achieve the purpose of using hardware to accelerate the solution of the Bellman equation for the optimal value.

[0033] like Figure 5 The figure shows a circuit diagram of a full hardware implementation of the Bellman equation according to an embodiment of the present invention. Figure 5 The project includes a three-layer array circuit implementation. The first and second layers are 25×25 memristor arrays. Their main purpose is to input reward values and calculate the output policy value, iterating until the value converges.

[0034] The output current value is then converted into a voltage signal and output to the third layer of the 25×100 memristor array. Matrix multiplication and addition operations are performed using Ohm's law and Kirchhoff's law, and the optimal value is finally output and returned to the host computer.

[0035] The final output value is used to update the conductance value of the three-layer array, and the calculation of the entire circuit is finally completed after 6 update operations.

[0036] It should be noted that the contents not described in detail in the embodiments of the present invention belong to the prior art known to those skilled in the art.

[0037] In summary, the Bellman equation is one of the important cornerstones of reinforcement learning and accounts for a large proportion of the system's calculation process. Therefore, it is possible to obtain an analytical solution by performing matrix transformation iteration on the Bellman equation. The analytical solution can be used to solve the optimal strategy of the Bellman equation by utilizing the inherent acceleration of matrix multiplication and addition of the memristor array. Using the memristor array to hardware it will greatly optimize the performance of the reinforcement learning hardware system.

[0038] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or any form of implication that the information constitutes prior art already known to those skilled in the art.

Claims

1. A hardware implementation method of the Bellman equation based on policy iteration, characterized in that: The method comprises: Step 1: Input the reward value into the Bellman expectation equation circuit and calculate the strategic value of the reward value; The Bellman expectation equation circuit is an iterative solution that converts the expectation form of the Bellman expectation equation into a matrix multiplication and addition form using the fixed point principle; The original Bellman equation is expressed as: ,in Represents the reward value of the input, represents the sum of future discounted rewards, γ is the discount factor, is the transition matrix from the current state to the next state; The Bellman equation indicates that the value function of the current state is calculated by the value function of the next state. After writing it in matrix form and solving and deducing it, its iterative analytical solution is: ; Based on this iterative analytical solution, the left and right parts of the equation are replaced by the matrix multiplication function of the memristor array itself, thus realizing the Bellman equation in hardware. Among them, the input voltage signal represents the reward value input by the system ; The conductance value of each memristor in the memristor array represents the state transition probability , by using the peripheral FPGA board circuit to perform reasoning and mapping of the memristor array; the output current value is converted into a voltage signal through a constant resistance array, which represents the output value, that is, the strategic value of the input reward value; The memristor array is formed by stacking single crossbar memristors in two-dimensional rows and columns, with each crosspoint on the array having a memristor, and each memristor having an adjustable conductance value; When voltage is input row by row into the memristor array, each memristor in the same row can obtain the current value at that point by multiplying the input voltage by the conductance of the memristor. When input is applied to multiple rows of the array at the same time, for each column, Kirchhoff's current law applies, and each column outputs the sum of the current values of all memristors in that column, achieving the purpose of matrix multiplication and addition. Step 2: Input the strategy value obtained in step 1 into the Bellman optimal equation circuit to perform strategy iteration and solve the optimal value; The Bellman optimal equation circuit repeatedly recursively operates the strategy value obtained by the Bellman expectation equation circuit, updates the value probability matrix using a greedy algorithm, and repeatedly solves the strategy value until it returns to a certain value, namely the optimal value; The circuit's input voltage signal is the strategy value matrix calculated by the Bellman expectation equation circuit. The array conductance value in the memristor array represents the value probability transfer matrix. After performing matrix multiplication and addition operations on the memristor array, the corresponding voltage value is output and then the input value is returned for repeated recursive operations. The algorithm updates the value probability matrix and outputs a new, optimized value. The fixed-point iteration method shows that after repeated iterations, the output value gradually stabilizes. Finally, the maximum value is obtained using a winner-takes-all circuit, which is the optimal value. Step 3: Map the optimal value obtained in step 2 to the strategy map composed of the memristor array to complete the optimal value solution for each state, and determine the movement direction of each state based on the size of the optimal value, so as to achieve the purpose of using hardware to accelerate the solution of the Bellman equation for the optimal value.

Images