A method for hardware acceleration of reinforcement learning based on memristor

Abstract

This invention discloses a hardware acceleration method for reinforcement learning based on memristors, relating to the field of memristor technology. First, 32 round paths are generated in batches, each batch. These paths are inferred and mapped onto a TiO2 memristor array to form a policy graph. The policy graph information is then imported column-by-column into a 32×32 memristor array. Parallel matrix multiplication and addition operations on the array are used to obtain the value of each round corresponding to a given column. This value is compared using a WTA circuit, and the maximum round value among the 32 paths is output. Finally, this value is compared with the maximum value obtained from the previous batch, and the largest value is retained. This process is iterated repeatedly until convergence. This method hardware-enables the reinforcement learning algorithm's iterative process using memristor array multiplication and addition, reducing the number of iterations in the reinforcement learning task and decreasing computer throughput.

Description

Technical Field

[0001] This invention belongs to the field of memristor technology, specifically relating to a hardware acceleration method for reinforcement learning based on memristors. Background Technology

[0002] With the continuous development of technology, memristor-based memory computing is a more efficient and low-power architecture that can implement matrix-vector multiplication (MVM) using "physical computing," thereby accelerating machine learning and reducing training power consumption. In recent years, memristor-based reinforcement learning has been widely studied, but most of the reported work only uses memristors to complete MVM, while the main computational iteration process is still performed in the CPU. In the process of reinforcement learning trial and error exploration, the computational resources for value iteration account for the largest proportion. How to use the matrix multiplication operation of memristor arrays to accelerate the convergence of reinforcement learning algorithms in hardware is a problem that must be solved in the hardware implementation of reinforcement learning systems. Summary of the Invention

[0003] To address the shortcomings of existing technologies, the present invention aims to provide a hardware acceleration method for reinforcement learning based on memristors, which solves the technical problem of poor convergence performance in existing technologies that use matrix multiplication operations of memristor arrays to accelerate reinforcement learning algorithms.

[0004] The objective of this invention can be achieved through the following technical solution: a hardware acceleration method for reinforcement learning based on memristors, the method comprising the following steps:

[0005] Obtain 32 round paths and output the round paths with action information. Infer and map them to a memristor array to form a strategy graph. Input a 32×32 memristor hardware array by column.

[0006] The reward value corresponding to each action in the generated strategy graph is used as the read voltage input memristor hardware array. The corresponding round value is obtained according to the array. The round value of each column is compared through the WTA winner-take-all circuit, and the largest round value among the 32 rounds is output.

[0007] The maximum round value obtained from the comparison is compared with the maximum round value obtained previously. The larger round value is retained. This process is repeated until convergence, and parallel computing is used to reduce the number of iterations.

[0008] Preferably, the strategy map formed by the memristor is to naturally replace the decay factor λ in the algorithm Sarsa(λ), which requires a large number of exponential decay operations, by using the physical decay function of the memristor.

[0009] Preferably, the round path includes information about state-action pairs.

[0010] Preferably, the memristor array uses different conductance values ​​to represent different state actions.

[0011] Preferably, the read voltage of the memristor hardware array is used to represent the reward information of each state-action pair, and the current value output column by column according to the algorithm value update formula is the round value.

[0012] Preferably, the algorithm value update formula is:

[0013]

[0014] Where t represents the number of steps in each round, a is the action information of the k-th step, represented as the conductance value in the memristor array, and R is the reward value for the corresponding step. The value is calculated using the matrix operation inherent in the memristor itself, and the memristor read voltage is set to R. t-k+1 The conductivity value on the array is a k The action information for each round is input into the array column by column, and then the voltage is input row by row. The sum of the currents in each column is read according to the multiplication-addition law of the array and Kirchhoff's current law, which is the value of the round.

[0015] Preferably, the parallel computing capability of the memristor array is utilized to perform 32 column operations simultaneously, outputting 32 values, which are then compared using the WTA winner-take-all circuit to output the highest round value.

[0016] Preferably, a path of 32 rounds is generated for each batch according to the Sarsa(λ) algorithm. By printing the action information of each round, which includes four actions (up, down, left, and right), the four different actions are mapped to the four conductance states of the memristor and then mapped to the TiO2 memristor array to form a feature map.

[0017] Preferably, an apparatus includes:

[0018] One or more processors;

[0019] Memory, used to store one or more programs;

[0020] When the one or more programs are executed by the one or more processors, the one or more processors implement a memristor-based reinforcement learning hardware acceleration method as described above.

[0021] Preferably, a storage medium includes acquisition machine executable instructions, which, when executed by an acquisition machine processor, are used to perform a memristor-based reinforcement learning hardware acceleration method as described above.

[0022] The beneficial effects of this invention are:

[0023] In its application, this invention acquires 32 round paths and outputs round paths with action information. These paths are then inferred and mapped onto a memristor array to form a policy graph. The graph is input column-wise into a 32×32 memristor hardware array. The reward value corresponding to each action in the generated policy graph is then used as a read voltage input to the memristor hardware array. The corresponding round value is obtained by array-wise comparison. Each column of round values ​​is compared using a WTA winner-take-all circuit, and the largest round value among the 32 rounds is output. This largest round value is then compared with the previously obtained largest round value, retaining the larger value. This process is iterated repeatedly until convergence, utilizing parallel computing to reduce the number of iterations. This method reduces the iterations of reinforcement learning algorithms by using a memristor array multiply-accumulate approach. The hardware memristor array circuit serves as the evaluation system, leveraging its parallel matrix multiply-accumulate function to accelerate the policy iteration speed of the algorithm and improve the fit between the hardware and software acceleration systems. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a schematic diagram of the hardware implementation method of the reinforcement learning algorithm based on memristors provided in an embodiment of the present invention;

[0026] Figure 2 This is a schematic diagram illustrating the implementation principle of the memristor-based reinforcement learning hardware acceleration system according to an embodiment of the present invention.

[0027] Figure 3 The attenuation curve and its corresponding function fitting graph of the Pt / TiO2 / Ti memristor described in the embodiment of the present invention are shown. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] like Figure 1 The diagram shows a flowchart of a hardware implementation method for a reinforcement learning algorithm based on memristors provided in an embodiment of the present invention. The method includes the following steps:

[0030] Obtain 32 round paths and output the round paths with action information. Infer and map them to a memristor array to form a strategy graph. Input a 32×32 memristor hardware array by column.

[0031] Further explanation is needed regarding the specific implementation process. Firstly, the Sarsa(λ) algorithm generates 32 rounds of paths per batch. By printing the action information (up, down, left, right) for each round, we map these four different actions to the four conductance states of the memristor. This is then mapped onto the TiO2 memristor array to form a feature map. In the TiO2 memristor array, changes in oxygen vacancy distribution due to oxygen diffusion cause conductance drift, i.e., the inherent attenuation factor of the device. We first measure the attenuation of the memristor conductance, such as... Figure 3 As shown, a function is then fitted onto the data. The four gray curves represent the results obtained from tests conducted at voltages between 1.2V and 1.5V, while the black curve represents the data obtained from the function fitting. The attenuation function after fitting is G = G0t. -θ Where G0 is the initial conductance, t is time, and θ≈0.084 is the drift coefficient. Therefore, we can replace the action state information with λ decay generated by the Sarsa(λ) algorithm with the physical decay function of the memristor to form a strategy graph, thereby improving the hardware fit of the software algorithm.

[0032] The reward value corresponding to each action in the generated strategy graph is used as the read voltage input memristor hardware array. The corresponding round value is obtained according to the array. The round value of each column is compared through the WTA winner-take-all circuit, and the largest round value among the 32 rounds is output.

[0033] It should be further explained that, in the specific implementation process, combined with this hardware acceleration system, the value update formula for the algorithm Sarsa(λ) is... Where t represents the number of steps in each round, a is the action information of the k-th step, represented as the conductance value in the memristor array, and R is the reward value for the corresponding step. This value can be calculated using the matrix arithmetic operations inherent in the memristor itself, i.e., setting the memristor read voltage to R. t-k+1 The conductivity value on the array is a k All action information for each round is input into the array column by column, and then the voltage is read row by row. According to the multiplication-addition law of the array and Kirchhoff's current law, the sum of the currents in each column is read as the value of that round. A WTA winner-take-all circuit is added to the output terminal to output the one with the largest voltage.

[0034] The memristor array is constructed by stacking individual crossbar memristors in rows and columns. Each crossbar on the array contains a memristor, and each memristor has an adjustable conductance. When a read voltage is applied to the memristor array, the current value at that point is obtained by multiplying the input voltage by the memristor's conductance. When multiple rows are input to the array, for each column, according to Kirchhoff's current law, the output of each column is the sum of the current values ​​of all memristors in that column, achieving matrix multiplication and addition. Here, we use a 1T1R structure for the memristor array, where one transistor corresponds to one memristor. The transistor drain is connected to the top electrode of the memristor to integrate the memristor array. Current crosstalk between different rows and columns is suppressed by controlling the transistor switching through gate voltage. This reduces read errors and improves the recognition accuracy of the hardware system.

[0035] The maximum round value obtained from the comparison is compared with the previously obtained maximum round value, and the round value with the larger value is retained. This process is repeated until convergence, and parallel computing is used to reduce the number of iterations. The entire reinforcement learning hardware acceleration system is as follows: Figure 2 As shown, in this invention, the system mainly utilizes memristors for parallel computation to reduce the number of iterations. Furthermore, to implement classic reinforcement learning algorithms for system verification, we fabricated TiO2 memristors to utilize their inherent decay characteristics as a physical decay function. The algorithm-generated paths are extracted and mapped to a policy array with self-decaying properties to improve the fit between the hardware and software systems. After extracting the round path information of this policy graph, it is input into a hardware memristor test array. Through repeated iterative comparisons and calculations until convergence, the system can significantly reduce the computer's throughput and the number of iterations, improving energy efficiency.

[0036] The above formulas are all numerical calculations after removing dimensions. The formulas are obtained by software simulation based on a large amount of data and are closest to the real situation. The preset parameters and preset thresholds in the formulas are set by those skilled in the art according to the actual situation or obtained by simulation based on a large amount of data.

[0037] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0038] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the present invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.

Claims

1. A method for hardware acceleration of reinforcement learning based on memristors, characterized in that, The method comprises the following steps: 32 round paths are obtained, and round paths with action information are output, and a strategy map is formed by mapping to a memristor array; Each action corresponding to a reward value in the generated strategy map is input as a read voltage to the memristor hardware array, and corresponding round values are obtained by array, each column of round values is compared through a WTA winner-takes-all circuit, and the maximum round value in 32 rounds is output; The maximum round value obtained by comparison is compared with the maximum round value obtained before, and the larger round value is retained, and iteration is repeated until convergence, and parallel computing is used to reduce the number of iterations; The function of parallel computing of the memristor array is used to simultaneously perform 32 column operations, 32 values are output, then a WTA winner-takes-all circuit is used for comparison, and the maximum round value is output.

2. The memristor-based reinforcement learning hardware acceleration method according to claim 1, wherein, The strategy map formed by the memristor naturally replaces the decay factor λ in the algorithm Sarsa(λ) which needs to perform a large number of exponential decay operations.

3. The memristor-based reinforcement learning hardware acceleration method according to claim 1, wherein, The round path contains state-action pair information.

4. The memristor-based reinforcement learning hardware acceleration method according to claim 1, wherein, The memristor array uses different conductance values to represent different state actions.

5. The memristor-based reinforcement learning hardware acceleration method according to claim 1, wherein, The read voltage of the memristor hardware array is used to represent the reward information of each state action pair, and the current value output according to the algorithm value update formula is the round value.

6. The memristor-based reinforcement learning hardware acceleration method according to claim 5, characterized in that, The algorithm value update formula is: Wherein, t represents t steps in each round, a is the action information of the kth step, represented as conductance value in the memristor array, R is the reward value of the corresponding step, the value is solved by matrix multiplication operation of the memristor itself, the memristor read voltage is set as , the conductance value on the array is , all the action information of each round is input into the array column by column, and then the read voltage is input row by row, and the current of each column read out according to the multiplication law of the array and the Kirchhoff current law is the round value.

7. The memristor-based reinforcement learning hardware acceleration method according to claim 1, wherein, According to the Sarsa(λ) algorithm, 32 round paths are generated for each batch, the action information containing four actions of up, down, left and right in each round is printed, four conductance states of the four different actions are obtained, and then a strategy map is formed in the TiO2 memristor array.

8. An apparatus, comprising: It comprises: One or more processors; Memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement a memristor-based reinforcement learning hardware acceleration method as claimed in any one of claims 1-7.

9. A storage medium containing computer-executable instructions, wherein: The computer executable instructions, when executed by a computer processor, are used to perform a memristor-based reinforcement learning hardware acceleration method as claimed in any one of claims 1-7.

Images