A crossbar level heterogeneous ReRAM accelerator configuration method and system for DNN model

Abstract

The application provides a Crossbar-level heterogeneous ReRAM accelerator configuration method and system for a DNN model, the application is oriented to an in-memory computing architecture based on ReRAM, the relationship between weight parameters in a model layer and the size of a Crossbar in performance is analyzed, and the idea of hierarchical heterogeneity is proposed; a reinforcement learning algorithm is used to automatically configure the model heterogeneity, and in the case of not losing the model accuracy, the Crossbar utilization can be effectively improved, and the system computing delay and energy consumption can be reduced.

Description

Technical Field

[0001] This invention relates to the field of computer science and artificial intelligence, and in particular to a method and system for configuring a Crossbar-level heterogeneous ReRAM accelerator for DNN models. Background Technology

[0002] Artificial intelligence (AI) technology has already played a significant role in numerous fields and has garnered considerable attention from both industry and academia. Deep neural networks (DNNs), as an evolved form of AI neural networks, have applications spanning various areas, such as image processing, speech recognition, natural language processing, and recommendation systems. However, to improve the performance of neural networks during training and inference, the number of layers and parameters in networks is constantly increasing, leading to a huge demand for computing resources. Traditional computing hardware, such as central processing units (CPUs) and graphics processing units (GPUs), often faces computational bottlenecks when processing complex DNN models, limiting the model's scale and performance. Secondly, the computational process of DNNs typically requires substantial data transfer and storage. Traditional storage solutions, such as dynamic random access memory (DRAM), can become performance bottlenecks when processing massive amounts of data. Furthermore, high-performance computing hardware usually requires a large power supply, resulting in high energy costs and adverse environmental impacts. Additionally, for mobile devices and embedded systems, high power consumption limits battery life and device portability. As DNN models continue to evolve, their scale and complexity are also rapidly increasing. This requires more storage and higher computing power, thus necessitating new hardware technologies to meet the challenges.

[0003] Applying rewritable resistive random access memory (ReRAM) to address the challenges of deep neural networks (DNNs) is a field of considerable interest. ReRAM is a type of in-memory computation (PIM) device that enables matrix-vector multiplication (MVM) to be performed in-situ, thus saving data transfer overhead. For example... Figure 1 As shown, in a ReRAM-based accelerator, the crossbar is a cross-computation array composed of ReRAM-based cells. By storing the DNN's parameter weights as conductances and converting the input data into voltages, the current obtained by multiplying these two values ​​according to Ohm's law is the result of one dot product in the neural network. Connecting the input word lines in parallel along the row direction and superimposing the output bit lines along the column direction is equivalent to completing one MVM operation. It has the following characteristics:

[0004] 1) Low power consumption: ReRAM has low power consumption characteristics because it does not require an additional power supply during state retention. This allows it to significantly reduce power consumption during inference, making it suitable for mobile devices and energy-constrained environments.

[0005] 2) High density and fast access: ReRAM enables high-density data storage and fast access speed. This allows it to effectively manage large-scale models and large datasets in DNNs.

[0006] 3) Programmability: ReRAM can store data by changing the resistance state, which makes it suitable for implementing weights and connections between neurons. This programmability makes ReRAM a potential hardware accelerator for accelerating DNN inference.

[0007] 4) Non-volatile: ReRAM is a type of non-volatile memory (NVM) that can store data without requiring a continuous power supply. This is crucial for the reliability and data persistence of DNN models.

[0008] Despite its immense potential, research on ReRAM-based DNN accelerators is still in its early stages, with many challenges yet to be overcome. Generally, improving crossbar utilization and reducing overall computational latency and energy consumption are key design considerations. Previous work typically employed homogeneous hardware designs, where all crossbar arrays had identical dimensions. However, in DNNs, the configurations of layers at different depths can vary. For example, the height and width of two-dimensional convolutional kernels can differ, as can the number of channels in each layer. This mismatch introduces fragmentation within the crossbars into the weight mapping results. Furthermore, in the current prevalent multi-level computational unit architecture—Processing Engine (PE)—Tile—Crossbar—the mapping strategy for a particular layer may exclusively occupy the selected tile, even if all weights for that layer have been mapped and there are still free crossbars within the tile, preventing their allocation to other layers. All of these factors indicate room for improvement in utilization.

[0009] Regarding system performance, past research has shown that the size of the crossbar affects system energy consumption. This is mainly due to the high energy consumption of external circuits (PCs), such as analog-to-digital converters (ADCs). Mapping a parameter matrix of the same size to a large crossbar can reduce energy consumption because a smaller crossbar will segment the parameter matrix more, thus introducing more ADCs. However, a large crossbar is more prone to internal fragmentation, reducing overall utilization and computational efficiency. These two factors constitute a trade-off regarding crossbar size. To better leverage the performance of ReRAM accelerators, it is worthwhile to consider selecting appropriate crossbar sizes for different layers in the network. However, with the increasing depth of DNN layers and the existence of data dependencies between layers, the search space becomes enormous and difficult to specify manually; furthermore, implementing a hierarchical heterogeneous architecture design is also a challenge. Summary of the Invention

[0010] The purpose of this invention is to address the problems of large differences in weights between different layers in DNNs and low hardware utilization when using homogeneous Crossbar mapping, and to propose a Crossbar-level heterogeneous ReRAM accelerator configuration method and system for DNN models.

[0011] The technical solution adopted in this invention is:

[0012] A method for configuring a Crossbar-level heterogeneous ReRAM accelerator for DNN models, characterized in that:

[0013] Obtain the structural information of each network layer of the DNN model as state information;

[0014] Two paired neural network models are selected as the real performer-target performer and the real commentator-target commentator, respectively. The optimal hierarchical heterogeneous Crossbar configuration is used as the learning objective, and several batches of iterative configurations are performed based on reinforcement learning.

[0015] In each batch, each iteration takes the structural information of the current network layer as input, learns valuable information from the environment through the real performer and the target performer, and outputs actions to influence the environment at each step; the action is the Crossbar size corresponding to a network layer of the DNN model; after all iterations are completed, the action policy is output; after all iterations are completed, the environment will reward according to the performance of the action policy; the real commentator and the target commentator output evaluations for each action output by the real performer and the target performer, respectively; the real commentator optimizes the network parameters by minimizing the loss values ​​of the real commentator and the target commentator as the loss function; the real performer optimizes the network parameters by minimizing the negative number of the output of the real commentator to obtain the policy gradient for parameter update; the target commentator and the target performer use soft updates; the reward function is a polynomial composed of the normalized Crossbar utilization u and the system energy e; the action policy output in the last batch is the final Crossbar-level heterogeneous ReRAM accelerator configuration result.

[0016] Furthermore, the structural information of each network layer includes one or more of the following: network layer number, network layer type, number of input channels, number of output channels, size of network layer convolution kernel, convolution stride, number of network layer weight parameters, and size of input feature map.

[0017] Furthermore, in each iteration, the combination of state, action, and reward is recorded as an experience sample and stored in the experience pool.

[0018] Furthermore, in each batch of iterative configurations, after all iterations are completed, a batch of sample data will be sampled from the experience pool to train and optimize the performers and commentators.

[0019] Furthermore, in the iterative configuration process based on reinforcement learning, the first few batches are a warm-up phase, and the actions are generated randomly.

[0020] Furthermore, the reward function is expressed as:

[0021] reward=h u *u+h e *(1-e)

[0022] Where h u and h e It is a hyperparameter that achieves a trade-off between utilization rate and energy consumption.

[0023] Furthermore, the loss function corresponding to the real commentator is expressed as follows:

[0024]

[0025] In the formula, S i ,a i S represents the structural information corresponding to the i-th layer of the DNN model and the action output by the real performer, respectively. i+1 ,a i+1 Q represents the structural information of the (i+1)th layer of the DNN model and the action output by the target performer, respectively. w (*) represents the evaluation of the current state-action by the real-world commentator, Q. w′ (*) indicates the target commenter's evaluation of the next layer's state-action; N represents the number of layers in the DNN model network.

[0026] Furthermore, it also includes combining and merging blocks based on the principle of best adaptation for the final Crossbar-level heterogeneous ReRAM accelerator configuration results.

[0027] Furthermore, the environment is implemented through a simulator, which receives the action strategy output by the performer, performs the mapping and inference process from the DNN model to the ReRAM Crossbar, and collects utilization and energy consumption performance data for feedback.

[0028] A Crossbar-level heterogeneous ReRAM accelerator configuration system for DNN models, used to implement the Crossbar-level heterogeneous ReRAM accelerator configuration method for DNN models, includes:

[0029] The data acquisition module is used to acquire the structural information of each network layer of the DNN model as state information;

[0030] The reinforcement learning module selects two paired neural network models as the real performer-target performer and the real commentator-target commentator, respectively. It uses the optimal hierarchical heterogeneous Crossbar configuration as the learning objective and performs several batches of iterative configurations based on reinforcement learning.

[0031] In each batch, each iteration takes the structural information of the current network layer as input, learns valuable information from the environment through the real performer and the target performer, and outputs actions at each step to influence the environment; the action is the Crossbar size corresponding to a network layer of the DNN model; after all iterations are completed, the action policy is output; after all iterations are completed, the environment will reward according to the performance of the action policy; the real commentator and the target commentator output evaluations for each action output by the real performer and the target performer, respectively; among them, the real commentator optimizes the network parameters by minimizing the loss value of the real commentator and the target commentator as the loss function; the real performer optimizes the network parameters by minimizing the negative number of the output of the real commentator to obtain the policy gradient for parameter update; the target commentator and the target performer use soft updates; the reward function is a polynomial composed of the normalized Crossbar utilization u and the system energy e; the action policy output in the last batch is the final Crossbar-level heterogeneous ReRAM accelerator configuration result;

[0032] The simulator is responsible for receiving action policies from the reinforcement learning module, executing the mapping and inference process from the DNN model to the ReRAM Crossbar, and collecting utilization and energy consumption performance data for feedback.

[0033] The beneficial effects of this invention are as follows: This invention is aimed at the in-memory computing architecture based on ReRAM, analyzes the relationship between the weight parameters and the Crossbar size in the model layer in terms of performance, and proposes the idea of ​​hierarchical heterogeneity; specifically, it adopts reinforcement learning agent to interact with ReRAM accelerator to realize automatic search and heterogeneous model configuration, which can effectively improve Crossbar utilization and reduce system computing latency and energy consumption without losing model accuracy. Attached Figure Description

[0034] Figure 1 This is a diagram of the ReRAM cell and Crossbar array structure;

[0035] Figure 2 This is a diagram of a Crossbar-level heterogeneous ReRAM accelerator architecture;

[0036] Figure 3This is a flowchart illustrating a Crossbar-level heterogeneous ReRAM accelerator configuration method for DNN models based on reinforcement learning, as described in this invention.

[0037] Figure 4 It is a diagram showing the correspondence between state-space symbols;

[0038] Figure 5 This is a schematic diagram of tile merging based on the Best-Fit principle;

[0039] Figure 6 These are experimental test results diagrams of embodiments of the present invention;

[0040] Figure 7 The figures show the experimental results before and after tile merging based on the best-fit principle on the VGG16-MNIST model test set.

[0041] Figure 8 This is a structural diagram of a Crossbar-level heterogeneous ReRAM accelerator configuration system for DNN models based on reinforcement learning, according to the present invention. Detailed Implementation

[0042] Figure 2 This is a diagram of a Crossbar-level heterogeneous ReRAM accelerator architecture. (See diagram for example.) Figure 2 As shown, the accelerator employs a classic hierarchical architecture to maintain versatility and computational parallelism. From top to bottom, the architecture consists of ReRAM banks, tiles, computation engines (PEs), and crossbars. A unified central controller manages the interaction between the accelerator and the method of this invention, as well as data movement on the accelerator. Each bank contains input and output buffers for storing input and output data, and an array of tiles for computation. A tile is the smallest allocation unit when performing DNN model network layer mapping, meaning that even if there is free space in a tile, it cannot be allocated to other layers. A tile mainly consists of a certain number of PEs, in addition to input and output buffers and pooling computation modules for handling pooling layer operations. Each PE contains a certain number of crossbars for parallel processing of different bits of a weight, and finally, the intermediate results are shifted and added using an adder tree. This invention provides a method and system for configuring a crossbar-level heterogeneous ReRAM accelerator for DNN models, capable of mapping different DNN model network layers to an optimally sized crossbar, achieving high crossbar utilization and energy efficiency. Figure 3This is a flowchart of a Crossbar-level heterogeneous ReRAM accelerator configuration method for DNN models based on reinforcement learning, as described in this invention. This method is based on the Deep Deterministic Policy Gradient (DDPG) algorithm and uses the Actor-Critic algorithm as its basic framework. Both the action function and evaluation function employ a dual neural network model architecture, serving as the real network and the target network respectively (using conventional neural network models, such as a two-layer perceptron), as the core of the agent. DDPG is a model-free policy algorithm for learning continuous actions, used to solve reinforcement learning problems in continuous action spaces. Simultaneously, this algorithm introduces an experience replay mechanism, using an experience pool to store experience samples of agent-environment interactions. Batch data samples are extracted for training, removing correlations and dependencies between samples, making the algorithm more prone to convergence. Specifically, the method of this invention is as follows:

[0043] The structural information of each network layer of the DNN model is obtained as the state information; two pairs of neural network models are selected as the real performer-target performer and the real commentator-target commentator, respectively. The optimal hierarchical heterogeneous Crossbar configuration is used as the learning target, and several batches of iterative configurations are performed based on reinforcement learning.

[0044] In each batch, each iteration takes the structural information of the current network layer as input, learns valuable information from the environment through real performers and target performers, and generates actions (defined as the crossbar size corresponding to a layer in the DNN model) at each step to influence the environment. After multiple actions, i.e., after all iterations are completed, the action policy (the combination of all actions, i.e., the crossbar size corresponding to each network layer of the DNN model) is output. The environment evaluates its performance by rewarding the actions performed. In this system, the real commentator and the target commentator output evaluations for each action performed by the real performer and the target performer, respectively. The real commentator optimizes the network parameters by minimizing the mean squared error loss between the evaluation values ​​output by the real commentator and the evaluation values ​​output by the target commentator as the loss function. The real performer provides the policy for the current state and optimizes the network parameters by combining the policy gradient obtained from the output of the real commentator with the policy gradient used for parameter updates. The target commentator and the target performer use soft updates, i.e., they periodically update the network parameters based on the network parameters of the real commentator and the real performer, respectively. To simultaneously evaluate the performance on Crossbar utilization and system energy consumption, and for ease of comparison, the reward function is set as a polynomial composed of the normalized Crossbar utilization u and the system energy e, i.e., reward = h. u *u+h e *(1-e), where h u and h eIt is a hyperparameter that achieves a trade-off between utilization and energy consumption. This process is iterative until the agent obtains a satisfactory optimal reward.

[0045] The loss function corresponding to the commenter is expressed as follows:

[0046]

[0047] In the formula, S i ,a i S represents the structural information corresponding to the i-th layer of the DNN model and the action output by the real performer, respectively. i+1 ,a i+1 Q represents the structural information of the (i+1)th layer of the DNN model and the action output by the target performer, respectively. w (*) represents the evaluation of the current state-action by the real-world commentator, Q. w′ (*) indicates the target commentator's evaluation of the next-layer state-action; N represents the number of layers in the DNN model network. The environment can be implemented using a simulator or a ReRAM accelerator. In one specific embodiment, a simulator is used. The simulator receives the performer's output action strategy, executes the mapping and inference process from the DNN model to the ReRAM Crossbar, and collects utilization and energy consumption performance data for feedback. Specifically:

[0048] like Figure 4 As shown, the state is defined as a vector consisting of eight elements. At the beginning of an iteration, the state space is updated according to the current state of the system, and then the agent generates an action based on the current state. After generating a corresponding action for each network layer of the DNN model, these actions form a heterogeneous action policy. The simulator performs mapping and DNN inference according to this policy and provides performance data feedback, including hardware utilization and energy consumption. The reinforcement learning module calculates and generates the corresponding reward value through a reward function based on the feedback metrics.

[0049] In one specific embodiment, each iteration further includes recording a combination of state, action, and reward as an experience sample and storing it in an experience pool; in each batch, the agent samples a batch of data from the experience pool and trains and optimizes the performer and commentator models in the background.

[0050] In one specific embodiment, the first few batches (e.g., 20 batches) are a warm-up phase, with actions generated randomly, and the experience samples obtained from these 20 batches serve as initial guidance.

[0051] In one specific embodiment, DNN mapping allocation at the tile granularity can lead to wasted Crossbar space within tiles. This invention provides a tile merging method to improve Crossbar utilization. After specifying the strategy for the current iteration, the central controller calculates the number of tiles required to map each DNN layer and combines tiles of the same Crossbar size according to the best fit principle, such as... Figure 5 As shown.

[0052] The effects of the present invention will be further illustrated below with reference to a specific embodiment:

[0053] The specific experiment is as follows:

[0054] Experimental setup:

[0055] (1) ReRAM accelerator simulator: MNSIM;

[0056] (1) Models: VGG16, AlexNet, ResNet152 standard code;

[0057] (2) Datasets: MNIST, ImageNet;

[0058] Final test results:

[0059] The mapping configuration method not using this invention: mapping all layers of the DNN to isomorphic crossbars, and not using the tile merging method; using the method of this invention: selecting five different sizes of crossbars (32*32, 36*32, 72*64, 288*256, 576*512) as the action range of the method of this invention. In each iteration, the Actor selects one from this range as the action for that step. Due to the existence of the experience replay mechanism, the crossbar size configuration selected by the Actor tends to improve the target defined by the reward function. This experiment uses the sum of the normalized utilization rate and the reciprocal of the energy consumption as the indicator to evaluate the performance of the architecture. The specific formula is as follows: Where U is the normalized utilization rate and E is the reciprocal of the normalized energy consumption.

[0060] like Figure 6As shown, the method of this invention improves the performance metrics by 1.9 times compared to the mapping configuration method without this invention. Experimental results show that the present invention outperforms the ResNet152 model on AlexNet and VGG16 models. This is because the ResNet152 model contains 104 convolutional layers of size 1x1, and mapping them to a square crossbar would obviously bring better utilization. However, this experiment only provides one square crossbar option, namely 32*32, which limits the system performance. In addition, since some layers of ResNet152 contain 7*7 convolutional kernels, they cannot be mapped to 32*32 and 36*32 crossbars, so the first two columns of ResNet152 are zero in the experimental results. In addition, under the VGG16+MNIST test configuration, this experiment tested the effect of the aforementioned tile merging scheme. Figure 7 As shown, compared to the scheme without tile merging, the merged scheme improved Crossbar utilization by 1.02 times and reduced energy consumption by 30%.

[0061] Corresponding to the aforementioned embodiments of the reinforcement learning-driven ReRAM-based automated heterogeneous configuration DNN acceleration method, the present invention also provides embodiments of the reinforcement learning-driven ReRAM-based automated heterogeneous configuration DNN acceleration system.

[0062] See Figure 8 This invention provides a reinforcement learning-driven, ReRAM-based, automated heterogeneous DNN acceleration system, comprising:

[0063] The data acquisition module is used to acquire the structural information of each network layer of the DNN model as state information;

[0064] The reinforcement learning module selects two paired neural network models as the real performer-target performer and the real commentator-target commentator, respectively. It uses the optimal hierarchical heterogeneous Crossbar configuration as the learning objective and performs several batches of iterative configurations based on reinforcement learning.

[0065] In each batch, each iteration takes the structural information of the current network layer as input, learns valuable information from the environment through the real performer and the target performer, and outputs actions at each step to influence the environment; the action is the Crossbar size corresponding to a network layer of the DNN model; after all iterations are completed, the action policy is output; after all iterations are completed, the environment will reward according to the performance of the action policy; the real commentator and the target commentator output evaluations for each action output by the real performer and the target performer, respectively; among them, the real commentator optimizes the network parameters by minimizing the loss value of the real commentator and the target commentator as the loss function; the real performer optimizes the network parameters by minimizing the negative number of the output of the real commentator to obtain the policy gradient for parameter update; the target commentator and the target performer use soft updates; the reward function is a polynomial composed of the normalized Crossbar utilization u and the system energy e; the action policy output in the last batch is the final Crossbar-level heterogeneous ReRAM accelerator configuration result;

[0066] The simulator is responsible for receiving action policies from the reinforcement learning module, executing the mapping and inference process from the DNN model to the ReRAM Crossbar, and collecting utilization and energy consumption performance data for feedback.

[0067] For the system embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0068] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A Crossbar-level heterogeneous ReRAM accelerator configuration method for a DNN model, characterized in that, Specifically: Obtain the structural information of each network layer of the DNN model as state information; Two paired neural network models are selected as the real performer-target performer and the real commentator-target commentator, respectively. The optimal hierarchical heterogeneous Crossbar configuration is used as the learning objective, and several batches of iterative configurations are performed based on reinforcement learning. In each batch, each iteration takes the structural information of the current network layer as input, learns valuable information from the environment through the real performer and the target performer, and outputs actions at each step to influence the environment; the action is the crossbar size corresponding to a network layer of the DNN model; after all iterations are completed, the action policy is output; after all iterations are completed, the environment will reward according to the performance of the action policy; the real commentator and the target commentator output evaluations for each action output by the real performer and the target performer, respectively; wherein, the real commentator optimizes the network parameters by minimizing the loss values ​​of the real commentator and the target commentator as the loss function; the real performer obtains the parameter update by minimizing the negative number of the output of the real commentator. The network parameters are optimized using policy gradients; soft updates are used for the target commentator and target performer; the reward function is a polynomial composed of the normalized Crossbar utilization u and the system energy e; the action policy output in the last batch is the final Crossbar-level heterogeneous ReRAM accelerator configuration result; in each iteration, the combination of state, action, and reward is recorded as an experience sample and stored in the experience pool; in each batch of iterative configuration, after all iterations are completed, a batch of sample data is sampled from the experience pool to train and optimize the real performer and real commentator; in the iterative configuration process based on reinforcement learning, the first few batches are a warm-up phase, and actions are randomly generated; the reward function is expressed as: wherein and are hyperparameters that enable trade-off between utilization and energy consumption.

2. The method of claim 1, wherein, The structural information of each network layer includes one or more of the following: network layer number, network layer type, number of input channels, number of output channels, size of network layer convolution kernel, convolution stride, number of network layer weight parameters, and size of input feature map.

3. The method according to claim 1, characterized in that, The loss function corresponding to the real commentator is expressed as follows: In the formula, These represent the structural information corresponding to the i-th layer of the DNN model and the action output by the real performer, respectively. These represent the structural information of the (i+1)th layer of the DNN model and the action output by the target performer, respectively. This represents the evaluation of the current state-action by a real-world commentator. This represents the target commenter's evaluation of the next level state-action. This indicates the number of network layers in the DNN model.

4. The method according to claim 1, characterized in that, It also includes combining and merging blocks based on the principle of best fit for the final Crossbar-level heterogeneous ReRAM accelerator configuration results.

5. The method according to claim 1, characterized in that, The environment is implemented through a simulator, which receives the action strategies output by the real performer and the target performer, executes the mapping and inference process from the DNN model to the ReRAM Crossbar, and collects utilization and energy consumption performance data for feedback.

6. A Crossbar-level heterogeneous ReRAM accelerator configuration system for DNN models, used to implement the Crossbar-level heterogeneous ReRAM accelerator configuration method for DNN models as described in any one of claims 1-5, characterized in that, include: The data acquisition module is used to acquire the structural information of each network layer of the DNN model as state information; The reinforcement learning module selects two paired neural network models as the real performer-target performer and the real commentator-target commentator, respectively. It uses the optimal hierarchical heterogeneous Crossbar configuration as the learning objective and performs several batches of iterative configurations based on reinforcement learning. In each batch, each iteration takes the structural information of the current network layer as input, learns valuable information from the environment through the real performer and the target performer, and outputs actions at each step to influence the environment; the action is the Crossbar size corresponding to a network layer of the DNN model; after all iterations are completed, the action policy is output; after all iterations are completed, the environment will reward according to the performance of the action policy; the real commentator and the target commentator output evaluations for each action output by the real performer and the target performer, respectively; among them, the real commentator optimizes the network parameters by minimizing the loss value of the real commentator and the target commentator as the loss function; the real performer optimizes the network parameters by minimizing the negative number of the output of the real commentator to obtain the policy gradient for parameter update; the target commentator and the target performer use soft updates; the reward function is a polynomial composed of the normalized Crossbar utilization u and the system energy e; the action policy output in the last batch is the final Crossbar-level heterogeneous ReRAM accelerator configuration result; The simulator is responsible for receiving action policies from the reinforcement learning module, executing the mapping and inference process from the DNN model to the ReRAM Crossbar, and collecting utilization and energy consumption performance data for feedback.

Images