A convolution operator mapping method for multi-core memristor storage and computing platform

Abstract

The present invention discloses a convolution operator mapping method for a multi-core memristor storage-computing integrated platform, and relates to the technical field of memristor storage-computing integrated platform. The locality of input data and the inter-core communication overhead are considered at the same time, and the total communication cost is used as the optimization target, and finally a mapping scheme with a relatively small overall communication overhead is obtained. The method proposed in the present invention comprehensively considers the communication cost brought by direct reading and writing of the memory and the communication cost brought by multi-core synchronization, can effectively reuse the input data, give play to the parallelism of the memristor array, and obtain a mapping scheme with a relatively small communication cost. The method proposed in the present invention is relatively simple to implement and has strong portability. It can be added to the back-end of a neural network compiler for execution to complete the convolution operator mapping for a memristor storage-computing integrated platform.

Description

Technical Field

[0001] The present invention relates to the technical field of memristor storage-computing integrated platform, and in particular to a convolution operator mapping method for a multi-core memristor storage-computing integrated platform. Background Art

[0002] As the scale of neural networks and the complexity of algorithms increase, the computation-centric von Neumann architecture is limited by memory bandwidth and data movement overhead, so researchers have proposed the concept of in-memory computing. Unlike the traditional von Neumann architecture, in-memory computing is data-centric and places computing logic in memory, or directly uses the characteristics of the memory itself for computing, thereby greatly reducing the communication overhead between the core and external storage. It is particularly suitable for deep learning neural networks, which are large-scale parallel applications with large amounts of data. Therefore, accelerators that integrate computing and storage have emerged, using non-volatile memory (NVM) such as memristors as computing units. This type of accelerator effectively solves the bottleneck of bandwidth and has the characteristics of low power consumption and high speed.

[0003] Resistive Random Access Memory (ReRAM) is a non-volatile memory with almost zero leakage power consumption. It stores information in a resistance state. This resistive storage principle can provide inherent computing power, so data storage and data processing functions can be integrated at the same physical unit address. As a cutting-edge memory technology, ReRAM is expected to replace the current Flash RAM in the future, and has the advantages of lower cost and more outstanding performance. The energy consumption of ReRAM memory chips can reach 1 / 20 of flash memory, and the upper limit of data erasure is 10 times that of the latter. In addition, non-volatility allows data to be stored directly in the on-chip system, realizing instant startup / shutdown without the need for additional off-chip memory. ReRAM supports a variety of in-memory operations, such as matrix-vector multiplication (MVM), search, bit operations, etc., and there are a large number of convolution operations involving MVM operations in convolutional neural networks, and the amount of operations can reach more than 95% of the entire network operation. This is due to the characteristics of ReRAM itself. ReRAM can reduce the complexity of matrix-vector multiplication operations from o() to o(1). Therefore, ReRAM-based accelerators can significantly improve the computing efficiency of neural networks. These excellent properties have enabled researchers to realize a variety of neural network accelerators based on ReRAM. Among the many in-memory computing designs, ReRAM has obvious advantages over designs based on traditional CMOS.

[0004] As the research on deep learning deepens, many deep learning software development frameworks have emerged, such as TensorFlow, PyTorch, etc. These frameworks simplify the development of deep learning models. Unlike the relatively unified front end, the hardware types in the application back end are diverse. Researchers have made a lot of efforts to efficiently map deep learning models, such as the highly optimized linear algebra libraries MKL and cuBLAS on general-purpose processors, and TensorRT for GPUs, which supports graphics optimization and has a large number of optimized kernels. The disadvantage of relying on libraries is that the development of libraries lags behind the development of deep learning models, so new models cannot be quickly applied to the back end.

[0005] To solve the above problems, the industry has proposed many deep learning compilers that can quickly map deep learning models to backend hardware, such as TVM, Tensor Comprehension, XLA, etc. The technical details of these compilers vary, but the basic processes are roughly similar. They all take the model defined by the front-end deep learning development framework as input, and the valid code of various back-end hardware as output. They use multi-layer IR and highly optimize the model and the generated code during conversion.

[0006] The basic concept of storage and computing can be traced back to the 1970s. Kautz et al. of Stanford Research Institute first proposed the concept of storage and computing computers in 1969. In 2010, Professor Williams' team at HP Labs proposed and verified the feasibility of ReRAM to implement simple Boolean logic functions. In 2016, Professor Xie Yuan's team at the University of California, Santa Barbara (UCSB) proposed using ReRAM to build a deep learning neural network based on storage and computing architecture (abbreviated as PRIME), which has attracted widespread attention in the industry. Test results show that compared with traditional solutions based on von Neumann computing architecture, PRIME can achieve about 20 times lower power consumption and about 50 times higher speed. Also in 2016, Ali Shafiee et al. of the University of Utah proposed ISAAC, a convolutional neural network accelerator based on ReRAM. Compared with the DaDianNao architecture, its throughput, energy consumption, and computing density are optimized by 14.8 times, 5.5 times, and 7.5 times, respectively. They also proposed an intra-block pipeline design, which improves the throughput of the accelerator. In 2019, Aayush Ankit and others from Purdue University proposed the memristor neural network accelerator PUMA.

[0007] Currently, there are few compilers or code generation tools for storage-computing integrated programming frameworks in the industry. ReRAM-based accelerators such as ISAAC, FloatPIM, and Atomlayer only explore the advantages of ReRAM arrays in computing efficiency in terms of architecture, and do not provide programming and compilation tools; although PRIME and PipeLayer provide software and hardware programming interfaces, they do not combine compilation tools with mainstream neural network frameworks. PUMA provides a supporting compiler, but it only performs simple mapping processing and does not make further optimizations.

[0008] Therefore, technicians in this field are committed to developing a convolution operator mapping method for a multi-core memristor storage and computing platform. Taking into account the locality of input data and the inter-core communication overhead, and taking the total communication cost as the optimization target, a mapping solution with a relatively low overall communication overhead is finally obtained. Summary of the invention

[0009] In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is that only a simple mapping is performed on the convolution operator, and the locality of the input data of the memristor array during the convolution operation is not fully utilized; and for larger-scale convolution operators, simple mapping may cause additional inter-core communication overhead.

[0010] To achieve the above object, the present invention provides a convolution operator mapping method for a multi-core memristor storage and computing integrated platform, comprising the following steps:

[0011] Step 1. Expand the weights W[OC][IC][KH][KW] of the convolutional layer into a weight matrix W[IC×KH×KW][OC]. All weights in each row can be multiplied with the same input in parallel, and the output is obtained in parallel by columns. The output on each column is the result of multiplying and accumulating [IC×KH×KW] weights with the corresponding input, that is, each column is a convolution kernel, and there are OC convolution kernels in total.

[0012] Step 2: divide the rows and columns of the matrix obtained in step 1 according to the size xbar_size of the memristor array. The obtained matrix is ​​represented by P, where each element in P represents a memristor array of a certain size, and P[i][j] represents the core ID to which the memristor array in the i-th row and i-th column belongs.

[0013] Step 3: Complete the mapping for the matrix P.

[0014] Furthermore, the multi-core memristor storage and computing integrated platform is divided into Core level and Crossbar level from top to bottom.

[0015] Furthermore, multiple cores of the multi-core memristor storage and computing integrated platform share a global memory through a bus.

[0016] Furthermore, the Core includes an instruction fetch and decoding module, a loading module, a storage module and a computing module.

[0017] Furthermore, the Core includes a data storage device.

[0018] Furthermore, the Crossbar unit and tensor ALU on the Core are core computing units.

[0019] Furthermore, the step 3 adopts a greedy strategy.

[0020] Furthermore, the communication overhead between the convolution operator and the Core external memory is:

[0021] Target=rd_factor(P)+sync_facotr(P)

[0022] Among them, the read and write cost rd_factor(P) is generated when the Core directly reads and writes the external memory to obtain input data; the synchronization cost sync_facotr(P) is generated when the Core transfers data through the external memory.

[0023] Furthermore, the communication overhead between the convolution operator and the Core external memory is:

[0024]

[0025] The convolution operation input is the feature map In, weight W, step length stride, and the output feature map is Out[B][OC][OH][OW]; N is the number of columns in the P matrix, diffrow(core i ) means core i The number of elements in different rows in the matrix P, diffcol(core i ) means core i The number of elements in different columns in the matrix P; reuse(core i , stride) represents the number of memristor arrays that can be reused for input data in the matrix P when the convolution step is stride, and L is the number of cores involved in the mapping.

[0026] Furthermore, in step 3, each Core has X memristor arrays. First, the number K of Cores required for mapping weight W is determined, and then all elements of the P array are initialized to K, that is, all arrays are logically assigned to core K first, and then the entire matrix P is traversed to calculate the Target when assigning elements to the current Core to be assigned or the adjacent Core, and the smallest solution is selected. This process is repeated for the unassigned elements until the number of elements assigned to Core K is no greater than X.

[0027] In a preferred embodiment of the present invention, the present invention first proposes an abstract hardware architecture of multi-core storage and computing, which has a certain versatility of storage and computing architecture. The present invention aims to optimize the performance of the roll operator on this architecture and reduce the number of communication instructions.

[0028] 1 Hardware Structure

[0029] The invention is a storage-computing integrated acceleration system for multiple computing cores. For universality considerations, the invention proposes a simple multi-core accelerator architecture. The abstract architecture of the invention is divided into Core level and Crossbar level from top to bottom. Multiple Cores share a global memory through a bus. Figure 1 The details of the Core computing unit in this article are shown. It mainly consists of four modules: Figure 1 In the figure, there are instruction fetch and decode module 101, load module 102, storage module 103 and calculation module 104. In addition, each Core includes a data memory used to (a) serve as a buffer for input data, (b) temporarily store intermediate results, and (c) store instructions executed by the Core. The Crossbar unit and tensor ALU on the Core are core computing units, responsible for processing matrix-vector multiplication operations and some other tensor ALU operations respectively.

[0030] From the data path, before the accelerator starts computing, it first copies the input data and weight data from the host to the global memory, and then executes the instructions required for each Core to copy the computing task (the instructions are statically generated by the compiler in advance). When the instruction copy is completed, the instruction prefetch module on the Core unit reads the current instruction and performs preliminary decoding, and sends the instruction to the loading, computing or storage module according to the decoding result. The instruction can be pipelined between the above modules.

[0031] With respect to the abstract architecture, the present invention can stably reduce the number of communication instructions generated when executing the convolution operator.

[0032] 2 Mapping strategy

[0033] 2.1 Problem Abstraction

[0034] Algorithm 1 is a simple convolution operation. Its input is feature map In, weight W, step length stride, and its output feature map is Out[B][OC][OH][OW]. By observing this algorithm, we can find that for W, the oc axis can be calculated in parallel, and the data of the ic, kh, and kw axes are multiplied and accumulated with the input feature map. Therefore, when mapping, the oc axis should be mapped to different columns of the crossbar, and the ic, kh, and kw axes should be mapped to different rows. At the same time, it can be found that there is some overlap between adjacent convolution windows. Using this part of the data can further reduce the number of read and write instructions.

[0035] Assume that the backend architecture has L cores, each core has X memristor arrays (crossbars), and the size of each crossbar is xbar_size×xbar_size. After the weight is divided by xbar_size, the data on the oc axis is allocated to different columns of the crossbar, and the data on the ic axis, kh axis, and kw axis are allocated to different rows of the crossbar. The characteristics of memristors can be used to accelerate convolution operations.

[0036] To further abstract the mapping problem, first expand the weight tensor into a matrix W[IC×KH×KW][OC], where IC is the number of input channels, OC is the number of output channels, KH and KW are the sizes of the convolution kernels, and then further divide the rows and columns of W according to the size of the memristor array xbar_size to obtain a matrix P[M][N], where P is a mapping scheme. This process is as follows: Figure 2 As shown. P[i][j] represents the number of the Core to which the memristor array at the i-th row and j-th column belongs. At this time, the problem is transformed into finding the matrix P that minimizes the objective function.

[0037]

[0038]

[0039] 2.2 Objective Function

[0040] The communication overhead between the convolution operator and the Core external memory is divided into two parts. One part is generated by the Core directly reading and writing the external memory to obtain input data, that is, the read and write cost rd_factor(P), and the other part is generated by the Core passing data through the external memory, that is, the synchronization cost sync_facotr(P). The total communication overhead of these two parts is the optimization target of the present invention:

[0041] Target=rd_factor(P)+sync_facotr(P)#(1)

[0042] In convolution operation, the read and write cost of a convolution window operation should be the sum of the read and write costs of each Core. Assuming that a communication operation is a Core reading or writing xbar_size data, for a mapping P, since the same input data is used, the elements that are equal and located in the same row will only bring one communication operation. The elements that are equal and located in the same column will bring the same number of communication operations as the number of elements without considering the data reuse in the Core. When considering the data reuse in the Core, the number of reusable input elements needs to be subtracted. So from

[0043] Where diffrow(core i ) means core i The number of elements in different rows in the matrix P, reuse(core i , stride) represents the number of memristor arrays that can be reused for input data in the matrix P when the convolution step is stride, and L is the number of cores involved in the mapping.

[0044] Similarly, the number of synchronizations of a convolution window operation should be the number of unequal elements in the same column of P minus 1. One synchronization will result in two communication operations, one read and one write. Therefore, from the perspective of matrix P,

[0045]

[0046] Observing equations (2) and (3), we can see that they have the same factors OH and OW. Further simplification yields:

[0047] 2.3 Mapping Algorithm

[0048] The direct mapping algorithm is divided into row-first mapping and column-first mapping, which traverse the matrix P by row or column respectively to assign elements to the corresponding Core in order. Row-first mapping minimizes the number of different elements on the same row, but increases the number of different elements on the same column, that is, it reduces rd_factor(P) and increases sync_facotr(P). Similarly, column-first mapping minimizes the number of different elements on the same column, but increases the number of different elements on the same row, that is, it increases rd_factor(P) and reduces sync_facotr(P).

[0049] The present invention is based on a greedy algorithm and combines the characteristics of row-first mapping and column-first mapping. According to the size of the weight matrix, a mapping scheme with relatively low communication cost can be obtained, as shown in Algorithm 2.

[0050]

[0051]

[0052] The algorithm input is the number of Cores CoreNums, the number of crossbars on each Core XbarNums, the crossbar size xbar_size, the weight W, the convolution step length stride, and the initial mapping scheme P output is the final mapping scheme P.

[0053] The main process is: first determine the number of cores K required for mapping weight W, then initialize all elements of P array to K, that is, logically assign all arrays to core K first, then traverse the entire matrix P, calculate the target when assigning the element to the current core to be assigned or its adjacent core, select the smallest solution, and repeat this process for unassigned points until the number of elements assigned to Core K is no more than X. The overall flow chart is shown in Figure 3 .

[0054] 3. Result evaluation

[0055] Let L = 30, X = 16, use the direct mapping algorithm as a reference, and arbitrarily select several convolutional layers for evaluation. The results are as follows Figure 4 ,The results show that compared with the direct mapping algorithm, ,the mapping algorithm proposed in this invention can stably obtain a relatively ,optimal mapping scheme and can reduce the communication cost by ,21% on average.

[0056] Compared with the prior art, the present invention has the following obvious substantial features and significant advantages:

[0057] 1. Compared with the direct mapping strategy, the method proposed in the present invention comprehensively considers the communication cost brought by directly reading and writing the memory and the communication cost brought by multi-core synchronization, can effectively reuse the input data, give full play to the parallelism of the memristor array, and obtain a mapping scheme with lower communication cost.

[0058] 2. The method proposed in the present invention is relatively simple to implement and has strong portability. It can be added to the back-end of the neural network compiler for execution to complete the convolution operator mapping for the memristor storage and computing integrated platform.

[0059] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is an abstract architecture of multi-core memristor storage and computing in one of the preferred embodiments of the present invention;

[0061] Figure 2 This is a mapping process of convolution weights to a memristor array in a preferred embodiment of the present invention;

[0062] Figure 3 This is a mapping algorithm flow based on a greedy strategy in a preferred embodiment of the present invention;

[0063] Figure 4 This is the mapping result of a preferred embodiment of the present invention.

[0064] Among them, 101 is an instruction fetch and decoding module, 102 is a loading module, 103 is a storage module, and 104 is a computing module. DETAILED DESCRIPTION

[0065] The following describes several preferred embodiments of the present invention with reference to the drawings in the specification, so that the technical content is clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the protection scope of the present invention is not limited to the embodiments mentioned in the text.

[0066] In the drawings, components with the same structure are indicated by the same numerical reference numerals, and components with similar structures or functions are indicated by similar numerical reference numerals. The size and thickness of each component shown in the drawings are arbitrarily shown, and the present invention does not limit the size and thickness of each component. In order to make the illustration clearer, the thickness of the components is appropriately exaggerated in some places in the drawings.

[0067] The present invention is mainly aimed at the compilation of a multi-core memristor storage and computing integrated platform, and mainly focuses on the mapping optimization of convolution operators during the compilation process.

[0068] The multi-core memristor storage and computing integrated platform is based on a memristor array and supports matrix-vector operations. When performing convolution operations, the convolution operations are decomposed into multiple matrix-vector multiplication operations, and then the weights of the convolution kernels are mapped to multiple memristor arrays. When the convolution scale is large, the weight data will be mapped to multiple computing cores, and multiple computing cores are required to jointly complete the operation of the convolution operator. Different mapping schemes will bring different communication costs, and the traditional direct mapping strategy does not consider the inter-core communication cost and data locality. To address this problem, the present invention proposes a mapping strategy that can be applied to the compilation stage to obtain a mapping scheme with a relatively small inter-core communication cost.

[0069] The mapping strategy proposed in the present invention is mainly applied to the back end of the neural network compiler. In the back end of the compiler, a better mapping scheme can be obtained according to the parameters of the convolution operator and the size of the memristor array. The mapping process is as follows:

[0070] 1. Expand the weights W[OC][IC][KH][KW] of the convolutional layer into a weight matrix W[IC×KH×KW][OC]. All weights in each row can be multiplied with the same input in parallel, and the output is obtained in parallel by column. The output on each column is the result of multiplying and accumulating [IC×KH×KW] weights with the corresponding input, that is, each column is a convolution kernel, and there are OC convolution kernels in total. Figure 2 The first step is shown.

[0071] 2. Divide the rows and columns of the matrix obtained in the previous step according to the size xbar_size of the memristor array, such as Figure 2 As shown in the second step, the obtained matrix is ​​represented by P, where each element in P represents a memristor array of a certain size, and P[i][j] represents the core ID to which the memristor array of the i-th row and j-th column belongs.

[0072] 3. Use Algorithm 2 for matrix P to complete the mapping.

[0073] A convolution operator mapping method for a multi-core memristor storage-computing integrated platform of the present invention comprises the following steps:

[0074] Step 1: Input the number of Cores (computing cores) CoreNums, the number of Crossbars (memristor arrays) on each Core XbarNums, the size of the Crossbar xbar_size, the weight matrix W (including its sizes OC, IC, KH, KW), and the stride of the convolution kernel stride.

[0075] Step 2: According to the method in 2.1 and the parameters in step 1, derive the size (M×N) of the mapping matrix P and the number of cores K required to map the current convolution operator.

[0076] Step 3: Initialize all elements of the P matrix to K.

[0077] Step 4 initializes the number curId of the current Core to be allocated to 1, and initializes the current optimal mapping solution curP to P.

[0078] Step 5 initializes the communication cost of the current mapping scheme to infinity.

[0079] Step 5.1 traverses the entire P matrix, and calculates the communication cost when replacing the element with value K with curId or curId+1 according to Formula 4 in 2.2, and retains the solution with the lowest communication cost in curP.

[0080] Step 5.2 If there is no crossbar available for allocation on the Core corresponding to number curId, curId is incremented by 1 and allocation starts for the next Core.

[0081] Step 5.3 updates P to curP.

[0082] In step 6, repeat step 5 when the number of elements with value K in P is greater than XbarNums.

[0083] Step 7P is the final mapping solution.

[0084] The present invention proposes a convolution operator mapping method for a multi-core memristor storage-computing integrated architecture.

[0085] The present invention proposes an evaluation function of communication cost for a convolution operator, which simultaneously considers the communication cost caused by direct memory access and the communication cost caused by multi-core synchronization, and provides a corresponding expression.

[0086] The method proposed in the present invention takes the reuse of input data into consideration when mapping weights, which can effectively reduce the communication cost and obtain a better mapping solution.

[0087] The greedy strategy adopted by the present invention is simple to implement, has strong portability, and is faster than other intelligent search algorithms.

[0088] The preferred specific embodiments of the present invention are described in detail above. It should be understood that ordinary technicians in the field can make many modifications and changes based on the concept of the present invention without creative work. Therefore, all technical solutions that can be obtained by technicians in the technical field based on the concept of the present invention through logical analysis, reasoning or limited experiments on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A convolution operator mapping method for a multi-core memristor storage and computing integrated platform, characterized in that: The following steps are involved: Step 1. Expand the weights W[OC][IC][KH][KW] of the convolutional layer into a weight matrix W[IC×KH×KW][OC]. All weights in each row are multiplied in parallel with the same input, and the output is obtained in parallel by columns. The output on each column is the result of multiplying and accumulating [IC×KH×KW] weights with the corresponding input, that is, each column is a convolution kernel, and there are OC convolution kernels in total. Step 2: divide the rows and columns of the matrix obtained in step 1 according to the size xbar_size of the memristor array. The obtained matrix is ​​represented by P, where each element in P represents a memristor array of a certain size, and P[i][j] represents the core ID to which the memristor array in the i-th row and j-th column belongs. Step 3, complete the mapping for the matrix P; The step 3 adopts a greedy strategy; The communication overhead between the convolution operator and the Core external memory is: Target=rd_factor(P)+sync_facotr(P) Among them, the read and write cost rd_factor(P) is generated when the Core directly reads and writes external memory to obtain input data; the synchronization cost sync_facotr(P) is generated when Cores transfer data through external memory; The communication overhead between the convolution operator and the Core external memory is: The convolution operation input is the feature map In, weight W, step length stride, and the output feature map is Out[B][OC][OH][OW]; N is the number of columns in the P matrix, diffrow(core i ) means core i The number of elements in different rows in the matrix P, diffcol(core i ) means core i The number of elements in different columns in the matrix P; reuse(core i , stride) represents the number of memristor arrays that can be reused for input data in the matrix P when the convolution step is stride, and L is the number of cores involved in the mapping; In step 3, each core has X memristor arrays. First, the number K of cores required for mapping weight W is determined. Then, all elements of the P array are initialized to K, that is, all arrays are logically assigned to core K first. Then, the entire matrix P is traversed to calculate the Target when assigning elements to the current Core to be assigned or the adjacent Core. The smallest solution is selected, and this process is repeated for the unassigned elements until the number of elements assigned to Core K is no greater than X.

2. The convolution operator mapping method for a multi-core memristor storage and computing integrated platform as claimed in claim 1, characterized in that: The multi-core memristor storage and computing integrated platform is divided into Core level and Crossbar level from top to bottom.

3. The convolution operator mapping method for a multi-core memristor storage and computing integrated platform as claimed in claim 1, characterized in that: Multiple cores of the multi-core memristor storage and computing integrated platform share a global memory through a bus.

4. The convolution operator mapping method for a multi-core memristor storage and computing integrated platform as claimed in claim 1, characterized in that: The Core includes an instruction fetch and decode module, a loading module, a storage module and a calculation module.

5. The convolution operator mapping method for a multi-core memristor storage and computing integrated platform as claimed in claim 1, characterized in that: The Core includes a data storage device.

6. The convolution operator mapping method for a multi-core memristor storage and computing integrated platform as claimed in claim 1, characterized in that: The Crossbar unit and tensor ALU on the Core are the core computing units.

Images