A device and method for in-memory neural network computing based on heterogeneous storage

Abstract

The present invention discloses an apparatus for in-memory neural network computing based on heterogeneous storage, comprising first and third computing modules. The first computing module has an in-memory computing memory for data storage and data operations in a computationally intensive layer, while the third computing module has an in-memory computing memory for data storage and data operations in a storage-intensive layer. The computationally intensive layer includes a layer in a neural network model where the amount of computation is greater than the amount of parameters, and the storage-intensive layer includes a layer in a neural network model where the amount of parameters is greater than the amount of computation. The computing speed of the first computing module is greater than the computing speed of the third computing module, and the storage capacity of the third computing module is greater than the storage capacity of the first computing module.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence chip technology, and in particular to a device and method for in-memory neural network computing based on heterogeneous storage. Background Art

[0002] Current artificial intelligence chips generally use the traditional von Neumann architecture, such as Figure 1 As shown in Figure 1, the architecture primarily consists of three parts: computing units, on-chip cache, and off-chip memory, with storage and computing separated. However, because AI algorithms such as deep neural networks rely on massive amounts of data and complex algorithmic models, current AI chips and traditional general-purpose processor architectures (such as CPUs and GPUs) require frequent and extensive data reads, writes, and transfers between off-chip memory, on-chip cache, and computing units when computing intelligent algorithms. As a result, in convolutional neural network and recurrent neural network algorithm models, off-chip memory reads and writes account for over 80% of the system's total processing energy consumption and latency, causing a memory wall problem that severely impacts the AI ​​chip's computational energy efficiency.

[0003] In order to solve the storage wall problem, the in-memory computing architecture came into being. The in-memory computing architecture integrates the computing unit into the storage array. The current representative in-memory computing architecture can be mainly divided into two categories: Figure 2a The digital in-memory computing architecture based on static random access memory (SRAM) shown in Figure 2bThe figure shows an analog in-memory computing architecture based on nonvolatile memory (NVM). As shown in the figure, the digital in-memory computing architecture adds basic logic circuits to the read-out end of the SRAM array, enabling bit-level multiplication operations within the memory array and performing multiplication-accumulation calculations through accumulation units close to the memory array. The advantages of digital in-memory computing are fast SRAM read and write speeds, and no precision loss in logic gate-based calculations. However, due to the limited capacity of SRAM, typically only in the KB range, when processing large-scale neural network models such as those in the MB range, algorithm weight data still needs to be stored in larger off-chip memory. Frequent weight data transfer and SRAM read and write operations are performed during the calculation process, resulting in a time overhead that can account for 50% or more of the total digital in-memory computing time. The analog in-memory computing architecture uses high-precision digital-to-analog and analog-to-digital converters at the read-in and read-out ends of the NVM array. During calculations, several rows of the memory array are opened at a time. By applying different input voltages to each row and reading the output current in each column, analog-domain matrix-vector multiplication calculations are performed within the memory array. The advantage of analog in-memory computing lies in the large capacity of NVM, typically in the MB to GB range, eliminating the need for off-chip memory for weight data storage. However, NVM has slow read and write speeds and high energy consumption. Furthermore, analog computing requires conversion between analog and digital signals and is susceptible to noise interference, leading to significant computational reliability issues. Summary of the Invention

[0004] In order to address some or all of the problems in the prior art, the present invention provides, on one hand, a device for in-memory neural network computing based on heterogeneous storage, comprising:

[0005] a first computing module having an in-memory computing memory for storing and computing data in a computationally intensive layer, wherein the computationally intensive layer comprises a layer in a neural network model where the amount of computation is greater than the amount of parameters, wherein the computing speed of the first computing module is greater than the computing speed of the third computing module; and

[0006] A third computing module has an in-memory computing memory for data storage and data operations of a storage-intensive layer, wherein the storage-intensive layer includes a layer in a neural network model in which the number of parameters is greater than the amount of computing, and wherein the storage capacity of the third computing module is greater than the storage capacity of the first computing module.

[0007] Furthermore, the first calculation module includes a first storage array, a first logic calculation unit and a first accumulation unit; and

[0008] The third calculation module includes a second storage array, a second logic calculation unit, a second accumulation unit and a nonlinear calculation unit.

[0009] Furthermore, the device further comprises:

[0010] A second computing module, configured to implement non-matrix-vector multiplication calculations; and

[0011] An on-chip cache is communicatively connected to the first, second, and third computing modules and is used to cache the weights of the neural network, input feature map data, and calculation results of the first and second computing modules.

[0012] Furthermore, the first storage array is used to store the weight matrix of the neural network, which includes an SRAM storage array, and the SRAM storage array includes i rows and j columns of storage cells, and k adjacent storage cells in the same row respectively store k bits of a weight value.

[0013] Furthermore, each column of the SRAM storage array is provided with an independent readout circuit as a readout terminal.

[0014] Furthermore, the first logic calculation unit is arranged at the read-out end of the first storage array, and is used to perform bit-by-bit logic AND calculation on the input feature map data and the weight data stored in the first storage array, and output the calculation result to the first accumulation unit for shift addition.

[0015] Furthermore, the second storage array includes a dynamic random access memory, and / or an NVM, and / or an SRAM, which includes m rows and n columns of storage cells.

[0016] Furthermore, the columns in the second memory array share one or more readout circuits as readout terminals.

[0017] Furthermore, the second logic calculation unit is arranged at the read-out end of the second storage array, and is used to perform bit-by-bit logic AND calculation on the input feature map data and the weight data stored in the second storage array, and output the calculation result to the second accumulation unit for shift addition.

[0018] Furthermore, the storage density of the second storage array is greater than that of the first storage array.

[0019] Furthermore, the on-chip cache is communicatively connected to the third computing module via a system bus.

[0020] Based on the aforementioned apparatus, another aspect of the present invention provides a method for in-memory neural network computing based on heterogeneous storage, comprising:

[0021] Storing the weights of the neural network model in a first storage array;

[0022] Read the input feature map data, perform convolution calculation of the computationally intensive layer through the first logic calculation unit and the first accumulation unit, and write the result to the on-chip cache;

[0023] Calculating other network layers other than the convolutional layer in the neural network model by a second computing module, and writing the calculation results into the on-chip cache; and

[0024] The calculation result in the on-chip cache is read, and the calculation of the storage dense layer is performed through the first logic calculation unit, the first accumulation unit and the nonlinear calculation unit, and the calculation result is output.

[0025] Furthermore, storing the weights of the neural network model in the first storage array includes:

[0026] Reading the weights and input feature map data of the neural network model from the second storage array into the on-chip cache for storage; and

[0027] The weights of the neural network model are written from the on-chip cache to the first storage array.

[0028] Furthermore, performing convolution calculation of the computationally intensive layer by the first logic calculation unit and the first accumulation unit includes:

[0029] Loading the input feature map data into the first logic calculation unit;

[0030] According to the bit correspondence, the input feature map data and the weight of the neural network model are subjected to bit-by-bit logical AND calculation to implement a bit-by-bit multiplication operation, and the result is input to the first accumulator unit; and

[0031] The matrix-vector multiplication calculation is completed by performing shift addition through the first accumulator unit.

[0032] The present invention provides an apparatus and method for in-memory neural network computing based on heterogeneous storage. First, at the system architecture level, different in-memory computing modules are used to implement the calculation of the computation-intensive layer and the storage-intensive layer. Specifically, high-speed SRAM is used to improve the computing efficiency of the computation-intensive layer with a high amount of neural network calculations, and large-capacity memory is used to solve the weight storage problem of the storage-intensive layer with a large amount of neural network parameters, thereby avoiding the time overhead caused by the transportation and writing of weight data. According to tests, it can improve the computing efficiency of the in-memory computing system by 60% to 80%. Secondly, at the computing unit level, the present invention adopts a digital domain in-memory computing component based on a low-cost logic unit. Compared with the existing analog domain in-memory computing component based on analog domain current accumulation and analog-to-digital conversion interface, it significantly reduces the power of the in-memory computing component and can ensure that there is no loss of accuracy in the calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] To further illustrate the above and other advantages and features of various embodiments of the present invention, a more detailed description of various embodiments of the present invention will be presented with reference to the accompanying drawings. It will be understood that these drawings depict only typical embodiments of the present invention and are not to be considered as limiting the scope thereof. In the drawings, for clarity, identical or corresponding components will be represented by the same or similar reference numerals.

[0034] Figure 1 A schematic diagram showing the structure of an artificial intelligence chip in the prior art;

[0035] Figure 2a and 2b A schematic diagram showing the structure of an in-memory computing architecture in the prior art is shown;

[0036] Figure 3 A schematic diagram illustrating the structure of an apparatus for in-memory neural network computing based on heterogeneous storage according to an embodiment of the present invention is shown;

[0037] Figure 4a A schematic diagram showing the structure of a first computing module according to an embodiment of the present invention is shown;

[0038] Figure 4b A schematic diagram showing the structure of a third computing module according to an embodiment of the present invention; and

[0039] Figure 5 A flow chart illustrating a method for in-memory neural network computing based on heterogeneous storage according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0040] In the following description, the present invention is described with reference to various embodiments. However, those skilled in the art will recognize that the various embodiments may be implemented without one or more of the specific details or with other alternative and / or additional methods or components. In other cases, well-known structures or operations are not shown or described in detail to avoid obscuring the inventive aspects of the present invention. Similarly, specific numbers and configurations are set forth for illustrative purposes in order to provide a comprehensive understanding of the embodiments of the present invention. However, the present invention is not limited to these specific details.

[0041] In this specification, reference to "one embodiment" or "the embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. The appearances of the phrase "in one embodiment" in various places in this specification are not necessarily all referring to the same embodiment.

[0042] It should be noted that the embodiments of the present invention describe the method steps in a specific order, but this is only for the purpose of illustrating the specific embodiment and does not limit the order of the steps. On the contrary, in different embodiments of the present invention, the order of the steps can be adjusted according to actual needs.

[0043] To address the shortcomings of existing in-memory computing architectures, the present invention proposes a method for constructing a heterogeneous in-memory computing architecture using high-speed, low-capacity static random access memory (SRAM) and low-speed, high-capacity memory, such as dynamic random access memory (DRAM) and magnetic random access memory (MRAM). Specifically, the present invention divides a neural network model into computation-intensive layers and storage-intensive layers based on the relative relationship between computational effort and parameter storage. Computation-intensive layers refer to layers in the neural network model where computational effort outweighs parameter effort, such as convolutional layers, while storage-intensive layers refer to layers in the neural network model where parameter effort outweighs computational effort, such as fully connected layers. In the present invention, computations in the computation-intensive layers are implemented using a digital in-memory computing architecture based on SRAM, while computations in the storage-intensive layers are implemented using an in-memory computing architecture based on high-capacity memory. Compared to existing digital in-memory computing architectures based on SRAM, the present invention incorporates both high-speed SRAM in-memory computing components and high-capacity memory in-memory computing components, avoiding the frequent reading and writing of large amounts of weight data between memory and SRAM, significantly improving the computational efficiency of the in-memory computing architecture. Compared to existing NVM-based analog in-memory computing architectures, the present invention can achieve data storage of the same scale while using digital domain computing to avoid analog domain noise interference, significantly improving the computational accuracy of the in-memory computing architecture. The following further describes the solution of the present invention in conjunction with the accompanying drawings of the embodiments.

[0044] Figure 3 FIG1 shows a schematic diagram of a device for in-memory neural network computing based on heterogeneous storage according to an embodiment of the present invention. Figure 3 As shown, a device for in-memory neural network computing based on heterogeneous storage includes a first computing module 001, a second computing module 002, a third computing module 003, and an on-chip cache 004. The first computing module 001, the second computing module 002, and the on-chip cache 004 are all arranged inside the chip, and the third computing module 003 is arranged outside the chip and is communicatively connected to the modules inside the chip through, for example, a system bus.

[0045] The first computing module 001 is an in-memory computing architecture, which is mainly used to perform computationally intensive layer operations. It includes a first storage array 101, a first logical computing unit 102, and a first accumulator 103. The first logical computing unit 102 is set at the read end of the first storage array 101 and is used for bit-level multiplication operations to complete multiplication operations while data is read. The first accumulator 103 is set next to the first storage array 101 and includes an addition tree, etc., which is used to accumulate the multiplication results to obtain a complete multiplication-accumulation calculation result, and then splice the matrix-vector multiplication calculation result.

[0046] As mentioned above, the calculation of the computationally intensive layer requires high computational efficiency. Therefore, in one embodiment of the present invention, the first computing module 001 is centered on an SRAM storage array, that is, the first storage array 101 uses an SRAM storage array. Figure 4a FIG. 1 is a schematic diagram showing the structure of the first computing module according to an embodiment of the present invention. Figure 4a As shown, the SRAM storage array includes i rows and j columns of storage cells w ij , where each storage unit can store one bit of data. In the case of a k-bit neural network weight matrix, k adjacent storage units in the same row can be used to store k bits of a weight value. Figure 4a As shown, in one embodiment of the present invention, an independent readout circuit is provided in each column of the SRAM storage array as a readout end. At the same time, each readout end is provided with an independent logic calculation unit, that is, the first logic calculation unit includes j logic calculation units, thereby improving the computational parallelism at the cost of a certain hardware area. Therefore, it can be applied to computationally intensive layers such as convolution layers where the computational amount is the main bottleneck.

[0047] The second computing module 002 is primarily used for operations on other non-convolutional layers in a neural network model, such as pooling layers, element-wise sum layers, and linear rectifier units (ReLUs). The second computing module 002 may be, for example, a vector computing unit, which is centered around a lookup table (LUT) and a dedicated computing unit, and which cooperates with the first computing module 001 to perform non-matrix-vector multiplication operations in neural network calculations, such as element-wise vector addition, pooling, and nonlinear function calculations.

[0048] The third computing module 003 is also an in-memory computing architecture, that is, a computing unit is set next to the memory array, thereby avoiding data transfer between it and the on-chip module. The third computing module 003 is mainly used to perform storage-intensive layer operations, which includes a second storage array 301, a second logic computing unit 302, a second accumulation unit 303 and a nonlinear computing unit 304. Among them, the second logic computing unit 302 is set at the read end of the second storage array 301, which is used for bit-level multiplication operations to complete multiplication operations while data is read out. The second accumulation unit 303 is set next to the second storage array 301, which includes an addition tree, etc., for accumulating the multiplication results to obtain a complete multiplication-accumulation calculation result, and then splicing out the matrix-vector multiplication calculation result. The nonlinear computing unit 304 is used to perform nonlinear computing functions such as the softmax logistic regression function in the neural network model.

[0049] As previously mentioned, the computational bottleneck of the storage-intensive layer lies in the amount of storage. Therefore, in one embodiment of the present invention, the third computing module 003 utilizes large-capacity off-chip memory as its core. By reducing the number of readout circuits and logical computing units, this improves its storage density while ensuring that computing power meets requirements, at the expense of a certain degree of computational parallelism. Examples of such large-capacity off-chip memory include dynamic random access memory (DRAM), NVM, and large-capacity SRAM. Figure 4b FIG. 1 is a schematic diagram showing the structure of the third computing module according to an embodiment of the present invention. Figure 4b As shown, the storage array of the third computing module includes m rows and n columns of storage units w mn , where each storage unit can store one bit of data. Figure 4b As shown, in one embodiment of the present invention, in order to improve storage density, the columns in the second storage array share one or more readout circuits as readout terminals, thereby reducing the number of logic calculation units. Since the second storage array 301 has a large storage capacity, the network weights of the dense layer are all stored in the second storage array 301 before the calculation begins. This allows the second logic calculation unit 302, the second accumulation unit 303, and the nonlinear calculation unit 304 to read the intermediate data from the on-chip cache 004, that is, the calculation results of the first and second calculation modules, and then start the calculation of the convolution and fully connected layers and output the calculation results of the neural network model.

[0050] The on-chip cache 004 is communicatively connected to the first, second, and third computing modules, and is primarily used to implement the following two functions: when the device is initialized, the weights and input feature map data of the neural network model are read from the second storage array 301 and stored in the on-chip cache 004, and then the weights are written from the on-chip cache 004 to the first storage array 101; and during the operation of the neural network, the first and second computing modules will read the input feature map data from the on-chip cache 004 and write the intermediate results to the on-chip cache 004 after completing the calculation. During the calculation of a multi-layer neural network, thanks to the design of the on-chip cache 004, the first and second computing modules do not need to frequently write and / or read the intermediate results to the off-chip second storage array, thereby significantly reducing system calculation latency.

[0051] Figure 5 FIG2 is a flow chart showing a method for in-memory neural network computing based on heterogeneous storage according to an embodiment of the present invention. Figure 5 As shown, a method for in-memory neural network computing based on heterogeneous storage includes:

[0052] First, in step 501, the calculation intensive layer operation is performed. The calculation intensive layer operation is implemented by the first calculation module. When the system starts, the first calculation module reads the weight parameters of the neural network model from outside the chip, and stores the weight parameters in the first storage array, and then reads the input feature map data from the outside or the on-chip cache. After the weight parameters and the input feature map data are prepared, the convolution calculation with the matrix-vector multiplication calculation as the core is completed by the first logic unit and the first accumulation unit, and the intermediate result of the convolution calculation is written to the on-chip cache for temporary storage of the result. In one embodiment of the present invention, the weight parameters are initially stored in the second storage array, and when the system starts, the weight parameters and input feature map data are read from the second storage array to the on-chip cache for storage, and then written to the first storage array by the on-chip cache. In another embodiment of the present invention, the convolution calculation of the calculation intensive layer by the first logic calculation unit and the first accumulation unit includes:

[0053] Loading the input feature map data into the first logic calculation unit;

[0054] According to the bit correspondence, the input feature map data and the weight of the neural network model are subjected to bit-by-bit logical AND calculation to implement a bit-by-bit multiplication operation, and the result is input to the first accumulator unit; and

[0055] Performing shift addition through the first accumulator unit to complete the matrix-vector multiplication calculation;

[0056] At the same time, in step 502, non-convolutional layer operations are performed. The second computing module is used to implement operations on other network layers in the neural network model. Specifically, the vector computing unit reads input data of network layers such as the pooling layer and the element-wise sum layer from the on-chip cache, and writes the calculation results back to the on-chip cache after the operations are completed; and

[0057] Finally, in step 503, the storage dense layer operation is performed. The storage dense layer operation is implemented by the third computing module, that is, the calculation of the fully connected layer is completed and the model calculation result of the neural network is output. The operation principle of the second logic unit and the second accumulation unit in the third computing module is basically the same as that of the first logic unit and the first accumulation unit. The difference is that the input of the third computing module includes the calculation results of the first and second computing modules read from the on-chip cache, and the weights of the storage dense layer stored in the second storage array. At the same time, compared with the high computational parallelism of the first computing unit, the number of logical computing units in the third computing unit is less than the number of columns of the second storage array. Therefore, its computational parallelism is lower, but the storage density is higher.

[0058] The present invention provides an apparatus and method for in-memory neural network computing based on heterogeneous storage. First, at the system architecture level, different in-memory computing modules are used to implement the calculation of the computation-intensive layer and the storage-intensive layer. Specifically, high-speed SRAM is used to improve the computing efficiency of the computation-intensive layer with a high amount of neural network calculations, and large-capacity memory is used to solve the weight storage problem of the storage-intensive layer with a large amount of neural network parameters, thereby avoiding the time overhead caused by the transportation and writing of weight data. According to tests, it can improve the computing efficiency of the in-memory computing system by 60% to 80%. Secondly, at the computing unit level, the present invention adopts a digital domain in-memory computing component based on a low-cost logic unit. Compared with the existing analog domain in-memory computing component based on analog domain current accumulation and analog-to-digital conversion interface, it significantly reduces the power of the in-memory computing component and can ensure that there is no loss of accuracy in the calculation.

[0059] Although various embodiments of the present invention have been described above, it should be understood that they are presented by way of example only and not limitation. It will be apparent to those skilled in the relevant art that various combinations, modifications, and variations may be made thereto without departing from the spirit and scope of the present invention. Therefore, the breadth and scope of the present invention disclosed herein should not be limited by the exemplary embodiments disclosed above, but should be defined solely in accordance with the appended claims and their equivalents.

Claims

1. A device for in-memory neural network computing based on heterogeneous storage, characterized in that: include: a first computing module having an in-memory computing memory for storing and computing data in a computationally intensive layer, wherein the computationally intensive layer comprises a layer in a neural network model where the amount of computation is greater than the amount of parameters, wherein the computing speed of the first computing module is greater than the computing speed of the third computing module; a third computing module having an in-memory computing memory for storing and computing data in a storage-intensive layer, wherein the storage-intensive layer comprises a layer in a neural network model having a greater number of parameters than a greater number of computations, wherein the storage capacity of the third computing module is greater than the storage capacity of the first computing module; a second computing module configured to perform operations other than matrix-vector multiplication; as well as An on-chip cache is communicatively connected to the first, second, and third computing modules and is configured to cache the weights of the neural network, input feature map data, and calculation results of the first and second computing modules.

2. The device according to claim 1, wherein: The first calculation module includes a first storage array, a first logic calculation unit and a first accumulation unit; and The third calculation module includes a second storage array, a second logic calculation unit, a second accumulation unit and a nonlinear calculation unit.

3. The device according to claim 2, wherein The first storage array is configured to store the weight matrix of the neural network, which includes an SRAM storage array. The SRAM storage array includes i rows and j columns of storage cells, and k adjacent storage cells in the same row respectively store k bits of a weight value, where i, j, and k are natural numbers.

4. The device according to claim 3, characterized in that Each column of the SRAM memory array is provided with an independent readout circuit as a readout terminal; and The first logic calculation unit includes j logic calculation units, and the j logic calculation units correspond to the readout ends one by one.

5. The device according to claim 2, wherein The first logic calculation unit is arranged at the read-out end of the first storage array, and is configured to perform bit-by-bit logic AND calculation on the input feature map data and the weight data stored in the first storage array, and output the calculation result to the first accumulation unit for shift addition.

6. The device according to claim 2, wherein The second storage array includes a dynamic random access memory, and / or an NVM, and / or an SRAM, and includes m rows and n columns of storage cells, where m and n are natural numbers.

7. The device according to claim 6, characterized in that Each column in the second memory array shares one or more readout circuits as a readout terminal.

8. The device according to claim 2, wherein The second logic calculation unit is arranged at the read-out end of the second storage array, and is configured to perform bit-by-bit logic AND calculation on the input feature map data and the weight data stored in the second storage array, and output the calculation result to the second accumulation unit for shift addition.

9. The device according to claim 2, wherein The storage density of the second storage array is greater than that of the first storage array.

10. The device according to claim 1, wherein The on-chip cache is communicatively connected to the third computing module via a system bus.

11. A method for in-memory neural network computing based on heterogeneous storage, characterized in that: Including steps: Storing the weights of the neural network model in a first storage array; Read the input feature map data, perform convolution calculation of the computationally intensive layer through the first logic calculation unit and the first accumulation unit, and write the result to the on-chip cache; Calculating other network layers other than the convolutional layer in the neural network model by a second computing module, and writing the calculation results into the on-chip cache; as well as The calculation result in the on-chip cache is read, and the calculation of the storage dense layer is performed through the second logic calculation unit, the second accumulation unit and the nonlinear calculation unit, and the calculation result is output.

12. The method according to claim 11, wherein Storing the weights of the neural network model in the first storage array comprises the steps of: Reading the weights and input feature map data of the neural network model from the second storage array into the on-chip cache for storage; as well as The weights of the neural network model are written from the on-chip cache to the first storage array.

13. The method according to claim 11, wherein Performing convolution calculation of the computationally intensive layer by the first logic calculation unit and the first accumulation unit includes the following steps: Loading the input feature map data into the first logic calculation unit; According to the bit correspondence, the input feature map data and the weight of the neural network model are subjected to bit-by-bit logical AND calculation to implement a bit-by-bit multiplication operation, and the result is input to the first accumulator unit; as well as The matrix-vector multiplication calculation is completed by performing shift addition through the first accumulator unit.

Images