Cache multiplexing method and system for non-volatile in-memory computing ai chip architecture
Patent Information
- Application Number
- CN202310654948.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-05
- Publication Date
- 2026-10-09
- Estimated Expiration
- 2043-06-05
AI Technical Summary
[0006]通过上述分析，现有技术存在的问题及缺陷为：每个存内计算单元负责运算的神经网络层不同，导致不同存内计算单元对缓存大小的要求不一致
[0027] First, the specific description is as follows:
Smart Images
Figure CN116775505B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of artificial intelligence technology, and in particular relates to a cache reuse method and system for non-volatile in-memory computing AI chip architecture. Background Technology
[0002] Currently, artificial intelligence / machine learning applications are widely deployed across various industries. As the parameters and complexity of AI application models continue to increase, the demand for chip computing power is also rising. AI applications are characterized by data access and communication intensity. The difference between memory access efficiency and processor speed causes the memory wall problem, making memory access and communication performance bottlenecks. To overcome the memory wall problem, academia and industry have proposed AI accelerator chip architectures based on non-volatile in-memory computing devices.
[0003] In the basic architecture of the in-memory computing AI accelerator chip, the chip consists of two parts: an on-chip network and in-memory computing units. The in-memory computing units are responsible for performing specific operations such as convolution, pooling, and activation, while the on-chip network is responsible for data stream transmission and result fusion. Each tile of the in-memory computing unit contains a convolution unrolling unit, an Xbar and its associated ADC, a cache for storing bias data, an activation unit (Act), and a pooling unit (Pool). The data stream entering the in-memory computing unit from the outside is divided into three categories: first, the convolution input data from the previous layer, called the cast-in data stream; second, the data to be fused from other tiles responsible for the current layer, called the merge-in data stream; and third, the data to be fused from branches parallel to the current model branch, called the gather-in data stream. The data streams sent out by a Tile are also divided into three categories: First, the final computation result of the current layer, which is to be passed to the next layer Tile as the convolution input of the next layer Tile, is called the cast-out data stream; second, the computation result of the current Tile, which is expected to be merged with the computation results of other Tile in the current layer, is called the merge-out data stream; and third, the final computation result of the current layer, which is expected to be merged with the computation results of parallel branches, is also called the cast-out data stream. Therefore, each in-memory computing unit is connected to the on-chip network through three independent network interfaces: cast, merge, and gather. These three interfaces are used to carry the three types of data streams, respectively.
[0004] The cast-in data stream is fed as input to the WL (Word Line) of the Xbar as input to the convolution operation. The convolution calculation result is obtained on the BL (Bit Line) of the Xbar. This calculation result is either transmitted to the outside of the Tile as a merge-out data stream, or it is added to the data in the Bias RAM within the Tile, and then added to the merge-in data stream to obtain the result of the current branch and the current layer. This result is added to the gather-in data stream, and then goes through activation and pooling operations in sequence. Finally, it is transmitted to the outside of the Tile as both the cast-out and gather-out data streams.
[0005] Each step in the above process has a register or input buffer, forming a pipeline. However, due to the operational characteristics of non-volatile in-memory computing devices, the weight configuration is costly. Therefore, non-volatile in-memory computing AI chips use a method of directly mapping the neural network to multiple in-memory computing units on the chip to complete the computation. These multiple in-memory computing units are interconnected through an on-chip network. For neural networks of different sizes, the sizes of the convolution unrolling unit buffer, pooling unit buffer, and router buffer in each in-memory computing unit are all different. When the feature map size and depth of the neural network layer are large, the buffer of the in-memory computing unit responsible for the operation of that layer is larger, and vice versa. This leads to different buffer size requirements for each in-memory computing unit. However, to adapt to neural networks of different sizes, more flexible allocation space for buffer resources is required to save area. Therefore, a buffer reuse method needs to be introduced into the architecture of non-volatile in-memory computing AI chips.
[0006] Based on the above analysis, the problems and defects of the existing technology are as follows: each in-memory computing unit is responsible for different neural network layers, which leads to inconsistent requirements for cache size for different in-memory computing units. Summary of the Invention
[0007] To address the problems existing in the prior art, this invention provides a cache reuse method and system for non-volatile in-memory computing AI chip architectures.
[0008] This invention is implemented as follows: A cache reuse method for non-volatile in-memory computing AI chip architecture includes:
[0009] Step 1: Introduce a cache reuse mechanism on the basis of the existing in-memory computing AI chip architecture;
[0010] Step 2: Cache and reuse adjacent in-memory computing units and on-chip network interfaces to maintain the original mesh on-chip network layout structure.
[0011] Step 3: The cache pool controller introduces an index cache of the same depth on the basis of the original cache to store the scrolling index of cache pointers of different in-memory computing units, thereby accelerating the scrolling of cache pointers.
[0012] Furthermore, the cache reuse mechanism in step one unifies the management of the convolution unrolling unit cache, pooling unit cache, and on-chip network interface cache of adjacent in-memory computing units, thereby increasing cache utilization and reducing cache area through the reuse mechanism.
[0013] Furthermore, in step three, each cache interface does not provide an address, supports backpressure, and the rolling cache pointer is uniformly maintained by the cache pool controller;
[0014] The cache interface has changed from the original SRAM interface to a FIFO type interface. The read latency of this interface is two clock cycles. In the first clock cycle, the index cache is accessed to obtain the next pointer position of the cache interface. In the second clock cycle, the data cache is accessed to obtain the data.
[0015] Furthermore, in the in-memory computing AI architecture, the rolling cache introduces a burst transfer interface to increase transfer efficiency.
[0016] Furthermore, in step three, each rolling cache in the buffer pool is precisely indexed using a head pointer and a cache index table;
[0017] When multiple in-memory computing units share a buffer pool, port arbitration is required. The number of read and write ports of a multi-port RAM determines the maximum parallelism of read and write operations.
[0018] Furthermore, in step three, port arbitration is the process of arbitrating the read channel and the write channel separately. If a memory with n write ports and m read ports is used (n, m≤2), and the number of in-memory computing units using the buffer pool is greater than the number of ports, the data is sent to the buffer pool pointer controller after port arbitration. The pointer controller drives the multi-port RAM to perform data read and write operations.
[0019] Furthermore, when a write port after arbitration writes data into the buffer pool, the tail pointer of the buffer pool moves and writes data. When the tail pointer is the same as the head pointer of a certain in-memory computing unit, the write-full signal of the buffer interface is pulled high.
[0020] When writing data, the index pointer position corresponding to the last write is changed in other memory to accelerate the scrolling process.
[0021] When a read port after arbitration reads data from the buffer pool, it first accesses the index cache based on the current head pointer and reads the scroll-accelerated index.
[0022] Another object of the present invention is to provide a cache reuse system for a non-volatile in-memory computing AI chip architecture, the cache reuse system for a non-volatile in-memory computing AI chip architecture comprising:
[0023] A cache reuse mechanism module is introduced to introduce a cache reuse mechanism on the basis of the original in-memory computing AI chip architecture;
[0024] The cache reuse module is used to cache and reuse adjacent in-memory computing units and on-chip network interfaces to maintain the original mesh on-chip network layout structure.
[0025] The index cache introduction module is used to introduce an index cache of the same depth on the basis of the original cache of the cache pool controller, and to store the scroll index of cache pointers of different in-memory computing units, thereby accelerating the scrolling of cache pointers.
[0026] Based on the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solution to be protected by this invention are as follows:
[0027] First, the specific description is as follows:
[0028] 1. Based on the traditional non-volatile in-memory computing AI chip architecture, this invention introduces cache reorganization technology, which can greatly save chip area without affecting the compatibility of neural network scale.
[0029] 2. Compared with the heterogeneous in-memory computing unit scheme, this scheme can achieve flexible cache reorganization and has stronger applicability.
[0030] 3. The buffer pool controller in this invention has a simple structure, is easy to design, occupies few resources, and introduces almost no additional overhead.
[0031] 4. This invention uses a rolling index cache (Index SRAM) to accelerate the access speed of the rolling cache.
[0032] Secondly, as supporting evidence of the inventiveness of this invention, it is also reflected in the following important aspects:
[0033] (1) The expected benefits and commercial value of the technical solution of the present invention after transformation are as follows: The present invention solves the key problem of insufficient cache capacity of non-volatile in-memory computing architecture, so that non-volatile in-memory computing architecture can be applied to neural networks of different sizes, thereby enabling the industrialization of non-volatile in-memory computing chips.
[0034] (2) Does the technical solution of this invention solve the technical problem that people have long desired to solve but have never been able to successfully solve: the conflict between the high efficiency and high resource utilization of non-volatile in-memory computing architectures? The compatibility problem of non-volatile in-memory computing architectures with convolutional neural networks of different sizes. Attached Figure Description
[0035] Figure 1 This is a flowchart of a cache reuse method for a non-volatile in-memory computing AI chip architecture provided in an embodiment of the present invention;
[0036] Figure 2 This is a schematic diagram of the cache reuse scheme provided in an embodiment of the present invention;
[0037] Figure 3 This is a schematic diagram illustrating the changes in the caching interface provided in this embodiment of the invention;
[0038] Figure 4 This is a schematic diagram of the merged rolling cache provided in an embodiment of the present invention;
[0039] Figure 5 This is a schematic diagram of the buffer pool controller provided in an embodiment of the present invention;
[0040] Figure 6 This is an example of the cache capacity of the in-memory computing unit provided in an embodiment of the present invention;
[0041] Figure 7 This represents the percentage reduction in cache capacity provided by the embodiments of the present invention compared to the original solution. Detailed Implementation
[0042] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0043] like Figure 1 As shown, the cache reuse method for non-volatile in-memory computing AI chip architecture provided by this embodiment of the invention includes:
[0044] The S101 introduces a cache reuse mechanism on the basis of the original in-memory computing AI chip architecture.
[0045] S102 caches and reuses adjacent in-memory computing units and on-chip network interfaces to maintain the original mesh on-chip network layout structure.
[0046] S103, the cache pool controller introduces an index cache of the same depth on the basis of the original cache, which is used to store the scroll index of the cache pointers of different in-memory computing units, thereby accelerating the scrolling of the cache pointers.
[0047] As a preferred embodiment of the present invention, a cache reuse method for non-volatile in-memory computing AI chip architecture is described. Specific implementation steps include:
[0048] S101: Based on the existing in-memory computing AI chip architecture, a cache reuse mechanism is introduced. This means that cache reuse is introduced as a new optimization technique into the in-memory computing AI chip architecture to improve computing efficiency and performance.
[0049] S102: Cache and reuse adjacent in-memory compute units and on-chip network interfaces to maintain the original mesh on-chip network layout. This means applying cache reuse to data transmission between in-memory compute units and on-chip network interfaces to reduce network communication overhead and latency, thereby improving computational efficiency and performance.
[0050] S103: The cache pool controller introduces an index cache of the same depth on top of the existing cache. This index cache is used to store the scrolling index of cache pointers for different in-memory computation units, thereby accelerating the scrolling of cache pointers. This means that the cache pool controller will introduce a new index cache to optimize the scrolling of cache pointers, thereby further improving the efficiency and performance of in-memory computation.
[0051] Overall, this cache reuse approach for non-volatile in-memory AI chip architectures can improve computational efficiency and performance by leveraging cache reuse techniques and indexed cache optimization, thereby achieving faster data processing and higher throughput. Furthermore, this approach can maintain the original mesh on-chip network layout, thus preserving the system's scalability and flexibility.
[0052] This invention introduces a cache reuse mechanism based on the existing in-memory computing AI chip architecture. This mechanism unifies the management of convolution unrolling unit caches, pooling unit caches, and adjacent on-chip network interface caches of neighboring in-memory computing units. This reuse mechanism increases cache utilization and reduces cache area. Figure 2 As shown.
[0053] By caching and reusing adjacent in-memory computing units (top, bottom, left, and right) and on-chip network interfaces, the original mesh on-chip network layout can be maintained. This increases cache utilization without adding extra burden to backend layout and routing.
[0054] The cache pool controller (SRAM Pool) has a simple structure. Based on the original cache, an index cache (Index SRAM) of the same depth is introduced to store the scrolling index of cache pointers for different in-memory computing units, thereby accelerating the scrolling of cache pointers.
[0055] Because of the use of the rolling scheme, each cache (SRAM) interface no longer needs to provide an address, but to support backpressure, the rolling cache pointer is maintained uniformly by the cache pool controller.
[0056] The cache interface has changed as shown in the figure below. It has changed from the original SRAM interface to a FIFO type interface. However, unlike FIFO, the read latency of this interface is two clock cycles. In the first clock cycle, the index cache is accessed to obtain the next pointer position of the cache interface. In the second clock cycle, the data cache is accessed to obtain the data.
[0057] In in-memory AI architecture, since memory access to the rolling cache is generally done in vector units, the rolling cache can also introduce a burst transfer interface to increase transfer efficiency.
[0058] like Figure 4 As shown, each rolling cache in the buffer pool is precisely indexed using a head pointer and a cache index table, and its structure resembles a circular queue. The buffer pool also needs to maintain a tail pointer.
[0059] First, when multiple in-memory computing units share a single buffer pool, port arbitration is required. The number of read and write ports in a multi-port RAM determines the maximum parallelism of read and write operations. Port arbitration is the process of arbitrating the read and write channels separately. Figure 5 The diagram shows a SRAM Pool Controller. If a memory with n write ports and m read ports (n, m≤2) is used, and the number of in-memory computing units using the buffer pool is greater than the number of ports, the data is sent to the buffer pool pointer controller after port arbitration. The pointer controller then drives the multi-port RAM to read and write data.
[0060] When a write port writes data to the buffer pool after arbitration, the buffer pool's tail pointer moves, and data is written. When the tail pointer matches the head pointer of a certain in-memory computation unit, the write-full signal (WFULL) of the buffer interface is pulled high. Simultaneously, during data writing, the index pointer positions corresponding to the last write operation of other in-memory computation units need to be changed to accelerate the scrolling process.
[0061] When a read port after arbitration reads data from the buffer pool, it first accesses the index cache (Index SRAM) based on the current head pointer to read the scrolling acceleration index. If the current head pointer cannot obtain the scrolling acceleration index, it means that the cache corresponding to the in-memory computation unit has been read, and the cache interface is pulled high with a read full signal (RFULL). After reading the scrolling acceleration index, the head pointer of the in-memory computation moves to the position pointed to by the scrolling acceleration index (i.e., the next valid cache position corresponding to the in-memory computation unit).
[0062] In this way, each in-memory computing unit achieves precise access to the rolling cache without interfering with each other. After merging the rolling caches of each in-memory computing unit using this scheme, if the cache pool capacity is less than the total capacity of the rolling caches of each in-memory computing unit before the merger, it is still necessary to control the data flow of each in-memory computing unit to prevent deadlock.
[0063] This cache reuse scheme can effectively save cache space because, in traditional schemes, each in-memory computing unit has its own cache of the same size to accommodate neural networks of different sizes. For example... Figure 6 As shown, a certain neural network uses a 2*3 in-memory computing unit array. The in-memory computing units 0 to 5 require 32KB, 16KB, 10KB, 10KB, 4KB, and 4KB of cache, respectively.
[0064] Using a traditional approach, each in-memory computing unit would require a 32KB cache, resulting in a total cache usage of 32KB * 8 = 256KB. However, using cache reuse technology, configuring two 64KB caches based on the cache capacities of CIMU0, CIMU1, CIMU4, and CIMU5 is sufficient, resulting in a total cache capacity of 128KB. Furthermore, this configuration is compatible with other neural networks of similar size, saving 50% of the cache.
[0065] A computer device, characterized in that the computer device includes a memory and a processor, the memory storing a computer program, and when the computer program is executed by the processor, causing the processor to perform the steps of the cache reuse method for a non-volatile in-memory computing AI chip architecture.
[0066] A computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the cache reuse method for a non-volatile in-memory computing AI chip architecture.
[0067] The embodiments of the present invention have achieved some positive results during the research and development or use process, and have indeed great advantages compared with the prior art. The following content describes them in conjunction with the data, charts and other information of the experimental process.
[0068] Using a non-volatile in-memory computation mapping toolchain, we tested five typical convolutional neural network models: ResNet-18, ResNet-50, VGG-16, GoogLeNet, and MobileNet. The percentage of cache savings compared to the original solution is as follows: Figure 7 .
[0069] from Figure 7 As can be seen from the above, among the five typical convolutional neural networks, the solution described in this invention saves significantly more cache resources compared to the original solution.
[0070] The advantages and positive effects of the cache reuse method for non-volatile in-memory computing AI chip architecture provided in this invention include:
[0071] 1) Improve computational efficiency and performance: By introducing cache reuse mechanisms and index cache optimization, redundant calculations and data transfers can be avoided, thereby improving computational efficiency and performance.
[0072] 2) Reduce network communication overhead and latency: By caching and reusing adjacent in-memory computing units and on-chip network interfaces, network communication overhead and latency can be reduced, thereby further improving computing efficiency and performance.
[0073] 3) Maintain the original mesh on-chip network layout: By caching and reusing adjacent in-memory computing units and on-chip network interfaces, the original mesh on-chip network layout can be maintained, thereby maintaining the scalability and flexibility of the system.
[0074] 4) Accelerate the rolling of cache pointers: By introducing index caches of the same depth, the rolling index of cache pointers of different in-memory computing units can be saved, thereby accelerating the rolling of cache pointers and improving computing efficiency and performance.
[0075] In summary, this cache reuse method for non-volatile in-memory computing AI chip architecture can improve computing efficiency and performance without increasing hardware costs, further reduce power consumption and latency, thereby achieving faster data processing and higher throughput.
[0076] It should be noted that embodiments of the present invention can be implemented in hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuitry such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., or by software executed by various types of processors, or by a combination of the above-described hardware circuitry and software, such as firmware.
[0077] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.
Claims
1. A cache reuse method for non-volatile in-memory computing AI chip architecture, characterized in that, include: Step 1: Based on the existing in-memory computing AI chip architecture, a cache reuse mechanism is introduced. The cache reuse mechanism unifies the management of the convolution unrolling unit cache, pooling unit cache, and on-chip network interface cache of the adjacent in-memory computing units, in order to increase cache utilization and reduce cache area through the reuse mechanism. Step 2: Cache and reuse adjacent in-memory computing units and on-chip network interfaces to maintain the original mesh on-chip network layout structure. Step 3: The cache pool controller introduces an index cache of the same depth on the basis of the original cache to store the scrolling index of the cache pointers of different in-memory computing units, thereby accelerating the scrolling of the cache pointers. In step three, each cache interface does not provide an address, supports backpressure, and the rolling cache pointer is maintained uniformly by the cache pool controller. The cache interface has changed from the original SRAM interface to a FIFO type interface. The read latency of this interface is two clock cycles. In the first clock cycle, the index cache is accessed to obtain the next pointer position of the cache interface. In the second clock cycle, the data cache is accessed to obtain the data. In step three, each rolling cache in the buffer pool is precisely indexed using a head pointer and a cache index table; When multiple in-memory computing units share a buffer pool, port arbitration is required. The number of read and write ports of a multi-port RAM determines the maximum parallelism of read and write operations. The port arbitration is a process of arbitrating the read channel and the write channel separately; If a memory with n write ports and m read ports (n, m≤2) is used, and the number of in-memory computing units using the buffer pool is greater than the number of ports, then after port arbitration, the data is sent to the buffer pool pointer controller, which drives the multi-port RAM to perform data read and write operations.
2. The cache reuse method for non-volatile in-memory computing AI chip architecture as described in claim 1, characterized in that, In the in-memory computing AI architecture, the rolling cache introduces the burst interface to increase transmission efficiency.
3. The cache reuse method for non-volatile in-memory computing AI chip architecture as described in claim 1, characterized in that, When a write port after arbitration writes data into the buffer pool, the tail pointer of the buffer pool moves and writes data. When the tail pointer is the same as the head pointer of a certain in-memory computing unit, the write-full signal of the buffer interface is pulled high. When writing data, the index pointer position corresponding to the last write is changed in other memory to accelerate the scrolling process. When a read port after arbitration reads data from the buffer pool, it first accesses the index cache based on the current head pointer and reads the scroll-accelerated index.
4. A cache reuse system for a non-volatile in-memory computing AI chip architecture, used to implement the cache reuse method for any one of claims 1 to 3, characterized in that, The cache reuse system for non-volatile in-memory computing AI chip architecture includes: A cache reuse mechanism module is introduced to introduce a cache reuse mechanism on the basis of the original in-memory computing AI chip architecture; The cache reuse module is used to cache and reuse adjacent in-memory computing units and on-chip network interfaces to maintain the original mesh on-chip network layout structure. The index cache introduction module is used to introduce an index cache of the same depth on the basis of the original cache of the cache pool controller, and to store the scroll index of cache pointers of different in-memory computing units, thereby accelerating the scrolling of cache pointers.
5. A computer device, characterized in that, The computer device includes a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the cache reuse method for a non-volatile in-memory computing AI chip architecture as described in any one of claims 1 to 3.
6. A computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the cache reuse method for a non-volatile in-memory computing AI chip architecture as described in any one of claims 1 to 3.