Shared last-stage on-chip storage architecture and method for heterogeneous AI accelerator

Abstract

A shared last-stage on-chip storage method for a heterogeneous AI accelerator comprises a GPGPU computing core, an NPU computing core, an on-chip network, a plurality of shared last-stage on-chip storage fragments and a memory channel. Shared last-stage on-chip storage resources are dynamically configured as a last-stage cache of the GPGPU or a scratch pad memory of the NPU; the operation performance of the artificial intelligence model can be improved according to the operator composition of the to-be-operated artificial intelligence model.

Description

Technical Field

[0001] This invention relates to a technology in the field of computer storage, specifically a shared final-level on-chip storage architecture and method for heterogeneous AI accelerators. Background Technology

[0002] In existing edge systems-on-chip (SoCs), general-purpose AI accelerators (such as GPGPUs) and dedicated AI accelerators (such as NPUs) employ independent on-chip memory architectures. When executing complex models requiring collaboration among multiple accelerators, the lack of on-chip data sharing channels forces data to be routed through off-chip memory, resulting in performance degradation. Furthermore, the statically isolated allocation of on-chip memory resources cannot adapt to dynamic changes in model workload, leading to inefficient utilization of on-chip memory resources and further performance degradation. Summary of the Invention

[0003] To address the aforementioned shortcomings of existing technologies, this invention proposes a shared final-level on-chip storage architecture and method for heterogeneous AI accelerators. This architecture dynamically configures the shared final-level on-chip storage resources as the final-level cache of the GPGPU or the note memory of the NPU, thereby improving the running performance of the AI ​​model based on the operator composition of the AI ​​model to be run.

[0004] This invention is achieved through the following technical solution:

[0005] This invention relates to a shared final-level on-chip storage architecture for heterogeneous AI accelerators, comprising: a GPGPU computing core, an NPU computing core, an on-chip network, several shared final-level on-chip storage shards, and memory channels. Each final-level on-chip storage shard is connected to both the GPGPU computing core and the NPU computing core via the on-chip network, and each final-level on-chip storage shard has its own corresponding memory channel. The GPGPU computing core performs calculations of artificial intelligence model operators based on the content of the computing program running on the GPGPU, obtains the calculation results of the operators, and sends memory access requests to the on-chip network during the calculation process. The NPU computing core performs calculations based on the content of the computing program running on the NPU. The system performs calculations on the content of the AI ​​model operators, obtains the calculation results, and sends memory access requests or note memory access requests to the on-chip network during the calculation process. The on-chip network, based on the data address contained in the memory access request or note memory access request issued by the GPGPU computing core or NPU computing core, sends the data access request of the GPGPU computing core to the shared final-level on-chip storage segment in the GPGPU final-level cache mode, and sends the data access request of the NPU computing core to the shared final-level on-chip storage segment in the NPU note memory mode. Each final-level on-chip storage segment operates independently in the GPGPU final-level cache mode or the NPU note memory mode.

[0006] The on-chip network includes: multiple network interface units, multiple routing nodes, and physical links connecting the routing nodes. The network interface units are respectively connected to the GPGPU computing core, the NPU computing core, and the shared last-level on-chip storage fragment, and are used to receive and send network data packets. The routing nodes are interconnected through physical links to form an interconnected topology, and are configured to route network data packets to the corresponding shared last-level on-chip storage fragment according to the data address in the network data packet.

[0007] This invention relates to a control method based on the aforementioned shared final-level on-chip memory architecture for heterogeneous AI accelerators, comprising:

[0008] Step 1: Construct and allocate operators, specifically including: allocating operators that run on the NPU with higher performance, power consumption, or performance-to-power ratio than those running on the GPGPU to the NPU; and allocating operators that cannot run on the NPU or whose performance, power consumption, or performance-to-power ratio on the NPU is lower than those running on the GPGPU to the GPGPU.

[0009] Step 2, Classification of Operation Modes: When all operators are divided into two parts in terms of timing, one part is allocated entirely to the GPGPU and the other part is allocated entirely to the NPU, the GPGPU and NPU run simultaneously in a pipeline manner and proceed to Step 4; when all operators are divided into two or more parts in terms of timing, and each part is fully allocated to either the GPGPU or the NPU, the GPGPU and NPU run alternately and proceed to Step 3.

[0010] Step 3: Initially partition the shared last-level on-chip memory into fragments and alternately switch between them, specifically including:

[0011] 3.1 Depending on whether the first operator is allocated to the GPGPU or the NPU, set the mode of all final-level on-chip memory slices to either the GPGPU final-level cache mode or the NPU note memory mode.

[0012] 3.2 Whenever GPGPU operation switches to NPU operation, the mode of all final-level on-chip memory slices is switched from GPGPU final-level cache mode to NPU note memory mode; whenever NPU operation switches to GPGPU operation, the mode of all final-level on-chip memory slices is switched from NPU note memory mode to GPGPU final-level cache mode.

[0013] Step 4: Fixedly divide the final-level on-chip storage into fragments and perform data copying, specifically including:

[0014] 4.1 Allocate a portion of the final-level on-chip memory fragments to the GPU in GPGPU final-level cache mode, and allocate the remaining final-level memory fragments to the NPU in NPU note memory mode.

[0015] 4.2 Whenever the GPGPU and NPU need to exchange data, the data is copied to the following two memory blocks: the memory to which the shared final-level on-chip storage fragment is connected in the GPGPU final-level cache mode, and the memory to which the shared final-level on-chip storage fragment is connected in the NPU note memory mode. Technical effect

[0016] This invention connects the GPGPU computing core and NPU computing core to multiple shared final-level on-chip storage shards via an on-chip network, and then to independent memory channels. Each shared final-level on-chip storage shard can switch between GPGPU final-level cache mode and NPU note-memory mode. Compared with existing technologies, this invention can retain data that needs to be transferred between the GPGPU and NPU within the shared final-level on-chip storage shard when switching modes, eliminating the need to transfer data through off-chip memory and thus improving performance. Furthermore, it can allocate all or part of the final-level on-chip storage shards to the GPU in GPGPU final-level cache mode, and allocate the remaining final-level storage shards to the NPU in NPU note-memory mode, enabling on-chip storage capacity and off-chip memory bandwidth to be allocated on demand, thereby improving performance. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the architecture of the present invention;

[0018] Figure 2 This is a schematic diagram of on-chip storage fragmentation at the final level;

[0019] Figure 3 This is a flowchart of an implementation example;

[0020] Figure 4 , 5 This is a schematic diagram illustrating the effects of the present invention. Detailed Implementation

[0021] like Figure 1 As shown, this embodiment relates to a shared final-level on-chip storage architecture for heterogeneous AI accelerators, including: a GPGPU computing core, an NPU computing core, an on-chip network, several shared final-level on-chip storage shards, and memory channels.

[0022] like Figure 2As shown, each final-level on-chip storage shard includes: a data array, a tag array and its management circuitry, an index array and its management circuitry, and a status table and its management circuitry. The data array stores data rows in a set-associative manner. The tag array and its management circuitry, upon receiving an instruction from the on-chip network that the NPU has finished running and needs to switch to GPGPU execution, switch to GPGPU final-level cache mode. Based on the GPGPU memory access request for the physical memory address, and after handling cache line hits and misses using a replacement strategy, it returns the result to the GPGPU computing core via the on-chip network and issues an access request to off-chip memory via a memory channel. The index array and its management circuitry... After receiving an instruction from the on-chip network that the GPGPU has finished running and needs to switch to NPU operation, its management circuit switches to NPU note memory mode. Based on the NPU access note memory address in the NPU note memory access request, it accesses the corresponding data row of the internal data array and returns the result to the NPU computing core through the on-chip network or issues a memory access request to the memory channel connected to the last-level on-chip storage segment based on the NPU memory access request. In GPGPU cache mode, the status table management circuit maintains the cache eviction status of data rows within the note memory address range and resets the status table when the GPGPU cache mode switches to NPU note memory mode.

[0023] The size of the address range for NPU access to the note memory is equivalent to the total size of the data array of all shared final-level on-chip storage fragments.

[0024] The NPU memory access requests include requests for NPU access to physical memory addresses that are not within the address range of the note storage.

[0025] The aforementioned tag array refers to a dedicated storage unit that stores tags in a set-associative manner, wherein the tag is generated as an address to uniquely distinguish different data rows within the same cache group.

[0026] In GPGPU cache mode, the tag array management circuit accesses data rows in a cached manner according to the tag array, compares a specific tag portion in the address of the GPU memory request with the tag array, and determines whether the cache hit or miss is processed according to a replacement strategy including but not limited to Least Recently Used (LRU).

[0027] Preferably, in NPU note memory mode, when the tag array management circuit receives a write request in the NPU note memory access request and the written data needs to be accessed after switching to GPGPU cache mode, it maintains the tag array by writing the tag part of the requested address into the tag array, so that if it switches to GPGPU cache mode later, the data row can be accessed without clearing the data array.

[0028] The index array refers to a dedicated storage unit that stores the indexes of all data row addresses within the address range of the note storage device in a group-associative manner, wherein the index is a one-to-one mapping information from the tag to the actual storage location of the data row in the data array.

[0029] In NPU note memory mode, the index array management circuit accesses data rows in note memory mode according to the index array, and maps the specific marker part in the address requested by the NPU note memory to the actual storage location in the data array according to the index array.

[0030] Preferably, in GPGPU last-level cache mode, when the index array management circuit receives a write request in a GPU memory access request and the range of the data address to be written is within the range of the note memory address, it updates the index array. That is, the index of the group containing the requested note memory address in the index array is still a one-to-one mapping, and the marker of the write request address is mapped to the storage location of the data row to be written in the data array. Specifically, the mapping information of the marker stored in the target group that is the marker of the write request address is denoted as A<->B; the mapping information of the storage location of the data array stored in the target group that is the storage location of the data row to be written in the data array is denoted as C<->D; and the index is updated to A<->D, C<->B.

[0031] The aforementioned status table refers to a dedicated storage unit that stores the cache eviction status of all data row addresses within the address range of the note storage in a group-associative manner, wherein the cache eviction status is a status value that records whether a data row has been evicted from the cache.

[0032] In GPGPU cache mode, when a data row within the address range of the note memory is evicted from the cache, the state table management circuit sets the corresponding entry in the state table to evicted. When switching from GPGPU cache mode to NPU note memory mode, all data rows set to evicted are read back from off-chip memory according to the state table, and the state table entry is restored to unevicted after being read back.

[0033] After verification in practical application scenarios, the shared final-level on-chip storage method for heterogeneous AI accelerators was specifically implemented and verified by building a co-simulation platform with high cycle accuracy. In the simulation platform, on the one hand, the operational behavior of the following three components was modeled using a full-system simulator coupled with an open-source convolutional neural network accelerator hardware model: the NPU computing core, the shared final-level on-chip storage segment running in NPU note-memory mode, and the memory channel connected to the shared final-level on-chip storage segment running in NPU note-memory mode. On the other hand, an open-source GPGPU simulator verified on actual edge devices was used to model the operational behavior of the following three components: the GPGPU computing core, the shared final-level on-chip storage segment running in GPGPU final-level cache mode, and the memory channel connected to the shared final-level on-chip storage segment running in GPGPU final-level cache mode. By adding functional models of the shared final-level on-chip storage segment and its control logic to the above two types of simulators, the hardware architecture and operation flow proposed in this invention were reproduced.

[0034] The NPU computing core was further configured as an NVDLA open-source convolutional accelerator with 6 TOPS of computing power and operating at a frequency of 1.5 GHz; the GPGPU computing core was configured with 4 streaming multiprocessors without matrix computation units, providing a total computing power of 3 TOPS and operating at a frequency of 1.5 GHz; the shared final-level on-chip storage was set to have 4 slices, totaling 4 MB in size; the memory was set to LPDDR5-6400 with 4 channels, totaling 16 GB in size; image classification models represented by EfficientNetv2, SE-ResNet, and MobileNetv3 were selected, and the simulation system was started with two configurations at input sizes of 224×224, 336×336, and 448×448, respectively: the baseline architecture (each with independent on-chip storage, data exchanged via off-chip memory) and the architecture of this invention (supporting on-chip data exchange). The Swish, Hardswish, and GlobalAvgPool operators in the model were assigned to the GPGPU for execution, and other operators were assigned to the NPU for execution.

[0035] The NPU computing core is further configured as two NVDLA open-source convolutional accelerators with 6 TOPS of computing power operating at 1.5 GHz; the GPGPU computing core is configured as having 8 streaming multiprocessors and matrix computing units, with a total computing power of 24 TOPS, operating at 1.5 GHz; the shared final-level on-chip storage is set to have 8 slices, totaling 8MB in size; the memory is set to LPDDR5-6400 with 8 channels, totaling 32GB in size; multi-view perception models represented by BEVFormer, PETR, BEVDet4D, and FastBEV are selected, with six 256x704 resolution images input each time. The inference cycle, i.e., the reciprocal of throughput, is compared between the baseline architecture (fixed even distribution of on-chip storage and memory channels) and the architecture of this invention (flexible partitioning of on-chip storage and memory channels). All operators of the task-specific header part of the model are assigned to the GPGPU for execution, and all operators of the visual backbone part are assigned to the NPU for execution.

[0036] Compared with existing technologies, this invention can reduce single inference latency by 8.7% to 11.8% under alternating operation conditions; and can reduce single inference cycle (the reciprocal of throughput) by approximately 4.9% to 16.5% under simultaneous operation conditions. The above experimental data demonstrate that, under given simulation environment and parameter conditions, the device / method proposed in this invention can achieve significant performance improvements.

[0037] The above-described specific implementations can be partially adjusted by those skilled in the art in different ways without departing from the principles and purpose of the present invention. The scope of protection of the present invention is defined by the claims and is not limited to the above-described specific implementations. All implementation schemes within the scope of the claims are bound by the present invention.

Claims

1. A shared final-level on-chip memory architecture for heterogeneous AI accelerators, characterized in that, include: The system comprises a GPGPU computing core, an NPU computing core, an on-chip network, several shared final-level on-chip storage shards, and memory channels. Each final-level on-chip storage shard is connected to both the GPGPU and NPU computing cores via the on-chip network, and each shard has its own corresponding memory channel. The GPGPU computing core performs AI model operator calculations based on the computational program running on the GPGPU, obtains the operator calculation results, and sends memory access requests to the on-chip network during the calculation process. The NPU computing core performs AI model operator calculations based on the computational program running on the NPU. The calculation process obtains the results of the operator's calculation and sends memory access requests or note memory access requests to the on-chip network during the calculation. The on-chip network, based on the data address contained in the memory access request or note memory access request issued by the GPGPU computing core or NPU computing core, sends the data access request of the GPGPU computing core to the shared last-level on-chip storage segment in the GPGPU last-level cache mode, and sends the data access request of the NPU computing core to the shared last-level on-chip storage segment in the NPU note memory mode. Each last-level on-chip storage segment operates independently in the GPGPU last-level cache mode or the NPU note memory mode.

2. The shared final-level on-chip memory architecture for heterogeneous AI accelerators according to claim 1, characterized in that, The on-chip network includes: multiple network interface units, multiple routing nodes, and physical links connecting the routing nodes. The network interface units are respectively connected to the GPGPU computing core, the NPU computing core, and the shared last-level on-chip storage fragment, and are used to receive and send network data packets. The routing nodes are interconnected through physical links to form an interconnected topology, and are configured to route network data packets to the corresponding shared last-level on-chip storage fragment according to the data address in the network data packet.

3. The shared final-level on-chip memory architecture for heterogeneous AI accelerators according to claim 1, characterized in that, Each final-level on-chip storage segment includes: a data array, a tag array and its management circuitry, an index array and its management circuitry, and a status table and its management circuitry. The data array stores data rows in a set-associative manner. The tag array and its management circuitry, upon receiving an instruction from the on-chip network indicating that the NPU has finished running and needs to switch back to GPGPU execution, switch to GPGPU final-level cache mode. Based on the GPGPU memory access request for the physical memory address, and using a replacement strategy to handle cache line hits and misses, it returns the result to the GPGPU computing core via the on-chip network and issues access requests to off-chip memory via memory channels. The index array and its management circuitry... After receiving an instruction from the on-chip network that the GPGPU has finished running and needs to switch to NPU operation, the processing circuit switches to NPU note memory mode. According to the NPU access note memory address in the NPU note memory access request, it accesses the corresponding data row of the internal data array and returns the result to the NPU computing core through the on-chip network or sends a memory access request to the memory channel connected to the last-level on-chip storage segment according to the NPU memory access request. In GPGPU cache mode, the status table management circuit maintains the cache eviction status of data rows within the note memory address range and resets the status table when the GPGPU cache mode switches to NPU note memory mode.

4. The shared final-level on-chip memory architecture for heterogeneous AI accelerators according to claim 3, characterized in that, The size of the address range for NPU access to the note memory is equivalent to the total size of the data array of all shared final-level on-chip storage fragments.

5. The shared final-level on-chip memory architecture for heterogeneous AI accelerators according to claim 3 or 4, characterized in that, The aforementioned tag array refers to a dedicated storage unit that stores tags in a set-associative manner, wherein the tag is generated by an address to uniquely distinguish different data rows within the same cache group; In GPGPU cache mode, the tag array management circuit accesses data rows in a cached manner according to the tag array, compares a specific tag portion in the address of the GPU memory request with the tag array, and determines whether the cache hit or miss is processed according to a replacement strategy including but not limited to Least Recently Used (LRU).

6. The shared final-level on-chip memory architecture for heterogeneous AI accelerators according to claim 5, characterized in that, In NPU note memory mode, when the aforementioned tag array management circuit receives a write request in the NPU note memory access request and the written data needs to be accessed after switching to GPGPU cache mode, it maintains the tag array by writing the tag portion of the requested address into the tag array. This allows the data row to be accessed without clearing the data array if the system switches to GPGPU cache mode later.

7. The shared final-level on-chip memory architecture for heterogeneous AI accelerators according to claim 3 or 4, characterized in that, The index array refers to a dedicated storage unit that stores the indexes of all data row addresses within the address range of the note storage device in a group-associative manner, wherein the index is a one-to-one mapping information from the tag to the actual storage location of the data row in the data array; In NPU note memory mode, the index array management circuit accesses data rows in note memory mode according to the index array, and maps the specific marker part in the address requested by the NPU note memory to the actual storage location in the data array according to the index array.

8. The shared final-level on-chip memory architecture for heterogeneous AI accelerators according to claim 7, characterized in that, In GPGPU final-level cache mode, when the index array management circuit receives a write request in a GPU memory access request and the range of the data address to be written is within the range of the note memory address, it updates the index array. That is, the index of the group containing the requested note memory address in the index array remains a one-to-one mapping, and the marker of the write request address is mapped to the storage location of the data row to be written in the data array. Specifically, the mapping information of the marker stored in the target group that is the marker of the write request address is denoted as A<->B; the mapping information of the storage location of the data array stored in the target group that is the storage location of the data row to be written in the data array is denoted as C<->D; and the index is updated to A<->D, C<->B.

9. The shared final-level on-chip memory architecture for heterogeneous AI accelerators according to claim 3 or 4, characterized in that, The aforementioned status table refers to a dedicated storage unit that stores the cache eviction status of all data row addresses within the address range of the notepad memory in a group-associative manner, wherein the cache eviction status is a status value that records whether a data row has been evicted from the cache. In GPGPU cache mode, when a data row within the address range of the note memory is evicted from the cache, the state table management circuit sets the corresponding entry in the state table to evicted. When switching from GPGPU cache mode to NPU note memory mode, all data rows set to evicted are read back from off-chip memory according to the state table, and the state table entry is restored to unevicted after being read back.

10. A control method based on the shared final-level on-chip memory architecture of any one of claims 1-9, characterized in that, include: Step 1: Construct and allocate operators, specifically including: allocating operators that run on the NPU with higher performance, power consumption, or performance-to-power ratio than those running on the GPGPU to the NPU; and allocating operators that cannot run on the NPU or whose performance on the NPU is lower than that on the GPGPU to the GPGPU. Step 2, Classification of Operation Modes: When all operators are divided into two parts in terms of timing, one part is allocated entirely to the GPGPU and the other part is allocated entirely to the NPU, then the GPGPU and NPU run simultaneously in a pipeline manner and proceed to Step 4; when all operators are divided into two or more parts in terms of timing, and each part is fully allocated to the GPGPU or NPU, then the GPGPU and NPU run alternately and proceed to Step 3. Step 3: Initially partition the shared last-level on-chip memory into fragments and alternately switch between them, specifically including: 3.1 Based on whether the first operator is allocated to GPGPU or NPU, set the mode of all final-level on-chip memory slices to GPGPU final-level cache mode or NPU note memory mode. 3.2 Whenever GPGPU operation switches to NPU operation, the mode of all final-level on-chip memory slices is switched from GPGPU final-level cache mode to NPU note memory mode; whenever NPU operation switches to GPGPU operation, the mode of all final-level on-chip memory slices is switched from NPU note memory mode to GPGPU final-level cache mode. Step 4: Fixedly divide the final-level on-chip storage into fragments and perform data copying, specifically including: 4.1 Allocate a portion of the final-level on-chip memory fragments to the GPU in GPGPU final-level cache mode, and allocate the remaining final-level memory fragments to the NPU in NPU note memory mode; 4.2 Whenever the GPGPU and NPU need to exchange data, the data is copied to the following two memory blocks: the memory to which the shared final-level on-chip storage fragment is connected in the GPGPU final-level cache mode, and the memory to which the shared final-level on-chip storage fragment is connected in the NPU note memory mode.

Images