On-chip off-chip collaborative hybrid cache architecture and large model acceleration chip

Abstract

According to the on-chip and off-chip collaborative hybrid cache architecture and the large model acceleration chip provided by the invention, through the collaboration of the on-chip storage layer and the off-chip extension layer, the effective cache capacity of the chip is improved from hundreds of megabytes to several GB, the data access frequency of the DRAM of the off-chip extension layer is reduced, and the total delay of large model reasoning is reduced. Compared with a traditional full SRAM on-chip architecture, the on-chip cache area is obviously reduced, the chip manufacturing cost is reduced, the static leakage current is reduced, and the overall power consumption is also reduced. The on-chip storage layer architecture can be realized based on a standard CMOS (Complementary Metal-Oxide-Semiconductor Transistor) process, and the off-chip extension layer supports mainstream DRAMs (Dynamic Random Access Memory) such as HBM3 / DDR5 and the like, can be adapted to various mainstream acceleration chip platforms and is high in compatibility. According to the method, the partition proportion of the on-chip storage layer and the off-chip extension layer can be dynamically adjusted according to large model types (such as LLM and CV models), and access features of different large model data are adapted.

Description

Technical Field

[0001] This application relates to the field of large model caching technology, and in particular to on-chip and off-chip collaborative hybrid cache architecture and large model acceleration chips. Background Technology

[0002] Currently, large-scale models that meet commercial requirements generally require models with hundreds of billions of parameters. The caching systems for current large-scale model acceleration chips (such as the hundreds of billions of parameters Transformer and BLOOM-176B) mainly fall into two categories:

[0003] (1) On-chip cache solution: Represented by NVIDIA H800 and Huawei Ascend 910, the L1-L3 three-level cache is built using a full SRAM architecture. The on-chip cache capacity is usually several hundred megabytes (such as the L3 cache of H800 is 512MB). It relies on the low latency (1-10ns) of SRAM to ensure data access efficiency.

[0004] (2) Off-chip expansion scheme: Off-chip storage expansion is achieved through DRAM such as DDR5 and HBM3, with a capacity of several GB to tens of GB (e.g., HBM3 single stack capacity of 8GB), used to store large-capacity data such as model cold weights and historical KV cache, but the access latency is high (the latency of HBM3 is about 100-200ns, and the latency of DDR5 is about 300-500ns).

[0005] (3) Hybrid storage exploration: Some solutions also attempt to combine on-chip SRAM and GC-eDRAM, but only focus on on-chip architecture; other solutions (such as AMD MI300) combine HBM with on-chip SRAM, but the on-chip is still all SRAM, which does not solve the contradiction between area and power consumption.

[0006] Therefore, existing technologies suffer from three core contradictions, failing to meet the caching requirements of large-scale models for "large capacity, low latency, and low power consumption":

[0007] (1) The traditional on-chip full SRAM cache capacity is only a few hundred megabytes, which cannot cover the GB-level data required for large model inference (such as BLOOM-176B inference which requires loading more than 2GB of weights at a time). It is necessary to frequently access off-chip DRAM, resulting in a very high proportion of data interaction delay in the total inference delay.

[0008] (2) The traditional on-chip full SRAM architecture has a high proportion of chip area (e.g., 512MB L3 cache accounts for more than 15% of the chip area), resulting in high manufacturing costs; and SRAM has a large static leakage current, with idle power consumption accounting for 22% of the total chip power consumption, thus conflicting with the low power consumption requirements of edge scenarios.

[0009] (3) The off-chip DRAM in the existing solution is only used as “passive storage” and cannot dynamically supplement the on-chip cache capacity according to the data access characteristics, resulting in frequent off-chip data loading and a surge in latency. Summary of the Invention

[0010] In view of the shortcomings of the prior art, the present invention provides an on-chip and off-chip collaborative hybrid cache architecture and a large model acceleration chip to solve the problem that the prior art cannot meet the cache requirements of large models, large capacity, low latency, and low power consumption.

[0011] To achieve the above and other related objectives, a first aspect of this application provides an on-chip and off-chip collaborative hybrid cache architecture, comprising: an on-chip storage layer for storing high-frequency and mid-frequency data in a large model data set; an off-chip extension layer for storing ultra-low-frequency data in the large model data set; and a scheduling control layer connected to the on-chip storage layer and the off-chip extension layer, respectively, for using a dynamic scheduling mechanism to control the dynamic migration of the large model data between the on-chip storage layer and the off-chip extension layer in real time.

[0012] In some embodiments of the first aspect of this application, the scheduling control layer includes: a data feature classifier, a dynamic migration controller, and a data prefetching unit; wherein, the data feature classifier is used to monitor the large model's inference task execution process in real time, classify the large model data according to the monitoring results, and obtain classification results; the dynamic migration controller is used to control the large model data to perform dynamic migration in real time using a dynamic scheduling mechanism; the data prefetching unit is used to prefetch the corresponding ultra-low frequency data in the off-chip extension layer to the transition area of ​​the on-chip storage layer using a preset prefetch window based on the data flow characteristics during the large model's inference task execution process.

[0013] In some embodiments of the first aspect of this application, the process of real-time monitoring of the large model's inference task execution process and classifying the large model data according to the monitoring results to obtain classification results includes: real-time monitoring of the large model's inference task execution process to obtain real-time data popularity feature parameters corresponding to each large model data; and classification based on preset classification rules according to the real-time data popularity feature parameters corresponding to each large model data to obtain classification results.

[0014] In some embodiments of the first aspect of this application, the on-chip storage layer includes: a high-frequency region, an intermediate-frequency region, and a transition region; wherein, the high-frequency region is used to store high-frequency data in the large model data; the intermediate-frequency region is used to store intermediate-frequency data in the large model data; and the transition region is used to temporarily store data dynamically migrated between the on-chip storage layer and the off-chip extension layer or prefetched ultra-low-frequency data.

[0015] In some embodiments of the first aspect of this application, the on-chip storage layer employs a hybrid array of SRAM and GC-eDRAM.

[0016] In some embodiments of the first aspect of this application, the high-frequency region uses 6T SRAM; the intermediate-frequency region uses 4T GC-eDRAM; and the transition region uses 3T GC-eDRAM.

[0017] In some embodiments of the first aspect of this application, the off-chip extension layer is DRAM.

[0018] In some embodiments of the first aspect of this application, the scheduling control layer further includes: an adaptive refresh unit; the adaptive refresh unit is used to execute a corresponding refresh strategy according to the data lifecycle corresponding to each large model data.

[0019] In some embodiments of the first aspect of this application, the real-time data popularity characteristic parameters include: data access frequency, data lifecycle, and data update cycle.

[0020] To achieve the above and other related objectives, a second aspect of this application provides a large-model acceleration chip, including the aforementioned on-chip and off-chip collaborative hybrid cache architecture.

[0021] As described above, the on-chip and off-chip collaborative hybrid cache architecture and large-model acceleration chip provided in this application have the following beneficial effects:

[0022] (1) This application improves the effective cache capacity of the chip from hundreds of megabytes to several gigabytes by working together with the on-chip storage layer and the off-chip extension layer, and reduces the data access frequency of the off-chip extension layer DRAM, thereby reducing the total latency of large model inference.

[0023] (2) Compared with the traditional all-SRAM on-chip architecture, the on-chip cache area is significantly reduced, which reduces the chip manufacturing cost, and the static leakage current is reduced, and the overall power consumption is also reduced.

[0024] (3) The on-chip memory layer architecture of this application can be implemented based on standard CMOS process, and the off-chip extension layer supports mainstream DRAM such as HBM3 / DDR5, which can be adapted to various mainstream acceleration chip platforms and has strong compatibility.

[0025] (4) This application supports dynamically adjusting the partition ratio of on-chip storage layer and off-chip extension layer according to the type of large model (such as LLM, CV model) to adapt to the access characteristics of different large model data. Attached Figure Description

[0026] Figure 1 The diagram shown is a structural schematic of an on-chip and off-chip collaborative hybrid cache architecture according to an embodiment of this application.

[0027] Figure 2 The diagram shown is a schematic representation of the structure of a scheduling control layer in one embodiment of this application. Detailed Implementation

[0028] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0029] Before providing a further detailed description of the present invention, the nouns and terms used in the embodiments of the present invention are explained, and the nouns and terms used in the embodiments of the present invention are subject to the following interpretations:

[0030] <1> On-chip cache: This is a high-speed memory integrated inside the chip, used to temporarily store data and instructions that are frequently accessed by the CPU, in order to reduce the latency of accessing main memory and improve system performance.

[0031] <2> DRAM (Dynamic Random Access Memory): A type of computer memory used to store data for fast access by the central processing unit.

[0032] <3> SRAM (Static Random-Access Memory) is a semiconductor memory based on bistable flip-flops. It stores data through transistor states and can maintain storage while powered on without the need for refresh circuits. Data is lost when power is off.

[0033] <4> KV: Key and Value data in the attention mechanism.

[0034] <5> HBM3: The latest version of HBM technology, representing the highest level of current high-bandwidth memory technology.

[0035] <6> Gain-cell eDRAM (GC-eDRAM) is an embedded memory architecture that combines two to four transistors to enable read, write, and random access functions. GC-eDRAM is fully compatible with CMOS logic processes and has a higher integration density than 6T-SRAM.

[0036] <7> BLOOM (Big Science Large Open-science Open-access Multilingual Language Model): BLOOM is a language model with 176 billion parameters trained on 46 natural languages ​​and 13 programming languages.

[0037] To facilitate understanding of the embodiments of this application, firstly, in conjunction with Figure 1 Detailed explanation. Figure 1 This illustration shows a schematic diagram of an on-chip and off-chip collaborative hybrid cache architecture according to an embodiment of the present invention. The on-chip and off-chip collaborative hybrid cache architecture 100 in this embodiment includes: an on-chip storage layer 110, an off-chip extension layer 120, and a scheduling control layer 130, wherein the scheduling control layer 130 is connected to the on-chip storage layer 110 and the off-chip extension layer 120, respectively.

[0038] The on-chip storage layer 110 is used to store high-frequency and medium-frequency data in the large model data; the off-chip extension layer 120 is used to store ultra-low-frequency data in the large model data; the scheduling control layer 130 is connected to the on-chip storage layer 110 and the off-chip extension layer 120 respectively, and is used to control the dynamic migration of the large model data between the on-chip storage layer 110 and the off-chip extension layer 120 in real time using a dynamic scheduling mechanism.

[0039] It should be noted that with the rapid development of artificial intelligence technology, the demand for high-performance computing for large models is constantly increasing, requiring dedicated chips to support their training and inference processes. In this embodiment, an on-chip and off-chip collaborative hybrid caching architecture is integrated on the large model chip to store data during the large model's inference task execution and to quickly access it during computation. The large model data includes, but is not limited to: tokens, KV cache generated during computation, various expert model weight parameters, intermediate data generated by multi-task parallel computation, and output data after computation, etc. The specific types are determined according to the task and different stages, and are not limited here.

[0040] In one embodiment of this application, the on-chip storage layer 110 adopts a hybrid array of SRAM and GC-eDRAM.

[0041] In one embodiment of this application, the off-chip extension layer 120 is DRAM. The off-chip extension layer may use DRAM memory such as HBM, GDDR, DDR, or LPDDR, and there is no limitation herein.

[0042] In one embodiment of this application, combined with Figure 1The scheduling control layer 130 includes: a data feature classifier 1301, a dynamic migration controller 1302, and a data prefetching unit 1303. The data feature classifier 1301 is used to monitor the large model's inference task execution process in real time, classify the large model data based on the monitoring results, and obtain classification results. The dynamic migration controller 1302 is used to control the dynamic migration of the large model data in real time using a dynamic scheduling mechanism. The data prefetching unit 1303 is used to prefetch the corresponding ultra-low frequency data from the off-chip extension layer to the transition area of ​​the on-chip storage layer using a preset prefetch window based on the data flow characteristics during the large model's inference task execution process.

[0043] In this embodiment, the scheduling control layer 130 adopts a hardware controller, which may be one or more of the following: central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), field-programmable gate array (FPGA), neural network processor (NPU), microcontroller unit (MCU), and application-specific integrated circuit (ASIC).

[0044] In some examples, the process of real-time monitoring of the large model's inference task execution and classification of the large model data based on the monitoring results includes: real-time monitoring of the large model's inference task execution to obtain real-time data popularity feature parameters corresponding to each large model data; and classification based on preset classification rules according to the real-time data popularity feature parameters corresponding to each large model data to obtain classification results.

[0045] Specifically, the inference task execution process of large models is monitored in real time to obtain real-time data popularity characteristic parameters corresponding to each large model's data. These real-time data popularity characteristic parameters include, but are not limited to, data access frequency, data lifecycle, and data update cycle. Based on the real-time data popularity characteristic parameters of each large model's data, all large model data are classified into three types according to preset classification rules: high-frequency data, medium-frequency data, and ultra-low-frequency data.

[0046] The preset classification rules include, but are not limited to: when the data lifecycle of a large model data is less than or equal to a first preset lifecycle threshold, and the data access frequency is greater than or equal to a first preset access frequency threshold, the large model data is classified as high-frequency data; when the data lifecycle of a large model data is greater than the first preset lifecycle threshold and less than or equal to a second preset lifecycle threshold, and the data access frequency is less than the first preset access frequency threshold but greater than or equal to the second preset access frequency threshold, the large model data is classified as medium-frequency data; when the data access frequency of a large model data is less than or equal to a third preset access frequency threshold, and the data lifecycle is greater than or equal to a third preset lifecycle threshold, the large model data is classified as ultra-low-frequency data. Specific classification rules are determined based on the task and different stages, and are not limited here.

[0047] For example, if the lifetime of activation values ​​and attention matrices in large model data is less than or equal to 10 μs and the data access frequency is greater than or equal to 100 times / ms, then activation values ​​and attention matrices are classified as high-frequency data. If the lifetime of hot model weights and recent key-value cache blocks in large model data is greater than or equal to 1ms and less than 100ms, and the data access frequency is in the range of 10-100 times / ms, then hot model weights and recent key-value cache blocks are classified as medium-frequency data. If the lifetime of cold model weights in large model data is greater than or equal to 10 min and the data access frequency is less than or equal to 1 time / ms, then cold model weights are classified as ultra-low-frequency data. Or, if the data access frequency of historical key-value cache cold data is less than or equal to 5 times / ms and the data lifetime is greater than or equal to 100ms, then historical key-value cache cold data are classified as ultra-low-frequency data.

[0048] The large model data after the above classification is stored according to type. The high-frequency and mid-frequency data in the large model data are stored in the on-chip storage layer, and the ultra-low-frequency data in the large model data is stored in the off-chip extension layer.

[0049] In one embodiment of this application, the on-chip storage layer 110 includes: a high-frequency region 1101, an intermediate-frequency region 1102, and a transition region 1103; wherein, the high-frequency region 1101 is used to store high-frequency data in the large model data; the intermediate-frequency region 1102 is used to store intermediate-frequency data in the large model data; and the transition region 1103 is used to temporarily store data dynamically migrated between the on-chip storage layer and the off-chip extension layer or prefetched ultra-low-frequency data.

[0050] In some examples, the high-frequency region 1101 uses 6T SRAM; the mid-frequency region 1102 uses 4T GC-eDRAM; and the transition region 1103 uses 3T GC-eDRAM.

[0051] It should be noted that after obtaining the classification results, the large model parameters are stored separately according to type. High-frequency data and intermediate-frequency data are stored in the on-chip storage layer 110, which further includes a high-frequency region 1101 and an intermediate-frequency region 1102. Specifically, the high-frequency region 1101 uses 6T SRAM to store high-frequency data such as activation values ​​and attention matrices, with a data access latency of less than or equal to 2ns in actual use. The intermediate-frequency region 1102 uses 4T GC-eDRAM to store intermediate-frequency data such as model hot weights and recent KV cache blocks. The bit density of 4T GC-eDRAM is 3.2 times that of 6T SRAM, resulting in a 65% reduction in static leakage current.

[0052] The transition region 1103 employs 3T GC-eDRAM as a temporary storage buffer for data exchange between the on-chip storage layer 110 and the off-chip extension layer 120, supporting dual-port parallel access and achieving a latency of less than 10ns. The transition region 1103 is used to temporarily store dynamically migrated data or prefetched ultra-low frequency data between the on-chip storage layer and the off-chip extension layer. It can also store large model data with access frequencies lower than intermediate frequency data but higher than ultra-low frequency data. The data access frequency threshold for dividing and storing large model data within the transition region is set according to actual needs and is not limited here.

[0053] For example, the cache capacity in the high-frequency area of ​​the on-chip memory layer is 16-64MB, the cache capacity in the mid-frequency area is 128-512MB, and the cache capacity in the transition area is 64-256MB. The off-chip expansion layer uses HBM3, DDR5, etc. as off-chip storage (capacity 8GB-64GB). By adopting an on-chip and off-chip collaborative architecture, the effective cache capacity can be expanded to several GB.

[0054] Furthermore, the process of dynamically controlling the dynamic migration of large model data in real time using a dynamic scheduling mechanism in the dynamic migration controller 1302 includes: if the real-time data heat characteristic parameters of the classified large model data change during real-time monitoring of the large model's inference task execution, it indicates that the classification result will also change, and the large model data stored in the on-chip storage layer and the off-chip extension layer will be dynamically migrated according to the changed classification result.

[0055] Specifically, when a large model data has a data access frequency that suddenly rises to a level greater than or equal to a first preset access frequency threshold in the intermediate frequency region 1102 within the on-chip storage layer 110, the type of the large model data changes from intermediate frequency data to high frequency data. At this time, a migration command is issued to migrate the large model data from the intermediate frequency region 1102 to the high frequency region 1101. Similarly, when the data access frequency of the large model data stored in the on-chip storage layer 110 drops to a level less than or equal to a third preset access frequency threshold, it is migrated to the off-chip extension layer 120 for storage. Experimental data shows that the single migration latency is less than or equal to 50 ns when migrating within the on-chip storage layer 110, and less than or equal to 200 ns when migrating between the on-chip storage layer 110 and the off-chip extension layer 120.

[0056] The scheduling control layer 130 also includes a data prefetching unit 1303, used to prefetch the corresponding ultra-low frequency data from the off-chip extension layer into the transition area of ​​the on-chip storage layer based on the data flow characteristics during the execution of inference tasks of a large model, using a preset prefetch window. That is, the data prefetching unit 1303 prefetches the ultra-low frequency data from the off-chip extension layer into the transition area of ​​the on-chip storage layer in advance, so that it can be called in a timely manner during large model calculations, thereby reducing the access waiting time of the off-chip extension layer. The data flow characteristics are data dependencies between Transformer layers, for example, the calculation processing of the current layer requires the output of other layers. The length of the preset prefetch window is set according to the actual situation and is not limited here.

[0057] It should be explained that the access latency of the off-chip extension layer is much higher than that of the on-chip storage layer. If large models frequently wait for data loading from the off-chip extension layer during computation, it will cause computing units (such as GPU cores) to be idle, resulting in a decrease in overall throughput. Through the prefetching technology of the data prefetching unit 1303, ultra-low frequency data can be preloaded to the on-chip storage layer.

[0058] For example, the Transformer model consists of stacked encoders and decoders, with each layer containing a self-attention mechanism and a feedforward neural network. In the self-attention mechanism, the output at each position depends on the data from all positions in the input sequence; the feedforward neural network transforms the output at each position independently. Based on the data dependencies between Transformer layers, the access patterns of ultra-low-frequency data during model computation are determined. A pre-fetch window is used to pre-fetch the ultra-low-frequency data required for model computation to the on-chip storage layer, which can transform off-chip access into on-chip access, fundamentally solving the memory access bottleneck in the training and inference of large models and reducing off-chip access waiting time.

[0059] In one embodiment of this application, as Figure 2As shown, the scheduling control layer 130 also includes an adaptive refresh unit 1304; the adaptive refresh unit 1304 is used to execute corresponding refresh strategies according to the data lifecycle of each large model data. High-frequency and mid-frequency data have longer lifecycles than ultra-low-frequency data. During the inference task of large models, high-frequency and mid-frequency data are frequently accessed, while ultra-low-frequency data is accessed less frequently. Therefore, using different refresh strategies for different data types can reduce power consumption caused by invalid refreshes. The specific setting of the refresh strategy depends on the actual task and is not limited here. For example, setting the refresh interval of large model data stored in the mid-frequency region to 500μs and the refresh interval of large model data stored in the transition region to 800μs can reduce the refresh power consumption of GC-eDRAM, which experiments have shown can reduce it by 35%.

[0060] Furthermore, when configuring the on-chip memory layer and the off-chip expansion layer, ECC verification circuits are respectively set at the interface connecting the on-chip memory layer and the off-chip expansion layer to compensate for accidental errors during data migration. The ECC verification circuits adopt ECC (Error Correcting Code) error correction technology, which can automatically correct errors when data errors occur, further improving data reliability. At the same time, the dynamic data migration between the on-chip memory layer and the off-chip expansion layer is performed in parallel with the large model calculation, thereby hiding migration latency.

[0061] This application also provides a large-scale acceleration chip, including the on-chip and off-chip collaborative hybrid cache architecture described above.

[0062] It should be noted that with the rapid development of artificial intelligence technology, the demand for high-performance computing for large models is constantly increasing, requiring dedicated acceleration chips to support their training and inference processes. As a key hardware component for implementing these technologies, the performance of acceleration chips directly impacts the efficiency and effectiveness of artificial intelligence applications. These acceleration chips can be dedicated computing acceleration chips (or accelerators) designed to handle heavy computational tasks, such as general-purpose graphics processing units (GPGPUs) and tensor processing units (TPUs). Of course, they can also be other processors designed for large model computational tasks; no limitation is made here.

[0063] To facilitate understanding of the on-chip and off-chip collaborative hybrid cache architecture of this application, the following specific embodiments are provided for illustration.

[0064] Example 1: Taking "Billion-Parameter Transformer Model (BLOOM-176B) Inference" as an example, the detailed implementation steps are as follows:

[0065] 1. Initialization phase (model loading).

[0066] 1.1 Model weights (total capacity approximately 352GB, including 8GB of hot weights and 344GB of cold weights) loading: Hot weights (data access frequency threshold 5-10 times / ms) are stored in the GC-eDRAM mid-frequency area of ​​the on-chip storage layer (512MB), and cold weights are stored in the off-chip extension layer HBM3 (64GB, divided into 8 stacks).

[0067] 1.2 Cache parameter configuration: SRAM high-frequency area capacity is 64MB (data access frequency threshold 100 times / ms), GC-eDRAM transition area is 256MB (data access frequency threshold 5 times / ms), HBM3 interface bandwidth is 1TB / s, and the preset prefetch window of the data prefetch unit is set to "lower 2 layer Transformer weight".

[0068] 2. Reasoning and calculation stage.

[0069] 2.1 Input prompt word processing: The generated activation value (data access frequency 120 times / ms, data lifetime 5μs) is stored in the high-frequency area of ​​SRAM for direct access by the computing unit;

[0070] 2.2 Attention Calculation: The generated KV cache block (initial data access frequency 80 times / ms) is first stored in the GC-eDRAM intermediate frequency region; after a 100ms observation period, the KV cache block with the data access frequency reduced to 8 times / ms is migrated to the transition region;

[0071] 2.3 Ultra-low frequency data prefetching: The data prefetching unit prefetches the cold weights of the next two layers (from the off-chip extension layer HBM3) into the transition area of ​​the on-chip storage layer according to the inter-layer scheduling of the Transformer, waiting for the computing unit to call them.

[0072] 3. Dynamic adjustment phase.

[0073] 3.1 Mid-frequency data upscaling: When the access frequency of a certain KV cache block increases to 110 times / ms due to context association, the dynamic migration controller migrates it from the GC-eDRAM mid-frequency zone to the SRAM high-frequency zone with a migration delay of 45ns;

[0074] 3.2 Transition Region Data Frequency Reduction: When the data access frequency of a certain cold weight stored in the transition region drops to 3 times / ms, it is migrated to the off-chip extension layer HBM3, and at the same time, body bias control is triggered (the voltage of the 3T GC-eDRAM in the transition region drops from 0.3V to 0.1V to reduce leakage current).

[0075] 3.3 Error Correction: The ECC verification circuit detects a data transmission error between the on-chip memory layer and the off-chip extension layer once, and corrects it through a local retransmission mechanism, so that the large model inference will not be interrupted.

[0076] 4. Performance test verification results. This architecture in BLOOM-176B inference:

[0077] 4.1 The on-chip storage layer has a cache capacity of 512MB (SRAM 64MB + GC-eDRAM 448MB) + the off-chip extension layer HBM3 has a capacity of 64GB, effectively covering 98% of the data required for large model inference;

[0078] 4.2 The data access frequency of the off-chip extension layer was reduced from 40 times / ms in the all-SRAM architecture to 16 times / ms, resulting in a 32% reduction in total inference latency;

[0079] 4.3 The chip area is reduced by 52% compared to the all-SRAM architecture solution, and the overall power consumption is reduced by 38%.

[0080] It should be noted that, through experiments, this application demonstrates that, by coordinating the on-chip storage layer and the off-chip extension layer, the effective cache capacity of the chip is increased from several hundred megabytes to several gigabytes, the data access frequency of the off-chip extension layer DRAM is reduced by 60%, and the total latency of large model inference is reduced by 25%-35% (taking BLOOM-176B as an example, the token generation rate is increased from 80 tokens / s to 95 tokens / s). Compared with the traditional all-SRAM on-chip architecture, this application reduces the on-chip cache area by 40%-60%, reduces chip manufacturing costs by 28%, reduces static leakage current by 65%, and reduces overall power consumption by 30%-45%.

[0081] It should be emphasized that the on-chip and off-chip collaborative hybrid caching architecture provided in this application has the following beneficial effects:

[0082] (1) This application improves the effective cache capacity of the chip from hundreds of megabytes to several gigabytes by working together with the on-chip storage layer and the off-chip extension layer, and reduces the data access frequency of the off-chip extension layer DRAM, thereby reducing the total latency of large model inference.

[0083] (2) Compared with the traditional all-SRAM on-chip architecture, the on-chip cache area is significantly reduced, which reduces the chip manufacturing cost, and the static leakage current is reduced, and the overall power consumption is also reduced.

[0084] (3) The on-chip memory layer architecture of this application can be implemented based on standard CMOS process, and the off-chip extension layer supports mainstream DRAM such as HBM3 / DDR5, which can be adapted to various mainstream acceleration chip platforms and has strong compatibility.

[0085] (4) This application supports dynamically adjusting the partition ratio of on-chip storage layer and off-chip extension layer according to the type of large model (such as LLM, CV model) to adapt to the access characteristics of different large model data.

[0086] In the embodiments of this application, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect, without limiting their order. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" do not necessarily imply that they are different.

[0087] It should be noted that, in the embodiments of this application, the words "exemplary" or "for example" indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0088] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0089] It should also be understood that the module division in the embodiments of this application is illustrative and only represents a logical functional division; in actual implementation, there may be other division methods. Furthermore, the functional modules in the various embodiments of this application can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0090] As used in this specification, the terms "component," "module," "system," etc., are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. As illustrated, applications running on computing devices and computing devices can both be components. One or more components may reside in a process and / or an execution thread, and components may be located on a single computer and / or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via signals).

[0091] Those skilled in the art will recognize that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0092] In the embodiments provided in this application, it should be understood that the disclosed systems and devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0093] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0094] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0095] In the above embodiments, the functions of each functional unit can be implemented entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. A computer program product includes one or more computer instructions (programs). When the computer program instructions (programs) are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs, DVDs), or semiconductor media (e.g., solid-state disks, SSDs, etc.).

[0096] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0097] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0098] In summary, the on-chip and off-chip collaborative hybrid cache architecture and large model acceleration chip provided in this application include: an on-chip storage layer for storing high-frequency and mid-frequency data in the large model data; an off-chip extension layer for storing ultra-low-frequency data in the large model data; and a scheduling control layer connected to the on-chip storage layer and the off-chip extension layer respectively, used to dynamically control the migration of large model data between the on-chip storage layer and the off-chip extension layer in real time using a dynamic scheduling mechanism. This application, through the collaboration of the on-chip storage layer and the off-chip extension layer, increases the effective cache capacity of the chip from hundreds of megabytes to several gigabytes, and reduces the data access frequency of the DRAM in the off-chip extension layer, thereby reducing the total latency of large model inference. Compared with the traditional all-SRAM on-chip architecture, this application significantly reduces the on-chip cache area, lowers chip manufacturing costs, reduces static leakage current, and lowers overall power consumption. The on-chip storage layer architecture of this application can be implemented based on standard CMOS technology, and the off-chip extension layer supports mainstream DRAMs such as HBM3 / DDR5, making it compatible with various mainstream acceleration chip platforms and exhibiting strong compatibility. This application supports dynamically adjusting the partition ratio of the on-chip storage layer and the off-chip extension layer according to the type of large model (such as LLM and CV models), adapting to the access characteristics of different large model data. Therefore, this application effectively overcomes the various shortcomings of the prior art and has high industrial application value.

[0099] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A hybrid on-chip and off-chip collaborative cache architecture, characterized in that, include: An on-chip storage layer is used to store high-frequency and mid-frequency data in large model data. An off-chip extension layer is used to store ultra-low frequency data in large model data; The scheduling and control layer is connected to the on-chip storage layer and the off-chip extension layer respectively, and is used to control the dynamic migration of large model data between the on-chip storage layer and the off-chip extension layer in real time using a dynamic scheduling mechanism.

2. The on-chip and off-chip collaborative hybrid cache architecture according to claim 1, characterized in that, The scheduling control layer includes: a data feature classifier, a dynamic migration controller, and a data prefetching unit; wherein... The data feature classifier is used to monitor the large model's inference task execution process in real time, classify the large model data based on the monitoring results, and obtain classification results. The dynamic migration controller is used to control the dynamic migration of large model data in real time using a dynamic scheduling mechanism. The data prefetching unit is used to prefetch the corresponding ultra-low frequency data in the off-chip extension layer to the transition area of ​​the on-chip storage layer based on the data flow characteristics during the inference task of the large model and using a preset prefetch window.

3. The on-chip and off-chip collaborative hybrid cache architecture according to claim 2, characterized in that, The process of real-time monitoring of the inference task execution of a large model, classifying the large model data based on the monitoring results, and obtaining the classification results includes: Real-time monitoring of the inference task execution process of large models is performed to obtain real-time data heat characteristic parameters corresponding to the data of each large model. Based on the real-time data popularity feature parameters corresponding to each large model, classification is performed according to preset classification rules to obtain classification results.

4. The on-chip and off-chip collaborative hybrid cache architecture according to claim 1, characterized in that, The on-chip memory layer includes: a high-frequency region, an intermediate-frequency region, and a transition region; wherein... The high-frequency zone is used to store high-frequency data from large model data; The intermediate frequency region is used to store intermediate frequency data in large model data; The transition area is used to temporarily store data that is dynamically migrated between the on-chip storage layer and the off-chip extension layer, or to prefetch ultra-low frequency data.

5. The on-chip and off-chip collaborative hybrid cache architecture according to claim 1, characterized in that, The on-chip storage layer uses a hybrid array of SRAM and GC-eDRAM.

6. The on-chip and off-chip collaborative hybrid cache architecture according to claim 4, characterized in that, The high-frequency region uses 6T RAM; the mid-frequency region uses 4T GC-eDRAM; and the transition region uses 3T GC-eDRAM.

7. The on-chip and off-chip collaborative hybrid cache architecture according to claim 1, characterized in that, The off-chip extension layer uses DRAM.

8. The on-chip and off-chip collaborative hybrid cache architecture according to claim 1, characterized in that, The scheduling control layer further includes an adaptive refresh unit; the adaptive refresh unit is used to execute the corresponding refresh strategy according to the data lifecycle of each large model data.

9. The on-chip and off-chip collaborative hybrid cache architecture according to claim 3, characterized in that, The real-time data popularity characteristic parameters include: data access frequency, data lifecycle, and data update cycle.

10. A large-scale model acceleration chip, characterized in that, include: The on-chip and off-chip collaborative hybrid caching architecture as described in any one of claims 1 to 9.

Images