Buffer memory structure based on HBM packaging and electronic equipment

Abstract

The invention relates to the field of chips, and discloses an HBM packaging-based cache structure and electronic equipment, and the structure is characterized in that a first surface of a silicon interposer is provided with a calculation bare chip and a first HBM stack, the first HBM stack is located outside a coverage area of the calculation bare chip, a second surface of the silicon interposer is provided with a second HBM stack and a third HBM stack, a part of a storage area of the second HBM stack is located in the coverage area, and the third HBM stack is located in the coverage area; the third HBM stack is completely located in the coverage area; the priority sensing module in the computing bare chip determines the priority of cache data, the address mapping module selects a storage area for mapping the cache data from the first HBM stack, the second HBM stack and the third HBM stack according to the priority of the cache data, the routing module allocates interconnection paths, the first HBM stack and the second HBM stack are located in the storage area outside a coverage area, and the first HBM stack and the second HBM stack are located in the storage area outside the coverage area. The second HBM stack is located in a storage area in the coverage area, and the priorities of the stored cache data of the third HBM stack are sequentially increased. And the high-priority cache data delay is reduced.

Description

Technical Field

[0001] This application relates to the field of chips, and more particularly to a cache structure and electronic device based on HBM packaging. Background Technology

[0002] In large-scale language models (LLMs) based on the Transformer architecture, traditional key-value cache data is often stored in the Dynamic Random Access Memory (DRAM) of the Graphics Processing Unit (GPU), such as High-Bandwidth Memory (HBM). HBM is a new memory technology designed to address memory bandwidth bottlenecks in high-performance computing and is widely used in GPUs, AI accelerators, and high-performance servers. Based on 2.5 / 3D advanced packaging technology, HBM vertically stacks multiple DRAM dies using through-silicon vias (TSVs) and packages them together with the GPU to form a high-capacity, high-bit-width Double Data Rate (DDR) array. Cache data is typically evenly distributed or statically mapped across multiple HBM stacks, and some cached data access does not meet low-latency requirements. Summary of the Invention

[0003] The purpose of this application is to provide at least one cache structure and electronic device based on HBM packaging, which can at least solve the problem that some cached data access does not meet the low latency requirements, and can at least reduce the latency of high-priority cached data.

[0004] In a first aspect, this application provides a cache structure based on HBM packaging, including: a silicon interposer; The silicon interposer has a first surface and a second surface opposite to each other. The first surface is provided with a computing die and at least one first HBM stack, which is located outside the coverage area of ​​the computing die. The second surface is provided with a second HBM stack and at least one third HBM stack, a portion of the storage area of ​​the second HBM stack is located within the coverage area of ​​the computing die, and the third HBM stack is completely located within the coverage area of ​​the computing die. The computing die includes a routing module, a priority awareness module, and an address mapping unit. The priority awareness module is used to determine the priority of cached data; The address mapping module is used to map the cached data to target storage areas in the first HBM stack, the second HBM stack, and the third HBM stack according to the priority of the cached data; wherein, the priority of the cached data mapped to the first HBM stack, the priority of the cached data mapped to the storage area of ​​the second HBM stack located outside the coverage area of ​​the computing die, the priority of the cached data mapped to the storage area of ​​the second HBM stack located within the coverage area of ​​the computing die, and the priority of the cached data mapped to the third HBM stack increase sequentially; The routing module is used to allocate interconnection paths between the target storage area and the computing die.

[0005] Optionally, the first surface is provided with a plurality of first HBM stacks, which are symmetrically arranged around the coverage area of ​​the computing die; the second surface is provided with a plurality of second HBM stacks, which are symmetrically arranged around the edge of the coverage area of ​​the computing die.

[0006] Optionally, the orthographic projection regions of the third HBM stacked on the silicon interposer, the second HBM stacked on the silicon interposer, and the first HBM stacked on the silicon interposer do not overlap.

[0007] Optionally, the computing die is perpendicularly docked to the third HBM stack via microbumps.

[0008] Optionally, the storage area of ​​the second HBM stack located in the coverage area of ​​the computing die includes an edge storage unit, which is directly connected to the computing die via microbumps.

[0009] Optionally, the computing die has I / O contacts that correspond one-to-one with each of the first HBM stacks, each of the second HBM stacks and the third HBM stack, and each of the first HBM stacks, each of the second HBM stacks and the third HBM stack is interconnected with the corresponding I / O contacts.

[0010] Optionally, the routing module is specifically used to connect a portion of the data channels of the computing die to the first HBM stack, the second HBM stack, and the third HBM stack in a round-robin fashion according to a preset scheduling strategy.

[0011] Optionally, the address mapping unit is further configured to, after each inference cycle of the LLM, re-determine the priority of the cached data based on the actual access hit statistics of the cached data, and update the storage area if the re-determined priority of the cached data does not match the current storage area.

[0012] Optionally, the priority awareness module is specifically used to determine the priority of the cached data based on the access frequency, context position, recentity, attention score, and model stage of the cached data.

[0013] Secondly, this application provides an electronic device including a cache structure based on HBM packaging as described above.

[0014] The beneficial effects of the embodiments of this application compared with the prior art are: In the HBM-based cache structure of this application, the silicon interposer has a first surface and a second surface opposite to each other. The first surface is provided with a computing die and at least one first HBM stack, which is located outside the coverage area of ​​the computing die. The second surface is provided with a second HBM stack and at least one third HBM stack. A portion of the storage area of ​​the second HBM stack is located within the coverage area of ​​the computing die, and the third HBM stack is completely located within the coverage area of ​​the computing die. The computing die contains a routing module, a priority-aware module, and an address mapping unit. The priority-aware module determines the priority of the cached data, and the address mapping module is used to determine the priority of the cached data based on its priority. The routing module maps the cached data to target storage areas in the first HBM stack, the second HBM stack, and the third HBM stack. The routing module allocates interconnection paths between the target storage areas and the compute die. Since the priority of the cached data mapped to the first HBM stack is higher than the priority of the cached data mapped to the storage area of ​​the second HBM stack located outside the coverage area of ​​the compute die, and the priority of the cached data mapped to the storage area of ​​the second HBM stack located within the coverage area of ​​the compute die and the third HBM stack, the high-priority cached data can be mapped to the storage area of ​​the HBM stack within the coverage area of ​​the compute die, reducing access latency.

[0015] It is understandable that the beneficial effects of the second aspect mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0016] One or more embodiments are illustrated by way of example with reference to the accompanying drawings, and these illustrative descriptions do not constitute a limitation on the embodiments.

[0017] Figure 1 This is a schematic diagram of a cache structure based on HBM encapsulation provided in one embodiment of this application. Figure 1 ; Figure 2 This is a schematic diagram of a cache structure based on HBM encapsulation provided in another embodiment of this application. Figure 2 . Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been provided in the various embodiments of this application to help readers better understand this application. However, the technical solutions claimed in this application can be implemented even without these technical details and various changes and modifications based on the following embodiments. The division of the various embodiments below is for the convenience of description and should not constitute any limitation on the specific implementation of this application. The various embodiments can be combined with and referenced by each other without contradiction.

[0019] To facilitate understanding of the embodiments of this application, relevant information about HBM will be introduced first.

[0020] In LLM based on the Transformer architecture, key-value cache data is used to store the keys and values ​​of the attention layer to accelerate the inference process. Traditional key-value cache data and other cached data are mostly stored in GPU DRAM, such as HBM. HBM is a new memory technology designed to address memory bandwidth bottlenecks in high-performance computing and is widely used in GPUs, AI accelerators, high-performance servers, and other scenarios. HBM is based on 2.5 / 3D advanced packaging technology, using TSV technology to vertically stack multiple DRAM dies and package them together with the GPU to form a high-capacity, high-bandwidth DDR array. For LLM key-value cache data storage, HBM's high bandwidth and low latency are its natural advantages. However, cached data is usually uniformly distributed or statically mapped to multiple HBM stacks, which cannot fully account for differences in access frequency and latency sensitivity, resulting in suboptimal memory access paths and long average inference times.

[0021] In LLM, the size of key-value cache data is linearly positively correlated with the sequence length. This relationship directly affects the memory usage, computational efficiency, and hardware resource requirements of model inference. Therefore, on the one hand, effective storage can be reduced through compression (such as quantization and sparse storage) or eviction strategies; on the other hand, key-value cache data can be dynamically divided into multiple priority levels and allocated to different memories based on metrics such as access frequency, context position, recentity, and attention score to improve access speed.

[0022] Mainstream packaging structures mostly use a single-sided silicon interposer. The spatial layout between the compute die and the memory stack fails to achieve tight coupling of high-priority paths, which limits the potential for latency optimization. High-priority key-value cache data access still does not meet the low-latency requirements.

[0023] Based on this, this application provides a cache structure based on HBM packaging, which combines a double-sided silicon interposer and an asymmetric HBM packaging structure to achieve a cache data priority-aware distribution method suitable for edge inference computing in LLM. By integrating multiple HBM stacks and compute dies on both sides of the silicon interposer, and allocating high-priority cache data to the HBM stack closest to the compute die, low-latency path access is achieved. This scheme, in conjunction with a priority-aware memory controller, enables hot identification of cached data, cache mapping control, and channel scheduling optimization.

[0024] The following is a detailed description of the implementation details of the HBM-based cache structure in this embodiment. The following content is only for the convenience of understanding the implementation details and is not necessary for implementing this solution.

[0025] This embodiment provides a cache structure based on HBM encapsulation, such as Figure 1 As shown, it includes: a silicon interposer 101; The silicon interposer 101 has a first surface and a second surface opposite to each other. The first surface is provided with a computing die 102 and at least one first HBM stack 103, which is located outside the coverage area of ​​the computing die 102. The second surface is provided with a second HBM stack 104 and at least one third HBM stack 105, a portion of the storage area of ​​the second HBM stack 104 is located within the coverage area of ​​the computing die 102, and the third HBM stack 105 is completely located within the coverage area of ​​the computing die 102. The computing die 102 is equipped with a routing module, a priority awareness module, and an address mapping unit. The priority awareness module is used to determine the priority of cached data; The address mapping module is used to map the cached data to target storage areas in the first HBM stack 103, the second HBM stack 104, and the third HBM stack 105 according to the priority of the cached data; wherein, the priority of the cached data mapped to the first HBM stack 103, the priority of the cached data mapped to the storage area of ​​the second HBM stack 104 located outside the coverage area of ​​the computing die 102, the priority of the cached data mapped to the storage area of ​​the second HBM stack 104 located within the coverage area of ​​the computing die 102, and the priority of the cached data mapped to the third HBM stack 105 increase sequentially; The routing module allocates an interconnect path between the target storage area and the computing die 102.

[0026] The target storage area is the mapped location used to store cached data.

[0027] Figure 1In this structure, the upper surface of the silicon interposer is the first surface, and the lower surface is the second surface. The silicon interposer has a double-sided packaging structure: both the upper and lower surfaces of the silicon interposer are processed with TSV vias and a redistribution layer (RDL); the upper surface is where a compute die is placed and an HBM stack, including a first HBM stack, is placed; the lower surface is where an HBM stack, including a second HBM stack and a third HBM stack, is placed.

[0028] HBM stacks consist of multiple stacked HBM dies.

[0029] The orthographic projection area of ​​the computing die onto the silicon interposer is the coverage area of ​​the computing die. The area of ​​the orthographic projection area of ​​the HBM stack onto the silicon interposer is smaller than the area of ​​the coverage area of ​​the computing die. The first HBM stack is positioned on the same side as the computing die, but outside the coverage area of ​​the computing die. The second HBM stack is positioned on the opposite side of the computing die, with a portion of its storage area within the coverage area and another portion outside. The third HBM stack is positioned on the opposite side of the computing die, completely within the coverage area of ​​the computing die, and is located directly below the center of the first surface of the silicon interposer, perpendicularly aligned with the computing die. This forms an asymmetric HBM package structure. The interconnect path lengths of the first HBM stack, the second HBM stack (with storage areas outside the coverage area of ​​the computing die), the second HBM stack (with storage areas within the coverage area of ​​the computing die), and the third HBM stack decrease sequentially from the computing die, resulting in progressively lower access latency.

[0030] Based on this, the priority of the cached data mapped by the first HBM stack, the priority of the cached data mapped by the storage area of ​​the second HBM stack located outside the coverage area of ​​the computing die, the priority of the storage area of ​​the second HBM stack located within the coverage area of ​​the computing die, and the priority of the cached data mapped by the third HBM stack increase sequentially. Thus, the storage area for high-priority cached data is either the storage area of ​​the third HBM stack or the storage area of ​​the second HBM stack located within the coverage area of ​​the computing die. The interconnection paths between the storage areas of the third HBM stack and the second HBM stack located within the coverage area of ​​the computing die and the computing die are shorter, resulting in lower access latency, thereby reducing the access latency of high-priority cached data.

[0031] A routing module, a priority-aware module, and an address mapping unit are configured within the compute die. For example, these modules are located within the memory controller of the compute die. However, they can also be located in other locations. The priority-aware module determines the priority of cached data and outputs this information to the address mapping unit. The address mapping unit maps the cached data to target memory areas in the first, second, and third HBM stacks based on the priority of the cached data and outputs this information to the routing module. The routing module allocates interconnect paths between the target memory areas and the compute die, allowing the cached data to be transmitted through these allocated interconnect paths.

[0032] For example, the first surface may have multiple first HBM stacks arranged symmetrically around the coverage area of ​​the computing die. For instance, the first surface may have two first HBM stacks arranged symmetrically outside the opposite sides of the coverage area of ​​the computing die. This allows for shorter interconnect paths between the second HBM stacks and the computing die, further reducing access latency.

[0033] For example, the second surface may have multiple second HBM stacks arranged symmetrically around the edge of the coverage area of ​​the computing die. For instance, the second surface may have two second HBM stacks arranged symmetrically on opposite sides of the coverage area of ​​the computing die. This allows for shorter interconnect paths between the second HBM stacks and the computing die, further reducing access latency.

[0034] like Figure 1As shown, the first surface of the silicon interposer 101 houses a computing die 102 and two first HBM stacks 103, denoted as HBM-D and HBM-E, located on the same side as the computing die, exhibiting the highest relative latency. The second surface houses a third HBM stack 105, denoted as HBM-A, located directly below the center of the first surface of the silicon interposer, vertically aligned with the computing die 102, resulting in the shortest interconnect path (i.e., the shortest physical distance) and the lowest access latency. The second surface also houses two second HBM stacks 104, denoted as HBM-B and HBM-C, located below the side of the computing die 102, exhibiting medium latency. The cached data priorities include high, medium, and low priority. For example, the address mapping unit employs a hierarchical mapping mechanism that supports static initialization. Specifically, it prioritizes the storage areas of HBM-A, HBM-B, or HBM-C that are covered by the computing die's coverage area as high-priority cache data storage areas; prioritizes the storage areas of HBM-B or HBM-C outside the computing die's coverage area as medium-priority cache data storage areas; and prioritizes the storage areas of HBM-D or HBM-E outside the computing die's coverage area as low-priority cache data storage areas. In this way, cache data is mapped to storage areas of different HBM stacks according to priority, reserving core area storage areas for high-priority cache data. Because these core areas are close to the computing die, access latency is reduced.

[0035] If the first and second HBM stacks are placed close to the compute die layout, routing congestion may occur. RDL, power supply, and thermal issues can lead to increased latency, especially with high-frequency access or large data throughput, and degraded signal transmission quality. Based on this, see [reference needed]. Figure 2 The diagram illustrates the relative positions of the calculated die and the orthographic projection areas of each HBM stack on the silicon interposer. The orthographic projection areas of the third HBM stack, the second HBM stack, and the first HBM stack on the silicon interposer do not overlap. This ensures stable and independent TSV and RDL paths, good management of wiring resources and heat dissipation, resulting in clear and stable signal transmission with controllable latency.

[0036] Vertical TSVs are arranged inside the silicon interposer. One or both of the upper and lower surfaces of the silicon interposer have RDLs. The I / O pins on the die on the two surfaces of the silicon interposer are rearranged and aligned to the TSV positions on the silicon interposer. This is mainly used for signal fan-out, connecting TSVs and micro-bumps, and realizing the interconnection of electrical signals between the upper and lower surfaces.

[0037] Each HBM die's microbump contacts extend to the center of the silicon interposer via RDL wiring, and are then connected to a compute die or other HBM die on the other surface via TSV.

[0038] In the HBM-based cache structure of this embodiment, the silicon interposer has a first surface and a second surface. The first surface is provided with a compute die and at least one first HBM stack, which is located outside the coverage area of ​​the compute die. The second surface is provided with a second HBM stack and at least one third HBM stack. A portion of the storage area of ​​the second HBM stack is located within the coverage area of ​​the compute die, and the third HBM stack is completely located within the coverage area of ​​the compute die. The compute die contains a routing module, a priority awareness module, and an address mapping unit. The priority awareness module determines the priority of cached data, and the address mapping module determines the priority of the cached data based on its priority. First, cached data is mapped to target storage areas in the first HBM stack, the second HBM stack, and the third HBM stack. The routing module allocates interconnection paths between the target storage areas and the compute die. Since the priority of the cached data mapped to the first HBM stack is higher than the priority of the cached data mapped to the storage area of ​​the second HBM stack located outside the coverage area of ​​the compute die, and the priority of the cached data mapped to the storage area of ​​the second HBM stack located within the coverage area of ​​the compute die and the third HBM stack, the high-priority cached data can be mapped to the storage area of ​​the HBM stack within the coverage area of ​​the compute die, thus reducing access latency.

[0039] In some embodiments, the computing die is perpendicularly mated to the third HBM stack via microbumps.

[0040] In the interconnect structure between the compute die and the silicon interposer, an asymmetric connection layout is adopted. The number of microbumps on the compute die is greater than the number of microbumps in a single HBM stack. Some regions of microbumps are directly connected to the HBMs storing high-priority cached data located directly below them through the vertical shortest path, such as HBM-A, which is called the Primary Access Stack. The remaining microbumps are laterally connected (fan-out) through the RDL of the silicon interposer to the bias-deployed HBM stacks, such as HBM-B, HBM-C, HBM-D, and HBM-E, thereby achieving interconnect path optimization based on the priority of the stored cached data.

[0041] In this embodiment, the central region of the die located on the first surface of the silicon interposer is vertically connected to the HBM stack in the central region of the other surface of the silicon interposer through microbump bonding, forming an ultra-short interconnect path for high-priority cached data, which further reduces access latency.

[0042] In some embodiments, the portion of the second HBM stack located in the coverage area of ​​the computing die includes an edge memory cell directly connected to the computing die via microbumps.

[0043] Given that the area of ​​the die's coverage region is greater than the area of ​​at least one HBM stack projected onto the silicon interposer, the package's coverage region extends to some edge banks of other HBM stacks. Through local microbumps and short-path direct connections, these edge banks have similar physical access advantages to the central HBM-A. This allows for the dynamic partitioning of some HBM-B / C sub-regions (i.e., some storage areas) into storage areas for high-priority cached data based on path latency.

[0044] In some embodiments, the computing die has an input / output (I / O) interface corresponding one-to-one with each of the first HBM stacks, each of the second HBM stacks and the third HBM stack, and each of the first HBM stacks, each of the second HBM stacks and the third HBM stack is interconnected with the corresponding I / O contact.

[0045] The compute die is physically connected to multiple HBM stacks. Each HBM stack has its own dedicated I / O contact on the compute die. Multiple point-to-point or bus-like interfaces are formed on the compute die through RDL and TSV on the silicon interposer. To avoid signal conflicts and latency bottlenecks, the data path between each compute die and the HBM stack is point-to-point with a fixed bandwidth.

[0046] By interconnecting each of the first HBM stack, each of the second HBM stack, and the third HBM stack with the corresponding I / O contacts, an independent access path between the cache and the computing die can be achieved, ensuring low latency for cached data.

[0047] In some embodiments, the routing module is specifically used to connect a portion of the data channels of the computing die to the first HBM stack, the second HBM stack, and the third HBM stack in a round-robin fashion according to a preset scheduling strategy.

[0048] The compute die is logically shared with multiple HBM stacks. A portion of the compute die's data channels are rotated among the channels of multiple HBM stacks according to a scheduling policy, such as through routing switching logic or MUX / DEMUX. For example, the routing module within the memory controller dynamically allocates channels connecting to multiple HBM stacks, enabling logical interconnect path rescheduling.

[0049] When the data channel between the compute die and the HBM stack is busy, a portion of the compute die's data channel can be routed to the busy HBM stack using a preset scheduling strategy to avoid access latency.

[0050] In some embodiments, the address mapping unit is further configured to, after each inference cycle of the LLM, redetermine the priority of the cached data based on the actual access hit statistics of the cached data, and update the storage area if the redetermined priority of the cached data does not match the current storage area.

[0051] In LLM, each generated token constitutes an independent inference cycle. The actual access hit statistics for cached data are the result of statistics on actual access hits to cached data.

[0052] After each inference cycle, the priority of cached data is re-determined, thereby implementing a periodic reordering mechanism for storage areas. Based on access behavior feedback during inference, data can be automatically migrated between low-, medium-, and high-priority cache storage areas. If the actual access hit statistics of cached data increase, its popularity rises, and its priority increases, then it is migrated to a higher-priority storage area. Conversely, if the actual access hit statistics of cached data decrease, its popularity decreases, and its priority decreases, then it is migrated to a lower-priority storage area. In this way, dynamic hot migration is achieved by re-evaluating the popularity of cached data.

[0053] In some embodiments, the priority-aware module is specifically used to determine the priority of the cached data based on the access frequency, context position, recentity, attention score, and model stage of the cached data.

[0054] For example, cached data includes key-value cached data. The priority awareness module has a built-in heat detection module that dynamically determines the priority level based on indicators such as the access frequency, context position, recentity, attention score, model stage, and historical token statistics of cached data. The method of priority determination can be set according to the actual situation. For example, high-priority cached data is given priority because it is about to be used or has a high hit rate. Cached data with a core context position has high priority, and so on.

[0055] In this embodiment, the priority of cached data can be determined by considering multiple factors to ensure that the cached data is mapped to a suitable storage area and meets the corresponding latency requirements.

[0056] The HBM-based cache structure provided in this solution offers the following advantages: it utilizes the space above and below the silicon interposer to achieve latency-layered data scheduling; it reduces high-priority cache data latency, improving model inference throughput; it reduces long-distance data transmission, lowering power consumption; and it provides a new physical-logical joint optimization path for AI chip architectures. Traditional vertical stacking, while physically close and with short TSV penetration, suffers from densely packed compute and storage units, potentially leading to wiring congestion. RDL, power supply, and thermal issues can increase latency, especially with high-frequency access or large data throughput, resulting in degraded signal transmission quality. In contrast, while this solution involves a longer physical distance, the TSV and RDL paths are more stable and independent, with better management of wiring resources and heat dissipation, resulting in clear, stable signal transmission and controllable latency.

[0057] This application also relates to an electronic device, including a cache structure based on HBM packaging as described in any of the above embodiments.

[0058] In this embodiment, the silicon interposer has a first surface and a second surface opposite to each other. The first surface is provided with a computing die and at least one first HBM stack, which is located outside the coverage area of ​​the computing die. The second surface is provided with a second HBM stack and at least one third HBM stack. A portion of the storage area of ​​the second HBM stack is located within the coverage area of ​​the computing die, and the third HBM stack is completely located within the coverage area of ​​the computing die. The computing die is provided with a routing module, a priority awareness module, and an address mapping unit. The priority awareness module determines the priority of cached data, and the address mapping module allocates cached data according to the priority of the cached data. Data is mapped to target storage areas in the first HBM stack, the second HBM stack, and the third HBM stack. The routing module allocates interconnection paths between the target storage areas and the compute die. Due to the increasing priority of cached data stored in the first HBM stack, the increasing priority of cached data stored in storage areas of the second HBM stack located outside the coverage area of ​​the compute die, the increasing priority of cached data stored in storage areas of the second HBM stack located within the coverage area of ​​the compute die, and the increasing priority of cached data stored in the third HBM stack, high-priority cached data can be mapped to storage areas of HBM stacks within the coverage area of ​​the compute die, reducing access latency.

[0059] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing this application, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of this application.

Claims

1. A cache structure based on HBM encapsulation, characterized in that, include: Silicon interposer; The silicon interposer has a first surface and a second surface opposite to each other. The first surface is provided with a computing die and at least one first HBM stack, which is located outside the coverage area of ​​the computing die. The second surface is provided with a second HBM stack and at least one third HBM stack, a portion of the storage area of ​​the second HBM stack is located within the coverage area of ​​the computing die, and the third HBM stack is completely located within the coverage area of ​​the computing die. The computing die includes a routing module, a priority awareness module, and an address mapping unit. The priority awareness module is used to determine the priority of cached data; The address mapping module is used to map the cached data to target storage areas in the first HBM stack, the second HBM stack, and the third HBM stack according to the priority of the cached data; wherein, the priority of the cached data mapped to the first HBM stack, the priority of the cached data mapped to the storage area of ​​the second HBM stack located outside the coverage area of ​​the computing die, the priority of the cached data mapped to the storage area of ​​the second HBM stack located within the coverage area of ​​the computing die, and the priority of the cached data mapped to the third HBM stack increase sequentially; The routing module is used to allocate interconnection paths between the target storage area and the computing die.

2. The cache structure based on HBM encapsulation according to claim 1, characterized in that, The first surface is provided with a plurality of first HBM stacks, which are symmetrically arranged around the coverage area of ​​the computing die; the second surface is provided with a plurality of second HBM stacks, which are symmetrically arranged around the edge of the coverage area of ​​the computing die.

3. The cache structure based on HBM encapsulation according to claim 1, characterized in that, The orthographic projection areas of the third HBM stacked on the silicon interposer, the orthographic projection areas of the second HBM stacked on the silicon interposer, and the orthographic projection areas of the first HBM stacked on the silicon interposer do not overlap.

4. The cache structure based on HBM encapsulation according to claim 1, characterized in that, The computing die is perpendicularly docked to the third HBM stack via microbumps.

5. The cache structure based on HBM encapsulation according to claim 1 or 4, characterized in that, The storage area of ​​the second HBM stack located in the coverage area of ​​the computing die includes edge storage, which is directly connected to the computing die via microbumps.

6. The cache structure based on HBM encapsulation according to claim 1, characterized in that, The computing die has I / O contacts that correspond one-to-one with each of the first HBM stacks, each of the second HBM stacks and the third HBM stack, and each of the first HBM stacks, each of the second HBM stacks and the third HBM stack is interconnected with the corresponding I / O contacts.

7. The cache structure based on HBM encapsulation according to claim 6, characterized in that, The routing module is specifically used to connect a portion of the data channels of the computing die to the first HBM stack, the second HBM stack, and the third HBM stack in a round-robin fashion according to a preset scheduling strategy.

8. The cache structure based on HBM encapsulation according to claim 1, characterized in that, The address mapping unit is also used to redetermine the priority of the cached data based on the actual access hit statistics of the cached data after each inference cycle of the LLM, and update the storage area if the redetermined priority of the cached data does not match the current storage area.

9. The cache structure based on HBM encapsulation according to claim 1, characterized in that, The priority awareness module is specifically used to determine the priority of the cached data based on the access frequency, context position, recentity, attention score, and model stage of the cached data.

10. An electronic device, characterized in that, Includes the HBM-based cache structure as described in any one of claims 1 to 9.

Images