Chip three-dimensional stacking structure, electronic equipment and thermal optimization method

Abstract

The invention provides a chip three-dimensional stacking structure, electronic equipment and a thermal optimization method, and belongs to the technical field of semiconductors. The memory module of the chip three-dimensional stacking structure is stacked on the upper surface of the logic chip, and the memory module comprises at least one memory chip which is sequentially stacked in the preset direction; the logic chip comprises a plurality of core groups, each core group comprises at least one performance core and an energy efficiency core, each performance core is used for processing calculation-intensive calculation tasks in the corresponding core group, and the energy efficiency core is used for processing memory-intensive calculation tasks in the corresponding core group; the energy efficiency core of each core group is connected with each performance core in the group, and the energy efficiency core of each core group serves as an agent core of the core group to which the energy efficiency core belongs to communicate with other core groups. According to the logic chip, the heterogeneous design of the performance core and the energy efficiency core is adopted, the power consumption of the logic chip is reduced, heat production of the logic chip is reduced, heat flow distribution is improved, and temperature distribution is more uniform.

Description

Technical Field

[0001] This application relates to the field of semiconductor technology, and in particular to a three-dimensional stacked chip structure, electronic device, and thermal optimization method. Background Technology

[0002] 3D (Three-Dimensional) stacking refers to the technology of vertically stacking multiple chips together, creating vertical conductive channels between the chips through through-silicon vias (TSVs) to achieve three-dimensional interconnection. Using 3D stacking technology, multiple DRAM (Dynamic Random Access Memory) chips can be vertically stacked on the top surface of a logic chip to obtain a three-dimensional chip stack structure. Hardware devices constructed with three-dimensional chip stack structures have higher integration and performance, significantly improving signal transmission efficiency, bandwidth, and power consumption control. They are widely used in various scenarios, such as the inference process of large language models.

[0003] Existing three-dimensional stacked logic chips consist of multiple performance cores (P-cores). These performance cores are characterized by high performance and high power consumption. While they can improve the computing power of the three-dimensional stacked structure, they also generate significant heat during operation. Due to the inherent structural characteristics of the three-dimensional stacked structure, the memory chips on the top surface of the stacked logic chip affect heat dissipation. Furthermore, the heat dissipation problem becomes more severe as the number of memory chips stacked vertically increases, resulting in uneven temperature distribution across the entire logic chip. This significantly impacts the reliability, efficiency, and lifespan of the three-dimensional stacked structure. Therefore, it is necessary to propose a novel three-dimensional stacked structure to address the temperature issues of existing three-dimensional stacked structures. Summary of the Invention

[0004] This application provides a three-dimensional stacked chip structure, an electronic device, and a thermal optimization method, which can solve the temperature problems of existing three-dimensional stacked chip structures. The technical solution is as follows: In a first aspect, a three-dimensional stacked chip structure is provided, the three-dimensional stacked chip structure comprising: a logic chip and a memory module; The memory modules are stacked on the upper surface of the logic chip along a preset direction, which is perpendicular to the upper surface of the logic chip. The memory module includes at least one memory chip stacked sequentially along the preset direction. Each memory chip includes multiple storage areas, and each storage area includes multiple storage blocks. Storage areas located at the same position on at least one memory chip constitute a storage area group. The logic chip includes multiple core groups, each core group including at least one performance core and one energy efficiency core. Each performance core is used to handle computationally intensive computing tasks within its core group, and the energy efficiency core is used to handle memory-intensive computing tasks within its core group. The energy efficiency core of each core group is connected to each performance core within the group, and the energy efficiency core of each core group acts as a proxy core for its core group to communicate with other core groups. Each core group corresponds to at least one storage area group, and the at least one storage area group corresponding to each core group serves as the local memory of the core group and can be accessed by any core within the group.

[0005] In a first possible implementation of the first aspect, for any core within any core group, the core includes a control unit, a memory controller, a local cache, a communication unit, and an array-based computing engine; The control unit is used to manage the computing tasks within the core according to the received instructions; The memory controller is used to manage the core's local memory; The local cache is used to cache the local data of the core; The communication unit is used to communicate with the core and other cores; The vector unit is used to perform calculations on matrix data smaller than a preset size read by the core; The array-based computing engine is used to perform calculations on matrix data larger than the preset size read by the core.

[0006] In a second possible implementation of the first aspect, when the core is a performance core, the array-based computing engine is a systolic array, the systolic array includes multiple processing units, the systolic array is used to read matrix data from the local cache of the performance core, use multiple processing units to perform parallel calculations on the read matrix data to obtain a first calculation result, and write the first calculation result into the local cache of the performance core or use the vector unit to post-process the first calculation result.

[0007] In a third possible implementation of the first aspect, the communication unit is used to provide a path for the performance core to communicate with the energy efficiency core within its core group.

[0008] In the fourth possible implementation of the first aspect, when the core is an energy efficiency core, the array-based computing engine is a multiply-accumulate tree array, the multiply-accumulate tree array includes multiple multiply-accumulate trees, each of the multiply-accumulate trees has a different weight value, each of the multiply-accumulate trees is used to read matrix data from any of the local cache, local memory or communication unit of the energy efficiency core, and to perform calculations on the read matrix data to obtain a second calculation result, and to write the second calculation result into the local cache of the energy efficiency core or to use the vector unit to post-process the second calculation result.

[0009] In a fifth possible implementation of the first aspect, the communication unit includes a router for communicating with energy efficiency cores of other core groups based on the router.

[0010] In a sixth possible implementation of the first aspect, the on-chip network bandwidth of any core within each core group of the three-dimensional stacked chip structure is matched with the memory bandwidth of the local memory of the core group to which it belongs.

[0011] In the seventh possible implementation of the first aspect, the ratio between the number of performance cores and the number of energy-efficient cores included in each core group is a preset ratio, which is determined based on the computing performance and temperature of the chip's three-dimensional stacked structure.

[0012] In a second aspect, an electronic device is provided, the electronic device including a processor and the chip three-dimensional stacked structure described in the first aspect, the processor and the chip three-dimensional stacked structure being connected via a bus.

[0013] Thirdly, a thermal optimization method is provided, which is applied to the electronic device described in the second aspect, wherein the three-dimensional stacked structure of the chips in the electronic device constitutes an inference platform for a large language model, and the method includes: During the large language model inference process based on the chip's three-dimensional stacked structure, for any core group in the chip's three-dimensional stacked structure, the shared memory bandwidth in the first memory bandwidth is determined according to the number of inference requests received. The first memory bandwidth is the memory bandwidth allocated to the performance core in the core group, and the shared memory bandwidth is the memory bandwidth shared by the performance core in the core group to the energy efficiency core. Based on the shared memory bandwidth and the second memory bandwidth, the first data is divided into the second data corresponding to the shared memory bandwidth and the third data corresponding to the second memory bandwidth, so as to minimize the difference between the ratio of the second data to the shared memory bandwidth and the ratio of the third data to the second memory bandwidth. The second memory bandwidth is the memory bandwidth allocated to the energy efficiency core in the core group, and the first data is the data that the energy efficiency core in the core group needs to read from local memory in response to the inference request. The energy efficiency core in the control core group uses the shared memory bandwidth to read the second data from local memory, and uses the second memory bandwidth to read the third data from local memory.

[0014] The beneficial effects of the technical solutions provided in this application are: The chip three-dimensional stacked structure in this application embodiment no longer simply adopts a homogeneous design of performance cores, but rather a heterogeneous design of performance cores and energy-efficiency cores. Performance cores are characterized by high power consumption and high performance, while energy-efficiency cores are characterized by low power consumption and low performance. Based on the characteristics of the performance cores and energy-efficiency cores, computational tasks on the chip three-dimensional stacked structure are allocated: computationally intensive tasks are assigned to performance cores, and memory-intensive tasks are assigned to energy-efficiency cores. Since some computational tasks on the chip three-dimensional stacked structure are processed by energy-efficiency cores, and energy-efficiency cores have lower power consumption, the overall power consumption of the chip three-dimensional stacked structure is reduced, heat generation on the logic chips is reduced, thereby improving heat flow distribution and making the temperature distribution more uniform. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of a three-dimensional chip stacking structure provided in an embodiment of this application; Figure 2 This is a schematic diagram illustrating the entire reduction operation process provided in an embodiment of this application; Figure 3 This is a schematic diagram of a microarchitecture for a performance core and an energy efficiency core provided in an embodiment of this application; Figure 4 This is a comparison chart of the thermal performance of a heterogeneous system under a large language model inference load, provided in an embodiment of this application. Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application; Figure 6 This is a flowchart of a thermal optimization method provided in an embodiment of this application; Figure 7 This is a schematic diagram of a KV cache partition provided in an embodiment of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0018] It is understood that the terms "each," "multiple," and "any" used in the embodiments of this application, etc., mean that "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the corresponding multiples. For example, multiple words include 10 words, and "each word" refers to each of the 10 words, while "any word" refers to any one of the 10 words.

[0019] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0020] With the development of artificial intelligence technology, large language models are being applied to various fields. A typical example is in natural language processing applications, such as chatbots, text generation, and summarization. The reasoning process of a large language model includes two operational stages: the prefill stage and the decoding stage.

[0021] The core task of the pre-filling stage is to perform global modeling of the input sequence. After the input text is converted into a token sequence by the token segmenter, the model maps it into a vector representation through the embedding layer. At this point, the model performs complete self-attention computation, with all input tokens interacting through attention to generate hidden states containing global information. During this process, the model generates key-value pair caches for each attention head, and these cached data will serve as the basis for computation in the subsequent decoding stage. In the encoder-decoder architecture, the pre-filling stage also includes cross-attention computation of the encoder output, and the decoder uses the encoder's key-value cache to establish cross-language associations.

[0022] The decoding phase employs autoregressive methods to generate output tokens one by one. Each generation step only needs to calculate the attention relationship between the current token and the preceding sequence, avoiding redundant calculations by reusing pre-generated key-value caches. When generating the Nth token, the model only needs to calculate the attention distribution between this token and the cached N-1 key-value vectors. This enhanced computation mode significantly reduces the computational complexity of a single step, but frequent memory accesses make this stage extremely sensitive to memory bandwidth, especially during batch processing, where differences in the generation progress of different sequences lead to fragmentation of GPU memory access.

[0023] In short, the main task of the pre-filling stage is to process the input prompts, convert them into the model's internal representation, and generate an initial key-value cache to prepare for the decoding stage. The pre-filling stage involves a large amount of matrix and activation function calculations, making it a computationally intensive task. The main task of the decoding stage is to autoregressively generate the next token based on the existing key-value cache, gradually outputting the complete text. This stage has a relatively smaller computational load, but requires frequent access to the key-value cache in GPU memory, making it a memory-intensive task.

[0024] Because the decoding stage of large language models is memory-intensive, and existing hardware architectures can only provide limited memory bandwidth, latency occurs during the inference process of large language models. To address the memory limitations of hardware architectures, engineers have explored architectures that can provide higher memory bandwidth, such as three-dimensional chip stacking structures. Three-dimensional chip stacking structures offer higher integration and performance, significantly improving signal transmission efficiency, bandwidth, and power consumption control. They are being applied in high-end GPUs (Graphics Processing Units), autonomous driving, supercomputing, and ultra-thin terminal devices.

[0025] While existing three-dimensional chip stacking structures offer numerous advantages, their logic chips employ a homogeneous design with performance cores, offloading all computational tasks to these cores. This design, while achieving faster computing speeds, also results in higher power consumption and significant heat generation from the logic chips. Furthermore, the inherent structural characteristics of the three-dimensional stacking structure itself limit heat dissipation, as the memory chips stacked on top of the logic chips hinder heat dissipation. The uneven temperature distribution across the logic chip surface means that the central logic chips receive heat not only from the performance chips in the center but also from surrounding performance chips, leading to higher heat flux density and uneven temperature distribution in the central area. This problem worsens with the increasing number of vertically stacked memory chips, severely impacting the reliability, efficiency, and lifespan of the three-dimensional chip stacking structure.

[0026] When running large language models on a three-dimensional stacked chip structure, the high parallelism of the inference workload exacerbates heat dissipation issues, necessitating cross-stack optimization. However, current solutions for heat dissipation in three-dimensional stacked chip structures largely focus on independent areas such as materials, cooling, or load scheduling, with few (or no) solutions addressing the thermal problem from a circuit architecture perspective. In fact, considering thermal factors during the design phase is crucial for three-dimensional stacked chip structures.

[0027] In view of this, the present application provides a three-dimensional stacked chip structure. The logic chip of the three-dimensional stacked chip structure adopts a heterogeneous design of performance core and energy efficiency core. By distributing different types of computing tasks to different types of chips, the overall power consumption of the three-dimensional stacked chip structure is reduced, the heat generation of the three-dimensional stacked chip structure is reduced, thereby improving the heat flux density of the three-dimensional stacked chip structure, and thus making the temperature distribution of the three-dimensional stacked chip structure more uniform.

[0028] This application provides a three-dimensional stacked chip structure, see [link]. Figure 1 The chip's three-dimensional stacked structure includes: a logic chip and a memory module.

[0029] The memory module provides memory space to store the data required for calculations by the logic chip. The memory module is stacked on the upper surface of the logic chip along a preset direction, which is perpendicular to the upper surface of the logic chip. The memory module and logic chip can be stacked face-to-face or back-to-back. Each memory module includes at least one memory chip stacked sequentially along the preset direction. Theoretically, the number of memory chips stacked in the memory module can be 4, 6, 8, etc. While increasing the number of stacked memory chips provides more storage space, it also increases the complexity of the manufacturing process. Considering manufacturing capabilities, in this embodiment, the memory module stacks 4 memory chips. Each memory chip can be a DRAM chip, an SRAM (Static Random-Access Memory) chip, etc.

[0030] Logic chips are used to perform computational tasks. A logic chip consists of multiple cores, such as... Figure 1The number of cores shown can be 48. The number of cores in a logic chip can be expanded, for example, to 60, 72, etc. As the number of cores increases, the computing power of the logic chip increases, but this is accompanied by a larger area and more severe thermal problems. Therefore, the number of cores in a logic chip needs to be designed based on computing power and heat dissipation capabilities. The multiple cores in the logic chip are divided into multiple core groups, each core group including at least one performance core (P core) and one energy-efficient core (E core). Performance cores and energy-efficient cores are two different types of cores in a processor. Performance cores are characterized by high performance and high power consumption, while energy-efficient cores are characterized by low performance and low power consumption. The design of these two types of cores can better balance the processor's performance and power consumption to meet the dual requirements of high performance and low power consumption for modern computing tasks. Based on the characteristics of the two types of cores, this embodiment assigns different types of computing tasks to different types of cores for processing. Specifically, computationally intensive tasks (such as matrix multiplication) are assigned to performance cores, while memory-intensive tasks (such as matrix-vector multiplication) are assigned to energy-efficient cores. This allows each performance core to execute computationally intensive tasks, and each energy-efficient core to execute memory-intensive tasks. When the chip's three-dimensional stacked structure serves as an inference platform for a large language model, the performance cores can handle computational tasks in the fully connected layers, and the energy-efficient cores can handle computational tasks in the attention layers.

[0031] Because the power consumption of the energy efficiency core is relatively low, it generates less heat during operation compared to the performance core. The heterogeneous design of the performance core and the energy efficiency core significantly reduces the temperature rise on the surface of the logic chip, resulting in a more uniform temperature distribution on the surface of the logic chip.

[0032] Furthermore, as the number of energy-efficient cores increases, the heat dissipation capacity of the chip's three-dimensional stacked structure improves, but the computing performance decreases. Conversely, as the number of performance cores increases, the computing performance of the chip's three-dimensional stacked structure improves, but the heat dissipation capacity decreases, and the temperature of the logic chip surface rises. This increased surface temperature, in turn, limits the improvement in the logic chip's computing performance. Therefore, it is necessary to determine a preset ratio between the number of performance cores and the number of energy-efficient cores within each core group to balance the computing performance and thermal limitations of the logic chip. In this embodiment, the preset ratio can be determined based on the computing performance and temperature of the chip's three-dimensional stacked structure. Specifically, the preset ratio between the number of performance cores and the number of energy-efficient cores within each core group can be determined experimentally. For example, we can assume that the preset ratio between the number of performance cores and the number of energy-efficient cores in each core group is 1:1. Logic chips are designed according to this preset ratio, and multiple memory chips are stacked on the top surface of the designed logic chips to obtain a three-dimensional stacked chip structure. Then, we test the computational performance of this three-dimensional stacked chip structure in actual business scenarios (such as large language model inference scenarios). Next, we assume that the preset ratio between the number of performance cores and the number of energy-efficient cores in each core group is 2:1. Logic chips are designed according to this ratio, and multiple memory chips are stacked on the top surface of the designed logic chips to obtain a chip stacking result. Then, we test the computational performance of this chip stacking result in actual business scenarios. And so on, through multiple tests, we can obtain the optimal value of the preset ratio.

[0033] In this embodiment, each memory chip and logic chip in the logic chip and memory module are connected via hybrid bonding technology. Hybrid bonding, also known as direct bonding, is a novel technology that has gradually emerged in the era of 3D packaging. The core principle of hybrid bonding is based on intermolecular forces (van der Waals forces), achieving high-density vertical interconnects between chips through the synergistic effect of copper-copper direct bonding and dielectric bonding. This technology eliminates the need for traditional copper pillars or solder balls and other bump structures, enabling ultra-fine interconnect spacing of less than 1 micrometer (compared to over 20 micrometers for traditional bump bonding). The interconnect density is extremely high, increasing the number of I / O (Input / Output) ports per unit area by more than a thousand times, significantly improving the data transmission bandwidth between chips. Hybrid bonding enables thinner wafer stacking, making the overall architecture more compact, which not only improves thermal management capabilities but also optimizes electrical performance. Hybrid bonding supports the vertical stacking of different functional units such as logic chips, memory chips, and sensors, facilitating 3D integration and enhancing the flexibility of heterogeneous designs. Based on hybrid bonding technology, high-density hybrid bonding vias and fine-pitch micro-silicon vias can also be used to achieve higher I / O parallelism and memory bandwidth.

[0034] In this embodiment, each memory chip in at least one memory chip includes multiple storage regions, such as 12, 16, or 20. Each storage region includes multiple storage blocks, such as 4. The number of storage regions included in each memory chip in at least one memory chip is the same. Storage regions located at the same position on at least one memory chip can form a storage region group. The number of core groups included in the logic chip can be the same as the number of storage regions included in the memory chips, such as... Figure 1 Each memory chip shown includes 12 storage regions, and the logic chip also includes 12 core groups. Each core group in the logic chip corresponds to at least one storage region group. This at least one storage region group, as the local memory of the core group, can be accessed by any core within the group. That is, any core within each core group can access all the storage blocks included in the storage region group corresponding to that core group. For example… Figure 1 The memory region group consists of memory regions located in the same location on four memory chips. Each memory region includes four memory blocks. Any core in each core group can access its corresponding 16 memory blocks.

[0035] In this embodiment, the logic chip employs a two-level network structure to connect heterogeneous cores. Within each core group, a local path exists between the performance cores and energy efficiency cores. The energy efficiency cores of each core group connect to each performance core within the group via their corresponding local paths, forming a tree-like network-on-chip (NoC), where performance cores act as leaf nodes and energy efficiency cores as root nodes. Between core groups, all core groups are interconnected via a two-dimensional mesh of NoCs. The energy efficiency core of each core group acts as a proxy core for its respective group, communicating with other core groups. By adopting this two-level interconnected network structure, not only is the network latency of the chip's three-dimensional stacked structure reduced, but communication overhead is also decreased.

[0036] It's important to clarify here that for any given core group, the energy efficiency core acting as the proxy core for that core group does not communicate with the proxy cores of other core groups. Instead, it communicates only with the energy efficiency cores within the core group that is adjacent to that core group on the surface of its logic chip. For example... Figure 1 The logic chip shown includes 12 core groups. The energy efficiency core in the first row and first column core group communicates with the energy efficiency core in the first row and second column core group and the energy efficiency core in the second row and second column core group. The energy efficiency core in the first row and second column core group communicates with the energy efficiency core in the first row and first column core group, the energy efficiency core in the first row and third column core group, and the energy efficiency core in the second row and second column core group.

[0037] In large language model reasoning scenarios, the communication patterns between the various cores are regular and consistent, mainly dominated by all reduction operations. Figure 2 A schematic diagram of the entire reduction operation process is shown below. Figure 2 Nodes a, b, c, and d each have four child nodes. Initially, the data in the four child nodes of node a are a0, a1, a2, and a3; the data in the four child nodes of node b are b0, b1, b2, and b3; the data in the four child nodes of node c are c0, c1, c2, and c3; and the data in the four child nodes of node d are d0, d1, d2, and d3. After the reduce-scatter step 1, the data in the fourth child node of node a becomes a3 + d3; the data in the first child node of node b becomes a0 + b0; the data in the second child node of node c becomes b1 + c1; and the data in the third child node of node d becomes c2 + d2. After reduce-scatter step 2, the data in the third child node of node a becomes a2 + c2 + d2, the data in the fourth child node of node b becomes a3 + b3 + d3, the data in the first child node of node c becomes a0 + b0 + c0, and the data in the second child node of node d becomes b1 + c1 + d1. After reduce-scatter step 3, the data in the second child node of node a becomes a1 + b1 + c1 + d1, the data in the third child node of node b becomes a2 + b2 + c2 + d2, the data in the fourth child node of node c becomes a3 + b3 + c3 + d3, and the data in the first child node of node d becomes a0 + b0 + c0 + d0.

[0038] After AllGather step 1, the data of the first child node of node a becomes a0 + b0 + c0 + d0, the data of the second child node of node b becomes a1 + b1 + c1 + d1, the data of the third child node of node c becomes a2 + b2 + c2 + d2, and the data of the fourth child node of node d becomes a3 + b3 + c3 + d3. After AllGather step 2, the data of the fourth child node of node a becomes a3 + b3 + c3 + d3, the data of the first child node of node b becomes a0 + b0 + c0 + d0, the data of the second child node of node c becomes a1 + b1 + c1 + d1, and the data of the third child node of node d becomes a2 + b2 + c2 + d2. After AllGather step 3, the data in the third child node of node a becomes a2 + b2 + c2 + d2, the data in the fourth child node of node b becomes a3 + b3 + c3 + d3, the data in the first child node of node c becomes a0 + b0 + c0 + d0, and the data in the second child node of node d becomes a1 + b1 + c1 + d1.

[0039] Figure 1 The chip three-dimensional stacking structure shown in this application embodiment is illustrated in [reference]. Figure 1 The chip features a three-dimensional stacked logic chip structure with four memory chips stacked on its top surface. All four memory chips are DRAM chips. Each DRAM chip includes 12 memory regions, each region contains four DRAM memory blocks, and each memory block has a memory size of 128 bits. The four stacked DRAM chips can provide a memory capacity of 12GB. Each memory region also includes a row decoder and a column decoder. The four DRAM chips have 768 DRAM memory channels, each DRAM memory channel is an independent memory channel with 128 I / O ports. The DRAM chips are connected to each other and to the logic chip via hybrid bonding pillars and through-silicon vias (TSVs). The three-dimensional stacked logic chip structure includes 48 cores, divided into 12 core groups. Each core group contains four cores: one energy efficiency core and three performance cores. Within each core group, the three performance cores are connected to the energy efficiency core, forming a fat tree-like on-chip network. Between core groups, all core groups are interconnected via a two-dimensional mesh NoC (NoC), with the energy efficiency core of each core group acting as a proxy core for communication with other core groups.

[0040] The logic chip with the three-dimensional stacked structure provided in this application adopts a heterogeneous design of performance cores and energy-efficiency cores. Computationally intensive tasks are offloaded to the performance core, while memory-intensive tasks are offloaded to the energy-efficiency core. Because the energy-efficiency core has lower power consumption, it generates less heat during operation compared to the performance core, resulting in a lower overall temperature rise of the logic chip. In particular, the middle part of the logic chip generates less heat itself, and the heat generated around it is also reduced. Consequently, less heat diffuses from the surrounding area to the middle part, significantly reducing the heat in the middle section. This results in a more uniform temperature distribution within the logic chip, improving the efficiency and reliability of the three-dimensional stacked structure and extending its lifespan.

[0041] In this embodiment, each core within a core group includes a control unit, a memory controller, a local cache, a communication unit, and an array-based computing engine. The control unit is connected to the communication unit and the memory controller. The array-based computing engine is connected to the vector unit. The memory controller, communication unit, vector unit, and array-based computing engine can access the local cache to write or read data. Specifically, the control unit manages the computing tasks within the core according to received instructions. The memory controller manages the local memory of its core group, including reading data from and writing the read data back to the local memory. The local cache caches local data, which may include data to be computed written by the memory controller and the communication unit, as well as data processed by the array-based computing engine and the vector unit. The communication unit enables communication between cores; specifically, it can send data from one core to another and receive data from other cores. The vector unit performs calculations on matrix data read by the core that is smaller than a preset size, which can be set by an expert. The array-based computing engine performs calculations on matrix data read by the core that is larger than the preset size. For each core group, the functions of the control unit, memory controller, and local cache included in the performance core and energy efficiency core within the core group are the same. The difference lies in the structure and specific functions of the communication unit and the array-based computing engine, which will be introduced separately below.

[0042] In the first scenario, the aforementioned core is a performance core. When the aforementioned core is a performance core, the array-based computing engine included in this core is a highly parallel systolic array. This systolic array can handle computationally intensive tasks, such as matrix multiplication operations in fully connected layers. The systolic array includes multiple processing units used to read matrix data from the performance core's local cache, perform parallel computations on the read matrix data using multiple processing units to obtain a first computation result, and then write the first computation result to the performance core's local cache or use vector units to post-process the first computation result.

[0043] When the aforementioned core is a performance core, its communication unit is a routerless NoC communication unit. This communication unit provides a direct and dedicated path to the energy efficiency core. The performance core communicates with the energy efficiency cores within its core group through this dedicated intra-group path. This communication unit can realize intra-core group communication, and therefore it is also called a local communication unit. This communication unit maintains two FIFO (First In First Out) queues: a FIFO receive queue and a FIFO send queue. The FIFO receive queue is used to receive data sent by the energy efficiency cores within the core group according to the first-in-first-out principle. The FIFO send queue is used to send the performance core's data to the energy efficiency cores within its core group according to the first-in-first-out principle.

[0044] In the second scenario, the aforementioned core element is an energy efficiency core. When the aforementioned core is an energy-efficient core, the array-based computing engine included in the energy-efficient core is a multiply-accumulate tree array. This energy-efficient core is optimized for efficiency and can handle memory-intensive computational tasks, such as matrix-vector multiplication operations in the attention layer. The multiply-accumulate tree array supports flexible data access through multiple paths, not only reading data from the local cache but also bypassing the local cache to read data directly from local memory or communication units. This architecture eliminates redundant data in the memory hierarchy and optimizes attention operations. The multiply-accumulate tree array includes multiple multiply-accumulate trees, each with different weight values. Each multiply-accumulate tree is used to read matrix data from any of the energy-efficient core's local cache, local memory, or communication units, perform calculations on the read matrix data to obtain a second calculation result, and then write the second calculation result into the energy-efficient core's local cache or use vector units to post-process the second calculation result.

[0045] When the aforementioned core is an energy efficiency core, its communication unit is equipped with a router. This communication unit is a router-based NoC communication unit. It communicates with energy efficiency cores in other core groups via the router, thereby managing global communication between groups. This communication unit can achieve communication not only within a core group but also between core groups; therefore, it is also called a local / global communication unit. This communication unit maintains two FIFO queue groups: a FIFO receive queue group and a FIFO send queue group. The FIFO receive queue group includes at least one FIFO receive queue, and multiple FIFO receive queues correspond to at least one performance core in the core group to which this performance core belongs. These queues are used to receive data sent by the corresponding performance core according to a first-in, first-out (FIFO) principle. The FIFO send queue group includes at least one FIFO send queue, and multiple FIFO send queues correspond to at least one performance core in the core group to which this performance core belongs. These queues are used to send data to the corresponding performance core according to a first-in, first-out (FIFO) principle.

[0046] In this embodiment, the on-chip network bandwidth of any core within each core group of the chip's three-dimensional stacked structure matches the memory bandwidth of the local memory of its respective core group. For example, Figure 1 Each core group corresponds to 16 storage blocks in its local memory, each block being 128 bytes in size. The total local memory size for each core group is 16 * 128 bytes = 2048 bytes, and correspondingly, the on-chip network bandwidth for each core group is 2048 bytes. Within each core group, the efficiency core can communicate with each performance core via a dedicated intra-group path, thereby achieving efficient bandwidth sharing.

[0047] Figure 3 The microarchitecture of the performance core and energy efficiency core is shown in [reference]. Figure 3The performance core (P-core) includes a control unit, memory controller, local communication unit, local cache, systolic array, and vector unit. The control unit is connected to the memory controller and local communication unit, and the systolic array is connected to the vector unit. The memory controller, local communication unit, vector unit, and systolic array can all access the local cache. After receiving a computation task, the control unit sends instructions to the memory controller. In response to the instructions sent by the control unit, the memory controller reads data from local memory and writes the read data to the local cache. For data written to the cache, if the data is matrix data smaller than a preset size, it can be read and processed by the vector unit; if the data is matrix data larger than the preset size, it can be read and processed by the systolic array. Data processed by the vector unit can be written to the local cache or provided as intermediate data to the systolic array for processing. Data processed by the systolic array can be written to the local cache or post-processed by the vector unit before being written to the local cache. The local communication unit reads data from its local buffer and writes it into the FIFO transmit queue. Then, following the first-in-first-out (FIFO) principle, it sends the data in the FIFO transmit queue to the energy efficiency cores within its core group. When it receives data from an energy efficiency core within its core group, it writes the received data into the FIFO receive queue, also following the FIFO principle.

[0048] The energy efficiency core (E-core) includes a control unit, a memory controller, a local / global communication unit, a local cache, a multiply-accumulate tree array, and vector units. The control unit is connected to the memory controller and the local / global communication unit, while the multiply-accumulate tree array is connected to the vector units. The memory controller, local / global communication unit, vector units, and multiply-accumulate tree array can all access the local cache. Upon receiving a computation task, the control unit sends instructions to the memory controller. In response to these instructions, the memory controller reads data from local memory and writes it to the local cache. For data written to the local cache, if the data is a matrix smaller than a preset size, it can be read and processed by the vector units; if the data is a matrix larger than the preset size, it can be read and processed by the multiply-accumulate tree array. Data processed by the vector units can be written to the local cache, or it can be post-processed by the vector units before being written to the local cache. The local / global communication unit reads data from its local cache and writes it into the corresponding FIFO send queue. Then, following a first-in-first-out (FIFO) principle, it sends the data in the FIFO send queue to the corresponding performance core. When it receives data from a performance core within its core group, it writes the received data into the corresponding FIFO receive queue, also following a FIFO principle.

[0049] The heterogeneous architecture of this application embodiment can rebalance power distribution and heat flow, effectively suppressing the formation of severe hot spots, thereby reducing peak temperature. Due to the improved thermal performance brought by the heterogeneous architecture, the thermal margin of reduced temperature can be converted into room for frequency increase, thereby improving the overall inference performance. As the number of cores increases, the total power also increases, leading to higher temperatures and more uneven heat distribution. In this case, the heterogeneous architecture of this application embodiment is significantly effective. Figure 4 This paper illustrates the temperature variations when running different LLM models using the heterogeneous architecture provided in this application and a traditional 3D architecture (such as Homo-3D) with different numbers of cores (48 cores, 60 cores, and 72 cores). Assuming the input to the LLM model is data with a batch size of 32 and a length of 512, when the number of cores is 48, the heterogeneous architecture provided in this application reduces the average peak temperature by 3.96°C compared to the traditional 3D architecture; when the number of cores is 60, the heterogeneous architecture provides a reduction of 6.10°C; and when the number of cores is 72, the heterogeneous architecture provides a reduction of 5.99°C.

[0050] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0051] Figure 5 This illustration shows a structural block diagram of an electronic device provided in an exemplary embodiment of this application. The electronic device 500 includes a processor 501 and... Figure 1 The illustrated chip 3D stacked structure 502 is connected to the processor 501 via a bus. The communication protocol used when the processor 501 and the chip 3D stacked structure 502 interconnect can be PCIe (Peripheral Component Interconnect Express, high-speed serial computer expansion bus), etc. In the field of computer hardware, PCIe is a widely used serial communication protocol for high-speed data transmission. PCIe offers higher bandwidth and lower latency and is widely used in various computer hardware components, including graphics cards, network adapters, and storage devices.

[0052] Among them, the three-dimensional stacked structure 502 of the chip can integrate an inference platform for inference on large language models.

[0053] The processor 501 can be implemented in at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 501 may also include a main processor and a coprocessor; the main processor is a processor for processing data in the wake-up state, and the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 501 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 501 may also include an artificial intelligence processor for handling computational operations related to machine learning.

[0054] This application provides a thermal optimization method, which is applied to... Figure 5 In the illustrated electronic device, the three-dimensional stacked structure of the chips constitutes the inference platform for the large language model. See also... Figure 6 The method flow provided in this application embodiment includes: 601. In the process of large language model inference based on chip three-dimensional stacked structure, for any core group in chip three-dimensional stacked structure, the shared memory bandwidth in the first memory bandwidth is determined according to the number of inference requests received.

[0055] This application categorizes computing types on electronic devices, offloading computationally intensive tasks to performance cores and memory-intensive tasks to energy-efficient cores. As described above, any core within each core group of a logic chip can access local memory, and the on-chip network bandwidth of any core within each core group matches the memory bandwidth of the local memory of its respective core group. Figure 1Taking the network structure shown as an example, each core group includes three performance cores and one energy efficiency core. Performance cores account for three-quarters of the total number of cores in their respective core groups, while energy efficiency cores account for one-quarter. Correspondingly, the bandwidth allocated to performance cores is in a 3:1 ratio to that allocated to energy efficiency cores. Performance cores have more bandwidth than energy efficiency cores, which handle memory-intensive computational tasks, requiring frequent data reads from local memory and thus necessitating greater bandwidth. When performance cores handle computational tasks in the fully connected layer, the bandwidth allocated to them may become idle due to the reusability of data in batch processing. In this case, the unused bandwidth in performance cores can be shared with energy efficiency cores to reduce their computation time, thereby improving the computational performance of the chip's three-dimensional stacked structure.

[0056] In this embodiment, bandwidth sharing occurs between performance cores and energy efficiency cores within the same core group, and this sharing is real-time. That is, when a performance core within a core group has idle bandwidth, it will share that idle bandwidth with an energy efficiency core within the same core group. Furthermore, at least one performance core within the core group will synchronously share bandwidth; that is, if at least one performance core within the core group has idle bandwidth, at least one performance core will synchronously share its idle bandwidth.

[0057] In this embodiment of the application, the bandwidth allocated to the performance cores and energy efficiency cores within the core group is fixed, for example... Figure 1 Each core group has an on-chip network bandwidth of 2048 bits. Each core group includes three performance cores and one energy efficiency core. Each core within a core group can be allocated one-quarter of the total on-chip network bandwidth. Data in batch processing can be reused. The more inference requests the electronic device receives, the more bandwidth the performance cores can share; the fewer inference requests the electronic device receives, the less bandwidth the performance cores can share. Therefore, the shared memory bandwidth within the first memory bandwidth can be determined based on the number of received inference requests. The first memory bandwidth is the memory bandwidth allocated to the performance cores within the core group. The shared memory bandwidth is the memory bandwidth shared by the performance cores within the core group with the energy efficiency cores.

[0058] 602. Based on the shared memory bandwidth and the second memory bandwidth, the first data is divided into second data corresponding to the shared memory bandwidth and third data corresponding to the second memory bandwidth, so as to minimize the difference between the ratio of the second data to the shared memory bandwidth and the ratio of the third data to the second memory bandwidth.

[0059] The second memory bandwidth refers to the memory bandwidth allocated to the energy efficiency cores in the core group. The first data consists of data that the energy efficiency cores in the core group need to read from local memory in response to inference requests. The second data consists of the first data that the energy efficiency cores need to read using shared internal memory bandwidth. The third data consists of the first data that the energy efficiency cores need to read using the second memory bandwidth. The sum of the first and second data constitutes the first data. In the large language model inference scenario, the first data is key-value cached data.

[0060] For each core group, assuming the shared memory bandwidth of each performance core within the core group is BWshared, the secondary memory bandwidth of the energy efficiency core is BWcore, the data read based on the shared memory bandwidth is KVP, and the data read based on the secondary memory bandwidth is KVE, then the time for the energy efficiency core to read data using the shared memory bandwidth is KVP / BWshared, the time for the energy efficiency core to read data using the secondary memory bandwidth is KVE / BWcore, and the time for the energy efficiency core to read data from local memory is the maximum value between KVP / BWshared and KVE / BWcore. Since the number of inference requests received by the electronic device is not fixed, the shared memory bandwidth shared by the performance cores to the energy efficiency cores in each core group is not fixed. To minimize the data reading time of the energy efficiency cores, the difference between KVP / BWshared and KVE / BWcore must be minimized. This requires KVP / KVE to be close to BWshared / BWcore, meaning the data that the energy efficiency cores need to read needs to be allocated according to the ratio of BWshared to BWcore. By adjusting the amount of data corresponding to the shared memory bandwidth and the second memory bandwidth, the energy efficiency core data reading time can be minimized even when the shared memory bandwidth changes dynamically.

[0061] Based on the above, in one possible implementation, the ratio of the shared memory bandwidth to the second memory bandwidth can be calculated based on the shared memory bandwidth and the second memory bandwidth. Then, the first data can be allocated according to this ratio to minimize the difference between the ratio of the shared memory bandwidth to the second memory bandwidth and the ratio of the second data to the third data, thereby ensuring that the difference between the ratio of the second data to the shared memory bandwidth and the ratio of the third data to the second memory bandwidth is minimized.

[0062] In another possible implementation, to adapt to changes in shared memory bandwidth, the ratio of shared memory bandwidth to second memory bandwidth can be preset. Then, based on this ratio, the first data is allocated into second and third data, ensuring that the ratio of shared memory bandwidth to second memory bandwidth is equal to or close to the ratio of second and third data. Adaptive key-value (KV) cache data is then set in both the second and third data. When shared memory bandwidth increases, the energy efficiency core can read more data using the shared memory bandwidth. In this case, the adaptive KV cache data in the third data can be read by the energy efficiency core using the shared memory bandwidth to adapt to the increase in shared memory bandwidth. Conversely, when shared memory bandwidth decreases, the adaptive KV cache data in the second data can be read by the energy efficiency core using the second memory bandwidth to adapt to the decrease in shared memory bandwidth.

[0063] Figure 7 This illustrates a KV cache partition under bandwidth sharing; see [link / reference]. Figure 7 When the ratio of shared memory bandwidth to second memory bandwidth is BWshared / BWcore, the energy efficiency core uses BWshared to read second data (Normal KV Cache in P-Core) and uses BWcore to read third data (Normal KV Cache in E-Core). When BWshared becomes MaxBWshared, and Max BWshared / BWcore is greater than BWshared / BWcore, the energy efficiency core uses Max BWshared to read the second data (Normal KV Cache in P-Core) and the adaptive KV cache data (Adaptive KV Cache for B batch) in the third data (Normal KV Cache in E-Core), and uses BWcore to read data in the third data excluding the adaptive KV cache data (Adaptive KV Cache for B batch). When BWshared becomes BWshared(BM), and BWshared(BM) / BWcore is less than BWshared / BWcore, the energy efficiency core uses BWshared(BM) to read the second data (Normal KV Cache in P-Core). The data in P-Core, excluding the Adaptive KV Cache for BM batch, is read using BWcore, and the Adaptive KV Cache for BM batch is also read from the third data and the second data.

[0064] 603. In the control core group, the energy efficiency core uses shared memory bandwidth to read the second data from local memory, and uses the second memory bandwidth to read the third data from local memory.

[0065] Once the second and third data are determined, the electronic device will control the energy efficiency core in the core group to read the second data from local memory using shared memory bandwidth, and then read the third data from local memory using the second memory bandwidth. The energy efficiency core uses a multiply-accumulate tree array to process the second and third data, thereby ensuring the normal operation of the inference process of the large language model.

[0066] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0067] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A chip three-dimensional stack structure, characterized by, The chip three-dimensional stacking structure comprises a logic chip and a memory module; The memory module is stacked on the upper surface of the logic chip along a preset direction, the preset direction is perpendicular to the upper surface of the logic chip, the memory module comprises at least one memory chip stacked in sequence along the preset direction, each memory chip comprises a plurality of storage areas, each storage area comprises a plurality of storage blocks, and the storage areas at the same position on the at least one memory chip constitute a storage area group; The logic chip comprises a plurality of core groups, each core group comprises at least one performance core and an energy-efficient core, each performance core is used for processing a calculation-intensive computing task in the core group, and the energy-efficient core is used for processing a memory-intensive computing task in the core group; the energy-efficient core of each core group is connected with each performance core in the core group, and the energy-efficient core of each core group serves as a proxy core of the core group and communicates with other core groups; each core group corresponds to at least one storage area group, and the at least one storage area group corresponding to each core group can be accessed by any core in the core group as the local memory of the core group.

2. The chip three-dimensional stack structure of claim 1, wherein, For any core in any core group, the core comprises a control unit, a memory controller, a local cache, a communication unit, a vector unit and an array-based computing engine; The control unit is used for managing the computing task in the core according to the received instruction; The memory controller is used for managing the local memory of the core; The local cache is used for caching the local data of the core; The communication unit is used for the core to communicate with other cores; The vector unit is used for computing the matrix data smaller than a preset size read by the core; The array-based computing engine is used for computing the matrix data larger than the preset size read by the core.

3. The chip three-dimensional stack structure of claim 2, wherein, When the core is a performance core, the array-based computing engine is a systolic array, the systolic array comprises a plurality of processing units, the systolic array is used for reading matrix data from the local cache of the performance core, performing parallel computation on the read matrix data by using the plurality of processing units to obtain a first calculation result, and writing the first calculation result into the local cache of the performance core or performing post-processing on the first calculation result by using the vector unit.

4. The chip three-dimensional stack structure of claim 3, wherein, The communication unit is used for providing a path for the performance core to communicate with the energy-efficient core in the core group.

5. The chip three-dimensional stack structure of claim 2, wherein, When the core is an energy-efficient core, the array-based computing engine is a multiply-accumulate tree array, the multiply-accumulate tree array comprises a plurality of multiply-accumulate trees, each multiply-accumulate tree has a different weight value, each multiply-accumulate tree is used for reading matrix data from any one of the local cache, the local memory or the communication unit of the energy-efficient core, performing computation on the read matrix data to obtain a second calculation result, and writing the second calculation result into the local cache of the energy-efficient core or performing post-processing on the second calculation result by using the vector unit.

6. The chip three-dimensional stack structure of claim 5, wherein, The communication unit comprises a router, and the communication unit is configured to communicate with energy-efficient cores of other core groups based on the router.

7. The chip three-dimensional stack structure according to any one of claims 1 to 6, wherein, The on-chip network bandwidth of any core in each core group of the chip three-dimensional stacked structure matches the memory bandwidth of the local memory of the core group.

8. The chip three-dimensional stack structure according to any one of claims 1 to 6, wherein, The ratio between the number of performance cores and the number of energy-efficient cores in each core group is a preset ratio, and the preset ratio is determined based on the computing performance and temperature of the chip three-dimensional stacked structure.

9. An electronic device, comprising: The electronic device comprises a processor and the chip three-dimensional stacked structure of any one of claims 1 to 8, and the processor is connected to the chip three-dimensional stacked structure through a bus.

10. A method of thermal optimization, characterized by, The method is applied to the electronic device of claim 9, the chip three-dimensional stacked structure in the electronic device constitutes an inference platform of a large language model, and the method comprises: During the inference process of the large language model based on the chip three-dimensional stacked structure, for any core group in the chip three-dimensional stacked structure, according to the number of received inference requests, a shared memory bandwidth in a first memory bandwidth is determined, the first memory bandwidth is the memory bandwidth allocated to the performance cores in the core group, and the shared memory bandwidth is the memory bandwidth shared by the performance cores in the core group to the energy-efficient cores; According to the shared memory bandwidth and a second memory bandwidth, first data is divided into second data corresponding to the shared memory bandwidth and third data corresponding to the second memory bandwidth, so that the difference between the ratio of the second data to the shared memory bandwidth and the ratio of the third data to the second memory bandwidth is minimized, the second memory bandwidth is the memory bandwidth allocated to the energy-efficient cores in the core group, and the first data is the data that the energy-efficient cores in the core group need to read from the local memory in response to the inference request; The energy-efficient cores in the core group are controlled to read the second data from the local memory using the shared memory bandwidth, and to read the third data from the local memory using the second memory bandwidth.

Images