Thermal-aware memory scheduling method and system for large language model inference

Abstract

The application provides a heat perception memory scheduling method and system for large language model inference, which can be applied to the field of integrated circuit chip heat management technology. The method is applied to a logic chip, the logic chip comprising a memory management module configured with a thermal physical mapping table, comprising: in response to a memory access request for a target key-value cache logical block, determining a first target physical storage bank corresponding to the target key-value cache logical block according to the thermal physical mapping table, and obtaining a thermal state field of the first target physical storage bank according to the thermal physical mapping table; in the case that the thermal state field is a first state, performing a memory access operation on the target key-value cache logical block; in the case that the thermal state field is a second state, determining a second target physical storage bank from a three-dimensional stacked dynamic random access memory, and migrating data corresponding to the target key-value cache logical block from the first target physical storage bank to the second target physical storage bank.

Description

Technical Field

[0001] This application relates to the field of integrated circuit chip thermal management technology, and more specifically to a thermally-aware memory scheduling method and system for large language model inference. Background Technology

[0002] Large language models heavily rely on key-value caches to store the context of previous terms during the inference phase. As sequence length increases, the size of the key-value cache expands dramatically, requiring massive memory capacity and high-speed bandwidth to maintain acceptable terms per second performance. To meet these bandwidth demands, modern computing architectures have shifted to three-dimensional stacked Dynamic Random Access Memory (DRAM), which vertically integrates multiple layers of DRAM chips on top of logic chips and connects them via through-silicon vias (TSVs). This significantly reduces the physical distance between memory and the processor, but also introduces severe thermal challenges. Traditional thermal management typically relies on dynamic voltage and frequency regulation or coarse-grained throttling. When the three-dimensional stacked memory modules reach a critical temperature, the system reduces the clock frequency of the entire memory interface. For large language model workloads, this can cause a global drop in inference speed and latency spikes even if only one memory module overheats. Therefore, a more granular scheduling scheme is urgently needed to maintain system performance under thermal constraints. Summary of the Invention

[0003] In view of the above problems, this application provides a hot-aware memory scheduling method and system for large language model inference.

[0004] According to a first aspect of this application, a thermally-aware memory scheduling method for large language model inference is provided. The method is applied to a logic chip, which includes a memory management module configured with a thermal-physical mapping table. The thermal-physical mapping table stores a thermal state field of a physical storage unit and a mapping relationship between key-value cache logic blocks and physical coordinates. The physical coordinates include a layer identifier, a storage unit group identifier, and a storage unit identifier. The thermal state field characterizes a comparison result between the real-time temperature of the corresponding physical storage unit and a preset threshold. The method includes: responding to a memory access request for a target key-value cache logic block, determining a first target physical storage unit corresponding to the target key-value cache logic block according to the thermal-physical mapping table, and obtaining the thermal state field of the first target physical storage unit according to the thermal-physical mapping table; performing a memory access operation on the target key-value cache logic block when the thermal state field is in a first state; and determining a second target physical storage unit from a three-dimensional stacked dynamic random access memory when the thermal state field is in a second state, and migrating the data corresponding to the target key-value cache logic block from the first target physical storage unit to the second target physical storage unit.

[0005] According to an embodiment of this application, the first state indicates that the real-time temperature of the first target physical storage is lower than a first preset threshold, the second state indicates that the real-time temperature of the first target physical storage is higher than or equal to the first preset threshold and lower than a second preset threshold, and the thermal state field further includes a third state, which indicates that the real-time temperature of the first target physical storage is higher than or equal to the second preset threshold. The method further includes: when the thermal state field is the third state, determining the physical storage that the attention head to be executed needs to access; determining the attention head that needs to access the first target physical storage as a delayed execution object; determining the attention head that needs to access the physical storage in the first state as a priority execution object; and adjusting the execution queue of each attention head according to the order in which the priority execution object takes precedence over the delayed execution object.

[0006] According to an embodiment of this application, determining the second target physical memory from the three-dimensional stacked dynamic random access memory includes: obtaining the thermal state fields corresponding to each of the multiple candidate physical memory according to the thermal physical mapping table; filtering out a set of available physical memory with a real-time temperature lower than the first preset threshold from the multiple candidate physical memory according to the thermal state fields of each candidate physical memory; and selecting the second target physical memory from the set of available physical memory.

[0007] According to an embodiment of this application, migrating the data corresponding to the target key-value cache logic block from the first target physical storage to the second target physical storage includes: copying the data corresponding to the target key-value cache logic block to the second target physical storage; updating the physical coordinates corresponding to the target key-value cache logic block in the thermal physical mapping table so that the updated physical coordinates point to the second target physical storage; marking the storage area corresponding to the target key-value cache logic block in the first target physical storage as a cooling state, under which the storage area is prohibited from being reallocated.

[0008] According to an embodiment of this application, the method further includes: in response to an allocation instruction for a new key-value cache logic block, determining an available physical storage unit from the three-dimensional stacked dynamic random access memory according to the aforementioned thermal-physical mapping table; mapping the new key-value cache logic block to the available physical storage unit; and adding an entry to the aforementioned thermal-physical mapping table to record the correspondence between the physical coordinates of the new key-value cache logic block and the available physical storage unit.

[0009] According to an embodiment of this application, the above method further includes: calculating the access frequency score of at least one key-value cache logic block; determining hot key-value cache logic blocks based on the access frequency score; fixing the hot key-value cache logic blocks to the physical storage in the physical layer of the three-dimensional stacked dynamic random access memory that is farthest from the logic chip, and prohibiting migration operations on the hot key-value cache logic blocks.

[0010] According to an embodiment of this application, calculating the access frequency score of each key-value cache logic block includes: determining whether the key-value cache logic block is accessed in the current sampling period; if the key-value cache logic block is accessed, calculating the access frequency score by adding a second smoothing factor to the weighted sum of a first smoothing factor and the access frequency score of the previous sampling period; if the key-value cache logic block is not accessed, calculating the access frequency score by adding a second smoothing factor to the weighted sum of the first smoothing factor and the access frequency score of the previous sampling period.

[0011] According to an embodiment of this application, determining the hot key-value cache logic block based on the access frequency score includes: identifying the key-value cache logic block corresponding to the identified system prompt word as the hot key-value cache logic block; and identifying the key-value cache logic block with an access frequency score higher than the dynamic threshold as the hot key-value cache logic block based on the comparison result between the access frequency score and the dynamic threshold; wherein the dynamic threshold is the sum of the average access frequency score of the active key-value cache logic blocks and the sensitivity adjustment coefficient.

[0012] According to an embodiment of this application, the method further includes: when the access frequency score of the hot key-value cache logic block is continuously lower than the degradation threshold for a predetermined time window, canceling the fixed allocation of the hot key-value cache logic block, wherein the degradation threshold is the difference between the mean and the standard deviation of the access frequency scores of all active key-value cache logic blocks.

[0013] One aspect of this application provides a thermally-aware memory scheduling system for large language model inference, comprising: a logic chip including at least one processing core and a memory management module; a three-dimensional stacked dynamic random access memory module integrated with the logic chip, comprising multiple vertically stacked storage layers, each storage layer including multiple physical memory blocks, the three-dimensional stacked dynamic random access memory module further including thermal sensors distributed in each physical memory block and through-silicon vias penetrating each storage layer; the memory management module is configured with a thermal-physical mapping table, the thermal-physical mapping table being used to store thermal state fields of physical memory blocks and mapping relationships between key-value cache logic blocks and physical coordinates, the thermal state fields being used to characterize the comparison result between the real-time temperature of the corresponding physical memory block and a predetermined threshold; the memory management module being configured to execute the above method.

[0014] According to an embodiment of this application, the memory management module includes: a thermal management unit configured to obtain the real-time temperature of each physical storage unit by polling the thermal sensors, and dynamically update the thermal state field corresponding to each physical storage unit in the thermal physical mapping table based on the comparison result of the real-time temperature and a predetermined threshold; and a thermal sensing scheduler, communicatively connected to the thermal management unit, configured to, in response to a memory access request for a target key-value cache logic block, determine a first target physical storage unit corresponding to the target key-value cache logic block according to the thermal physical mapping table, and obtain the thermal state field of the first target physical storage unit according to the thermal physical mapping table; when the thermal state field is in a first state, perform a memory access operation on the target key-value cache logic block; when the thermal state field is in a second state, determine a second target physical storage unit from the three-dimensional stacked dynamic random access memory, and migrate the data corresponding to the target key-value cache logic block from the first target physical storage unit to the second target physical storage unit.

[0015] According to embodiments of this application, by introducing a thermal-physical mapping table, the mapping relationship between key-value cache logical blocks and physical coordinates, as well as the real-time temperature status of each physical storage unit, are integrated into a unified data structure. This allows the memory management module to simultaneously obtain the physical location and temperature status of the target logical block when responding to memory access requests. Based on this architecture, differentiated processing is achieved through a state-based flow mechanism: when the target physical storage unit is in the first state (safe state), memory access operations are executed normally, and the system maintains peak performance; when the target physical storage unit is in the second state (warning state), the memory management module migrates the data corresponding to the target key-value cache logical block from the overheated storage unit to the second target physical storage unit with a lower temperature. Because the page attention mechanism divides the key-value cache into non-contiguous logical blocks and the large language model inference engine accesses data through logical identifiers rather than physical addresses, the thermal physical mapping table can update physical coordinates without the inference engine's awareness. This ensures that data migration does not interrupt the inference process and avoids the reduction of memory interface clock frequency caused by dynamic voltage and frequency adjustment of the entire graphics processor. This eliminates the root cause of overall performance degradation caused by local hotspots in traditional solutions, maintaining the stability of inference latency and the overall throughput of the system while ensuring the thermal safety of the 3D stacked DRAM. Attached Figure Description

[0016] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0017] Figure 1 An architecture diagram of a heat-aware memory scheduling system according to an embodiment of this application is shown.

[0018] Figure 2 A schematic diagram of a thermophysical mapping table according to an embodiment of this application is shown.

[0019] Figure 3 A flowchart of a hot-aware memory scheduling method for large language model inference according to an embodiment of this application is shown.

[0020] Figure 4 A flowchart illustrating the heat-sensing assignment of newly generated lexical units according to an embodiment of this application is shown.

[0021] Figure 5 A flowchart of active migration according to an embodiment of this application is shown.

[0022] Figure 6 A structural block diagram of a heat-aware memory scheduling system for large language model inference according to an embodiment of this application is shown. Detailed Implementation

[0023] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0024] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms "comprising," "including," etc., as used herein indicate the presence of the above-described features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0025] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0026] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0027] First, the terminology appearing in the embodiments of this application will be explained:

[0028] Logic chip: The core chip for performing large language model inference computation, comprising at least one processing core and a memory management module. In this application, the logic chip is the physical carrier of the memory management module, responsible for running the large language model inference computation and executing the heat-sensing scheduling logic.

[0029] 3D stacked dynamic random access memory (DRAM): This is a memory structure in which multiple layers of DRAM chips are vertically stacked on top of logic chips and connected via through-silicon vias (TSVs). Compared to traditional planar memory, 3D stacked DRAM can significantly shorten the physical distance for data access and provide higher bandwidth, but it also brings more severe heat dissipation challenges, with the layers closer to the logic chips getting hotter. Common high-bandwidth memories belong to this type of architecture.

[0030] Storage layer: A single vertical layer in a three-dimensional stacked DRAM. Each layer is an independent DRAM chip, vertically connected to other layers via through-silicon vias (TSVs). In the three-dimensional stacked structure, different layers have different heat dissipation conditions: the bottom layer (closest to the logic chip) has the highest temperature, and the top layer (farthest from the logic chip) has the lowest temperature.

[0031] Physical storage unit: The basic storage hardware unit in the storage layer. Each storage layer is divided into multiple physical storage units, and each physical storage unit contains a large number of storage cells (the intersection of rows and columns). The physical storage unit is the actual physical space for data storage and also the smallest monitoring unit of the temperature sensor. The temperature sampling and thermal state fields in this application are both implemented at the physical storage unit level. A single physical storage unit can store data from multiple key-value cache logical blocks.

[0032] Thermal sensors: Temperature sensing elements (such as thermistors) distributed throughout each physical memory module. The thermal sensors monitor the temperature of the corresponding physical memory module in real time and transmit the temperature data to the thermal management unit.

[0033] Through-Silicon Vias (TSVs): Vertical conductive channels that penetrate through each memory layer. A TSV is a vertical conductor formed by drilling holes in a silicon chip and filling them with metal (such as copper or tungsten), used to transmit electrical signals and data between memory layers. In this application, a TSV is a vertical transmission channel for data migration between physical memory layers; data migration from the hot zone at the bottom layer to the cold zone at the top layer is accomplished through a TSV.

[0034] The thermal-physical mapping table, a dynamic data structure maintained in the memory management module, is one of the core features of this application. This table stores two types of information: (1) the mapping relationship between key-value cache logical blocks and physical coordinates; and (2) the thermal state field of each physical storage unit. This achieves the integration of logical memory management and physical thermal management, making data migration completely transparent to the large language model inference engine.

[0035] Physical coordinates: A composite address used to uniquely identify the location of a physical memory bank in a three-dimensional stacked DRAM. Physical coordinates consist of three parts: Layer ID (indicating which layer), Bank Group ID (indicating which bank group), and Bank ID (indicating which bank in the group).

[0036] Key-value cache logical blocks: the basic data unit managed by the page attention mechanism. The key-value cache data generated by the large language model during inference is divided into multiple fixed-size logical blocks, each with a unique logical identifier (e.g., "Block 0", "Block 5"). The large language model inference engine references a specific logical block through its logical identifier, regardless of the actual physical storage location of that logical block. Logical blocks are the core objects of scheduling in this application.

[0037] Page attention mechanism: A key-value cache memory management mechanism that divides the key-value cache into non-contiguous logical blocks. Each logical block can be stored independently in any physical location, and a page table records the mapping relationship between logical blocks and physical addresses. The page attention mechanism solves the memory fragmentation problem caused by traditional contiguous memory allocation.

[0038] Thermal Status Field: A data field in the thermal physical mapping table used to record the temperature status of the physical storage unit. The value of the thermal status field (first status / second status / third status) is dynamically updated by the thermal management unit based on the comparison between the real-time temperature of the physical storage unit and a preset threshold, and is used by the thermal sensing scheduler for querying.

[0039] First state: i.e. safe state, in this application it represents that the real-time temperature of the corresponding physical storage is lower than the first preset threshold. In this state, the temperature of the physical storage is in a safe range and can be read and written normally.

[0040] The second state, also known as the warning state, indicates in this application that the real-time temperature of the corresponding physical storage unit is higher than or equal to a first preset threshold and lower than a second preset threshold. In this state, the temperature of the physical storage unit has entered the warning range, and active migration should be initiated to move the data away.

[0041] The third state, or critical state, in this application indicates that the real-time temperature of the corresponding physical memory is higher than or equal to the second preset threshold. In this state, the temperature of the physical memory has entered the critical range, posing a risk of triggering hardware overheat protection. Therefore, an attention head reordering strategy should be adopted instead of a migration strategy.

[0042] Attention Head: A computational unit in the multi-head attention mechanism of a large language model. A large language model contains multiple attention heads, each of which needs to access specific key-value cache logic blocks during inference. This application allows for heat dissipation time for hot storage by adjusting the execution order (reordering) of different attention heads.

[0043] In this embodiment, the reasoning process of the large language model includes a pre-filling stage and a decoding stage. In the pre-filling stage, the model processes all user input tokens in parallel at once, generating a corresponding key-value cache (KVCache) for each token and storing it in memory. In the decoding stage, the model generates output tokens sequentially, repeatedly reading existing key-value caches and continuously adding key-value caches for the currently generated tokens. The page attention mechanism divides the key-value cache into fixed-size logical blocks, each of which can be independently stored in any physical location within a three-dimensional stacked dynamic random access memory. A page table records the mapping relationship between logical blocks and physical addresses. This allows the key-value cache to be stored non-contiguously, avoiding memory fragmentation problems.

[0044] Three-dimensional stacked dynamic random access memory (DRAM) vertically stacks multiple layers of DRAM chips on top of logic chips, with the layers connected by through-silicon vias (TSVs). Each layer is divided into multiple physical memory cells, each of which is a basic hardware unit for data storage. In the three-dimensional stacked structure, the temperature of the bottom layer (hot zone) closer to the logic chips is significantly higher than that of the top layer (cold zone) farther away from the logic chips, forming a vertical temperature gradient.

[0045] Because the page attention framework in related technologies allocates logic blocks to physical memory addresses solely based on memory availability, it ignores the physical temperature of the underlying DRAM memory. Traditional thermal management typically relies on dynamic voltage and frequency regulation or coarse-grained throttling. When the 3D stacked memory modules reach a critical temperature, the system reduces the clock frequency of the entire memory interface or graphics processor logic. For large language model workloads, this can cause a global drop in inference speed and latency spikes even if only one memory bank overheats.

[0046] Based on the above-mentioned technical problems, embodiments of this application provide a thermally-aware memory scheduling method for large language model inference. The method includes: responding to a memory access request for a target key-value cache logic block, determining a first target physical storage unit corresponding to the target key-value cache logic block according to the aforementioned thermal-physical mapping table, and obtaining a thermal state field of the first target physical storage unit according to the aforementioned thermal-physical mapping table; performing a memory access operation on the target key-value cache logic block when the thermal state field is in a first state; and determining a second target physical storage unit from a three-dimensional stacked dynamic random access memory when the thermal state field is in a second state, and migrating the data corresponding to the target key-value cache logic block from the first target physical storage unit to the second target physical storage unit.

[0047] Figure 1 An architecture diagram of a heat-aware memory scheduling system according to an embodiment of this application is shown. Figure 2 A schematic diagram of a thermophysical mapping table according to an embodiment of this application is shown.

[0048] like Figure 1As shown, the system includes a GPU Logic Die and a three-dimensional stacked DRAM memory module. The logic die contains at least one processing core and a memory management module. The three-dimensional stacked DRAM memory module is integrated with the logic die and contains multiple vertically stacked memory layers, each containing multiple physical memory blocks. Thermal sensors and through-silicon vias (TSVs) penetrating each memory layer are also distributed within the three-dimensional stacked DRAM memory module. A thermally-aware scheduling module communicates with the GPU and memory to manage workloads based on thermal data. The memory management module is configured with a thermal-physical mapping table. The thermal-physical mapping table is a dynamic data structure maintained in the memory management module, used to store the thermal state fields of physical memory blocks and the mapping relationship between key-value cache logic blocks and physical coordinates. The physical coordinates are used to uniquely identify the location of a physical memory block in the three-dimensional stacked dynamic random access memory. The physical coordinates include layer identifiers (…). ), storage group identifier ( ) and storage identifier ( ),Right now The layer identifier indicates the vertical position of the physical memory in the three-dimensional stack (e.g., layer 0 represents the bottom layer, and layer 7 represents the top layer); the memory group identifier indicates the memory group to which the physical memory belongs; the memory identifier indicates the specific number of the physical memory within its memory group.

[0049] Key-value caching logic blocks are the basic data units managed by the page attention mechanism. For example... Figure 2 As shown, multiple key-value cache logic blocks (denoted as logic block 0, logic block 1, logic block 2, logic block 3, etc.) managed by the page attention mechanism are illustrated. These logic blocks are data units from the perspective of the large language model inference engine. Each logic block has a unique logical identifier (such as "block 0" or "block 1"). The large language model inference engine references specific key-value cache data through the logical identifier, without caring about the actual physical location where the data is stored. Figure 2 The left side shows the physical structure of a three-dimensional stacked dynamic random access memory (DRAM), comprising multiple vertically stacked physical layers (denoted as layer 0 to layer N, where layer 0 is the bottom layer closest to the logic chip, and layer N is the top layer / cold zone furthest from the logic chip). Each physical layer is further divided into multiple physical memory banks. Based on the real-time temperature of each physical memory bank, the three-dimensional stacked DRAM is physically divided into hot and cold zones: the bottom layer closest to the logic chip (including layer 0 and adjacent layers) has the highest temperature and is designated as the hot zone. Figure 2(Indicated by cross-shading); the top layer (Nth layer and adjacent layers) furthest from the logic chip has the lowest temperature and is marked as the cold zone. Physical memory in the hot zone is prone to localized hot spots due to its proximity to the high-power logic chip and long heat dissipation path; physical memory in the cold zone is significantly cooler than the hot zone because it is furthest from the logic chip and has the best heat dissipation conditions.

[0050] Figure 2 The dashed arrows in the diagram illustrate the mapping between key-value cache logical blocks and physical storage. For example, logical block 1 is mapped to a physical storage device in the cold storage area via the dashed arrow. This indicates that the data in this logical block is actually stored in the cold storage area where the temperature is lower. Logical block 3 is mapped to a physical storage device in the hot storage area via the dashed arrow, indicating that the data in this logical block is actually stored in the hot storage area where the temperature is higher. The same logical block can be mapped to different physical storage devices at different times (e.g., after active migration), but the large language model inference engine accesses the data through logical identifiers and is completely unaware of the change in physical mapping.

[0051] Figure 2 The bottom shows an exemplary data structure for the thermal-physical mapping table. The thermal-physical mapping table maintains the mapping relationship between each key-value cache logical block and its physical coordinates, as well as the thermal state of the corresponding physical memory, in tabular form. Specifically, each row (i.e., one entry) of the thermal-physical mapping table corresponds to a key-value cache logical block and includes the following fields: logical ID (e.g., "Block 0", "Block 1", "Block 2"), used to uniquely identify a logical block; physical coordinates (composed of layer ID, memory group ID, and memory ID), used to indicate the location of the physical memory currently mapped to the logical block; and thermal state fields (e.g., "SAFE", "WARN", "CRITICAL"), used to indicate the current temperature state of the physical memory. The thermal management unit obtains real-time temperatures by polling thermal sensors distributed in each physical memory and dynamically updates the corresponding thermal state fields in the thermal-physical mapping table based on the comparison results between the real-time temperature and a preset threshold. Figure 2 The core architecture of this application, a logical-physical two-layer mapping, is clearly presented: the upper layer is a logical block view visible to the large language model inference engine, and the lower layer is a physical storage view of the three-dimensional stacked dynamic random access memory. A thermal-physical mapping table establishes and maintains the dynamic mapping relationship between the two. Through this architecture, the memory management module can migrate data from hot physical storage to cold physical storage without changing the logical view of the large language model inference engine—simply updating the corresponding physical coordinates in the thermal-physical mapping table, which the large language model inference engine is completely unaware of. Simultaneously, the introduction of the thermal state field makes temperature information an integral part of the mapping table, providing a real-time data foundation for subsequent scheduling decisions such as thermally-aware allocation, proactive migration, and attention head reordering.

[0052] Figure 3 A flowchart of a hot-aware memory scheduling method for large language model inference according to an embodiment of this application is shown.

[0053] like Figure 3 As shown, the memory management module executes operations S210 to S230.

[0054] In operation S210, in response to a memory access request for the target key-value cache logic block, the first target physical storage body corresponding to the target key-value cache logic block is determined according to the aforementioned thermal-physical mapping table, and the thermal state field of the first target physical storage body is obtained according to the aforementioned thermal-physical mapping table.

[0055] In one example, when the large language model inference engine needs to read or write a key-value cache block (i.e., the target key-value cache block), it issues a memory access request to the memory management module. This request contains the logical identifier of the target key-value cache block (e.g., "Block 5"). Upon receiving this request, the memory management module queries the thermal-physical mapping table. The thermal-physical mapping table records the mapping relationship between each key-value cache block and its physical coordinates. Based on the logical identifier of the target key-value cache block, the memory management module searches for the corresponding entry in the thermal-physical mapping table to determine the physical storage block currently mapped to, i.e., the first target physical storage block. Simultaneously, the memory management module reads the thermal status field of the first target physical storage block from the thermal-physical mapping table. The thermal status field is dynamically updated by the Thermal Management Unit (TMU) based on a comparison between the real-time temperature of the physical storage block and a preset threshold. Therefore, by reading the thermal status field, the memory management module can determine the current temperature status of the first target physical storage block.

[0056] In operation S220, when the hot state field is in the first state, a memory access operation is performed on the target key-value cache logic block.

[0057] In one example, if the thermal state field of the first target physical storage is in the first state, it indicates that the real-time temperature of the physical storage is within a safe range, and normal read and write operations can be performed. In this case, the memory management module directly performs the requested memory access operation (i.e., read or write operation) on the target key-value cache logical block. The first state, or SAFE state, indicates that the real-time temperature is below a first preset threshold ( This meets the conditions for high-frequency read access and accepting new write requests.

[0058] In operation S230, when the above-mentioned thermal state field is in the second state, a second target physical storage body is determined from the three-dimensional stacked dynamic random access memory, and the data corresponding to the above-mentioned target key-value cache logic block is migrated from the above-mentioned first target physical storage body to the above-mentioned second target physical storage body.

[0059] In one example, if the thermal state field of the first target physical memory is in the second state, it indicates that the real-time temperature of the physical memory has risen to the warning range. In this case, if read and write operations continue on the physical memory, the temperature may rise further, triggering the hardware-level Dynamic Voltage and Frequency Scaling (DVFS) mechanism, causing the memory interface clock frequency to decrease. The memory management module initiates an active migration mechanism. First, a second target physical memory with a lower temperature is determined from the three-dimensional stacked dynamic random access memory. The specific determination method can be found in subsequent embodiments. The data corresponding to the target key-value cache logic block is copied from the first target physical memory to the second target physical memory. After the migration is completed, the physical coordinates corresponding to the target key-value cache logic block in the thermal physical mapping table are updated to point to the second target physical memory. The second state, i.e., the WARN state, indicates that the real-time temperature is higher than or equal to the first preset threshold ( And below the second preset threshold ( ( ), which is marked as an active migration.

[0060] This application embodiment integrates the logical block management of the page attention mechanism with the physical temperature management of three-dimensional stacked DRAM by introducing a thermal-physical mapping table. Traditional page attention frameworks allocate physical addresses solely based on memory availability, completely ignoring temperature factors. This embodiment queries the thermal-physical mapping table to obtain the thermal state of the target physical memory during each memory access and performs differentiated processing based on the thermal state—normal access when the temperature is safe, and proactive data migration when a temperature warning is issued. This technical solution brings at least the following technical effects:

[0061] First, it avoids performance loss caused by memory interface frequency reduction. Traditional solutions trigger the DVFS mechanism when any storage bank overheats, causing the clock frequency of the entire memory interface to drop, and all attention heads to slow down synchronously. This embodiment actively migrates data away from hot storage banks, eliminating hot spots and thus maintaining stable system performance. Second, it utilizes the logical-physical address decoupling characteristic of the page attention mechanism. Since the page attention mechanism treats key-value cache as non-contiguous logical blocks, the hot-physical mapping table can modify the physical coordinates corresponding to logical blocks without the large language model inference engine's awareness—for the large language model, the data is still "block 5", but the hot-physical mapping table has updated the corresponding physical address from the hot zone to the cold zone, making data migration completely transparent to upper-layer applications.

[0062] The following will describe in detail the three states of the hot state field and the attention head reordering in the third state, with reference to the embodiments of this application.

[0063] According to an embodiment of this application, the first state indicates that the real-time temperature of the first target physical storage is lower than a first preset threshold, the second state indicates that the real-time temperature of the first target physical storage is higher than or equal to the first preset threshold and lower than a second preset threshold, and the thermal state field further includes a third state, which indicates that the real-time temperature of the first target physical storage is higher than or equal to the second preset threshold. The method further includes: when the thermal state field is the third state, determining the physical storage that the attention head to be executed needs to access; determining the attention head that needs to access the first target physical storage as a delayed execution object; determining the attention head that needs to access the physical storage in the first state as a priority execution object; and adjusting the execution queue of each attention head according to the order in which the priority execution object takes precedence over the delayed execution object.

[0064] In one example, the thermal status field includes three states: State 1, State 2, and State 3. State 1 indicates that the real-time temperature of the first target physical memory is below a first preset threshold. In this state, the temperature of the physical memory is within a safe range, and normal read / write operations can be performed. State 2 indicates that the real-time temperature of the first target physical memory is above or equal to the first preset threshold but below the second preset threshold. In this state, the temperature of the physical memory has entered a warning range, and active migration should be initiated. State 3 indicates that the real-time temperature of the first target physical memory is above or equal to the second preset threshold. In this state, the temperature of the physical memory has entered a critical range, posing a risk of triggering hardware overheat protection. The second preset threshold is lower than the hardware-level forced frequency reduction or shutdown temperature to ensure that user configuration does not exceed hardware safety limits. In practical applications, the specific values ​​of the first and second preset thresholds can be preset by the system based on the thermal characteristics, heat dissipation conditions, and performance requirements of the 3D stacked dynamic random access memory, and can be flexibly configured through interfaces such as memory-mapped I / O registers. The thermal management unit obtains the real-time temperature of each physical memory module by polling the thermal sensors and continuously compares it with the two thresholds mentioned above, dynamically updating the thermal status field in the thermal physical mapping table, thereby realizing fine-grained monitoring and management of the thermal status of the three-dimensional stacked DRAM.

[0065] In one example, when the hot state field is in the third state, it indicates that the first target physical storage is overheated. Continuing data migration at this point could exacerbate the overheating due to additional read / write operations during the migration process. Therefore, this embodiment employs an attention head reordering strategy instead of a migration strategy. Specifically, the physical storage that the attention head to be executed needs to access is first determined. The inference process of a large language model involves the parallel or concurrent execution of multiple attention heads. Each attention head needs to access specific key-value cache logic blocks during computation, and these logic blocks are stored in different physical storage units. The memory management module first determines which physical storage units each attention head to be executed needs to access. Then, attention heads that need to access the first target physical storage unit are identified as delayed execution objects. After determining the correspondence between each attention head and physical storage unit, the memory management module identifies those attention heads that need to access the first target physical storage unit (i.e., the hot storage unit in the third state). If these attention heads continue to execute, they will access the overheated physical storage unit, potentially triggering hardware protection mechanisms or causing data errors. Therefore, these attention heads are marked as delayed execution objects. Next, attention heads that need to access physical storage in the first state are identified as priority execution objects. Simultaneously, the memory management module identifies attention heads that need to access physical storage in the first state (safe state). The physical storage accessed by these attention heads is at a normal temperature and can be safely executed. Therefore, these attention heads are marked as priority execution objects. The execution queue of each attention head is adjusted according to the order of priority execution objects over delayed execution objects. The memory management module rearranges the execution order of the attention heads, moving priority execution objects to the front of the execution queue and delayed execution objects to the back of the execution queue. Attention heads accessing cold storage are executed first to maintain hardware utilization; attention heads accessing hot storage are executed late to allow overheated storage time to dissipate heat.

[0066] This application provides an alternative thermal mitigation method by reordering attention heads when data migration is not feasible (e.g., memory is full, migration costs are too high). This technical solution brings at least the following technical effects: Traditional solutions trigger the DVFS mechanism when the memory reaches a critical temperature, causing a decrease in the clock frequency of the entire memory interface. This embodiment provides an idle heat dissipation window for hot memory by adjusting the execution order of attention heads without reducing the memory interface clock frequency. It maintains hardware utilization. This application does not simply suspend all operations accessing hot memory, but rather executes attention heads accessing cold memory in advance, allowing the GPU to continue executing computing tasks while the hot memory is cooling down, avoiding the waste of idle hardware resources. It complements proactive migration. Proactive migration is used when the memory is in the second state (warning state), and attention head reordering is used when the memory is in the third state (critical state). The two strategies cover different temperature ranges, forming a complete thermal management closed loop.

[0067] Figure 4 A flowchart illustrating the heat-sensing assignment of newly generated lexical units according to an embodiment of this application is shown.

[0068] According to an embodiment of this application, determining the second target physical memory from the three-dimensional stacked dynamic random access memory includes: obtaining the thermal state fields corresponding to each of the multiple candidate physical memory according to the thermal physical mapping table; filtering out a set of available physical memory with a real-time temperature lower than the first preset threshold from the multiple candidate physical memory according to the thermal state fields of each candidate physical memory; and selecting the second target physical memory from the set of available physical memory.

[0069] In one example, such as Figure 4As shown, when it is necessary to allocate physical storage for a new key-value cache logic block or select a target physical storage for migration data, the memory management module first queries the thermal-physical mapping table to initialize the physical layer (i.e., layer N, top layer / cold zone) farthest from the logic chip in the three-dimensional stacked dynamic random access memory as the target query layer L (i.e., L=N). Then, the memory management module queries the thermal state field corresponding to each physical storage in layer L in the thermal-physical mapping table to determine if there is a usable physical storage in layer L with a real-time temperature lower than a first preset threshold (i.e., in the first state / safe state). If a usable physical storage exists in layer L, it is determined as the target physical storage for allocation or migration (i.e., the second target physical storage), and the process ends. If no usable physical storage exists in layer L, the memory management module decreases the query layer number to L-1 (i.e., queries upwards to the next coldest layer) and continues to query whether a usable physical storage exists in layer L-1 until a usable physical storage is found. If no usable physical storage is found even at the bottom layer (layer 0), the result of no usable storage is returned. Figure 4 The "Lth layer" and "Nth layer" in the text identify the physical layers of the three-dimensional stack, where the Nth layer is the top layer (cold zone) furthest from the logic chip, and the 0th layer is the bottom layer (hot zone) closest to the logic chip.

[0070] based on Figure 4 The flowchart shown illustrates the detailed process of "determining a second target physical memory from a three-dimensional stacked dynamic random access memory" in this application embodiment. When the thermal state field of the first target physical memory is in the second state (warning state), the memory management module needs to select a physical memory with a lower temperature from the three-dimensional stacked dynamic random access memory as the migration destination (i.e., the second target physical memory).

[0071] The memory management module uses the physical layer (i.e., the top layer / Nth layer) furthest from the logic chip in the three-dimensional stacked dynamic random access memory as the starting query layer. Since the top layer is furthest from the logic chip, it has the best heat dissipation and the lowest temperature, so it prioritizes searching for available physical memory starting from the top layer. The memory management module obtains the thermal state field of each physical memory in the current physical layer according to the thermal physical mapping table, and determines whether there is a physical memory with a real-time temperature lower than a first preset threshold. Physical memory with a real-time temperature lower than the first preset threshold is in a first state (safe state) and can be used as a migration destination. If there are one or more physical memory cells with a real-time temperature lower than the first preset threshold in the current physical layer, the memory management module selects one of them as the second target physical memory cell. The selection strategy can include: selecting the physical memory cell with the lowest temperature, selecting any available physical memory cell in that layer, or selecting a physical memory cell that meets other predetermined conditions. If no physical storage device in the current physical layer has a real-time temperature lower than the first preset threshold (i.e., all physical storage devices in this layer are in the second or third state), the memory management module moves the query layer upwards (from layer L to layer L-1) and continues to query whether there is a usable physical storage device in this layer. This process of querying upwards layer by layer continues until a usable physical storage device is found or all physical layers have been queried. This method of querying downwards layer by layer from the coldest layer ensures that the physical storage device with the lowest temperature and best heat dissipation is selected as the migration destination, while avoiding unnecessary query overhead.

[0072] This application embodiment determines the migration destination by querying layer by layer starting from the coldest layer, ensuring that data is preferentially migrated to the physical storage with the lowest temperature. Physical storage with a real-time temperature below a first preset threshold is selected using a thermal status field, ensuring that the migration destination is always in a safe state and avoiding data migration to physical storage that is already in a warning or critical state. By querying from the coldest layer (top layer), the physical storage with the lowest temperature and best heat dissipation conditions is preferentially selected as the migration destination, migrating data from hot areas to cold areas, maximizing the inherent heat dissipation advantages of the three-dimensional stacked structure.

[0073] Figure 5 A flowchart of active migration according to an embodiment of this application is shown.

[0074] According to an embodiment of this application, migrating the data corresponding to the target key-value cache logic block from the first target physical storage to the second target physical storage includes: copying the data corresponding to the target key-value cache logic block to the second target physical storage; updating the physical coordinates corresponding to the target key-value cache logic block in the thermal physical mapping table so that the updated physical coordinates point to the second target physical storage; marking the storage area corresponding to the target key-value cache logic block in the first target physical storage as a cooling state, under which the storage area is prohibited from being reallocated.

[0075] In one example, such as Figure 5 As shown, when the thermal state field of the first target physical storage is in the second state (warning state), the memory management module performs an active migration, moving the data corresponding to the target key-value cache logical block from the first target physical storage to the second target physical storage. The memory management module copies the data corresponding to the target key-value cache logical block from the first target physical storage to the second target physical storage via the memory controller or direct memory access engine. The copying operation transmits data between the layers of the 3D stack through through-silicon vias. During the copying process, the execution of the attention head of the page to be migrated may be briefly paused, but other attention heads (accessing other logical blocks) are unaffected and continue to execute. This ensures that the impact of the migration operation on the overall system performance is minimized. After the data copying is complete, the memory management module updates the entry corresponding to the target key-value cache logical block in the thermal physical mapping table, updating the physical coordinates of the logical block from the coordinates of the first target physical storage to the coordinates of the second target physical storage. Since the page attention mechanism treats the key-value cache as a non-contiguous logical block, the large language model inference engine references the data through logical identifiers, without caring which physical storage the data is actually stored in. Therefore, after updating the physical coordinates in the thermal physics mapping table, when the large language model inference engine accesses the logical block again, the memory management module will automatically route the access to the second target physical storage, and the inference engine will be completely unaware that the data has been migrated.

[0076] In one example, after the data migration is complete, the original data in the first target physical storage still exists, but it is no longer referenced by any logical block. The memory management module marks this storage area as a cooled-down state. In the cooled-down state, this storage area cannot be reallocated to store new data. This is to allow sufficient time for the storage area to cool down—it has undergone intensive read and write operations and its temperature is high; if it is immediately reallocated and new data is written, it will not be able to cool down effectively. Only when the temperature of the storage area drops to a safe range (i.e., returns to the first state / SAFE state) can the storage area be removed from the cooled-down state and rejoined the pool of available physical storage for subsequent data allocation.

[0077] According to an embodiment of this application, the method further includes: in response to an allocation instruction for a new key-value cache logic block, determining an available physical storage unit from the three-dimensional stacked dynamic random access memory according to the aforementioned thermal-physical mapping table; mapping the new key-value cache logic block to the available physical storage unit; and adding an entry to the aforementioned thermal-physical mapping table to record the correspondence between the physical coordinates of the new key-value cache logic block and the available physical storage unit.

[0078] In one example, during the decoding phase of the large language model, whenever the model generates a new lexical, a new key-value cache logic block needs to be allocated to store its key-value cache data. When the large language model inference engine needs to allocate a new key-value cache logic block, it issues an allocation instruction to the memory management module. After receiving the instruction, the memory management module queries the thermal physical mapping table to determine an available physical memory from the three-dimensional stacked dynamic random access memory. In a preferred embodiment of this application, the memory management module preferentially selects an available physical memory from the physical layer (i.e., the cold zone) farthest from the logic chip in the three-dimensional stacked dynamic random access memory. This is because the cold zone has the lowest temperature, and writing new data directly to the cold zone can avoid hot spots from the source. The specific method for determining the available physical memory can refer to the three-step screening process described in Embodiment 3: obtaining the thermal state field of the candidate physical memory, screening out the set of available physical memory with a real-time temperature lower than a first preset threshold, and selecting the target physical memory from the set of available physical memory.

[0079] The memory management module maps the new key-value cache logical block to the identified available physical storage. This means that the data of the new logical block will be written to that physical storage. The memory management module adds a new entry to the hot-physical mapping table, recording the correspondence between the logical identifier of the new key-value cache logical block and the physical coordinates of the physical storage it is mapped to. Simultaneously, the hot status field of this entry is initialized to the current hot status of the physical storage. Subsequently, when the large language model inference engine needs to access this new key-value cache logical block, it can determine its physical storage and obtain its hot status by querying the newly added entry in the hot-physical mapping table.

[0080] Traditional page attention frameworks employ first-come, first-served or random allocation strategies, potentially unintentionally deploying high-load pages to the hottest lower-level areas. This application's embodiment prioritizes allocating new logic blocks to cold zones (top-level areas), suppressing overheating issues at the source through page layout allocation. If a new logic block is allocated to a hot zone, subsequent active migration may be triggered due to temperature increases. This application's embodiment, through "one-time allocation," directly places new logic blocks in cold zones, reducing the number of subsequent migrations and overhead.

[0081] According to an embodiment of this application, the above method further includes: calculating the access frequency score of at least one key-value cache logic block; determining hot key-value cache logic blocks based on the access frequency score; fixing the hot key-value cache logic blocks to the physical storage in the physical layer of the three-dimensional stacked dynamic random access memory that is farthest from the logic chip, and prohibiting migration operations on the hot key-value cache logic blocks.

[0082] In one example, pages in the key-value cache are not all accessed to the same degree. For instance, the key-value cache block corresponding to the system prompt word is accessed in every query, resulting in extremely high access frequency. This embodiment calculates access frequency scores and determines hot key-value cache logic blocks accordingly, then permanently assigns these hot blocks to cold zones. Logic blocks identified as hot key-value cache logic blocks are permanently assigned to the physical storage in the 3D stacked dynamic random access memory (DRAM) in the physical layer furthest from the logic chip (i.e., the cold zone / top layer), and migration operations are prohibited for these hot key-value cache logic blocks. Once hot key-value cache logic blocks are fixed in the cold zone, they will not be actively migrated to other physical layers due to temperature reasons. Since these pages are constantly accessed, permanently placing them in the top layer area with the lowest ambient temperature and best heat dissipation ensures that the basic thermal load remains within a controllable range, reducing the possibility of triggering migration.

[0083] According to an embodiment of this application, calculating the access frequency score of each key-value cache logic block includes: determining whether the key-value cache logic block is accessed in the current sampling period; if the key-value cache logic block is accessed, calculating the access frequency score by adding a second smoothing factor to the weighted sum of a first smoothing factor and the access frequency score of the previous sampling period; if the key-value cache logic block is not accessed, calculating the access frequency score by adding a second smoothing factor to the weighted sum of the first smoothing factor and the access frequency score of the previous sampling period.

[0084] In one example, the memory management module calculates the access frequency score for each key-value cache block. The access frequency score measures how frequently a key-value cache block is accessed; a higher score indicates that the block is accessed more frequently. The access frequency score is calculated using an exponentially weighted moving average method, as shown in formula (1):

[0085] + (1).

[0086] in, For the current moment Access frequency score; This is the second smoothing factor (value range 0 < α < 1), used to balance historical visit popularity with instantaneous surge popularity. It is the first smoothing factor; This is an indicator function. It is 1 if the logic block is accessed during the current sampling period, and 0 otherwise.

[0087] The memory management module monitors whether each key-value cache logical block is accessed during each sampling period. The length of the sampling period can be preset according to system requirements. If a key-value cache logical block is accessed, an access frequency score is calculated by adding a second smoothing factor to the weighted sum of the access frequency score from the previous sampling period and a first smoothing factor. If the key-value cache logical block is accessed during the current sampling period (i.e.,...), the access frequency score is determined. =1), then the access frequency score The first smoothing factor is used to weight the access frequency score of the previous sampling period, and the second smoothing factor is used to weight the access indicator value of the current sampling period. If the key-value cache block is not accessed, the access frequency score is calculated based on the weighted sum of the first smoothing factor and the access frequency score of the previous sampling period. If the key-value cache block is not accessed in the current sampling period (i.e., ...), the access frequency score is calculated. If =0), then the access frequency score is 0. + 0 = The access frequency score of unaccessed logical blocks will gradually decrease over time.

[0088] According to an embodiment of this application, determining the hot key-value cache logic block based on the access frequency score includes: identifying the key-value cache logic block corresponding to the identified system prompt word as the hot key-value cache logic block; and identifying the key-value cache logic block with an access frequency score higher than the dynamic threshold as the hot key-value cache logic block based on the comparison result between the access frequency score and the dynamic threshold; wherein the dynamic threshold is the sum of the average access frequency score of the active key-value cache logic blocks and the sensitivity adjustment coefficient.

[0089] In one example, hot key-value cache logic blocks are determined based on access frequency scores, including two determination methods: The first method directly identifies the key-value cache logic block corresponding to a system prompt word as a hot key-value cache logic block. System prompt words are instructions or background information entered by the user at the start of the large language model dialogue. The key-value cache block corresponding to the system prompt word is loaded during the pre-filling stage and will be accessed in all subsequent inference steps, with an access probability approaching 1. Therefore, once the key-value cache logic block corresponding to the system prompt word is identified, it is directly identified as a hot key-value cache logic block without needing to compare scores with thresholds.

[0090] The second category: Based on the comparison between the access frequency score and the dynamic threshold, key-value cache logic blocks with access frequency scores higher than the dynamic threshold are identified as hot key-value cache logic blocks. For key-value cache logic blocks other than system prompts (i.e., ordinary key-value cache blocks generated during inference), the memory management module compares their access frequency scores with the dynamic threshold. If the access frequency score is higher than the dynamic threshold, the logic block is identified as a hot key-value cache logic block. Dynamic threshold Determined based on statistical values ​​of access frequency scores for all active key-value cache logical blocks. Specifically, dynamic threshold. ,in The average access frequency score for all active key-value cache logical blocks. This is the sensitivity adjustment coefficient (typically ranging from 1.5 to 3.0). The dynamic threshold is adjusted in real time according to the statistical distribution of the access frequency scores of all active logic blocks, adaptively reflecting the current access popularity distribution.

[0091] According to an embodiment of this application, the method further includes: when the access frequency score of the hot key-value cache logic block is continuously lower than the degradation threshold for a predetermined time window, canceling the fixed allocation of the hot key-value cache logic block, wherein the degradation threshold is the difference between the mean and the standard deviation of the access frequency scores of all active key-value cache logic blocks.

[0092] In one example, the fixed allocation of hot key-value cache blocks is not permanent. When the access frequency of a hot key-value cache block decreases over time, and its access frequency score remains below the degradation threshold for a predetermined time window, the memory management module revokes the fixed allocation of that hot key-value cache block. The degradation threshold is the difference between the mean and standard deviation of the access frequency scores of all active key-value cache blocks, i.e. - .in The mean, The standard deviation is given when the access frequency score of a certain hot key-value cache logical block is consistently lower than the standard deviation. - Reaching a predetermined time window (e.g., N consecutive sampling periods) indicates that the access frequency of this logic block has significantly decreased, and it no longer needs to be fixed in the cold zone. At this point, its fixed state is revoked, and it is converted into a normal page, allowing dynamic adjustment between physical layers based on real-time access frequency (including migration to hot or warm zones).

[0093] Based on the aforementioned hot-aware memory scheduling method for large language model inference, this application also provides a hot-aware memory scheduling system for large language model inference. The following will combine... Figure 6 The system is described in detail.

[0094] Figure 6 A structural block diagram of a heat-aware memory scheduling system for large language model inference according to an embodiment of this application is shown.

[0095] like Figure 6 As shown, the thermally-aware memory scheduling system for large language model inference in this embodiment includes a logic chip and a three-dimensional stacked dynamic random access memory (DRAM) module. The logic chip (GPU Logic Die) contains at least one processing core and a memory management module. The processing core performs inference computations for the large language model, including the computation of attention mechanisms. The memory management module manages the storage and access of key-value cache data in the DRAM. The DRAM module is integrated with the logic chip and contains multiple vertically stacked storage layers. Each storage layer includes multiple physical memory blocks. The DRAM module also includes thermal sensors distributed in each physical memory block and through-silicon vias (TSVs) penetrating each storage layer. The thermal sensors are used to monitor the temperature of each physical memory block in real time. Distributed across the layers of the three-dimensional stack, the thermal sensors provide memory-level temperature data. The TSVs are vertical conductive channels penetrating each storage layer, used for high-speed data and signal transmission between layers. Data migration between physical memory blocks is achieved through TSVs, enabling vertical data transmission. The memory management module is configured with a thermal-physical mapping table. A thermal-physical mapping table is used to store the thermal state field of physical memory blocks and the mapping relationship between key-value cache logical blocks and physical coordinates. The thermal state field is used to characterize the comparison result between the real-time temperature of the corresponding physical memory block and a predetermined threshold. The memory management module is configured to execute the method of the above embodiments.

[0096] Specifically, in response to a memory access request for a target key-value cache logical block, the memory management module determines the first target physical storage unit corresponding to the target key-value cache logical block based on a thermal-physical mapping table, and obtains the thermal state field of the first target physical storage unit based on the thermal-physical mapping table. If the thermal state field is in the first state, a memory access operation is performed on the target key-value cache logical block. If the thermal state field is in the second state, a second target physical storage unit is determined from the three-dimensional stacked dynamic random access memory, and the data corresponding to the target key-value cache logical block is migrated from the first target physical storage unit to the second target physical storage unit.

[0097] The memory management module includes a Thermal Management Unit (TMU) and a thermal sensing scheduler. The TMU is configured to acquire the real-time temperature of each physical memory module by polling thermal sensors and dynamically update the thermal status field of each physical memory module in the thermal physical mapping table based on a comparison between the real-time temperature and a predetermined threshold. Specifically, the TMU polls the thermal sensors distributed across each physical memory module of the 3D stacked dynamic random access memory at a predetermined sampling period (e.g., every few milliseconds) to acquire the real-time temperature reading of each physical memory module. The TMU compares the real-time temperature of each physical memory module with preset thresholds. These preset thresholds include a first preset threshold (active migration threshold) and a second preset threshold (critical throttling threshold). The first preset threshold is lower than the second preset threshold, and the second preset threshold is lower than the hardware-level forced downclocking or shutdown temperature. The TMU updates the determined thermal status to the corresponding thermal status field in the thermal physical mapping table for querying by the thermal sensing scheduler.

[0098] The thermal sensing scheduler is communicatively connected to the thermal management unit and is configured to, in response to a memory access request for a target key-value cache block, determine the first target physical storage unit corresponding to the target key-value cache block according to a thermal-physical mapping table, and obtain the thermal state field of the first target physical storage unit according to the thermal-physical mapping table. When the thermal state field is in the first state, the thermal sensing scheduler performs a memory access operation on the target key-value cache block. When the thermal state field is in the second state, the thermal sensing scheduler determines a second target physical storage unit from a three-dimensional stacked dynamic random access memory and migrates the data corresponding to the target key-value cache block from the first target physical storage unit to the second target physical storage unit. Furthermore, the thermal sensing scheduler is also configured to perform the operations described in the above embodiments, including attention head reordering, new key-value cache block allocation, access frequency score calculation, hotspot key-value cache block determination, fixed allocation, and degradation processing.

[0099] By dividing the memory management module into two functional units—a thermal management unit and a thermal sensing scheduler—the responsibilities of "temperature monitoring and status updating" and "scheduling decision-making and execution" are separated. The thermal management unit focuses on acquiring temperature data and maintaining thermal status fields, ensuring the real-time nature and accuracy of temperature information. The thermal sensing scheduler focuses on making scheduling decisions based on thermal status, ensuring the timeliness and effectiveness of these decisions. The two units interact with each other through a thermal physics mapping table, forming an efficient closed-loop thermal management system.

[0100] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0101] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0102] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

Claims

1. A hot-sensing memory scheduling method for reasoning in large language models, characterized in that, The method is applied to a logic chip, which includes a memory management module. The memory management module is configured with a thermal-physical mapping table, which stores the thermal state field of physical memory blocks and the mapping relationship between key-value cache logic blocks and physical coordinates. The physical coordinates include a layer identifier, a memory group identifier, and a memory block identifier. The thermal state field characterizes the comparison result between the real-time temperature of the corresponding physical memory block and a preset threshold. The method includes: In response to a memory access request for a target key-value cache logic block, the first target physical storage corresponding to the target key-value cache logic block is determined according to the thermal physical mapping table, and the thermal state field of the first target physical storage is obtained according to the thermal physical mapping table. When the hot state field is in the first state, a memory access operation is performed on the target key-value cache logic block; When the thermal state field is in the second state, a second target physical storage is determined from the three-dimensional stacked dynamic random access memory, and the data corresponding to the target key-value cache logic block is migrated from the first target physical storage to the second target physical storage; Wherein, the first state indicates that the real-time temperature of the first target physical storage is lower than the first preset threshold, the second state indicates that the real-time temperature of the first target physical storage is higher than or equal to the first preset threshold and lower than the second preset threshold, and the thermal state field also includes a third state, the third state indicating that the real-time temperature of the first target physical storage is higher than or equal to the second preset threshold. The method further includes: When the hot state field is in the third state, determine the physical storage that the attention head to be executed needs to access; The attention head that needs to access the first target physical storage is identified as the object to be executed in a deferred manner. The attention head that needs to access the physical storage in the first state is identified as the priority execution object; The execution queue of the attention head is adjusted according to the order in which the priority execution objects take precedence over the delayed execution objects.

2. The method according to claim 1, characterized in that, The step of determining the second target physical memory from the three-dimensional stacked dynamic random access memory includes: The thermal state fields corresponding to each of the multiple candidate physical storage units are obtained according to the thermal physical mapping table. Based on the thermal state field of each candidate physical storage, a set of usable physical storages with real-time temperatures below a first preset threshold is selected from the plurality of candidate physical storages; Select the second target physical storage from the set of available physical storage.

3. The method according to claim 1, characterized in that, Migrating the data corresponding to the target key-value cache logical block from the first target physical storage to the second target physical storage includes: Copy the data corresponding to the target key-value cache logical block to the second target physical storage; Update the physical coordinates corresponding to the target key-value cache logic block in the thermal physical mapping table so that the updated physical coordinates point to the second target physical storage; The storage area in the first target physical storage corresponding to the target key-value cache logic block is marked as a cool state, and the storage area is prohibited from being reallocated in the cool state.

4. The method according to claim 1, characterized in that, The method further includes: In response to the allocation instruction for a new key-value cache logic block, an available physical memory block is determined from the three-dimensional stacked dynamic random access memory according to the thermal-physical mapping table; Map the new key-value cache logic block to the available physical storage; Add a new entry to the thermophysical mapping table to record the correspondence between the new key-value cache logic block and the physical coordinates of the available physical storage.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: Calculate the access frequency score of at least one key-value cache logical block; Determine the hotspot key-value cache logic block based on the access frequency score; The hotspot key-value cache logic block is fixedly allocated to the physical storage in the physical layer of the three-dimensional stacked dynamic random access memory that is farthest from the logic chip, and migration operations are prohibited for the hotspot key-value cache logic block.

6. The method according to claim 5, characterized in that, The calculation of the access frequency score for each key-value cache logical block includes: Determine whether the key-value cache logic block has been accessed during the current sampling period; When the key-value cache logic block is accessed, the access frequency score is calculated by adding a second smoothing factor to the weighted sum of the access frequency score of the first smoothing factor and the access frequency score of the previous sampling period. If the key-value cache logic block is not accessed, the access frequency score is calculated based on the weighted sum of the first smoothing factor and the access frequency score of the previous sampling period.

7. The method according to claim 5, characterized in that, The logic block for determining hot key-value cache based on the access frequency score includes: The key-value cache logic block corresponding to the identified system prompt words is identified as the hot key-value cache logic block; Based on the comparison between the access frequency score and the dynamic threshold, key-value cache logic blocks with access frequency scores higher than the dynamic threshold are identified as hot key-value cache logic blocks; wherein, the dynamic threshold is the sum of the average access frequency score of active key-value cache logic blocks and the sensitivity adjustment coefficient.

8. The method according to claim 5, characterized in that, The method further includes: If the access frequency score of the hot key-value cache logic block continues to be lower than the degradation threshold for a predetermined time window, the fixed allocation of the hot key-value cache logic block will be revoked. The degradation threshold is the difference between the mean and the standard deviation of the access frequency scores of all active key-value cache logic blocks.

9. A heat-aware memory scheduling system for reasoning in large language models, characterized in that, The system includes: A logic chip, comprising at least one processing core and a memory management module; A three-dimensional stacked dynamic random access memory module, integrated with the logic chip, includes multiple vertically stacked memory layers, each memory layer including multiple physical memory cells, and the three-dimensional stacked dynamic random access memory module also includes thermal sensors distributed in each physical memory cell and through-silicon vias penetrating each memory layer. The memory management module is configured with a thermal-physical mapping table, which is used to store the thermal state field of the physical storage and the mapping relationship between the key-value cache logic block and the physical coordinates. The thermal state field is used to characterize the comparison result between the real-time temperature of the corresponding physical storage and a predetermined threshold. The memory management module is configured to perform the method as described in any one of claims 1 to 8.

10. The system according to claim 9, characterized in that, The memory management module includes: The thermal management unit is configured to obtain the real-time temperature of each physical memory by polling the thermal sensor, and dynamically update the thermal state field corresponding to each physical memory in the thermal physical mapping table according to the comparison result of the real-time temperature and a predetermined threshold. A thermal sensing scheduler, communicatively connected to the thermal management unit, is configured to, in response to a memory access request for a target key-value cache logic block, determine a first target physical storage unit corresponding to the target key-value cache logic block according to the thermal physical mapping table, and obtain the thermal state field of the first target physical storage unit according to the thermal physical mapping table; if the thermal state field is in a first state, perform a memory access operation on the target key-value cache logic block; if the thermal state field is in a second state, determine a second target physical storage unit from the three-dimensional stacked dynamic random access memory, and migrate the data corresponding to the target key-value cache logic block from the first target physical storage unit to the second target physical storage unit.

Images