A cache access method, a DRAM cache, and a cache management system

Abstract

The application provides a cache access method, a DRAM cache, a cache management system and a computing device. The cache access method comprises receiving a memory access request, the request specifying a data size to be accessed; determining a corresponding granularity cache area in the DRAM cache according to the data size; and the granularity cache area is obtained by dividing the DRAM cache area according to different granularity sizes. Thus, the corresponding granularity cache area is selected according to the data size of the request, the appropriate cache block can be positioned more accurately, the opportunity of DRAM cache hit is increased, and the memory access delay is reduced. The proportion of useful data in the DRAM cache is improved, and the cache space utilization is improved.

Description

Technical Field

[0001] This application relates to the field of cache management technology, and in particular to a cache access method, a DRAM cache, and a cache management system. Background Technology

[0002] In current system architectures, emerging high-speed interconnect technologies (such as Compute Express Link, CXL) make it possible to build shared memory pools within a rack. These memory pools can be shared by multiple hosts within the same rack, serving as an extension of host memory capacity. In this way, hosts only need to reserve a small amount of memory (around GiB) for storing the operating system and frequently accessed data, while large-scale application data can be stored in the shared memory pool, thus reducing the host's memory requirements. For example, in network communication scenarios, hosts can configure their network interface cards (NICs) to use buffers located in the memory pool, and network packets will be stored directly in the memory pool. Hosts can efficiently access these buffers using Load / Store (LD / ST) synchronization commands, enabling fast data reading and writing.

[0003] However, in this shared memory pool architecture, latency is a significant issue due to network transmission delays between the host and the memory pool. When the host accesses data in the remote memory pool, although interconnect technology accelerates transmission, the LD / ST synchronization instructions consume critical computing resources of the host processor (such as reserved stations), limiting the number of concurrent memory access requests. This limited concurrency cannot effectively mask memory access latency, thus restricting the host's throughput when accessing block data. Therefore, an optimized cache access method is needed to reduce latency, improve memory access efficiency, and enable the shared memory pool to more efficiently support various application requests from the host. Summary of the Invention

[0004] This application provides a cache access method, a DRAM cache, a cache management system, and a computing device, which can reduce memory access latency and improve DRAM cache space utilization.

[0005] Firstly, a cache access method is provided, which includes: receiving a memory access request, the request including the size of the data to be accessed; determining the corresponding granularity cache region in the DRAM cache according to the size of the data to be accessed; the granularity cache region is obtained by dividing the DRAM cache region according to different granularity sizes.

[0006] The method provided in this application divides the DRAM cache into multiple buffers for granular management, allocating appropriate granular cache regions for requests of different sizes, thus avoiding wasted time and resources due to cache regions being too large or too small. By selecting the corresponding granular cache region based on the size of the requested data, suitable cache blocks can be located more accurately, thereby increasing the chance of DRAM cache hits and reducing memory access latency. This increases the proportion of useful data in the DRAM cache and improves cache space utilization.

[0007] In one possible implementation, before initiating a memory access request, the application first sends a memory allocation request, which includes the size of the data to be allocated. Based on the required data size, the runtime allocates memory blocks of appropriate granularity from a memory pool. The memory pool is divided into multiple memory regions, each divided into memory blocks of different granularities, each memory block having a specific granularity. The sum of the sizes of the memory blocks is not less than the required data size. A virtual address-to-physical address mapping is performed on the found memory blocks of the appropriate granularity; the physical address is the physical location of the memory block of the appropriate granularity. The mapped virtual address is then returned to the application.

[0008] In this implementation, the partitioning and granularity of the memory pool are flexible. Different memory regions support different granularities, which can accommodate memory allocation requests of various sizes and achieve on-demand allocation and mapping of memory blocks.

[0009] In one possible implementation, memory access is performed via a physical address; the physical address is obtained by mapping the virtual address; based on the granularity flag in the physical address, the DRAM cache determines the granularity required by the memory access request; based on the granularity, a granular cache region in the DRAM cache is determined; and a cache block lookup is performed in the determined granular cache region.

[0010] In this implementation, granularity information is directly obtained through the granularity flag, eliminating the need for additional table lookups and reducing processing latency. Based on the size of the memory access request and the process of mapping virtual addresses to physical addresses, the required memory granularity can be determined. Through the granularity flag in the physical address, the DRAM cache can quickly locate the required granularity cache region.

[0011] In one possible implementation, if no cache block identifier is found that matches the data to be accessed, the size of the granular memory block to be loaded is determined based on the size of the data in the memory access request; the granular memory block is located in the memory pool according to the granular memory block size corresponding to the data size and the mapped physical address; and the located granular memory block is loaded into the granular cache area of ​​the DRAM cache.

[0012] In this implementation, data of the corresponding granularity is returned according to the size of the requested data, reducing unnecessary data transfer and cache usage. By determining the required granular memory block size based on the size of the memory access request and the mapped physical address, the loaded memory block is ensured to better match the granularity requirements of the access request. The located granular memory block is then loaded into the granular cache area of ​​the DRAM cache, improving the cache hit rate.

[0013] In one possible implementation, the method further includes: identifying each memory region in the memory pool at runtime and determining the different granularities supported by each memory region; writing the granularity information of each memory region and its corresponding physical address range into the DRAM cache to indicate the granularity configuration of each granularity cache region in the DRAM cache.

[0014] In this implementation, the granularity information of each memory region and its corresponding physical address range are written into the DRAM cache, enabling the DRAM cache to perform more precise cache management based on the configuration information of different granularity regions. The granularity of the cache regions in the DRAM cache is consistent with the granularity of the memory pool partitioning, avoiding the loading of irrelevant data and improving the space utilization of the DRAM cache.

[0015] In one possible implementation, the runtime allocates memory blocks of the appropriate granularity from the memory pool, including: based on the required data size, first searching the memory pool for the smallest granularity memory block that matches the size; wherein, when selecting a granularity memory block, the granularity memory block whose location is closest to the memory region of the request initiator is selected; if the size of a single granularity memory block cannot meet the required data size, then multiple granularity memory blocks are searched in the memory pool, and the total size of the selected memory blocks is not less than the required data size.

[0016] In this implementation, the matching granularity of memory regions is dynamically determined based on the data size, and suitable memory blocks are searched from the memory pool to ensure that memory allocation accurately matches application requirements. By allowing the combination of multiple granular memory blocks when a single memory block cannot meet data needs, the system has greater flexibility and can handle complex memory allocation scenarios. Even if the application's memory requests are large or small, the system can flexibly allocate memory according to the actual situation.

[0017] In one possible implementation, each granularity cache region is implemented through a multi-way group mapping structure; each granularity cache region includes multiple cache groups, wherein each cache group includes multiple cache blocks, and the size of each cache block is the granularity size of the corresponding granularity cache region.

[0018] In this implementation, the size of each cache block is consistent with the granularity of the corresponding cache region, making memory management simple and efficient. The multi-way group mapping structure within each granularity cache region can quickly locate the target cache block, thereby shortening cache lookup time, reducing cache access latency, and improving overall system performance. Data requests of different granularities can directly locate the corresponding group, reducing lookup latency. The multi-way group design allows parallel access to data of different granularities, improving performance.

[0019] In one possible implementation, the identifier of the cache block is stored in on-chip memory so that the latency of reading the identifier is controlled within ten cycles.

[0020] In this implementation, by storing the identifier of the cache block separately in on-chip memory, the latency of cache lookup can be significantly reduced.

[0021] Secondly, a DRAM cache is provided, comprising: a request receiving module for receiving a memory access request, the request specifying the size of data to be accessed; a granularity matching module for determining a corresponding granularity cache region in the DRAM cache based on the data size; the granularity cache region is obtained by dividing the DRAM cache region according to different granularity sizes; a cache block is selected in the granularity cache region, and it is checked whether the accessed data matches the identifier of the cache block.

[0022] Thirdly, a memory pool-based cache management system is provided, comprising: a memory pool management module for managing storage space in the memory pool, pre-dividing the storage space into multiple storage regions, each storage region being divided into memory blocks of the same size according to a preset granularity; a runtime module for receiving memory allocation requests, allocating memory blocks of the corresponding granularity from the memory pool according to the data size, and performing virtual address to physical address mapping on the found memory blocks of the corresponding granularity; the physical address being the physical location of the memory block of the corresponding granularity; and returning the mapped virtual address to the application; and a DRAM cache module for receiving memory access requests, specifying the size of the data to be accessed; determining the corresponding granularity cache region in the DRAM cache according to the data size; the granularity cache region being obtained by dividing the DRAM cache region according to different granularity sizes; selecting a cache block in the granularity cache region, and checking whether the accessed data matches the identifier of the cache block.

[0023] Fourthly, a computing device is provided, including a memory and a processor, wherein the memory stores instructions that, when executed by the processor, cause the cache access method according to the embodiments of this application to be implemented.

[0024] Fifthly, a computer program product containing instructions is provided, which, when executed by a cluster of computer devices, causes the cluster of computer devices to perform the method provided in the first aspect.

[0025] The beneficial effects of the second to fifth aspects can be referred to the introduction of the beneficial effects of the first aspect above, and will not be repeated here. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of a DRAM cache system based on a memory pool architecture;

[0027] Figure 2 This is a schematic diagram of a cache access structure using a data prefetcher in related technologies.

[0028] Figure 3 This is a schematic diagram of the cache hierarchy;

[0029] Figure 4 This is a schematic diagram of a data access structure based on fixed granularity in related technologies.

[0030] Figure 5 This paper illustrates a system architecture diagram of a cache access method provided in an embodiment of this application.

[0031] Figure 6 A schematic diagram illustrating a memory allocation and memory access process provided in an embodiment of this application;

[0032] Figure 7 A schematic diagram of the storage region granularity partitioning structure of a memory pool and DRAM cache provided in an embodiment of this application;

[0033] Figure 8 This is a schematic diagram of the interaction process between a DRAM cache and a memory pool provided in an embodiment of this application;

[0034] Figure 9 This is a flowchart illustrating a cache access method provided in an embodiment of this application;

[0035] Figure 10 A schematic diagram illustrating a cache access method provided in an embodiment of this application;

[0036] Figure 11 A schematic diagram illustrating a memory allocation scheme provided in an embodiment of this application;

[0037] Figure 12 This is a diagram illustrating the percentage of cache space utilization when using fixed-granularity access to DRAM cache in related technologies.

[0038] Figure 13A schematic diagram illustrating the percentage of cache space utilization using dynamic granular access to the DRAM cache, provided for embodiments of this application;

[0039] Figure 14 A schematic diagram of a cache structure provided in an embodiment of this application;

[0040] Figure 15 A schematic diagram of a memory pool-based cache management system provided in an embodiment of this application;

[0041] Figure 16 A computing device provided in an embodiment of this application. Detailed Implementation

[0042] The solutions provided in the embodiments of this application will now be described with reference to the accompanying drawings. In the embodiments of this application, "multiple" refers to two or more objects, and "various types" refers to two or more types. Terms such as "first," "second," etc., are only used to distinguish similar objects and are not necessarily used to describe a specific order or number of objects.

[0043] To facilitate understanding of the solutions provided in the embodiments of this application, the technical terms that may be involved in the embodiments of this application will be introduced first.

[0044] Miss Status Handling Registers (MSHR): These registers track incomplete memory access requests (such as cache misses) and record their status information, including the target address, request type, and return data location. They temporarily store this information while the processor waits for memory access results, allowing it to efficiently continue processing other requests.

[0045] Load / Store (LD / ST) are two basic memory access operations in computer instruction set architecture. Load: reads data from memory into processor registers. Store: writes data from registers back into memory.

[0046] Dynamic Random Access Memory (DRAM): A commonly used type of random access memory, widely used in computers and other electronic devices as main memory.

[0047] Static Random Access Memory (SRAM) is a high-speed random access memory. SRAM has a faster read and write speed than DRAM and is typically used in scenarios with high latency requirements, such as CPU cache (L1, L2, L3 cache).

[0048] DRAM Cache: A cache layer that uses DRAM (Dynamic Random Access Memory) as a cache, typically located between main memory (or persistent storage) and the processor, to accelerate data access. DRAM Cache can also be called DRAM cache.

[0049] CXL (Compute Express Link) is a high-performance, high-bandwidth interconnect protocol used to provide high-speed, low-latency data transfer between processors, memory, and accelerators (such as GPUs and FPGAs).

[0050] Memory Management Unit (MMU): A component in computer hardware used to manage and control memory allocation in a computer system, mapping virtual addresses to physical addresses to support the implementation of virtual memory.

[0051] Page Table: A key data structure used by the operating system to implement virtual memory management, mapping virtual addresses to physical addresses.

[0052] A memory pool is a shared memory resource specifically designed for the centralized storage and management of large-scale data across multiple hosts. By using a memory pool, a system can effectively expand the host's memory capacity, allowing the host to reserve only a small amount of local memory for the operating system and frequently accessed data, while offloading most data requests to the shared memory pool.

[0053] For example, such as Figure 1As shown, multiple independent memory modules (such as DRAM and persistent memory) form a shared memory pool, centrally connected through a CXL (Compute Express Link) switch to form a unified memory pool. The CXLS switch connects to multiple hosts, each of which directly accesses data in the memory pool via LD / ST (Load / Store) instructions. All hosts can access the shared storage resources through the CXL switch without needing to allocate large amounts of local memory for each host. However, because access between hosts and the memory pool requires forwarding through the CXL switch, data requests across hosts and memory modules generate significant network latency. For example, the latency of a memory pool based on the CXL protocol is between 170ns and 250ns, more than four times that of local memory. Since the LD / ST synchronization instructions consume critical computing resources of the host processor (reservation stations, cache MSHR, etc.), a host can only initiate a maximum of 64 memory access requests concurrently. Because this limited concurrency cannot mask the memory access latency, the throughput of host accessing block data is limited by the memory access latency. Especially when multiple hosts access the memory pool concurrently at high frequency, this latency will increase significantly, thereby limiting the host's efficient access to large data blocks, resulting in a decrease in the overall data throughput of the system and affecting the running performance of the application.

[0054] In one solution, a data prefetcher is introduced at the host processor's cache level to address latency issues and improve the throughput of host access to block data. Figure 2 This is a schematic diagram illustrating the structure of cache access using a data prefetcher in related technologies. For example... Figure 2 As shown, the prefetcher increases the likelihood of cache line hits when the processor requests memory by loading data from the memory pool into the cache in advance. To achieve this, the prefetcher needs to learn the application's access patterns, including the specific data accessed and the time intervals, so that it can load data into the cache within the appropriate time frame. If data is prefetched too early, it may be replaced by other data; if it is prefetched too late, the processor will be blocked waiting for data. Therefore, the prefetcher typically assumes that the memory access latency is constant to precisely control the timing of data loading. However, because the CXL protocol supports multi-level CXL switch interconnects, the latency of the memory pool may be non-uniform, which increases the difficulty of prefetcher design.

[0055] In another solution, to address latency issues and improve the throughput of host access to block data, a DRAM cache was introduced. Figure 3 This is a schematic diagram of the cache hierarchy, such as Figure 3As shown, the L1 cache is located closest to the CPU core, making it the fastest but with a smaller capacity. It stores the most frequently accessed data to reduce latency when the CPU accesses main memory. The L2 cache is larger than the L1 cache but slightly slower. The L3 cache is a shared cache with the largest capacity. The DRAM cache (also known as DRAMCache) is located between the CPU and main memory, typically serving as a high-performance buffer layer for main memory. It reduces the number of times the CPU directly accesses main memory by prefetching and storing frequently accessed data.

[0056] During cache hits, data access follows a hierarchical order, starting with the fastest L1 cache. If the required data exists in the L1 cache, the access returns a result immediately, achieving minimal latency. If L1 misses, the L2 cache is queried, and so on, checking larger cache layers in turn. If L2 also misses, the L3 cache is searched, serving as the last level of on-chip cache. If L3 also misses, the system forwards the request to the DRAM cache. The DRAM cache, acting as a buffer layer before main memory, significantly reduces the latency of main memory access by storing recently or frequently accessed main memory data. Only when the DRAM cache also misses is the data finally retrieved from main memory.

[0057] As an additional cache layer besides L1, L2, and L3, similar to the processor's internal SRAM cache (such as L1, L2, and L3), the DRAM cache triggers a cache miss when the processor first accesses data, and then loads data from the memory pool at a page granularity (typically 4KiB). Due to its large capacity, the DRAM cache can more effectively cache large blocks of data and leverages the locality of reference in application data access, ensuring that subsequent accesses to adjacent data will hit the cache, thus reducing frequent accesses to the memory pool. Utilizing the locality of reference in application data access, subsequent accesses to adjacent data by the processor will hit the DRAM cache. In this solution, the DRAM cache caches hot data at a fixed page granularity, which can lead to space waste when facing different application access granularities. For example, in network application scenarios, the granularity of network packets varies widely (from 0 to the Maximum Transmission Unit (MTU), with a default value of 1500 bytes). When the DRAM cache caches small packets with a fixed granularity (e.g., 4KiB), a large amount of unused space can exist in unfilled pages, resulting in wasted cache space. In other words, when an application requests, for example, 1500 bytes of data, the DRAM cache loads the data from the memory pool in 4KiB units, reading the entire 4KiB data block from the memory pool. However, the application only needs 1500 bytes, so the remaining 2596 bytes, although loaded into the DRAM cache, are not requested. This situation leads to wasted cache space because only a small portion of the 4KiB space allocated by the DRAM cache is needed by the application; the rest is useless in this request. DRAM cache cannot dynamically adjust the size of cache cells according to actual needs, resulting in low buffer space utilization when processing small-granularity data. This increases data access latency, reduces overall system performance, and leads to unnecessary data transfers, increasing data loading time and wasting cache space. Data that truly needs to be cached may be replaced, increasing the probability of cache misses. Each cache miss triggers a new loading process, reading from the memory pool again, resulting in even higher latency.

[0058] Figure 4 This is a schematic diagram of a data access structure based on fixed granularity in related technologies. For example... Figure 4As shown, the memory pool's storage space is divided into uniform, fixed-granularity pages (e.g., 4KiB pages), each page being a fixed allocation unit. When an application requests memory, it specifies the size of the data it needs to access (e.g., 128B or 256B). The system allocates memory based on the data size, aligning to the smallest 4KiB unit, and selects one or more 4KiB data blocks to satisfy the memory requirement. These data blocks are stored in the memory pool. After allocation, the application can directly access the allocated memory space for storing and manipulating the required data.

[0059] For example, an application issues a read / write request for 1500 bytes of data. The request first searches the L1, L2, and L3 caches sequentially. If no match is found, the request is transferred to the DRAM cache. The DRAM cache checks its current contents to see if it contains a fixed-size cache block (e.g., 4KiB) covering the requested range. If the DRAM cache misses, the corresponding data block is loaded from the memory pool into the DRAM cache via the CXL switching device, and then the data is returned. The DRAM cache loads data from the memory pool in fixed 4KiB page granularity. The 1500 bytes of data requested by the application actually only occupy a small portion of the 4KiB page; the remaining 2596 bytes (4096-1500) are unused. Because the cache contains a large amount of invalid data (2596 bytes), the cache density of truly useful data is reduced. The DRAM cache is easily filled with wasted space, causing truly hot data to be replaced from the cache, thus increasing the probability of cache misses. It's important to clarify that fixed granularity refers to DRAM cache managing data loading, storage, and checking in fixed-size cache blocks (e.g., 4KiB, 1KiB, etc.). This is at the DRAM cache's data organization and logical management level, independent of the specific implementation of hardware access granularity. This differs from the actual transfer granularity of hardware access to memory, which is typically 64 bytes, determined by the hardware interface and DRAM's transfer mechanism. The relationship between the two is that hardware does indeed typically load data from lower-level storage layers to higher-level cache layers in fixed-size units (e.g., 64 bytes, also called cache lines). A cache line is the basic unit of storage and operation in the cache. A cache block, on the other hand, is the logical management unit of the DRAM cache. In a fixed-page granularity architecture, this logical management unit size is, for example, 4KiB. 4KiB can be understood as the data granularity managed by the DRAM cache's logical layer.

[0060] Because it's impossible to allocate memory according to the precise needs of the application, especially when the requested memory amount is not an integer multiple of 4KiB. As the number of requests increases, even small amounts of accumulated waste can significantly impact the overall system memory utilization. When data size is less than the page size, a fixed 4KiB granularity causes the DRAM cache to load additional useless data, increasing cache bandwidth consumption, reducing effective cache space, lowering cache hit rate, and consequently affecting access performance. Figure 4 The diagram shows several 4KiB data blocks with fixed granularity, illustrating memory waste under different request ratios. Some data blocks use 1500 bytes but waste 2596 words. Of these, the 1500 bytes actually needed will be hit, while the remaining 2596 bytes will not be hit because they are not requested. This results in low cache space utilization. In addition, in the case of a miss, additional data needs to be loaded from the memory pool, increasing access latency.

[0061] In view of this, this application proposes a cache access method that dynamically matches the cache granularity according to the size of the actual data request, effectively utilizes cache space, reduces space waste when dealing with different data needs, and improves memory access efficiency while reducing latency.

[0062] Next, the cache access method based on granular information provided in the embodiments of this application will be described. Figure 5 A system architecture diagram of a cache access method is shown. Figure 5 As shown, the system architecture includes a memory pool, memory pool management software running on Fabric Manager, DRAM cache, CXL Switches, and hosts connected to the memory pool via CXLSwitches.

[0063] The host can be any type of server, such as a rack server, blade server, or tower server. The host may have no local memory, or only a small portion of physical memory dedicated to the runtime environment. The host typically connects to the shared memory pool using a Peripheral Component Interconnect Express (PCIe) bus or a CXL-based bus. The CXL protocol provides consistent caching and memory access between the host and the memory pool. The host includes a processor, runtime environment, L1 / L2 / L3 caches, DRAM cache, and applications running on the host. The processor can be a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or other dedicated accelerators.

[0064] The memory pool consists of multiple memory modules interconnected via high-speed CXL interconnects, forming a shared memory resource accessible to multiple hosts. The memory modules may include dual inline memory modules (DIMMs), dynamic random access memory (DRAM), or synchronous dynamic random access memory (SDRAM). This application does not limit the number or specific form of the memory modules.

[0065] CXL Switches serve as the connection hub between the memory pool and each host, responsible for routing memory access requests from multiple hosts to the corresponding memory modules within the memory pool. CXL Switches support multi-level interconnect structures, allowing the memory pool to expand to the entire rack level, providing a unified memory access interface for all connected hosts. CXL switches can connect multiple hosts and the memory pool via the CXL bus.

[0066] The DRAM cache has a large capacity and is connected externally via CXL switches. It is used to buffer and store data loaded from the memory pool. Host memory access (LD / ST requests) only needs to access the memory pool through the CXL switches when a cache miss occurs. CXL switches can support multiple memory modules, for example, 4096.

[0067] Fabric Manager can be a standalone hardware or control node running outside the host. As the core management module running on Fabric Manager, the memory pool management module monitors memory pool usage and dynamically allocates memory pool resources to meet host access requests. Working in conjunction with the DRAM cache of each host, the memory pool management module ensures granular optimization at the cache level, supporting higher cache hit rates and lower access latency. It communicates with each host and the memory pool through CXL Switches. CXL Switches support multiple storage resources (e.g., 4096 storage devices) and route memory access requests to the memory pool.

[0068] When the system starts, the memory pool management software divides the memory pool's memory space into multiple regions. Each region represents a large segment of physical memory, facilitating subsequent management and allocation. Each region is further divided into smaller, fixed-size buffers (i.e., cache regions, with buffer sizes such as 512B, 1KiB, etc.), and each region contains a configurable number of N buffers. This division allows the memory pool to flexibly allocate buffers when faced with data requests of varying sizes, adapting to diverse data needs. For example, some memory blocks may be small (128B or 512B), while others may be large (e.g., 4KiB, 8KiB, etc.).

[0069] It's important to note that the granularity of memory pool storage space isn't limited to the Region level; Regions can be further divided into smaller units called Sections. A Section is a more granular unit of memory allocation than a Region, used to meet different application needs or data size requirements. For some applications, the required memory space doesn't need to reach the size of a Region; in this case, allocating a smaller Section unit is sufficient. The host then divides the Section into smaller buffers to support access at different granularities.

[0070] When the host starts up, it needs to request memory blocks from the memory pool management module and register them in the host's physical space. Specifically, during host startup or application execution, the host runtime requests buffers of different granularities from the memory pool management module in advance. Based on the application's memory allocation requests, the runtime finds suitable physical memory space from the memory pool, that is, it allocates different granularity cache regions (e.g., multiple buffers of different granularities such as 128B, 1KiB, etc.) from the memory pool to the application. For example, the runtime might request 100 1KiB buffers, 1000 512B buffers, and 1000 128B buffers.

[0071] When an application starts running, it first requests memory allocation. The application issues a memory allocation request (such as an `alloc` request) to check if there are sufficient memory blocks of appropriate size and granularity in the memory pool. At this point, the application sends the memory allocation request to the runtime. The runtime is the software layer responsible for the dynamic management of the memory pool, connecting the application and the memory pool management module (Fabric Manager). The runtime determines the granularity of the memory blocks required by the application based on the application's memory allocation request (e.g., 1 KiB, 4 KiB, 128B, etc.). As mentioned earlier, memory blocks in the memory pool are divided into different granularities. The runtime selects an appropriate size memory block based on the application's needs. If the memory pool is insufficient, the runtime requests more memory blocks from the Fabric Manager through the CXL switching device to ensure that the application's needs are met. Once the application enters the execution phase, it issues memory access requests, such as reading or writing data. Each time the application needs to access memory, it issues a memory access request to the processor. At this time, the processor checks the cache layers (L1, L2, L3) to determine if the data already exists in the cache. If an L1, L2, or L3 cache hit occurs, the processor retrieves the data directly from the cache and returns it to the application.

[0072] If a cache miss occurs (i.e., no data is found in L1, L2, or L3), the processor will continue to query the DRAM cache to check if the data is already there. If a DRAM cache miss occurs, the DRAM cache will communicate with the memory pool via the CXL switching device to request the data to be loaded from the Fabric Manager.

[0073] During this process, the memory pool retrieves a memory block of the corresponding granularity based on the size of the requested data and loads it into the DRAM cache. After successful data loading, the DRAM cache returns the corresponding buffer block to the processor, which then delivers the data to the application.

[0074] Throughout the process, the granularity of data delivery remains consistent with the size of the data requested by the application. In other words, data is returned according to the size specified by the application to ensure efficient data access and processing.

[0075] Figure 5 The device structure shown does not constitute a limitation on the computer device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0076] Figure 6This is a schematic diagram illustrating a memory allocation and memory access process provided in an embodiment of this application. Figure 6 As shown,

[0077] The memory pool management module divides the memory pool into multiple Regions. Upon host startup, each host requests a Region from the memory pool management module and further divides each Region into memory blocks (buffers) of the same size based on different granularities. Each Region and its memory blocks are allocated a unique physical address range. Simultaneously, a portion of the physical address contains a granularity identifier to distinguish memory blocks of different granularities. This allows for direct identification of the memory block's granularity and its Region during memory access or allocation, avoiding additional lookups. For example, one Region might be divided into multiple memory blocks with a smaller granularity (e.g., 128B) for small data storage; another Region might be divided with a larger granularity (e.g., 4KiB) to meet larger memory requirements.

[0078] Based on the configuration of these Regions, a predefined physical address range can be set for the DRAM cache, and the starting address and granularity of each Region can be recorded to support efficient access to subsequent data caches. When an application issues a memory allocation request (such as alloc), the runtime will parse the data size in the request.

[0079] At runtime, the system locates the most suitable Region in the memory pool based on the data size and allocates memory blocks of granularity that meet the data size requirements.

[0080] When the runtime processes a memory allocation request (such as `alloc`), it finds a range of virtual addresses in the application's virtual address space and uses this address to map the physical address of the requested physical memory. Each application is allocated an independent virtual address space in the operating system. The runtime allocates a range of virtual addresses in the application's virtual address space, establishes a mapping relationship between it and the physical address of physical memory, and records the mapping information in the page table.

[0081] The virtual address space is divided into fixed-size pages, each typically 4 KiB. At runtime, the number of virtual pages required is calculated based on the total memory allocation demand, and the corresponding physical pages (page frames) are allocated accordingly. This part describes the typical implementation; this explanation is merely adaptive.

[0082] Virtual address pages and physical address pages are mapped in a one-to-one relationship. A page frame can hold one or more memory blocks (buffers), depending on the granularity of the buffer: if the memory block is smaller than 4KiB (e.g., 1KiB), multiple buffers will share a single page frame; if the memory block is larger than 4KiB (e.g., 6KiB), it will span multiple page frames. The mapping between pages and page frames is performed according to common technical solutions, and this application does not impose any limitations on it.

[0083] At runtime, the mapping relationship is recorded in the page table. The page table is the core structure of the memory management system, used to record the mapping relationship between virtual addresses and physical addresses.

[0084] Applications access memory using virtual addresses allocated at runtime. At runtime, these virtual addresses are translated into physical addresses using page tables. During this translation process, virtual pages are mapped to corresponding physical page frames.

[0085] The translated physical address is passed to the DRAM cache. The DRAM cache determines whether a cache hit occurs based on the physical address: In the DRAM cache, the physical address needs to be resolved according to the division of the address field. Specifically, several bits of the address (such as bits [14:15]) are used to identify the granularity, which can indicate different granularity types (such as 1KiB, 4KiB, 128B, etc.). For example, address bits [14:15] being 00 indicates a granularity of 1KiB, and address bits [14:15] being 01 indicates a granularity of 4KiB. The lower bits are used to identify the tag, index, and offset, which determine the location of the address in the cache. The DRAM cache determines the corresponding granularity cache region for cache lookup based on the granularity information in the physical address.

[0086] Figure 7 This diagram illustrates a storage region granularity partitioning structure for a memory pool and DRAM cache according to an embodiment of this application. Figure 7 As shown, the memory pool is divided into multiple Regions, and each Region allocated to the host is divided into buffers with different granularities. The physical address range and granularity of each Region are marked (e.g., the address range of Region 1 is 0x0000_4000 to 0x0000_7FFF, with a granularity of 128B). For example, Region 1 is divided into multiple buffers, each with a size of 128B.

[0087] In the DRAM cache, the cache area corresponding to each region is divided into multiple smaller regions, and each region has corresponding cache blocks. The physical addresses of these regions and their buffer granularity are written to the DRAM cache configuration register at runtime. For example:

[0088] Region 1 starting address: 0x0000_0000, granularity 128B

[0089] Region 2 starting address: 0x0000_4000, granularity 512B

[0090] Region 3 starts at address 0x0000_8000, with a granularity of 1KiB. For example, the granularity region partitioning of the DRAM cache is performed at runtime through write operations to the configuration register. The runtime requests memory space from the memory pool management module and obtains the partitioned memory regions of different granularities (e.g., 1KiB, 128B, 4KiB, etc.). The runtime writes this granularity information to the DRAM cache's configuration register. Specifically, the configuration register records the identification information of different granularity regions. For example, if the system supports three granularities: 1KiB, 512B, and 128B, after writing this granularity information to the DRAM cache's configuration register at runtime, when a memory access request occurs, the DRAM cache will automatically select the appropriate cache region for data operations based on the requested granularity information.

[0091] When an application initiates a memory access request, the DRAM cache checks the granularity information previously written to the configuration register. The configuration register indicates the granularity of the current request. For example, the configuration register might indicate that the current request's granularity is 1 KiB. Based on the granularity, the DRAM cache selects a cache region that matches that granularity. After finding a suitable cache region, it performs the memory access, reading or writing data.

[0092] Both the DRAM cache and the memory pool need to create regions with the same granularity, each responsible for storing data of a specific granularity. For example, if the memory pool has regions with granularities of 4KiB, 1KiB, and 512B, the DRAM cache will also contain corresponding 4KiB, 1KiB, and 512B regions. Each granularity will have its own independently created region in the DRAM cache for storing cache blocks of different sizes. In this way, the DRAM cache can efficiently store data blocks from the memory pool within the corresponding granularity cache regions.

[0093] Please continue reading. Figure 6When an application calls the `alloc` function to request memory of a specified granularity, the runtime allocates a virtual address block VA1 of appropriate size for the request. It then searches the memory pool for a free physical memory block PA1 with a matching granularity and establishes a mapping between the virtual address block VA1 and the physical address block PA1 using a page table. The allocated virtual address block VA1 is mapped to the corresponding physical memory block PA1. When the application accesses data via the virtual address, the runtime uses the page table to find the corresponding physical address and passes it to the DRAM cache.

[0094] For example, if an application issues a request for address 0x0000_4800, the processor will execute a Load(LD) instruction. This request will be searched in the DRAM cache after the virtual address is mapped to the physical address. Granularity fields (such as bits [14:15] of the physical address) are used for granularity identification, and are defined as follows:

[0095] 00:128B particle size

[0096] 01:512B particle size

[0097] 10:1 KiB granularity

[0098] 11:4 KiB granularity

[0099] A physical address includes a tag, index, offset, and granularity identifier. The granularity identifier identifies the data's granularity (e.g., 1 KiB, 2 KiB, etc.), the tag identifies different cache lines, and the index locates the specific set of DRAM cache lines. The offset determines the data's position within the cache line.

[0100] For example, the lower-order bits of the Tag are used to store granularity information because these bits are more likely to change within the address space, while the higher-order bits change relatively little. The lowest two bits of the Tag represent the granularity. The remaining Tag bits and other fields retain their original functions. Taking a common cache configuration as an example: Cache line size: 64 bytes (i.e., 2^6, requiring 6 offset bits). 8 bits are required for the index, which is used to locate a specific cache set; for example, 1024 cache lines require 256 sets when concatenated in a 4-way cache. Number of Tag bits: 32 - number of index bits - number of offset bits = 18 bits.

[0101] Specifically, 0x0000_4800 is converted to 32-bit binary: 0000 0000 0000 0000 0100 1000 00000000. The granularity bits are the low-order bits (bits 14 and 15) of the tag bits, used to distinguish data granularity. The tag bits (31:16) are the high-order bits remaining after removing the granularity bits, used to identify the main memory address of the cache block. The index bits (13:6) are used to locate the cache line. The offset bits (5:0) are used to identify the location of the specific data within the block. A granularity bit of 01 indicates a granularity of 512B. The DRAM cache identifies the requested data granularity as 512B from the granularity flag of the requested address.

[0102] In the DRAM cache, configuration registers store the starting address and granularity information of each region in the memory pool. Based on this configuration information, the DRAM cache divides the cache space into cache regions of different granularities. For example, there are 128B granularity cache regions, 512B granularity cache regions, 1KiB granularity cache regions, and 4KiB granularity cache regions.

[0103] After determining the cache region in the DRAM cache based on the granularity flag in the physical address, the target cache group is further searched within that granularity region using the group index, which determines a set of cache lines within that granularity region. Finally, the existence of the corresponding data block within the group is verified using the tag to confirm whether a cache hit has occurred.

[0104] If a DRAM cache hit occurs, the CXL switch returns the data from the appropriate cache region in the DRAM cache to the processor. The data is then loaded into registers in the processor using the LD instruction, and the application can directly use the data loaded into these registers.

[0105] When an application needs to write data, the processor executes a Store(ST) instruction to initiate a write request. The processor sends the write request to the DRAM cache. If the write request misses (i.e., the data is not in the cache), the CXL switch routes the request to the corresponding physical address in the memory pool. To reduce latency for subsequent accesses, the runtime loads the data block or page being written into the appropriate granularity region of the DRAM cache.

[0106] When a cache hit occurs, the appropriate granularity cache region is selected from the DRAM cache according to the granularity information for data access, thereby reducing space waste and improving the cache hit rate.

[0107] In the event of a cache miss, data in the memory pool can be accessed directly through the mapped physical address, enabling efficient data reading and writing.

[0108] Figure 8 This is a schematic diagram illustrating the interaction process between a DRAM cache and a memory pool, provided as an embodiment of this application. Figure 8 As shown, it includes:

[0109] S1, at runtime, configures the physical address range of the Region and the granularity of the Region in the DRAM Cache.

[0110] S2, the application submits a memory allocation request to the runtime.

[0111] When an application (e.g., application 1, application 2, application 3) makes a memory allocation request, the runtime allocates buffers of different granularities according to the size of the request.

[0112] S3, at runtime, performs virtual address to physical address mapping on the selected granularity region and sends the physical address to the DRAM cache.

[0113] At runtime, page tables and the MMU (Memory Management Unit) are used to map virtual addresses to physical addresses.

[0114] S4, DRAM Cache determines whether a cache hit occurs by performing a cache lookup based on the physical address.

[0115] DRAM cache uses granularity flags in the physical address to determine the appropriate granularity. For example, bits N through N+1 are used as granularity flags to distinguish different buffer granularities. Each granularity (e.g., 4KiB, 1KiB, 512B) corresponds to a binary value combination. For instance, a granularity flag of 10 indicates a 4KiB granularity, while a flag of 01 indicates a 1KiB granularity. By decoding the granularity flags, the DRAM cache can directly determine the granularity cache region where the corresponding data block resides, without needing to search through each block individually. This method enables rapid data block location and reduces search time.

[0116] Once the size of the data block is confirmed by the granularity flag (e.g., a 4KiB region), the DRAM cache will directly jump to the corresponding granularity cache region without needing to search across all granularities.

[0117] Based on the size of the requested data (e.g., 128B, 1KiB, or 512B), the granularity region in the DRAM cache is determined for data lookup. The DRAM cache is divided into multiple independent regions with different granularities (e.g., 4KiB, 1KiB, 512B), each with a multi-way set-associative cache structure. This allows direct location of the corresponding region based on the request size, improving lookup efficiency. The physical address tag is compared with the tag stored in the cache line to determine if the data block is already cached. For example, with a multi-way set-associative cache structure (e.g., 4-way set-associative), the search can be performed among the four possible cache blocks in that granularity region.

[0118] If a matching cache block is found, the application can directly access the data in the cache without reloading it from the memory pool.

[0119] S5. If no matching cache block is found (i.e., cache miss), load the corresponding data block from the memory pool and place it into the specified granularity area of ​​the DRAM Cache.

[0120] Specifically, the CXL Loader can be used to load memory blocks of the corresponding granularity from the memory pool into the DRAM Cache. The CXL Loader locates the memory blocks in the memory pool based on their physical addresses and loads them into the DRAM Cache according to the set granularity (e.g., 4KiB or 512B). For example, the functionality of the CXL Loader can be embedded within the DRAM Cache.

[0121] Next, the flow of the cache access method based on granular information provided in this application embodiment will be described in conjunction with the system architecture. The flow of the cache access method based on granular information is as follows: Figure 9 As shown in the diagram, an access to the DRAM cache is triggered only when all L1, L2, and L3 caches are missed (not shown in the diagram). At this point, based on the granularity information in the memory access request, it is determined whether the data already exists in the DRAM cache. If a hit occurs, the request is returned directly; otherwise, a matching memory block is loaded from the memory pool into the DRAM cache.

[0122] During the memory pool granularity partitioning phase, at system startup, the memory pool management module divides its storage area into multiple Regions, each with a size of 1GB, and further divides it into multiple fixed-size Buffers according to a specified granularity (e.g., 1KiB, 128B, 4KiB, 512B). The starting and ending physical address ranges of each Region and Buffer are recorded in preparation for receiving requests from runtime.

[0123] During the host's request for memory pool buffers phase, upon startup and runtime, the host pre-requests and allocates buffers of different granularities from the memory pool management module. Multiple buffers are requested in advance according to different granularities (e.g., 1 KiB, 2 KiB, 4 KiB, etc.), specifying the required number of buffers for each granularity. The memory pool management module allocates the corresponding buffers based on the granularity and quantity, and records the physical address of each buffer. Afterwards, the application issues a memory allocation request.

[0124] Step 901: The application sends a memory allocation request to the runtime. During the application startup phase, the application first sends a memory allocation request. Applications (such as application 1, application 2, and application 3) send memory allocation requests to the runtime according to their own needs.

[0125] The following section explains memory allocation and memory request in applications using real-world usage scenarios.

[0126] For example, recommendation systems use embedding tables to represent user and item features. Each feature is typically embedded as a fixed-size vector. For example, each vector is 512 bytes in size (i.e., 128 floating-point numbers, each of which occupies 4 bytes).

[0127] An embedding table typically contains thousands to tens of thousands of items representing features of a user or item. Each item in an embedding table is independent and has a fixed size of 512 bytes. All items in the table are the same size, so 512 bytes can be considered the granularity of this data structure. A data structure is a logical way of organizing and storing data. It defines how data is arranged, stored, and accessed. In recommender systems, an embedding table is a common data structure used to store high-dimensional feature vectors. An embedding table is a two-dimensional array or table where each row represents an embedding vector, corresponding to a feature of a user or item. Each column corresponds to a feature dimension.

[0128] The granularity of a data structure refers to the size of each individual element within that data structure during operation or storage. For an embedding table, the granularity is the size of each embedding vector, which is 512 bytes, as this is the basic unit that the application processes or accesses. When an application needs to process the embedding features of a specific user or item—that is, when the application is processing this data structure—each memory access request will be made in units of this granularity, specifically in 512-byte units. The size of each request is consistent with the memory granularity.

[0129] When an application starts, it initiates an alloc request based on the characteristics of the data structure it needs to process (such as an Embedding Table). For example, for an Embedding Table where each embedding vector is a fixed 512 bytes and there are a total of 10,000 rows, the application knows that the granularity is 512 bytes and needs to allocate a total of 5 MiB of memory space.

[0130] Step 902: The runtime receives the request, determines the required data size, and searches for a suitable Region and Buffer from the memory pool management module.

[0131] Upon receiving an alloc request, the runtime checks if there are memory blocks of sufficient size and appropriate granularity in the memory pool. If available resources are in the memory pool, the runtime directly allocates the required memory block from the pool and returns the allocation result to the application. For example, if the application requests a 512B granularity, the runtime can select a 512B granularity buffer.

[0132] Step 903: After determining the allocated buffer, the runtime and the host MMU cooperate to establish a mapping from the application's virtual address to the physical address.

[0133] Step 904: During the application's memory access phase, the application issues a memory access request.

[0134] Applications initiate memory access requests via virtual addresses, which are then translated into physical addresses using a mapping table established during the memory allocation phase. The translated physical addresses are then used to access the DRAM cache.

[0135] When an application processes data (such as calculating user-item recommendation ratings), it will issue memory access requests (memory access requests sent to the processor) for specific rows (i.e., embedding vectors). For example, if an application needs to access row 1234 (512 bytes) of the EmbeddingTable, the request size is 512 bytes, consistent with the allocation granularity.

[0136] When the processor receives an access request, it first checks the caches (L1, L2, L3): (not shown in the diagram) Cache hit: If the data already exists in the cache, the processor directly returns the data to the application. Cache miss: If the data is not in the cache, the processor further checks the DRAM cache.

[0137] Step 905: The DRAM Cache performs a cache lookup based on the granularity information in the physical address.

[0138] The DRAM cache identifies the required cache region granularity based on the granularity flag in the physical address and locates the appropriate granularity cache region.

[0139] Step 906: The DRAM Cache extracts the Tag from the physical address. In the selected granularity region, the DRAM Cache compares the Tag of the physical address with the Tag of the cache block in that region to determine whether the corresponding data has been cached.

[0140] In one possible implementation, tags are stored in on-chip memory (SRAM), typically serving as metadata for the DRAM cache, used to record information such as cache line address mappings and validity bits. The DRAM cache data is stored in off-chip DRAM chips. Because the metadata (such as tags and validity bits) is stored in on-chip high-speed memory (SRAM), access speed is very fast, avoiding the high latency operations caused by off-chip DRAM. Therefore, the latency of reading metadata can typically be controlled to within ten cycles.

[0141] Step 907: If the buffer is hit, the data is returned directly from the buffer.

[0142] Step 908: If a buffer miss occurs, the DRAM Cache triggers a data loading process to load data from the memory pool.

[0143] Step 909: Load the data directly into the DRAM cache.

[0144] If the memory pool pre-allocated memory blocks to the host are insufficient, the runtime requests more memory blocks from the memory pool management module via the CXL switching device. The pre-allocated memory blocks in the memory pool may have different granularities (e.g., 512B, 128B). The runtime ensures that the selected memory block matches the requested granularity (512 bytes).

[0145] Throughout the process, the memory blocks (512 bytes) used for program allocation and the size of data access requests remain consistent with the granularity of the Embedding Table. This granular design ensures efficient memory allocation and data access, reduces latency, and improves DRAM cache space utilization.

[0146] Figure 10 This is a schematic diagram of a cache access method provided in an embodiment of this application. For example... Figure 10 As shown, this embodiment can be applied to DRAM cache. The cache access method includes at least the following steps:

[0147] Step 1001: Receive a memory access request, the request including the size of the data to be accessed.

[0148] In some possible implementations, before receiving memory access requests, the process includes receiving memory allocation requests: the request includes the size of the data blocks to be allocated, specifically the number of memory blocks (data blocks) and the size of each memory block; based on the number and size of the memory blocks, one or more granular regions are matched in the memory pool. The memory pool is divided into several memory regions, and each memory region is further divided into smaller memory blocks (granular regions, buffers) according to a certain granularity, with all memory blocks within the same memory region having the same granularity.

[0149] Based on the requested number and size of memory blocks, a granular region is matched in the memory pool, where the sum of the sizes of the granular regions is not less than the required data size. A virtual address-to-physical address mapping is performed on the found granular regions; the physical address is the physical location of the corresponding granular region. The mapped virtual address is then returned to the application.

[0150] In some possible implementations, the runtime allocates memory blocks of the appropriate granularity from the memory pool, including: based on the required data size, first searching the memory pool for the smallest granularity memory block that matches the size; wherein, when selecting a granularity memory block, the granularity memory block whose location is closest to the memory region of the request initiator is selected; if the size of a single granularity memory block cannot meet the required data size, then multiple granularity memory blocks are searched in the memory pool, and the total size of the selected memory blocks is not less than the required data size.

[0151] For example, in a memory pool, each memory block has a different granularity. Granularity is the smallest unit of memory allocation; for example, there might be memory blocks with different granularities such as 512B or 128B. The runtime first attempts to find the smallest granularity that matches the size of the requested data. Different memory blocks in the memory pool may reside in different memory regions. For instance, if a memory region is closer to the host that initiated the request, selecting memory blocks from that region can reduce latency caused by long-distance access.

[0152] For example, if the smallest granularity memory block found is not large enough to satisfy the data size requested by the application (e.g., the application requests 7 KiB of data, but the largest granularity memory block found at runtime is only 4 KiB), the request will be satisfied by combining memory blocks of multiple granularities.

[0153] Step 1002: Determine the corresponding granularity cache region in the DRAM cache according to the size of the data to be accessed; the granularity cache region is obtained by dividing the DRAM cache region according to different granularity sizes.

[0154] In some possible implementations, the runtime in the host identifies each memory region in the memory pool and determines the different granularities supported by each memory region; the granularity information of each memory region and its corresponding physical address range are written into the DRAM cache to indicate the granularity configuration of each granularity cache region in the DRAM cache.

[0155] For example, the granularity information and corresponding physical address range of each memory region are written into the configuration register of the DRAM cache. After the host starts up, the host runtime sets the configuration register according to the granularity information and physical address range of each memory region in the memory pool. The configuration register includes: granularity information, indicating the granularity size of each memory region, which may be 512B, 1KiB, 4KiB, etc.; and physical address range: the start and end physical addresses of each memory region. The physical address range in the configuration register ensures that memory access can accurately select the granular cache region. In this way, subsequent data access can directly locate the corresponding physical address range. When the application initiates a memory access request, the DRAM cache controller matches and schedules the request according to the granularity information and physical address range stored in the configuration register. Specifically, the data size in the memory access request is compared with the information in the configuration register to determine the granular cache region to be accessed and optimize the data loading process.

[0156] In some possible implementations, each granularity cache region is implemented using a multi-way set mapping structure; each granularity cache region includes multiple cache groups, where each cache group includes multiple cache blocks, and the size of each cache block is the granularity size of the corresponding granularity cache region. A cache block is the basic storage unit in the cache. Under the multi-way set mapping structure, the size of the cache block corresponds to the granularity of the granularity cache region.

[0157] In some possible implementations, each granularity cache region can also be implemented using a fully associative mapping structure, a direct mapping structure, etc.

[0158] In some possible implementations, the cache block identifier is stored in on-chip memory to keep the latency of reading the identifier within ten cycles. To optimize cache access performance, the cache block identifier (tag) is stored in on-chip memory. On-chip memory offers higher access speeds and lower latency compared to off-chip memory. This design allows the cache block identifier to be read within ten cycles, enabling rapid determination of cache hits. This not only improves cache lookup efficiency but also reduces the additional waiting time that may occur during cache lookups, providing better performance support for high-frequency memory access scenarios.

[0159] The following details the memory pool's storage allocation strategy. At startup, the memory pool pre-allocates some memory space to the host to satisfy common memory requests. When the runtime receives a new allocation request, it first checks the granularity of the request and searches the space pre-allocated to the host in the memory pool for a matching buffer. If a matching buffer is found, it further verifies its existence in the allocated space; if it exists, it is used directly. If no suitable buffer is found, the runtime dynamically requests a new buffer from the memory pool's storage space to ensure the application's request is satisfied.

[0160] When an application issues a memory request, such as alloc(n,m), the runtime selects a buffer of appropriate granularity from the buffers of each Region. In alloc(n,m), n represents the number of memory blocks required, and m represents the size (granularity) of each memory block. Based on the granularity of m, a buffer of appropriate granularity is searched in the memory pool for allocation.

[0161] First, calculate the requested granularity m, and then round it up to the nearest size that matches the system granularity; for example, if m is 3.75 KiB, round it up to a 4 KiB granularity.

[0162] Search the memory pool for a buffer granularity that matches m (rounded down). If a buffer matching the granularity is found, and the memory pool storage area pre-allocated to the host also includes buffers of that granularity, then allocate n buffers of that granularity directly.

[0163] n=1 indicates that only a buffer of one granularity is allocated. When the application issues an alloc(1,m) request, it means that it only needs a memory block of size m.

[0164] If n>1, it means that multiple buffers with the same granularity are allocated. When an application issues an alloc(n,m) request, where n>1, it means that it needs multiple memory blocks of size m.

[0165] If there is no buffer in the memory pool that matches the rounded m-granularity, select the largest available buffer and combine multiple smaller buffers to satisfy the request.

[0166] For example, the number of buffers in the memory pool is:

[0167] Region 0 (1 KiB granularity) is divided into 100 1 KiB buffers.

[0168] Region 1 (128B granularity) is divided into 200 128B buffers.

[0169] Region 2 (4KiB granularity) is divided into 50 4KiB buffers.

[0170] Region 3 (512B granularity) is divided into 150 512B buffers.

[0171] When an application requests a 2.5KiB memory block, the requested data size is rounded up to a size that matches a preset granularity. Since there is no directly matching 2.5KiB buffer, it is rounded up to a 4KiB granularity, as this is the closest available granularity to 2.5KiB. A buffer with a 4KiB granularity is searched in the memory pool. A buffer with a 4KiB granularity is found in Region 2, which meets the requirement. Further checks are performed to see if the buffer has been pre-allocated to the host. If the 4KiB buffer has been pre-allocated to the host, it is used directly to fulfill the request. If the buffer has not been pre-allocated to the host, meaning it is not found in the pre-allocated storage area, a new 4KiB buffer is dynamically requested and allocated from the memory pool. Then, a virtual address space is allocated for the 4KiB buffer, and a mapping relationship between virtual and physical addresses is established in the page table.

[0172] For example, when an application makes four 512B memory requests (i.e., alloc(4, 512)), the operating system first allocates virtual address space for these four 512B elements, reserving four contiguous 512B address regions within the virtual address space, totaling 2KiB of virtual address space. It then searches the memory pool for memory regions with a 512B granularity. Region 3 is found to be a 512B granularity region. The runtime then finds four free 512B buffers in Region 3 and allocates them to the request. Based on the found buffers, a mapping is established for each 512B virtual address segment in the virtual address space: the first 512B region in the virtual address space is mapped to the first buffer, the second 512B virtual address region is mapped to the second physical buffer, and so on, until the mapping of all four elements is completed. The page table records the mapping relationship between each virtual address and physical address, ensuring that the application can correctly translate virtual addresses to physical addresses.

[0173] When an application accesses any 512B data block, the runtime looks up the corresponding physical address using the page table. For example, when the application accesses the virtual address of the third element, the page table will point to the third 512B buffer in Region 3, ensuring that the data can be correctly read or written to physical memory.

[0174] In some possible implementations, the region with lower latency is selected from those regions that satisfy part or all of the data access size. For example, when multiple 1KiB buffers with the same granularity can satisfy the data access requirements, the runtime will choose the buffer with lower latency. For instance, if there are three 1KiB buffers in Region 0 (located at physical locations 0x0000_1000, 0x0000_2000, and 0x0000_3000 respectively), and the physical location of 0x0000_1000 is closer to the host, the runtime will preferentially choose the buffer at that location to complete the request faster.

[0175] The method provided in this application achieves precise and dynamic cache matching by dividing storage areas and managing caches in the memory pool based on granularity, thus satisfying the granularity of different application requests. When an application requests memory access, a buffer of appropriate granularity is matched in the memory pool according to the size of the requested data, realizing the mapping from virtual address to physical address. Through the granularity identification mechanism of DRAM Cache, the cache hit rate is effectively improved, and data waste in the cache is reduced. Even when a cache miss occurs, the required data can be loaded from the memory pool and stored in the corresponding granularity cache area, thereby providing efficient data access performance and stable low-latency response.

[0176] Figure 11 This is a schematic diagram illustrating a memory allocation scheme provided in an embodiment of this application. Figure 11 As shown,

[0177] The memory pool is pre-divided into different Regions based on different granularities. Each Region stores a buffer of a specific size to accommodate memory requests of different granularities.

[0178] Region 0: 1 KiB granularity, divided into 100 1 KiB buffers. Several bits in the physical address of each 1 KiB buffer contain specific flag bits to identify that these buffers belong to the 1 KiB granularity.

[0179] Region 1: 512B granularity, divided into 150 512B buffers. Several bits in the physical address of each 512B buffer contain identifier bits indicating that these buffers belong to the 512B granularity.

[0180] Region 2: 128B granularity, divided into 200 128B buffers. Several bits of the physical address of each 128B buffer contain corresponding flag bits indicating that these buffers belong to the 128B granularity.

[0181] Region 4: 4KiB granularity, divided into 50 4KiB buffers. Several bits of the physical address of each 4KiB buffer contain corresponding flag bits indicating that these buffers belong to the 4KiB granularity.

[0182] When the host is running, it requests several buffers of each of the above granularities from the memory pool management module of the memory pool.

[0183] Application requests are divided into two phases, which will be explained below. Memory request phase:

[0184] The application requests memory via alloc(1, 8.5KiB), initiating a memory allocation request of 8.5KiB to the runtime. The runtime confirms the request size; since the 8.5KiB request exceeds the size of a single 4KiB buffer, the runtime combines buffers from multiple Regions.

[0185] Select two adjacent 4-kiB buffers from Region 4 to meet the data requirement of the first 8-kiB. Assume the physical addresses are 0x1000_4000 to 0x1000_4FFF and 0x1000_5000 to 0x1000_5FFF.

[0186] Select a 512B buffer from Region 1 to satisfy the remaining 512B of data requirement. Assume the physical addresses are 0x1000_6000 to 0x1000_61FF.

[0187] Once the physical address is determined at runtime, a virtual address region is allocated from the application's allocated virtual address space, and a mapping relationship is established between it and the physical address. This virtual address region is dynamically allocated by the runtime based on the current memory allocation requirements to meet the 8.5 KiB requirement (e.g., starting address 0x3000_0000).

[0188] Page table entries create mappings that map virtual address pages to allocated physical buffers, as follows:

[0189] Buffer 1: Maps virtual addresses 0x3000_0000 to 0x3000_0FFF to physical addresses 0x1000_4000 to 0x1000_4FFF.

[0190] Buffer 2: Maps virtual addresses 0x3000_1000 to 0x3000_1FFF to physical addresses 0x1000_5000 to 0x1000_5FFF.

[0191] Buffer 3: Maps virtual addresses 0x3000_2000 to 0x3000_21FF to physical addresses 0x1000_6000 to 0x1000_61FF.

[0192] After the mapping is completed, the application can directly read and write 8.5 KiB of data continuously through the virtual address 0x3000_0000.

[0193] Memory access phase: During actual memory access, the application can transparently access data through the allocated virtual address.

[0194] When an application accesses 8.5 KiB of data starting at address 0x3000_0000, the runtime works with the page table to obtain the physical address mapped from the virtual address.

[0195] The first two 4KiB pages are mapped to the physical address spaces 0x1000_4000 and 0x1000_5000. The final 512B of data is mapped to the physical address region 0x1000_6000. At runtime, the mapped physical addresses, along with the data request, are passed to the DRAM cache. The DRAM cache uses the granularity of the physical address to determine the cache unit size.

[0196] For the address ranges 0x1000_4000 and 0x1000_5000, the DRAM Cache is identified as 4KiB granularity.

[0197] For the 0x1000_6000 address range, the DRAM cache is identified as 512B granularity.

[0198] The following are several scenarios:

[0199] 1. If the data is accessed for the first time and is not yet in the DRAM Cache, the DRAM Cache will handle the initial load of the data according to the following steps: Since the cache does not store the requested 4KiB and 512B data blocks, the DRAM Cache will read the data from the memory pool at these physical addresses. Based on the identified granularity information, the DRAM Cache loads the 4KiB and 512B data blocks respectively, allocates cache space for each data block, and stores the corresponding 4KiB and 512B data in the cache block. During the allocation process, the DRAM Cache sets a tag for each cache block. Granularity information is also stored in the control field of the cache block during allocation, enabling subsequent accesses to identify the size of the cache block.

[0200] 2. The DRAM cache has already cached the situation where this data is read again:

[0201] When the application accesses the address ranges 0x1000_4000, 0x1000_5000, and 0x1000_6000 again, DRAMCache will first check whether the data at these addresses has already been cached.

[0202] DRAM cache determines the size of each data block based on the granularity marker in the physical address. For example:

[0203] 0x1000_4000 and 0x1000_5000 are 4 KiB granularity.

[0204] 0x1000_6000 represents a 512B granularity.

[0205] The DRAM cache uses a tag field to check whether the required data block has already been stored in each cache line:

[0206] At this point, the labels for addresses 0x1000_4000 and 0x1000_5000 are already in the cache, indicating that these two 4KiB data blocks have been cached, so the cached data is read directly from the cache.

[0207] Similarly, the existence of the label for the 512B data block 0x1000_6000 indicates that the 512B granular data has also been cached, and the data in the cache was read.

[0208] For a hit data block, the DRAM cache reads the required data from the cache line and returns it directly to the application:

[0209] For 4KiB data blocks of 0x1000_4000 and 0x1000_5000, the DRAM cache reads directly from the cache line and returns the entire 4KiB data block. For 4KiB data blocks of 0x1000_6000, the DRAM cache reads only the 512B data block and returns it.

[0210] In another possible implementation, the application issues a data request specifying that it needs 8.5 KiB of data. The runtime receives the allocation request and confirms that the data size is 8.5 KiB. Since there is no buffer with a granularity of 9 KiB in the memory pool to satisfy the request, the memory block to be accessed is split into multiple smaller buffers for allocation.

[0211] To verify that the method provided in this application can improve the cache space utilization of DRAM cache, data access was performed using 4KiB fixed granularity and 4kiB dynamic granularity, and the cache space utilization was observed in each case. Figure 12This is a diagram illustrating the percentage of cache space utilization when using fixed-granularity access to DRAM cache in related technologies. Figure 13 This diagram illustrates the percentage of cache space utilization using dynamic granular access to the DRAM cache, as provided in an embodiment of this application. Figure 12 and Figure 13 As shown,

[0212] This application's embodiments statistically analyze the probability of accessing data by several mainstream applications, namely: CG - Conjugate Gradient (linear solver for computational fluid dynamics), BT - Block Tri-diagonal (block tridiagonal matrix solver), IS - Integer Sort (integer sorting algorithm), BHS - Blackhole Simulation (black hole simulation), and SP - Scalar Pentadiagonal (scalar pentadiagonal matrix solver).

[0213] UA - Unstructured Adaptive, PR - PageRank, VGG - Visual Geometry Group (image classification model, commonly used in deep learning), LU - Lower-Upper Symmetric Gauss-Seidel (LU decomposition, a method for solving linear equations), MG - MultiGrid (multigrid algorithm, commonly used for solving partial differential equations), CANU - Canu (genome assembly tool), LV - LeViT: A Vision Transformer in ConvNet's Clothing for Faster Inference, MC - Monte Carlo (Monte Carlo method, commonly used in probabilistic simulations).

[0214] In scenarios with fixed-granularity 4KiB access, cache space is typically allocated in fixed granularity (e.g., 4KiB pages). When an application reads data, regardless of the actual amount of data needed, the entire 4KiB page is loaded from the memory pool into the DRAM cache each time. This fixed-granularity caching method results in significant space waste when dealing with fluctuating access demands.

[0215] Each time an application requests data, the system loads a 4KiB page into the DRAM cache. To accurately analyze the actual usage of the cache space, we further divide each 4KiB page into 16 smaller blocks of 256B each to track more granular access patterns.

[0216] During the test, each time the application accessed data, as long as it accessed any 256B block in the 4KiB page, that block was marked as "accessed".

[0217] During multiple loading and accesses of the 4KiB page, the number of small blocks accessed each time was counted. Based on the utilization statistics of each access, the proportion of small blocks actually accessed in different block ranges (1-4, 5-8, etc.) was recorded.

[0218] 1-4 blocks: This indicates that within the loaded 4KiB page, the application accessed any 1 to 4 256B blocks.

[0219] 5-8 blocks: This indicates that within the loaded 4KiB page, the application accessed any 5 to 8 256B blocks.

[0220] 9-12 blocks: This indicates that within the loaded 4KiB page, the application accessed any 9 to 12 256B blocks.

[0221] 13-16 blocks: This indicates that within the loaded 4KiB page, the application accessed any 13 to 16 256B blocks.

[0222] Statistical results show that in the fixed-granularity caching mode, in most cases, only 1-4 256B blocks of the 4KiB page loaded into the DRAM cache are actually accessed by the application. In other words, in most scenarios, the application only needs a small part of the 4KiB page (1-4 256B blocks, about 256B to 1KiB), and the rest (12-15 blocks, accounting for 3-3.75KiB) is not accessed, but still occupies cache space.

[0223] This result indicates that applications often only use a few small chunks each time a 4KiB page is loaded, leaving most of the cache space idle and inefficiently utilized. Applications don't need the entire 4KiB page, but only a small portion of it. Because the DRAM cache loads pages in fixed 4KiB granularities, many unaccessed chunks are wasted in the cache, consuming valuable cache space and reducing cache utilization efficiency.

[0224] After using the dynamic granularity caching method provided in this application embodiment, the DRAM cache can dynamically adjust the cache granularity according to the actual data size requested by the application, instead of caching at a 4KiB page granularity. Test results show:

[0225] With dynamic granularity caching, more small blocks are actually accessed in each 4KiB page accessed by the application. The access rate of 1-4 small blocks decreases significantly, while the access frequency of 5-8 / 9-12 / 13-16 small blocks increases significantly. This indicates that with dynamic granularity caching, a larger proportion of the data loaded by the application is actually used each time, rather than just accessing 1-4 small blocks. This change shows that cache space is utilized more fully, waste is reduced, and the effective storage efficiency of DRAM cache is improved. The dynamic granularity caching method effectively reduces the number of unused small blocks in DRAM cache, significantly improving cache space utilization.

[0226] Because each loaded data block is closer to the actual application requirements, the amount of invalid data in the DRAM cache is significantly reduced, and the proportion of valid data in the cache is higher. The data loaded by the application is utilized more fully, reducing cache waste. Through dynamic granularity caching, the DRAM cache can more flexibly adapt to the application's access needs. Compared to fixed-granularity methods, dynamic granularity caching significantly improves cache utilization, resulting in more efficient use of storage space, reduced storage waste, and lower access latency.

[0227] See Figure 14 This application provides a schematic diagram of the structure of a cache 1400. For example... Figure 14 As shown, the cache 1400 includes: a request receiving module 1410, used to receive memory access requests, the requests specifying the size of the data to be accessed; and a granularity matching module 1420, which determines the corresponding granularity cache region in the DRAM cache according to the data size; the granularity cache region is obtained by dividing the DRAM cache region according to different granularity sizes.

[0228] See Figure 15 This application provides a schematic diagram of the structure of a memory pool-based cache management system 1500. For example... Figure 15 As shown, the cache management system 1500 includes:

[0229] The memory pool management module 1510 manages the storage space in the memory pool, pre-dividing the storage space into multiple storage regions, each of which is divided into memory blocks of the same size according to a preset granularity. The runtime module 1520 receives memory allocation requests, allocates memory blocks of the corresponding granularity from the memory pool according to the data size, performs virtual address to physical address mapping on the found memory blocks of the corresponding granularity, where the physical address is the physical location of the memory block of the corresponding granularity, and returns the mapped virtual address to the application.

[0230] The DRAM cache module 1530 is used to receive memory access requests, which specify the size of the data to be accessed; based on the data size, it determines the corresponding granularity cache region in the DRAM cache; the granularity cache region is obtained by dividing the DRAM cache region according to different granularity sizes; it selects a cache block in the granularity cache region and checks whether the accessed data matches the identifier of the cache block.

[0231] This application also provides a computing device 1600. For example... Figure 16 As shown, the computing device 1600 includes a bus 1602, a processor 1604, a memory 1606, and a communication interface 1608. The processor 1604, the memory 1606, and the communication interface 1608 communicate with each other via the bus 1602. The computing device 1600 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 1600.

[0232] Bus 1502 can be a Peripheral Component Interconnect (PCI) bus or a Compute Express Link (CXL) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 16 The bus 1602 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 1602 may include a path for transmitting information between various components of the computing device 1600 (e.g., memory 1606, processor 1604, communication interface 1608).

[0233] Processor 1604 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0234] Memory 1606 may include volatile memory, such as random access memory (RAM). Memory 1506 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0235] The memory 1606 stores executable program code, and the processor 1604 executes this executable program code to implement the functions of the aforementioned request receiving module 1410 and granularity matching module 1420, thereby achieving... Figure 7 The method shown. That is, the memory 1606 stores the information for executing... Figure 5 The instructions for the method shown.

[0236] The communication interface 1608 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 1600 and other devices or communication networks.

[0237] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform... Figure 7 The method shown.

[0238] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a host migration device such as a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute... Figure 7 The method shown.

[0239] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cache access method, characterized in that, The method includes: Receive a memory access request, the request including the size of the data to be accessed; The granularity of the cache region in the DRAM cache is determined based on the size of the data to be accessed; the granularity of the cache region is obtained by dividing the DRAM cache region according to different granularity sizes.

2. The method according to claim 1, characterized in that, Before an application initiates a memory access request, it first sends a memory allocation request, which includes the size of the data to be allocated. Based on the required data size, the runtime allocates memory blocks of appropriate granularity from the memory pool. The memory pool is divided into multiple memory regions, and each memory region is divided into memory blocks of different granularities. Each memory block has a specific granularity. The sum of the sizes of the memory blocks is not less than the required data size. Perform virtual address to physical address mapping on the found memory blocks of the corresponding granularity; the physical address is the physical location of the memory block of that granularity. The mapped virtual address is returned to the application.

3. The method according to claim 2, characterized in that, Based on the required data size, determine the corresponding granularity cache region in the DRAM cache, including: Memory access is performed via physical addresses; the physical addresses are obtained by mapping the virtual addresses. Based on the granularity flag in the physical address, the DRAM cache determines the granularity required for the memory access request; Based on the granularity, determine the granularity cache region in the DRAM cache; Perform a cache block lookup within the defined granularity cache region.

4. The method according to claim 3, characterized in that, If no cache block identifier is found that matches the data to be accessed, Based on the data size of the memory access request, determine the size of the memory block to be loaded. Based on the granular memory block size corresponding to the data size and the mapped physical address, locate the granular memory block in the memory pool; Load the located granular memory block into the granular cache area of ​​the DRAM cache.

5. The method according to claim 2, characterized in that, Also includes: At runtime, each memory region in the memory pool is identified, and the different granularities supported by each memory region are determined. The granularity information of each memory region and its corresponding physical address range are written into the DRAM cache to indicate the granularity configuration of each granularity cache region in the DRAM cache.

6. The method according to claim 2, characterized in that, At runtime, memory blocks of appropriate granularity are allocated from the memory pool, including: Based on the required data size, first search the memory pool for the smallest granularity memory block that matches that size; when selecting the granularity memory block, select the granularity memory block whose location is closest to the memory region that initiated the request; If the size of a single granular memory block is insufficient to meet the required data size, multiple granular memory blocks are searched in the memory pool, and the total size of the selected memory blocks is not less than the required data size.

7. The method according to claim 1, characterized in that, The identifier of the cache block is stored in on-chip memory so that the latency of reading the identifier is controlled within ten cycles.

8. The method according to claim 1, characterized in that, Also includes: Each granularity cache region is implemented through a multi-way group mapping structure; each granularity cache region includes multiple cache groups, where each cache group includes multiple cache blocks, and the size of each cache block is the granularity size of the corresponding granularity cache region.

9. A DRAM cache, characterized in that, include: The request receiving module is used to receive memory access requests, wherein the request specifies the size of the data to be accessed; The granularity matching module determines the corresponding granularity cache region in the DRAM cache based on the data size; the granularity cache region is obtained by dividing the DRAM cache region according to different granularity sizes.

10. A memory pool-based cache management system, characterized in that, include: The memory pool management module is used to manage the storage space in the memory pool and pre-divide the storage space into multiple storage areas. The runtime module is used to request storage areas from the memory pool and divide the storage areas into cache areas of different granularities; Upon receiving a memory allocation request, allocate a memory block of appropriate granularity from the memory pool based on the data size, and perform virtual address to physical address mapping on the found memory block of appropriate granularity; the physical address is the physical location of the memory block of appropriate granularity; return the mapped virtual address to the application. A DRAM cache module is used to receive memory access requests, the requests specifying the size of data to be accessed; based on the data size, it determines the corresponding granularity cache region in the DRAM cache; the granularity cache region is obtained by dividing the DRAM cache region according to different granularity sizes; a cache block is selected in the granularity cache region, and it is checked whether the accessed data matches the identifier of the cache block.

Images