Methods for handling memory commands in computing systems

Abstract

A method for processing memory commands in a computing system includes: transmitting a plurality of subcommands from a consistency center to a downstream cache subsystem, each subcommand corresponding to a cache line size; receiving at least one of the plurality of subcommands in the downstream cache subsystem; determining, in response to receiving the at least one subcommand, a complete address range of a data block by the downstream cache subsystem; and performing optimized memory operations for the downstream memory subsystem by the downstream cache subsystem.

Description

[Technical Field] This invention relates generally to computing system architecture, and more specifically to methods and systems for improving memory access efficiency in cache-consistent systems. [Background Technology] In modern System-on-a-Chip (SoC) architectures, multiple processing units, such as the central processing unit (CPU), graphics processing unit (GPU), neural processing unit (NPU), and media processing unit (MPU), typically share a common main memory, such as dynamic random-access memory (DRAM). To ensure data integrity within the system, cache coherency mechanisms are usually employed. This is managed by the hardware system, such as a coherency infrastructure.

[0003] The fundamental operation of a consistency infrastructure is maintaining consistency with the CPU cache. The CPU cache is managed in fixed-size blocks called cache lines, typically 64 bytes. Therefore, when a requester (such as a GPU, NPU, or MPU) requests a large block of data (i.e., a "long burst" access) larger than 64 bytes, the consistency infrastructure must break the original long burst command into multiple smaller subcommands, each aligned to a 64-byte boundary. This splitting is performed so that the consistency infrastructure can perform consistency checks on a per-cache-line basis, verifying that the data in each 64-byte segment resides in the CPU cache.

[0004] However, this process presents significant challenges to system performance. After the original memory access commands are split and processed by the consistency center, the corresponding subcommands are affected by internal processing logic, such as state machines and bus arbitration. As a result, subcommands often become out of order, losing their original sequential relationship. There is no guarantee which subcommand will exit the consistency center first. Therefore, when downstream memory controllers receive these 64-byte subcommands, they treat them as a series of small and unrelated requests.

[0005] This loss of context prevents the memory controller from reconstructing the original long burst memory access. Unable to recognize the original access intent, the memory controller cannot perform efficient, sequential long burst accesses to memory. Instead, it must issue multiple independent accesses, which reduces memory efficiency and fails to utilize data locality.

[0006] Therefore, a technical mechanism is needed to overcome the inefficiency caused by command splitting in a consistent infrastructure, thereby improving the performance of system memory for long burst accesses. [Summary of the Invention] In view of the above, one objective of the present invention is to provide a method and system for processing memory commands that overcomes the aforementioned disadvantages of the prior art and achieves efficient long burst memory operations by preserving the context of the original memory access intent within a consistent infrastructure. Specifically, embodiments of the present invention rely on a sequence hint indicating the size of the data block associated with the original long burst memory access command. This sequence hint is appended to each subcommand split from the original memory access command, thereby enabling a downstream caching subsystem to perform optimized memory operations. In some embodiments, the optimized memory operation may include a read prefetch operation, wherein the downstream caching subsystem proactively reads the full address range of the data block from system memory. In some embodiments, the optimized memory operation may include a write merging operation, wherein the downstream caching subsystem accumulates data from multiple write subcommands and performs a single write operation to system memory to write the accumulated data.

[0008] According to one embodiment of the present invention, a method for processing memory commands in a computing system is provided. The method includes: transmitting a plurality of subcommands from a consistency center to a downstream cache subsystem, each of the plurality of subcommands corresponding to a cache line size; receiving at least one of the plurality of subcommands in the downstream cache subsystem; determining, by the downstream cache subsystem, the complete address range of a data block in response to receiving the at least one subcommand; and performing optimized memory operations for the downstream memory subsystem by the downstream cache subsystem.

[0009] According to one embodiment, a computing system is provided. The computing system includes: a consistency center and a downstream cache subsystem. The consistency center is configured to transmit multiple subcommands, each subcommand corresponding to a cache line size. The downstream cache subsystem is configured to receive at least one subcommand, and in response to receiving the at least one subcommand, determine the complete address range of a data block; and perform optimized memory operations for a downstream memory subsystem.

[0010] These, and other objectives of the invention, will become apparent to those skilled in the art upon reading the following detailed description and accompanying drawings. [Attached Image Description] Brief description of the drawings Figure 1 A block diagram of a computing system according to an embodiment of the present invention is shown.

[0012] Figure 2A memory access command processing procedure according to an embodiment of the present invention is illustrated.

[0013] Figure 3 A memory access command processing procedure according to another embodiment of the present invention is shown.

[0014] Figure 4 This is a table illustrating an example encoding scheme for the sequence hint bits according to an embodiment of the present invention.

[0015] Figure 5 The diagram illustrates the flow of an end-to-end data path for memory access commands processed by a consistency center and a downstream caching subsystem, according to an embodiment of the present invention.

[0016] Figure 6 A sequence diagram of an example write-merge operation according to an embodiment of the present invention is shown.

[0017] Figure 7 A sequence diagram of an example read prefetch operation according to an embodiment of the present invention is shown.

[0018] Figure 8 A sequence diagram of an example read-prefetch operation in a partial probe hit scenario is shown according to an embodiment of the present invention.

[0019] Figure 9 A flowchart illustrating a method for processing memory commands in a computing system according to an embodiment of the present invention is shown.

Detailed Implementation Methods

[0021] Throughout this specification, the phrase "one embodiment" or "one example" means that a particular feature, structure, or characteristic associated with that embodiment or example is included in at least one embodiment of the present invention. Therefore, the phrases "in one embodiment" or "in one example" appearing in different places in the specification do not necessarily refer to the same embodiment. Furthermore, a particular feature, structure, or characteristic may be combined in any suitable combination and / or sub-combination in one or more embodiments.

[0022] In the following detailed description, the terms "upstream cache subsystem," "downstream cache subsystem," and "system memory subsystem" are used to denote components based on their relative position in the memory hierarchy. While specific examples in the specification may associate these terms with "CPU cache," "system-level cache (SLC)," and "main memory," respectively, it should be understood that this is for illustrative purposes only and is not restrictive. As the adjectives "upstream" and "downstream" imply, these terms describe a relative relationship. "Upstream" generally refers to a hierarchy level closer to the CPU, while "downstream" refers to a hierarchy level further away from the CPU and closer to the final data repository. Therefore, those skilled in the art will understand that the upstream cache subsystem, downstream cache subsystem, and system memory subsystem may correspond to different types or levels of cache and memory in various alternative architectures.

[0023] Figure 1 A block diagram of a computing system with an architectural environment according to an embodiment of the present invention is shown. As shown, the computing system may include a central processing unit 102 (CPU), a CPU cache 104 (which may serve as an upstream cache subsystem), a consistency center 110, a probe filter 120, a system-level cache (SLC) subsystem 130 (which may serve as a downstream cache subsystem, i.e., DCS), a main memory subsystem 140 (which may serve as both system memory and downstream memory subsystem, i.e., DMS), a graphics processing unit (GPU), a neural processing unit 160, and a media processing unit 170.

[0024] The consistency center 110 coordinates data flow between multiple request initiators and a hierarchical memory system consisting of CPU cache 104, SLC subsystem 130, and main memory subsystem 140. The request initiators are diverse processing units, each with different workload characteristics. They include GPU 150, NPU 160, and MPU 170.

[0025] CPU 102 serves as the system's general-purpose computing engine, executing operating system tasks and various applications. It is tightly coupled with CPU cache 104 in the hierarchical memory system and must maintain consistency with the rest of the computing system 100.

[0026] The GPU 150 is a processor specifically designed to accelerate graphics rendering and parallel computing. The NPU 160 is a processor specifically designed as a hardware accelerator for artificial intelligence (AI) and machine learning (ML) workloads. The MPU 170 is a processor specifically designed to handle media-related tasks such as video encoding / decoding, image processing, and audio processing.

[0027] All these request initiators share access to the hierarchical memory system. This shared access requires a consistency infrastructure to resolve data consistency issues that arise when multiple processors read from and write to the same memory location. In this architecture, the consistency infrastructure relies on a consistency hub 110, which is configured to facilitate communication between the request initiators and the memory hierarchy. The consistency hub 110 coordinates all data transfers and maintains data consistency across different components.

[0028] To optimize consistency, consistency center 110 is coupled with probe filter 120. Probe filter 120 can be a cache that tracks the state of cache lines in CPU cache 104. Its primary function is to mitigate the high-performance cost of broadcasting probes to CPU cache 104 to handle every memory access command from GPU 150, NPU 160, and MPU 170. By first checking probe filter 120, consistency center 110 can completely bypass the expensive upstream probing operation in the event of a "probe miss," indicating that the requested data is definitely not present in CPU cache 104.

[0029] It should be understood that Figure 1 The CPU cache 104 is simplified to a single block for illustration, but it is not limited to a single cache. In various embodiments, the CPU cache 104 can represent a multi-level cache hierarchy associated with the CPU 102. This cache hierarchy may include, for example, private L1 and L2 caches dedicated to each individual core within the CPU 102, and a shared last-level cache (L3 cache) accessible to all CPU 102 cores.

[0030] Figure 2 A memory access command processing procedure according to an embodiment of the present invention is illustrated. In this embodiment, the consistency center 110 processes and splits a long burst of 256-byte read commands.

[0031] As shown in the figure, the request initiator (which can be one of GPU 150, NPU 160, and MPU 170) first issues a single 256-byte read command to the consistency center 110, requesting a 256-byte data block. Once the consistency center 110 receives the 256-byte read command, it splits the command and performs a consistency check on the probe filter 120. Specifically, the consistency center 110 is responsible for breaking the 256-byte read command into four 64-byte read subcommands (labeled A, B, C, and D) to align with the cache line size corresponding to CPU cache 104. Then, the consistency center 110 performs a consistency check on each read subcommand to check the probe filter 120. During this process, the consistency center 110 further generates and appends a sequence hint to each 64-byte read subcommand A, B, C, and D by adding the sequence hint bit to specific fields of the 64-byte read subcommands A, B, C, and D.

[0032] Based on the consistency check response returned by probe filter 120, consistency center 110 can perform a read operation to access an upstream cache subsystem (e.g., CPU cache 104) when a probe hit occurs. Alternatively, in response to a probe miss, consistency center 110 can issue one or more 64-byte read subcommands to a downstream cache subsystem (e.g., SLC cache subsystem 130) or a downstream memory subsystem (e.g., main memory subsystem 140) to retrieve the data requested by the 256-byte read command.

[0033] If a read miss occurs downstream, the downstream caching subsystem will perform a single optimized read operation based on the original payload, with the help of sequence hints. For example, it will read 64 bytes of data for the requested data (requested by a single subcommand) and prefetch 192 bytes of data for the remainder of the 256-byte data block (hinted by sequence hints).

[0034] Figure 3 A memory access command processing procedure according to another embodiment of the present invention is illustrated. In this embodiment, multiple smaller commands are generated directly by the request initiator.

[0035] In this embodiment, the request initiator (which may be one of GPU 150, NPU 160, and MPU 170) directly generates four independent 64-byte read subcommands to achieve the intent of a 256-byte data block. Specifically, before issuing the four independent 64-byte read subcommands to the consistency center 110, the request initiator generates and appends order hints to each of the four 64-byte read subcommands. In this embodiment, consistency-related splitting is performed at the initiator level.

[0036] Once the consistency center 110 receives four pre-split 64-byte read subcommands, each carrying an order hint, the consistency center 110 does not need to perform a split operation. It only needs to perform a consistency check on the probe filter 120 to check each of the four 64-byte read subcommands.

[0037] Based on the consistency check response returned by probe filter 120, consistency center 110 can perform a read operation to access an upstream cache subsystem (e.g., CPU cache 140) when a probe hit occurs. Alternatively, in response to a probe miss, consistency center 110 can issue one or more 64-byte read subcommands to a downstream cache subsystem (e.g., SLC cache subsystem 130) or a downstream memory subsystem (e.g., main memory subsystem 140) to retrieve the data requested by the 256-byte read command.

[0038] If a read miss occurs downstream, the downstream caching subsystem will perform a single optimized read operation based on the original payload, with the help of sequence hints. For example, it will read 64 bytes of data for the requested data (requested by a single subcommand) and prefetch 192 bytes of data for the remainder of the 256-byte data block (hinted by sequence hints).

[0039] Figure 2 and Figure 3 The main difference between the two embodiments shown lies in which component is responsible for breaking down large memory access intentions into cache line-sized subcommands and generating sequence hints. In summary, both figures demonstrate the same result: multiple 64-byte subcommands, each carrying sequence hints, are passed downstream to optimize memory operations.

[0040] Figure 4This is a table illustrating an example encoding scheme for sequence hints according to an embodiment of the present invention. In this embodiment, 3-bit values ​​are used to represent various combinations of write / read commands within a 256-byte aligned data block. The 256-byte aligned data block is conceptually divided into four 64-byte blocks. The sequence hint bits precisely define the size and relative position of the original memory access command relative to the 256-byte aligned data block.

[0041] For raw memory access commands that are 64-byte read / write commands, the sequence hint bits are assigned to 0b000, regardless of their relative position within a 256-byte aligned data block.

[0042] For a raw memory access command that is a 128-byte read / write command, if it covers the first and second blocks within a 256-byte aligned data block, the sequence hint bit is assigned to 0b001. If the 128-byte read / write command covers the second and third blocks, the sequence hint bit is assigned to 0b010. If the 128-byte read / write command covers the third and fourth blocks, the sequence hint bit is assigned to 0b011.

[0043] For a raw memory access command that is a 192-byte read / write command, if it covers the first, second, and third blocks within a 256-byte aligned data block, the sequence hint bit is assigned to 0b100. On the other hand, if the 192-byte read / write command covers the second, third, and fourth blocks, the sequence hint bit is assigned to 0b101.

[0044] For raw memory access commands that are 256-byte read / write commands, the sequence hint bit is assigned to 0b110. Additionally, the sequence hint bit 0b111 is reserved for indicating error events or other undefined conditions.

[0045] This encoding scheme enables the consistency center 110 or the request initiator to efficiently pass the context of the original long burst memory access command to the downstream cache subsystem or downstream memory subsystem using a minimum number of bits. Upon receiving the subcommand with sequence hints, the downstream cache subsystem or downstream memory subsystem can accurately reconstruct the size and location of the original memory access command within its 256-byte aligned data block.

[0046] Please note that the above encoding scheme is for illustrative purposes only. According to various embodiments of the invention, sequence hints can be represented using fewer or more bits. Furthermore, in some embodiments, the encoding scheme for sequence hints may not be based on 256-byte alignment.

[0047] Figure 5This diagram illustrates an end-to-end data path for a memory access command, processed by a consistency center 110 and a downstream caching subsystem, consistent with one embodiment of the invention. The diagram details the internal stages of the consistency center 110.

[0048] First, when the receiver block (i.e., consistency receiver 111) within consistency center 110 receives a memory access command, which can be a read or write command with a payload size ranging from 1 to 256 bytes, the receiver block internally generates a sequence hint (i.e., Seq-hint) bit corresponding to the received memory access command according to a specific encoding scheme. The consistency checker block (i.e., consistency checker 112) splits the received memory access command into one or more 64-byte aligned subcommands for consistency checking. Furthermore, the generated sequence hint is appended to each 64-byte aligned subcommand.

[0049] After the consistency check, the transmitter block of the consistency center 110 (i.e., consistency transmitter 113) sends the result subcommands to the receiver block of the downstream cache subsystem (i.e., DCS receiver), with each subcommand carrying an order hint bit.

[0050] Upon receiving a subcommand carrying a sequence hint bit, the downstream caching subsystem performs optimized memory operations based on the additional sequence hint bit, such as read prefetch operations or write merge operations.

[0051] This flowchart provides a clear overview of the invention process, showing how memory access commands flow through the internal stages of the consistency center, where sequence hint bits are appended and they are broken down into subcommands, ultimately enabling the downstream caching subsystem to perform optimized memory operations.

[0052] Figure 6 A detailed sequence diagram is shown to illustrate an exemplary write merging operation in one embodiment of the invention. This embodiment demonstrates a scenario where a long burst write command causes all its subcommand probes to miss.

[0053] First, the request initiator of computing system 100 issues a write command with non-allocation attributes (e.g., a "write allocation" command) for writing 128 bytes of data to a target address of 0x00-0x7F, sent to consistency center 110 (operation a1). Upon receiving the 128-byte write command from the request initiator, consistency center 110 returns a write acknowledgment response to the request initiator (operation b1). For consistency checks, consistency center 110 splits the 128-byte write command into two 64-byte write subcommands. Consistency center 110 performs a first consistency check on the first 64-byte write subcommand, at addresses 0x00-0x3F, checking the probe filter 120 (operation c1), and a second consistency check on the second 64-byte write subcommand, at addresses 0x40-0x7F, checking the probe filter 120 (operation d1).

[0054] The probe filter 120 then returns a probe miss response to the consistency center 110 at addresses 0x00-0x3F, indicating that the data at addresses 0x00-0x3F does not exist in the upstream cache subsystem (operation e1). In response to the probe miss response at addresses 0x00-0x3F, the consistency center 110 issues the first 64-byte write subcommand to the downstream cache subsystem (i.e., DCS), with non-allocation attributes (addresses 0x00-0x3F) and carrying an order hint (operation f1).

[0055] The downstream caching subsystem receives the first 64-byte write subcommand. Based on sequence hints, the downstream caching subsystem understands that the payload data in the first 64-byte write subcommand is part of a long burst write operation. To perform merging, the downstream caching subsystem ignores the non-allocation attribute and performs a forced allocation operation, caching the payload data of the first 64-byte write subcommand for a later write merging operation (operation g1). Then, the downstream caching subsystem returns a write acknowledgment response at addresses 0x00-0x3F to the consistency center 110 (operation h1).

[0056] Subsequently, probe filter 120 returns a probe miss response at addresses 0x40-0x7F to consistency center 110, indicating that the data at addresses 0x40-0x7F does not exist in the upstream cache subsystem (operation i1). In response to the probe miss response at addresses 0x40-0x7F, consistency center 110 issues a second 64-byte write subcommand to the downstream cache subsystem, with non-allocation attributes (addresses 0x40-0x7F) and carrying an order hint (operation j1).

[0057] The downstream caching subsystem receives the second 64-byte write subcommand. Based on sequence hints, the downstream caching subsystem understands that the payload data in the second 64-byte write subcommand is another part of the aforementioned long burst write operation. To achieve merging, the downstream caching subsystem ignores the non-allocation attribute and performs a forced allocation operation, caching the payload data in the second 64-byte write subcommand for a later write merging operation (operation k1). Then, the downstream caching subsystem returns a write acknowledgment response at addresses 0x40-0x7F to the consistency center 110 (operation l1).

[0058] After a period of time, the downstream cache subsystem has accumulated the payload data of the first and second 64-byte write subcommands. The downstream cache subsystem then performs an optimized memory operation (i.e., a write-merge operation). It sends a single 128-byte write command to the downstream memory subsystem (DMS), writing the accumulated data to addresses 0x00-0x7F via a single long burst access (operation m1). Finally, after completing the 128-byte write command, the downstream memory subsystem returns a final write acknowledgment response corresponding to addresses 0x00-0x7F to the downstream cache subsystem (operation n1).

[0059] Figure 7 A sequence diagram is shown, detailing an exemplary read prefetch operation according to an embodiment of the present invention. This embodiment illustrates a scenario where a long burst read command results in all its subcommand probes missing.

[0060] First, the request initiator of computing system 100 sends a read command to consistency center 110, requesting a 256-byte data block with a target address of 0x00-0xFF (operation a2). Upon receiving the 256-byte read command from the request initiator, consistency center 110 performs a consistency check and splits the 256-byte read command into four 64-byte read subcommands. Specifically, consistency center 110 performs a first consistency check on the first 64-byte read subcommand (address 0x00-0x3F) against probe filter 120 (operation b2); a second consistency check on the second 64-byte read subcommand (address 0x40-0x7F) against probe filter 120 (operation c2); a third consistency check on the third 64-byte read subcommand (address 0x80-0xBF) against probe filter 120 (operation d2); and a fourth consistency check on the fourth 64-byte read subcommand (address 0xC0-0xFF) against probe filter 120 (operation e2).

[0061] Due to the out-of-order nature caused by state machine or bus arbitration, probe filter 120 first returns a consistency miss response corresponding to addresses 0xC0-0xFF to consistency center 110, indicating that the data corresponding to addresses 0xC0-0xFF does not exist in the upstream cache subsystem (operation f2). In response to the consistency miss response corresponding to addresses 0xC0-0xFF, consistency center 110 issues a fourth 64-byte read subcommand to the downstream cache subsystem, with addresses 0xC0-0xFF, carrying an order hint (operation g2).

[0062] The downstream caching subsystem receives the fourth 64-byte read subcommand. Based on the carried order hints, the downstream caching subsystem understands that the original request from the requester was for a 256-byte data block. Therefore, the downstream caching subsystem immediately performs an optimized memory operation (e.g., a read prefetch / read supplement operation) targeting addresses 0x00-0xFF by issuing a 256-byte read command to the downstream memory subsystem at addresses 0x00-0xFF (operation h2). This read prefetch / read supplement operation is initiated proactively by the downstream caching subsystem without waiting for the other three 64-byte read subcommands to be received. Subsequently, the downstream memory subsystem serves the 256-byte read command and returns the entire 256-byte data block to the downstream caching subsystem (operation m2).

[0063] Meanwhile, since the read supplement / read prefetch operation is transparent to the probe filter 120, the probe filter will sequentially return consistency miss responses corresponding to addresses 0x80-0xBF, 0x40-0x7F, and 0x00-0x3F to the consistency center 110, indicating that the data corresponding to the above addresses does not exist in the upstream cache subsystem (operations i2, k2, n2). In response to the consistency miss responses corresponding to addresses 0x80-0xBF, 0x40-0x7F, and 0x00-0x3F, the consistency center 110 will sequentially issue the third, second, and first 64-byte read subcommands for addresses 0x80-0xBF, 0x40-0x7F, and 0x00-0x3F, each carrying an order hint, to the downstream cache subsystem (operations j2, l2, o2).

[0064] When the downstream cache subsystem receives 256 bytes of read data (64 bytes on demand and 192 bytes prefetched) from the downstream memory subsystem, it can begin servicing a single 64-byte read subcommand received from the consistency center 110. That is, the downstream cache subsystem returns data to the consistency center 110 in 64-byte blocks, with the data sequence corresponding to addresses 0xC0-0xFF, 0xC0-0xFF (operation p2), 0x80-0xBF (operation q2), 0x40-0x7F (operation r2), and 0x00-0x3F (operation s2). This sequence corresponds to the order of the 64-byte read subcommands sent by the consistency center 110.

[0065] Finally, after the consensus center has collected all the requested data, it returns the entire 256-byte data block corresponding to addresses 0x00-0xFF to the request initiator, completing the process (operation t2).

[0066] Figure 8 A sequence diagram is shown, detailing an exemplary read prefetch operation in a partial probe hit scenario, according to an embodiment of the invention.

[0067] At the outset, the requester issues a read command to the consistency center 110 of computing system 100, requesting a 256-byte data block with a target address of 0x00-0xFF (operation a3). Upon receiving the 256-byte read command from the requester, the consistency center 110 performs a consistency check and splits the 256-byte read command into four 64-byte read subcommands. Specifically, the consistency center 110 performs a first consistency check on the first 64-byte read subcommand (corresponding to addresses 0x00-0x3F) and compares it with the probe filter 120 (operation b3); performs a second consistency check on the second 64-byte read subcommand (corresponding to addresses 0x40-0x7F) and compares it with the probe filter 120 (operation c3); performs a third consistency check on the third 64-byte read subcommand (corresponding to addresses 0x80-0xBF) and compares it with the probe filter 120 (operation d3); and performs a fourth consistency check on the fourth 64-byte read subcommand (corresponding to addresses 0xC0-0xFF) and compares it with the probe filter 120 (operation e3).

[0068] Due to out-of-order characteristics caused by state machine or bus arbitration, probe filter 120 first returns a probe miss response to consistency center 110 corresponding to address 0xC0-0xFF, indicating that the data corresponding to address 0xC0-0xFF is unavailable in the upstream cache subsystem (operation f3). In response to the probe miss response, consistency center 110 issues a fourth 64-byte read subcommand to the downstream cache subsystem, targeting address 0xC0-0xFF, and carrying an order hint (operation g3).

[0069] The downstream caching subsystem receives the fourth 64-byte read subcommand. Based on the carried order hints, the downstream caching subsystem understands that the original memory access command from the request initiator is for a 256-byte data block. Therefore, the downstream caching subsystem immediately executes an optimized memory operation (e.g., read prefetch / read fill operation) targeting addresses 0x00-0xFF by issuing a 256-byte read command to the downstream memory subsystem targeting addresses 0x00-0xFF (operation h3). Subsequently, the downstream memory subsystem processes the 256-byte read command and returns the entire 256-byte data block to the downstream caching subsystem (operation l3).

[0070] Meanwhile, since the data corresponding to addresses 0x80-0xBF, 0x40-0x7F, and 0x00-0x3F are available in the upstream cache subsystem, the probe hit response and the data corresponding to address 0x80-0xBF (operation i3), the probe hit response and the data corresponding to address 0x40-0x7F (operation j3), and the probe hit response and the data corresponding to address 0x00-0x3F (operation k3) will be returned to the consistency center 110, where the data is returned directly from the upstream cache subsystem.

[0071] Although the downstream cache subsystem receives the entire 256 bytes of read data from the downstream memory subsystem, it only returns the data block corresponding to addresses 0xC0-0xFF (operation m3) because only the data corresponding to addresses 0xC0-0xFF caused the probe to miss.

[0072] Finally, the Consistency Center 110 collects all the requested data from the four data blocks (i.e., three data blocks returned from the upstream cache subsystem due to probe hits and one data block returned from the downstream memory subsystem due to probe misses), assembles and returns the entire 256-byte data block corresponding to addresses 0x00-0xFF to the request initiator, completing the process (operation n3).

[0073] Because write / read commands may have specific attributes, such as allocation attributes (e.g., read allocation or write allocation), non-allocation attributes (e.g., read non-allocation or write non-allocation), and non-cached attributes, the following description explains how the downstream caching subsystem handles these various attributes.

[0074] For any command with non-cached attributes, the downstream caching subsystem ignores the order hints in the corresponding subcommand. Read / write commands with non-cached attributes are treated as simple demand accesses, and the aforementioned read prefetch / read fill and write merge operations will not be performed by the downstream caching subsystem.

[0075] For any read command with an allocation attribute (e.g., read allocation), the downstream caching subsystem considers and processes the sequence hints in the corresponding subcommand. A read command with an allocation attribute is treated as a long burst request when it arrives at the downstream caching subsystem. The downstream caching subsystem prefetches all data corresponding to the full address range indicated by the sequence hints and allocates the retrieved data to the downstream caching subsystem.

[0076] For any read command with a non-allocated attribute, the order hint in the corresponding read subcommand allows the downstream caching subsystem to override its non-allocated attribute. Due to the order hint, read subcommands with non-allocated attributes will be forced by the downstream caching subsystem to be treated as read subcommands with allocated attributes. However, the corresponding allocated cache lines in the downstream caching subsystem will be marked with a low-priority attribute, indicating that they should be the first candidates to be evicted (preferred eviction). This allows the downstream caching system to perform read prefetching without wasting downstream cache memory by storing data that is not intended to be stored long-term.

[0077] For any write command with an allocation attribute, the downstream caching subsystem does not perform any additional special processing on its sequence hints.

[0078] For any write command with a non-allocation attribute, the order hint in the associated read subcommand allows the downstream caching subsystem to override its non-allocation attribute. Similar to handling write commands with non-allocation attributes, the order hint in write subcommands with non-allocation attributes allows the downstream caching subsystem to override its non-allocation attribute and perform a write merge operation accordingly on the write subcommand. Furthermore, the corresponding allocated cache lines in the downstream caching subsystem are marked with a low-priority attribute, indicating that they should be evicted first (preferred eviction). In some embodiments, an additional non-discardable attribute can be applied to the allocated cache lines. This non-discardable attribute ensures that data, once allocated in the cache for merge purposes, is not discarded until successfully written back to the downstream memory subsystem. This guarantees data integrity during the write merge operation.

[0079] Figure 9A simplified flowchart illustrating a method for processing memory commands in a computing system according to an embodiment of the present invention is shown. As illustrated, the method includes the following steps: S101: Multiple sub-commands are transmitted from the consistency center to the downstream cache subsystem, each sub-command corresponding to a cache line size; S102: Receive at least one of a plurality of subcommands in the downstream caching subsystem; S103: In response to receiving at least one subcommand, the downstream cache subsystem determines the complete address range of the data block; and S104: Optimized memory operations for the downstream memory subsystem are performed by the downstream cache subsystem.

[0080] Since the principles and specific details of the above steps have been clearly described in the above embodiments, they will not be repeated here. It should be noted that by adding other additional steps or making appropriate modifications and / or adjustments, the above process can achieve better performance of the memory and caching subsystem.

[0081] In summary, this invention provides a novel and efficient method and system for enhancing memory performance in consistent computing architectures. By introducing a sequence hinting mechanism, the disclosed embodiments effectively overcome the inherent system memory efficiency loss when long burst commands are split for consistency checks in prior art systems. This invention restores the lost context of the original memory access intent, enabling downstream caching subsystems to perform optimized memory operations, such as read prefetching and write merging. The described mechanism is robust and flexible, adaptable to various command types, including those with non-allocation attributes, and supports scenarios where sequence hints are generated by the consistency center or directly by the request initiator. The result is significantly improved memory bandwidth utilization, reduced access latency, and enhanced overall system performance, particularly for data-intensive workloads generated by modern processors such as GPUs and NPUs.

[0082] The embodiments described herein can be implemented as devices, methods, or computer program products. Therefore, these embodiments can be implemented entirely in hardware, entirely in software, or in a combination of software and hardware aspects, all of which are generally referred to herein as “modules” or “systems.” Furthermore, these embodiments can take the form of computer program products, embodied in any tangible medium having computer-usable program code. In terms of hardware, the invention can be implemented by applying the following techniques or related combinations: individual operating logic with logic gates capable of performing logical functions based on data signals, and the application of application-specific integrated circuits (ASICs), programmable gate arrays (PGAs), or field-programmable gate arrays (FPGAs) with appropriate combinational logic.

[0083] The flowcharts and block diagrams illustrate the architecture, functionality, and operation of systems, methods, and computer program products according to various embodiments of this work. In this regard, each block in a flowchart or block diagram may represent a module, segment, or code portion, including one or more executable instructions for implementing a specified logical function. It should also be noted that each block in the block diagram and / or flowchart description, and combinations of blocks in the block diagram and / or flowchart description, can be implemented by a dedicated hardware system or a combination of dedicated hardware and computer instructions that performs the specified function or behavior. These computer program instructions may be stored in a computer-readable medium, instructing a computer or other programmable data processing device to operate in a particular manner, causing the instructions stored in the computer-readable medium to produce an article of manufacture containing instruction means that implement the function / behavior specified in the flowchart and / or block diagram blocks.

[0084] Those skilled in the art will readily observe that many modifications and alterations can be made to the apparatus and methods while retaining the teachings of the present invention. Therefore, the foregoing disclosure should be interpreted only within the scope of the appended claims.

Claims

1. A method for processing memory commands in a computing system, the method comprising: Multiple subcommands are transmitted from the consistency center to the downstream cache subsystem, each of which corresponds to a cache line size; The downstream caching subsystem receives at least one of the multiple subcommands; In response to receiving the at least one subcommand, the downstream caching subsystem determines the complete address range of the data block; and The downstream caching subsystem performs optimized memory operations for the downstream memory subsystem.

2. The method of claim 1, wherein each of the plurality of subcommands includes a target address and a sequence hint, and the sequence hint indicates the size of the data block corresponding to the subcommand with respect to the respective original memory access command.

3. The method of claim 2, wherein the step of determining the complete address range of the data block includes: The downstream caching subsystem determines the complete address range of the data block based on the sequence hint and the target address of the at least one subcommand.

4. The method of claim 2, further comprising: One or more bits of this sequence hint are determined based on the size and starting address of the predefined aligned data block.

5. The method of claim 2, further comprising: The consensus center receives raw memory access commands from the request initiator. The consistency center breaks down the original memory access command into multiple sub-commands; The sequence hint is generated by the consistency center; as well as The consistency center will append the sequence hint to each of the multiple subcommands.

6. The method of claim 2, further comprising: These multiple sub-commands are generated by the request initiator; The sequence suggestion is generated by the request initiator; The requester shall append the sequential prompt to each of the multiple subcommands; as well as The consensus center receives the multiple sub-commands from the request initiator prior to the transmission.

7. The method of claim 1, wherein the optimized memory operation corresponds to the full address range of the data block.

8. The method of claim 1, wherein the original memory access command is a read command, and the optimized memory operation is a read prefetch operation; and the step of performing the optimized memory operation includes: Actively read the complete address range of data blocks from the downstream memory subsystem, including the data block portion corresponding to subcommands that have not yet been received by the downstream cache subsystem.

9. The method of claim 8, wherein the original memory access command has a non-allocation attribute, and the method further comprises: The downstream caching subsystem will treat these multiple subcommands as having allocation attributes; as well as The downstream caching subsystem allocates a cache line for each piece of data corresponding to the subcommand.

10. The method of claim 9, further comprising: The downstream caching subsystem associates the allocated cache line with a low-priority attribute, indicating that the cache line is a candidate for early eviction.

11. The method of claim 1, wherein the original memory access command is a write command; Furthermore, this optimized memory operation is a write-merge operation, and the steps to perform this optimized memory operation include: The downstream caching subsystem accumulates data from two or more of the multiple subcommands; as well as Perform a single write operation on the downstream memory subsystem to write the accumulated data.

12. The method of claim 11, wherein the original memory access command has a non-allocation attribute, and the method further comprises: The downstream caching subsystem will treat these multiple subcommands as having allocation attributes; as well as The downstream caching subsystem allocates a cache line for each piece of data corresponding to the subcommand to accumulate data from the subcommand.

13. The method of claim 12, further comprising: The downstream caching subsystem associates the allocated cache line with a low-priority attribute, indicating that the cache line is a candidate for early eviction; as well as Associate the allocated cache line with a non-discardable attribute to ensure that the accumulated data is written to the downstream memory subsystem.

14. The method of claim 1, further comprising: Before transmission, the consistency center performs consistency checks on multiple subcommands to determine whether the payload data in the subcommands is available in the upstream cache subsystem; and the transmission of the subcommands to the downstream cache subsystem is performed in response to the probe miss response of the consistency check of the subcommands.

15. A computing system for processing memory commands in a computing system, comprising: A consistency center is configured to transmit multiple subcommands, each corresponding to a cache line size; And a downstream caching subsystem configured to receive at least one subcommand, and in response to receiving at least one subcommand, determine the full address range of a data block, and perform optimized memory operations for the downstream memory subsystem, wherein the optimized memory operations correspond to the full address range of the data block.

16. The computing system of claim 15, wherein each subcommand includes a target address and a sequence hint, the sequence hint indicating the size of the data block associated with the corresponding original memory access command.

17. The computing system of claim 16, wherein the downstream caching subsystem is further configured to determine the complete address range of a data block based on the sequential prompts and target addresses of at least one subcommand.

18. The computing system of claim 15, wherein optimized memory operations correspond to the full address range of the data block.

19. The computing system of claim 15, wherein the original memory access command is a read command, the optimized memory operation is a read prefetch operation; and the downstream cache subsystem is further configured to actively read the complete address range of a data block from the downstream memory subsystem, including the data block portion corresponding to the subcommand that has not yet been received by the downstream cache subsystem.

20. The computing system of claim 19, wherein the original memory access command has a non-allocation attribute, and the downstream caching subsystem is further configured to: treat multiple subcommands as having an allocation attribute; and allocate a cache line for the data corresponding to each subcommand.

Images