Handling page request interface support in direct memory access by caching host memory address translation data

Abstract

The invention relates to page request interface support when handling direct memory accesses by caching host memory address translation data. In one embodiment, a method includes buffering descriptors associated with translation units of an LBA-based direct memory access (DMA) read command of a host system in a descriptor queue, each descriptor relating to a pointer including a physical destination of data associated with a respective translation unit. The method includes sending an address translation request for a pointer of a respective translation unit to an address translation circuit and detecting, at a cache of the address translation circuit, an address translation request miss for a first pointer of a first translation unit linked to a first descriptor of the plurality of descriptors. The method includes causing a translation miss message to be sent to a page request interface (PRI) handler, the translation miss message containing a virtual address of a first pointer and to trigger the PRI handler to send a page miss request to a translation agent of a host system.

Description

Technical Field

[0001] The present disclosure relates generally to memory systems, and more particularly, to handling host submission queues and completing automated page request interface support associated with caching host memory address translation data in a memory subsystem. Background Art

[0002] The memory subsystem may include one or more memory components that store data. The memory components may be, for example, nonvolatile memory components and volatile memory components. In general, the host system may utilize the memory subsystem to store data at the memory components and retrieve data from the memory components. Summary of the invention

[0003] According to one aspect of the present application, a memory subsystem controller is provided. The memory subsystem controller includes: a host interface circuit system for interacting with a host system and a memory device, the host interface circuit system including: a data path interface circuit including a descriptor queue; and an address translation circuit coupled to the data path interface circuit, the address translation circuit for handling address translation requests of the host system and including a cache for storing address translations; and wherein the data path interface circuit is used to: buffer a plurality of descriptors associated with a translation unit of a direct memory access (DMA) memory command of the host system in the descriptor queue, each descriptor for interacting with a corresponding translation unit in the descriptor queue; The invention relates to a method for transmitting a DMA memory command to a host system through a cache memory device, wherein the DMA memory command is one of a DMA write command or a non-logical block address (non-LBA) DMA read command; sending an address translation request for the pointer of a corresponding translation unit of the DMA memory command to the address translation circuit; triggering a page request interface (PRI) handler to send a page miss request to a translation agent of the host system after the address translation request for the first pointer of the first translation unit that missed at the cache, wherein the page miss request includes a virtual address of the first translation unit; and discarding the plurality of descriptors to abort execution of the DMA memory command.

[0004] According to another aspect of the present application, a memory subsystem controller is provided. The memory subsystem controller includes: a host interface circuit system for interacting with a host system and a memory device, the host interface circuit system including: a data path interface circuit including a descriptor queue; and an address translation circuit coupled to the data path interface circuit, the address translation circuit for handling address translation requests of the host system and including a cache for storing address translations; and wherein the data path interface circuit is used to: buffer a plurality of descriptors associated with a translation unit of a direct memory access (DMA) read command of the host system in the descriptor queue, each descriptor for interacting with a corresponding translation unit The invention relates to a method for transmitting a DMA read command to a memory device for transmitting data to a corresponding translation unit of the DMA read command, wherein the DMA read command is associated with a pointer to a destination of the data in the memory device, wherein the DMA read command is associated with a logical block address; sending an address translation request for the pointer of the corresponding translation unit of the DMA read command to the address translation circuit; detecting, at the cache, a miss of the address translation request for a first pointer of a first translation unit linked to a first descriptor of the plurality of descriptors; and causing a translation miss message to be sent to a page request interface (PRI) handler, the translation miss message containing a virtual address of the first pointer and used to trigger the PRI handler to send a page miss request to a translation agent of the host system.

[0005] According to another aspect of the present application, a method is provided. The method includes: buffering a plurality of descriptors associated with a translation unit of a direct memory access (DMA) read command of a host system in a descriptor queue of a data path interface circuit of a processing device, each descriptor being used to relate to a pointer of a destination of data associated with a corresponding translation unit in a memory device, wherein the DMA read command is associated with a logical block address; sending an address translation request for the pointer of the corresponding translation unit of the DMA read command to an address translation circuit; detecting at a cache of the address translation circuit an address translation request miss for a first pointer of a first translation unit linked to a first descriptor of the plurality of descriptors; and causing a translation miss message to be sent to a page request interface (PRI) handler, the translation miss message containing a virtual address of the first pointer and used to trigger the PRI handler to send a page miss request to a translation agent of the host system. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The present disclosure will be more fully understood from the detailed description given below and from the accompanying drawings of various embodiments of the present disclosure.

[0007] Figure 1An example computing environment is described for page request interface support when caching host memory address translation data in a memory subsystem according to some embodiments of the present disclosure.

[0008] Figure 2 is a schematic block diagram of a system (or device) implementing Peripheral Component Interface Express (PCIe) and Non-Volatile Memory Express (NVMe) functionality within which the disclosed page request interface supports operation according to some embodiments.

[0009] Figure 3 is a schematic diagram illustrating an embodiment of a controller memory buffer (CMB) manager of a storage device.

[0010] Figure 4 is an example physical host interface between a host system and a memory subsystem that implements caching of host memory address translation data according to some embodiments.

[0011] Figure 5 A system is provided in which a memory subsystem controller contains an address translation cache (ATC) according to some embodiments.

[0012] Figure 6 A memory subsystem is provided for page request interface support when caching host memory address translation data of a plurality of host interface circuits according to some embodiments.

[0013] Figure 7 is an example memory subsystem controller including address translation circuitry that implements caching of host memory address translation data in accordance with some embodiments.

[0014] Figure 8 is a flow chart of an example method of caching host memory address translation data in a memory subsystem according to some embodiments.

[0015] Fig. 9 is a block diagram of a system including host queue interface circuitry interacting with a page request interface handler in accordance with some embodiments.

[0016] Fig.10 is a schematic block diagram of a system including processing bridge circuitry to implement page request interface support when enabling caching of host memory address translation data in a memory subsystem, in accordance with some embodiments.

[0017] Fig.11 is a flow chart of an example method for page request interface support when enabling caching of host memory address translation data in a memory subsystem in accordance with some embodiments.

[0018] Fig.12is a flow chart of an example method of handling page request interface support of a host submission queue while caching host memory address translation data in a memory subsystem in accordance with some embodiments.

[0019] Fig.13 is a schematic block diagram of a system including completion queue interface circuitry implementing page request interface support when enabling caching of host memory address translation data in a memory subsystem, in accordance with some embodiments.

[0020] Fig.14 is a flow chart of an example method of handling completion automation of page request interface support while caching host memory address translation data in a memory subsystem in accordance with some embodiments.

[0021] Fig.15 is a block diagram of host interface circuitry including a host data path interface (HDPA) circuit and a page fetch interface circuit (PFIC) for handling data transfer operations, according to some embodiments.

[0022] Fig.16A is a diagram of an exemplary command (eg, a write command or a non-logical block address (LBA) read command) in response to a translation miss during a pointer fetch or data transfer, according to some embodiments.

[0023] Fig. 16B is a chop command and different actions that have been split into the PFIC (or other host interface circuitry) in response to a translation miss during a pointer fetch according to some embodiments Fig.16A Figure 1 shows a diagram of exemplary commands for .

[0024] Fig.17 is a flow chart of an example method for handling a translation miss during pointer fetch of a memory command in accordance with some embodiments.

[0025] Fig.18 is a flow chart of an example method for handling translation misses during data transfer of direct memory access (DMA) memory commands, according to some embodiments.

[0026] Fig.19 is a diagram of an exemplary LBA read command that has been split into a chop command and different actions for the PFIC (or other host interface circuitry) in response to a translation miss during a pointer fetch, according to some embodiments.

[0027] Fig. 20 is a flow chart of an example method for handling a translation miss during pointer fetch of an LBA read command, according to some embodiments.

[0028] Fig.21is a flow chart of an example method for handling translation misses during data transfer of LBA-based DMA read commands in accordance with some embodiments.

[0029] Fig. 22 is a block diagram of an example computer system in which embodiments of the present disclosure may operate. DETAILED DESCRIPTION

[0030] Aspects of the present disclosure relate to page request interface support when caching host memory address translation data in a memory subsystem. The memory subsystem may be a storage device, a memory module, or a mixture of a storage device and a memory module. Figure 1 Examples of storage devices and memory modules are described. In general, a host system may utilize a memory subsystem that includes one or more memory components (also referred to hereinafter as a "memory device"). The host system may provide data stored at the memory subsystem and may request data retrieved from the memory subsystem.

[0031] When requesting data to be written to or read from a memory device, the host system typically generates a memory command (e.g., an erase (or unmap) command, a write command, or a read command) that is sent to a memory subsystem controller (e.g., a processing device or "controller"). The controller then executes these memory commands to perform the erase (or unmap) operation, write operation, or read operation at the memory device. Because the host operates on logical addresses, which are referred to as virtual addresses (or guest physical addresses) in the context of virtual machines (VMs) running on the host system, the host system includes a root complex that serves as a connection between the physical and virtual components of the host system and a Peripheral Controller Interconnect Express (PCIe) bus. This PCIe root complex can generate transaction requests (including address translation requests) on behalf of entities of the host system (e.g., a virtual processing device in one of the VMs).

[0032] The host system typically further includes a translation agent (TA) that performs virtual address to physical address translation on behalf of the controller. To this end, the TA is configured to communicate with the translation request / response through the PCIe root complex. In some systems, the TA is also referred to as an input / output memory management unit (IOMMU) executed by a hypervisor or virtual machine manager running on the host system. Thus, the TA can be a hardware component or software (IOMMU) with a dedicated driver.

[0033] The controllers in these systems may be configured to include address translation circuitry, more specifically referred to as an address translation service (ATS), for requesting that the TA perform certain address translations from virtual (or logical) addresses to available (or assigned) physical addresses of the memory devices. In this manner, the address translation circuitry (or ATS) dynamically determines address translations based on the virtual addresses located in the corresponding memory commands queued within the host memory. The various aspects of the ATS do not require a fixed amount of physical memory associated with an application program run by the host system.

[0034] Particularly to support multiple non-volatile memory express (NVMe) devices, the need to constantly request the TA to perform address translations is a bottleneck and impacts performance in terms of speed, latency, and quality of service in fulfilling memory commands. As the commit, completion, I / O, and management queues located in host memory grow larger and the media speeds of memory devices increase, performance is increasingly impacted. For example, the multiple address translation requests and responses for command queues and direct memory access (DMA) addresses can be slowed down by having to move back and forth across the PCIe bus, which also creates additional I / O traffic that slows down the entire memory subsystem.

[0035] Aspects of the present disclosure address the above and other shortcomings by implementing an address translation cache (ATC) within the address translation circuitry of a host interface circuitry within a controller, the ATC storing address translations corresponding to incoming address translation requests from a host interface (HIF) circuit of the host interface circuitry. The ATC can store address translations associated with address translation requests for future access by the host interface circuitry. For example, these address translation requests can be related to the processing of memory commands and the handling of DMA operations or commands. In this way, when the cached address translation matches a subsequent (or later) address translation request from the HIF circuitry (e.g., a cache hit), the address translation circuitry can retrieve the cached address translation and return it to the HIF circuitry without having to request the TA to perform the translation on behalf of the controller.

[0036] In some embodiments, for each memory command within a submission queue of the host memory, the address translation circuitry may store a first address translation in an ATC corresponding to a current page targeted by the memory command (involved in the address translation request) and a second address translation in an ATC for subsequent pages that sequentially follow the current page according to virtual address numbers. This look-ahead buffering of address translations for a predetermined number of submission queues in the ATC can significantly reduce the number of misses at the ATC while keeping the size of the ATC reasonable given the overhead of the availability of cache memory (e.g., static random access memory (SRAM)) at the controller. When command (and other) queues in the host memory are arranged to store memory commands sequentially according to virtual addresses, the hit rate at the cache can be further improved by this approach.

[0037] In most devices or systems, DMA operations cannot handle page faults, e.g., there is no way to resolve page faults (e.g., via paging) to successfully execute a given DMA command. To prevent page faults for DMA operations, the host system pins pages in host memory (either the source or destination of the DMA) before the driver submits I / O data to the device. Pinning pages to the host address space of the memory ensures that DMA operations are executed immediately when triggered by an interrupt. If the I / O device supports a large number of pre-allocated queues and data buffers, large blocks of host memory are reserved in advance, even if the I / O device does not use all of this fixed memory at the same time. For most applications and workloads, all that fixed memory is usually idle.

[0038] Therefore, although the address translation circuitry appears to be able to forgo memory pinning (by accessing the ATC), there are instances of misses at the ATC (or the cache of the ATS), which will create the need for paged memory. But because DMA operations do not support page faults, the address translation circuitry will still be an incomplete solution for DMA operations. To address this problem, in some embodiments, an additional page interface request (PRI) handler is added to the host interface design, which automates, from the perspective of the memory controller, sending PRI-related page miss requests to the host system to request the host system to re-pin the physical memory pages to the virtual addresses of each corresponding page miss request, so that these pages can be used again in the host memory.

[0039] In some embodiments, for example, the PRI handler tracks translation miss messages received from the host interface circuit, each translation miss message including a virtual address of a miss at the cache. The PRI handler may further remove duplicate translation miss messages having the same virtual address, for example due to overlapping virtual addresses from multiple host interface circuits concurrently. The PRI handler may further create page miss requests from non-duplicate translation miss messages categorized into page request groups. For example, the page request group corresponds to one of the host interface circuits.

[0040] In these embodiments, the PRI handler further queues the page request group to be sent to the host system's translation agent, e.g., for handling the page miss request and sending back an acknowledgement in the form of a page request response. In some embodiments, the page request response triggers the PRI handler to issue a restart message to the host interface circuitry and identify which hardware pipeline portion or queue may be restarted in view of the pages being available in memory. In these embodiments, the address translation of the re-pinned page may be further stored in the ATC for future access by anyone in the HIF circuitry.

[0041] Thus, advantages of systems and methods implemented in accordance with some embodiments of the present disclosure include, but are not limited to, improved performance of the memory subsystem in terms of speed, latency, and throughput of handling memory commands. The performance improvement is due in part to reduced I / O traffic on the PCIe bus of the memory subsystem and at the host TA. The disclosed address translation circuitry may also reduce the likelihood that previously cached translations will become invalid and must be re-fetched from the host system's TA. Additional advantages of the PRI handler participating in caching host memory address translation data in the memory subsystem include avoiding the need to pin large amounts of host memory to support DMA operations or the like by requesting re-pinning of pages as needed. Other advantages will be apparent to those skilled in the art of address translation within the memory subsystem, which will be discussed below. The following discussion of Figures 1 to 19 Provide additional details on these techniques.

[0042] Figure 1 An example computing environment 100 is illustrated that includes a memory subsystem 110 according to some embodiments of the present disclosure. Memory subsystem 110 may include media such as one or more volatile memory devices (such as memory device 140), one or more non-volatile memory devices (such as memory device 130), or a combination thereof.

[0043] The memory subsystem 110 may be a storage device, a memory module, or a mixture of a storage device and a memory module. Examples of storage devices include solid-state drives (SSDs), flash drives, universal serial bus (USB) flash drives, embedded multimedia controller (eMMC) drives, universal flash storage (UFS) drives, and hard disk drives (HDDs). Examples of memory modules include dual in-line memory modules (DIMMs), small outline DIMMs (SO-DIMMs), and non-volatile dual in-line memory modules (NVDIMMs).

[0044] The computing environment 100 may include a host system 120 coupled to one or more memory subsystems 110. In some embodiments, the host system 120 is coupled to memory subsystems 110 of different types. Figure 1 An example of a host system 120 coupled to one memory subsystem 110 is illustrated. The host system 120 uses the memory subsystem 110, for example, to write data to the memory subsystem 110 and read data from the memory subsystem 110. As used herein, "coupled to" generally refers to a connection between components, which can be an indirect communication connection or a direct communication connection (e.g., without intervening components), whether wired or wireless, including connections such as electrical, optical, magnetic, etc.

[0045] The host system 120 may be a computing device such as a desktop computer, a laptop computer, a network server, a mobile device, an embedded computer (such as an embedded computer included in a carrier, an industrial device, or a networked commercial device), or such computing devices including a memory and a processing device. The host system 120 may be coupled to the memory subsystem 110 via a physical host interface. Examples of the physical host interface include, but are not limited to, a serial advanced technology attachment (SATA) interface, a compute express link (CXL) interface, a peripheral component interconnect express (PCIe) interface, a universal serial bus (USB) interface, a fiber channel, a serial attached SCSI (SAS), etc. The physical host interface may be used to transfer data between the host system 120 and the memory subsystem 110. When the memory subsystem 110 is coupled to the host system 120 through a physical host interface (such as a PCIe or CXL bus), the host system 120 may further utilize an NVM express (NVMe) interface to access memory components (such as the memory device 130). The physical host interface may provide an interface for transferring control, address, data, and other signals between the memory subsystem 110 and the host system 120.

[0046] The memory devices may include any combination of different types of non-volatile memory devices and / or volatile memory devices. Volatile memory devices, such as memory device 140, may be, but are not limited to, random access memory (RAM), such as dynamic random access memory (DRAM) and synchronous dynamic random access memory (SDRAM).

[0047] Some examples of non-volatile memory devices, such as memory device 130, include NAND-type flash memory and write-in-place memory, such as three-dimensional cross-point ("3D cross-point") memory. A 3D cross-point memory device is a cross-point array of non-volatile memory cells that can perform bit storage based on body resistance changes in combination with a stackable cross-gate data access array. In addition, in contrast to many flash-based memories, cross-point non-volatile memories can perform write-in-place operations, where non-volatile memory cells can be programmed without first erasing the non-volatile memory cells.

[0048] Each of the memory devices 130 may include one or more memory cell arrays, such as single-level cells (SLC), multi-level cells (MLC), three-level cells (TLC), or four-level cells (QLC). In some embodiments, a particular memory component may include an SLC portion and an MLC portion, a TLC portion, or a QLC portion of a memory cell. Each of the memory cells may store one or more data bits used by the host system 120. In addition, the memory cells of the memory device 130 may be grouped to form pages, which may refer to a unit of a memory component for storing data. For some types of memory, such as NAND, pages may be grouped to form blocks. Some types of memory, such as 3D crosspoints, may group pages across dies and channels to form management units (MUs).

[0049] Although non-volatile memory components such as NAND-type flash memory and 3D crosspoint are described, the memory device 130 may be based on any other type of non-volatile memory, such as read-only memory (ROM), phase-change memory (PCM), magnetic random access memory (MRAM), or non-NOR flash memory, electrically erasable programmable read-only memory (EEPROM).

[0050] The memory subsystem controller 115 may communicate with the memory device 130 to perform operations such as reading data, writing data, or erasing data at the memory device 130, and other such operations. The memory subsystem controller 115 may include hardware, such as one or more integrated circuits and / or discrete components, buffer memory, or a combination thereof. The hardware may include digital circuitry with dedicated (i.e., hard-coded) logic for performing the operations described herein. The memory subsystem controller 115 may be a microcontroller, dedicated logic circuitry (e.g., a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), etc.), or other suitable processor.

[0051] The memory subsystem controller 115 may include a processor (processing device) 117 configured to execute instructions stored in a local memory 119. In the illustrated example, the local memory 119 of the memory subsystem controller 115 includes an embedded memory configured to store instructions for executing various processes, operations, logic flows, and routines that control the operation of the memory subsystem 110, including handling communications between the memory subsystem 110 and the host system 120.

[0052] In some embodiments, local memory 119 may include memory registers for storing memory pointers, fetch data, etc. Local memory 119 may also include read-only memory (ROM) for storing microcode. Figure 1 The example memory subsystem 110 in FIG. 1 is illustrated as including a memory subsystem controller 115, but in another embodiment of the present disclosure, the memory subsystem 110 may not include the memory subsystem controller 115, but may rely on external control (e.g., provided by an external host or by a processor or controller separate from the memory subsystem).

[0053] In general, the memory subsystem controller 115 may receive commands or operations from the host system 120 and may convert the commands or operations into instructions or appropriate commands to achieve the desired access to the memory device 130. The memory subsystem controller 115 may be responsible for other operations such as wear leveling operations, garbage collection operations, error detection and error correction code (ECC) operations, encryption operations, cache operations, and address translation between logical block addresses and physical block addresses associated with the memory device 130. The memory subsystem controller 115 may further include host interface circuitry for communicating with the host system 120 via a physical host interface. The host interface circuitry may convert commands received from the host system into command instructions to access the memory device 130 and convert responses associated with the memory device 130 into information for the host system 120.

[0054] The memory subsystem 110 may also include additional circuitry or components not illustrated. In some embodiments, the memory subsystem 110 may include a cache or buffer (e.g., DRAM) and address circuitry (e.g., row decoders and column decoders) that may receive addresses from the memory subsystem controller 115 and decode the addresses to access the memory device 130.

[0055] In some embodiments, the memory device 130 includes a local media controller 135 that operates in conjunction with the memory subsystem controller 115 to perform operations on one or more memory cells of the memory device 130. In some embodiments, the memory device 130 is a managed memory device, which is a raw memory device combined with a local controller for memory management within the same memory device package, such as the local media controller 135. An example of a managed memory device is a managed NAND (MNAND) device.

[0056] The memory subsystem 110 includes an address translation circuit 113 and an address translation cache (or ATC 116) that may be used to perform caching of host memory address translation data for queues, physical page regions (PRPs), scatter-gather lists (SGLs), and data transfers in the memory subsystem 110. For example, the address translation circuit 113 may receive an address translation request from a HIF circuit that is handling a memory command, requesting that the TA perform a translation of a virtual address located within the memory command, and upon receiving the physical address, store a mapping between the virtual address and the physical address (also referred to as an L2P mapping) in the ATC 116. Upon receiving a subsequent address translation request containing the same virtual address, the address translation circuit 113 may verify that the virtual address hits at the ATC 116 and copy the corresponding address translation from the ATC directly to the pipeline of the address translation circuit 113, which returns the corresponding address translation to the requesting HIF circuit. Similar caching of address translations may also be performed for DMA operations, which will be discussed in more detail. Further details regarding the operation of address translation circuit 113 and ATC 116 are described below.

[0057] Figure 2is a schematic block diagram of a system 200 (or device) implementing Peripheral Component Interface Express (PCIe) and Non-Volatile Memory Express (NVMe) functionality within which the disclosed page request interface supports operation in accordance with some embodiments. In various embodiments, the system 200 includes a host system 220 (e.g., host system 120), a memory subsystem 210 (e.g., memory subsystem 110), which in turn includes a controller 215 (e.g., controller 115) or more memory devices 130 and DRAM 222. In some embodiments, aspects of the controller 215 (including hardware and / or firmware functionality) are included in the local media controller 135.

[0058] In an embodiment, the host system 220 includes a central processing unit (CPU) 209 connected to a host memory 212 (e.g., DRAM or other main memory). Applications may be stored in the memory space 212 for execution by components of the host system 220. The host system 220 includes a bus 207 (e.g., a memory device interface) that interacts with a host interface 208, which may include a media access control (MAC) and physical layer (PHY) components of the memory device 130 for communication from the host system 220 to the memory device 130 and communication from the memory device 130 to the host system 220. The bus 207 and the host interface 208 operate under a communication protocol such as a peripheral component interface express (PCIe) serial communication protocol or other suitable communication protocols. Other suitable communication protocols include Ethernet, serial attached SCSI (SAS), serial AT attachment (SATA), any protocol related to remote direct memory access (RDMA) (e.g., Infiniband, iWARP, or RDMA over Converged Ethernet (RoCE)), and other suitable serial communication protocols.

[0059] The memory device 130 may also be connected to the host system 220 through a switch or a bridge. A single host system 220 is shown connected to the memory device 130 and supports the PCI-SIG Single Root I / O Virtualization and Sharing Specification (SR-IOV) single host virtualization protocol, as discussed in more detail below, where the memory device 130 may be shared by multiple hosts, where the multiple hosts may be virtualizing a physical function 211 (PF) and one or more virtual functions 205 (VF) of a single physical host system. In other embodiments, it is contemplated that the SR-IOV standard for virtualizing multiple physical hosts may be implemented with features of the disclosed systems and methods.

[0060] In an embodiment, the non-volatile memory array of memory device 130 (or NVM 206) may be configured for long-term information storage as a non-volatile memory space and retain information after power on / off cycles. Figure 1 Describing the same way, Figure 2 The NVM 206 in may include reference Figure 1 One or more dies of NAND-type flash memory or other memory are discussed.

[0061] The memory subsystem 210 includes a controller 215 (e.g., a processing device) that manages the operation of the memory device 130 (e.g., writing to and reading from the NVM 206). The controller 215 may include one or more processors 217, which may be multi-core processors. The processor 217 may typically handle or interact with the components of the memory device 130 through firmware code.

[0062] The controller 215 may operate under the NVM Express (NVMe) protocol, but other protocols may be applicable. The NVMe protocol is a communication interface / protocol developed to enable SSDs to operate on a host and memory device linked by a PCIe interface. The NVMe protocol provides a command queue and completion path for accessing data stored in the memory device 130 by the host system 220.

[0063] The controller 215 also includes a controller memory buffer (CMB) manager 202. The CMB manager 202 may be connected to a DRAM 222, a static random access memory (SRAM) 224, and a read-only memory (ROM) 226. The CMB manager 202 may also communicate with the NVM 206 through a media interface module 228. The DRAM 222 and the SRAM 224 are volatile memories or cache buffers used for short-term storage or temporary storage during operation of the memory device 130. In some embodiments, the SRAM 224 also includes a tightly coupled memory. Volatile memory does not retain stored data when power is removed. DRAM typically requires periodic refreshes of stored data, while SRAM does not require refreshes. Although SRAM typically provides faster data access than DRAM, it may also be more expensive.

[0064] The controller 215 executes computer-readable program code (e.g., software or firmware) executable instructions (referred to herein as “instructions”). The instructions may be executed by various components of the controller 215, such as the processor 218, logic gates, switches, application specific integrated circuits (ASICs), programmable logic controllers, embedded microcontrollers, and other components of the controller 215.

[0065] Instructions for implementing the embodiments described herein that can be executed by the controller 215 are stored in a non-transitory computer-readable storage medium. In some embodiments, the instructions are stored in a non-transitory computer-readable storage medium of the memory device 130, such as in a read-only memory (ROM) or NVM 206. The instructions stored in the memory device 130 can be executed without additional input or guidance from the host system 220. In other embodiments, the instructions are transmitted from the host system 220. The controller 215 is configured with hardware and instructions to perform the various functions described herein and shown in the figures.

[0066] The controller 215 may also include other components, such as an NVMe controller 203, a media interface module 228 coupled between the NVMe controller 203 and the memory device 130, and an error correction module 234. In an embodiment, the NVMe controller 203 includes an SRAM 204, an address translation circuit 213 (ATS) having an address translation cache 216, a direct memory access (DMA) module 230, a host data path automation (HDPA) circuit 232, a command parser 236, a command executor 238, and a control path 240. In various embodiments, the address translation circuit 213 is the same as the address translation circuit 113 and the address translation cache 216 is the same as the address translation cache 116, all of which will be discussed in more detail below. The SRAM 204 may be an internal SRAM of the NVMe controller 203 that is separate from the SRAM 224. The CMB manager 202 may be directly coupled to the NVMe controller 203 such that the NVMe controller 203 may interact with the CMB manager 202 to access the DRAM 222 and the SRAM 224 .

[0067] In an embodiment, the media interface module 228 interacts with the NVM 206 for read and write operations. The DMA module 230 performs data transfers between the host system 220 and the memory device 130 without involving the CPU 209. The HDPA circuit 232 controls the data transfer while activating the control path 240 for extracting PRP / SGL, issuing completions and interrupts, and activating the DMA 230 for the actual data transfer between the host system 220 and the memory device 130. The error correction module 234 corrects the data extracted from the memory array in the NVM 206. The command parser 236 parses the commands to the command executor 238 for execution on the media interface module 228.

[0068] In an embodiment, the controller 215 further includes a page request interface (PRI) handler 219 coupled to or integrated with the processor 217 and the host interface 208, as will be discussed in greater detail. The PRI handler 219 may be processing logic that may include hardware (e.g., a processing device, circuitry, dedicated logic, programmable logic, microcode, hardware of a device, an integrated circuit, etc.), firmware (e.g., instructions running or executed on the processor 217), or a combination thereof.

[0069] In various embodiments, the PRI handler 219 automates the sending of page miss requests to the host system 120 and the detection and handling of page request responses returned from the host system 120 from the perspective of the host interface 208 of the memory controller 215. Page miss requests request the translation agent of the host system 120 to repin the physical pages of the host memory 212 to the virtual addresses of each corresponding page miss request so that these pages can be used again in the host system 220. In some cases, the repinning of pages in the host memory 212 involves finding a valid virtual address to map to an existing physical address (where the data is still stored) and completing the mapping whose address translation can be returned to the controller 215. Each page request response may include confirmation that the response page has been repinned in memory and the address translation of the pinned page.

[0070] Figure 3 Yes Description Figure 2 2 is a schematic diagram of an embodiment of a CMB manager 202 of a system 200, but other systems are possible. The CMB manager 202 manages data transactions between a host system 220 and a memory device 130 having a controller memory buffer (CMB) 300. The CMB 300 is a controller memory space that may span one or more of the DRAM 222, SRAM 224, and / or NVM 206. The contents of the CMB 300 are generally not survivable across power cycles, so the CMB manager 202 rebuilds the CMB 300 after the system 200 is powered on.

[0071] One or more types of data structures defined by the NVMe protocol may be stored in CMB 300 by CMB manager 202 or may be stored in host memory 212 ( Figure 2). As described in more detail below, the host system 220 may initialize the CMB 300 before the CMB manager 202 stores the NVMe data structures in the CMB 300. During the initialization phase, the memory device 130 may advertise the capacity and size of the CMB 300 to the host system 220 and may advertise which NVMe data structures may be stored in the CMB 300. For example, the memory device 130 may store one or more of the NVMe data structures in the CMB 300, including NVMe queues 304, such as submission queues (SQs) 306, completion queues (CQs) 308, PRP lists 312, SGL segments 314, write data 320, read data 318, and combinations thereof.

[0072] The NVMe protocol standard is based on a paired submission and completion queue mechanism. Commands are placed in a submission queue (SQ) by the host software. Completions are placed in an associated completion queue (CQ) by the controller 215. The host system 220 (or device) may have multiple pairs of submission and completion queues for different types of commands. In response to a notification from the host system 220, the controller 215 extracts commands from a submission queue (e.g., one of the SQ 306). Thereafter, the controller 215 processes the command, such as performing internal command selection, executing a command (e.g., performing a write or read), and the like. After processing the command, the controller 215 places an entry in a completion queue, where the entry indicates that the command execution has been completed. The controller 215 then generates an interrupt to the host device indicating that the entry has been placed on the completion queue. The host system 220 reviews the entry of the completion queue and then notifies the controller 215 that the entry of the completion queue has been reviewed. As will be discussed in more detail, the address translation circuit 213 can help perform these functions of the controller 215.

[0073] Typically, submission and completion queues are allocated within host memory 212, where each queue may be physically located contiguously or non-contiguously in host memory. However, the CMB feature, such as supported in the NVMe standard, enables the host system 220 to place submission queues, completion queues, physical page region (PRP) lists, scatter-gather list (SGL) segments, and data buffers in controller memory instead of host memory 212.

[0074] Controller 215( Figure 2 ) also generates an internal mapping table 221 for use by CMB manager 202 to map PF and VF data to the correct CMB location in controller memory DRAM 222 or SRAM 224. Mapping table 221 itself is typically stored in flip-flops or DRAM 222 to reduce or eliminate any latency issues. In one embodiment, for example, the mapping table may have entries for the PF and each VF.

[0075] The NVMe standard supports NVMe virtualization environments. Virtualization environments can use NVM systems and multiple controllers to provide direct input / output (I / O) access to virtual or physical hosts (also referred to herein as virtual or physical functions). The NVM system includes a primary controller and a secondary controller, where the secondary controller depends on the primary controller for dynamic resource assignment. The host can issue an identification command specifying a list of secondary controllers to the primary controller to discover the secondary controllers associated with the primary controller. SR-IOV defines an extension of PCI Express that allows multiple system images (SIs) (such as virtual machines running on a hypervisor) to share PCI hardware resources (see Figure 4 ).

[0076] A physical function (PF) is a PCIe function that supports SR-IOV capabilities, which in turn allows it to support one or more dependent virtual functions (VFs). These PFs and VFs can support an NVMe controller that shares a basic NVM subsystem with multipath I / O and namespace sharing capabilities. In this virtualization environment, the physical function is sometimes referred to as the primary function, and each virtual function is assigned its own CMB, which is a portion of the total controller memory available for use by the CMB. As used herein, the term physical function refers to a PCIe function that supports SR-IOV capabilities, where a single physical host is divided into a physical function and multiple virtual functions that each communicate with a controller of a memory device. The terms physical function and primary function are used interchangeably herein.

[0077] In an embodiment, the controller 215 advertises CMB 300 availability only to physical functions (PFs) of a virtualized host system (e.g., host system 220) that has a single physical function 211 and one or more virtual functions 205 (or VFs). Moreover, the advertised CMB 300 availability may be in the form of the total CMB size available to all functions (physical and any virtual functions), so that the physical function 211 may selectively assign any desired portion of the advertised total available CMB size to itself and all other virtual functions 205.

[0078] The controller 215 may then store the portion of the CMB 300 selected by the physical function of the available CMB 300 in the NMVe registers dedicated to each physical function 211 and the virtual function, respectively. The virtual function may store different relative portion sizes of the advertised CMB size in each NVMe register to account for the different needs of the physical function 211 to ensure itself and each virtual function. Once the physical function 211 allocates different amounts and regions of the advertised CMB available for host access (e.g., for direct access by the primary and virtual functions) during the boot phase, these settings may be managed by the controller 215 to provide access to the corresponding primary or virtual function during operation of the memory device.

[0079] The controller buffer manager 202 may include a transaction classifier module 322 for classifying host write transactions received to the CMB 300. Host write transactions to the CMB 300 may be associated with host write commands and host read commands. In certain embodiments, the transaction classifier module 322 may classify host write transactions into one of three NVM data structure groups of the NVMe queue 304, pointers 310, and data buffers 316. The NVMe queue 304 includes a host submission queue (SQ) 306 and a host completion queue (CQ) 308. The pointers 310 may include a physical region page (PRP) list 312 and a scatter gather list (SGL) segment 314. The PRP list 312 contains pointers indicating physical memory pages that are filled or to be filled with user data, such as for a read or write command in the NVMe queue 304. The SGL segment 314 includes pointers indicating physical addresses of the host memory 212 from which data should be transferred for a write command and to which data should be transferred for a read command. Data buffer 316 may contain write data 320 written to NVM 206 associated with a write command and / or contain read data 318 read from memory device 130 associated with a read command.

[0080] In certain embodiments, the NVMe queue 304, pointer 310, and data buffer 316 associated with a particular command may be stored in the CMB 300 by the CMB manager 202 to reduce command execution latency for the memory device 130. For example, if the SQ is located in the host memory 212, then the host command entry written to the SQ-implemented CMB 300 avoids fetching the host command entry through the PCIe fabric that may include multiple switches. If the PRP list and SGL segment are located in the host memory space, then the PRP list 312 and SGL segment 314 written to the CMB 300 of the memory device 130 avoid fetching the PRP list 312 and SGL segment 314 separately through the PCIe fabric. The write data 320 written to the CMB 300 of the memory device 130 avoids having the memory device 130 fetch the write data from the host memory 212.

[0081] The address translation circuit 213 may communicate with components of the host system 220 and the memory subsystem 210 through the host interface 208. The address translation circuit 213 may also be at least partially incorporated within the host interface 208, as will be discussed in greater detail. The address translation circuit 213 may also retrieve commands from the SQ 306, handle the commands to include retrieving address translations from the ATC 216 (if present), and submit completion notifications to the CQ 308 of the host system 220. Thus, in at least some embodiments, the address translation circuit 213 may include or be integrated with the command parser 236, the command executor 238, and the DMA 230.

[0082] Figure 4 is an example physical host interface between the host system 220 and a memory subsystem that implements caching host memory address translation data according to some embodiments. In at least some embodiments, the example physical host interface also implements NVMe direct virtualization, as shown in reference Figures 2 to 3 In one embodiment, the controller 415 of the memory subsystem is coupled to the host system 220 through a physical host interface such as a PCIe bus 411A. In one embodiment, the NVMe control module 405 running on the controller 415 generates and manages a number of virtual NVMe controllers 402 to 408 within the controller 415. The virtual NVMe controllers 402 to 408 are virtual entities that appear as physical controllers to other devices (such as the host system 220) connected to the PCIe bus 411A by means of physical functions 412 to 418 associated with each virtual NVMe controller 402 to 408. Figure 4 Three virtual NVMe controllers 402 to 408 and three corresponding physical functions 412 to 418 are illustrated. However, in other embodiments, there may be any other number of NVMe controllers, each with a corresponding physical function.

[0083] Each of the virtual NVMe controllers 402 to 408 manages storage access operations of the basic memory device 130. For example, the virtual NVMe controller 402 may receive a data access request from the host system 220 via the PCIe bus 411A, including a request to read, write, or erase data. In response to the request, the virtual NVMe controller 402 may identify a physical memory address in the memory device 130 that is related to the virtual memory address in the request, perform the requested memory access operation on the data stored at the physical address, and return the requested data and / or confirmation or error message to the host system 220 as appropriate. The virtual NVMe controllers 404 to 408 may operate in the same or similar manner with respect to data access requests for one or more memory devices 130.

[0084] In an embodiment, the NVMe control module 405 associates one of the physical functions 412 to 418 with each of the virtual NVMe controllers 402 to 408 so as to allow each virtual NVMe controller 402 to 408 to appear as a physical controller on the PCIe bus 411A. For example, the physical function 412 may correspond to the virtual NVMe controller 402, the physical function 414 may correspond to the virtual NVMe controller 404, and the physical function 418 may correspond to the virtual NVMe controller 408. The physical functions 412 to 418 are full-function PCIe functions that can be discovered, managed, and manipulated like any other PCIe device and can therefore be used to configure and control PCIe devices (such as virtual NVMe controllers 402 to 408). Each physical function 412 to 418 may have a certain number of virtual functions (VFs) associated with it. A VF is a lightweight PCIe function that shares one or more resources with a physical function and a virtual function associated with the physical function. Each virtual function has a PCI memory space which is used to map its register set. The virtual function device driver operates on the register set to implement its functionality and the virtual function appears as an actual PCIe device accessible by the host system 220 through the PCIe bus 411A.

[0085] In at least some embodiments, the controller 415 is further configured to control the execution of memory operations associated with memory commands from the host system 220 at one or more memory devices 130 and one or more network interface cards (NICs) 450 (which are actual physical memory devices). In these embodiments, the controller 415 communicates with the memory devices 130 through the second PCIe bus 411B and communicates with the NICs 450 through the third PCIe bus 411C. Each memory device 130 may support one or more physical functions 432 and each NIC 450 may support one or more physical functions 436. Each physical function 432-436 may also have a number of virtual functions (VFs) associated therewith.

[0086] In these embodiments, each physical function 412-418 and 432-436 may be assigned to any of the virtual machines VM(0) to VM(n) in the host system 220. When I / O data is received from the virtual machine at the virtual NVMe controllers 402-408, the virtual machine driver (e.g., NVMe driver) provides a guest physical address for the corresponding read / write command. The NVMe control module 405 may translate the physical function number into a bus, device, and function (BDF) number and then add the command to a direct memory access (DMA) operation to perform a DMA operation on the guest physical address. In one embodiment, the controller 115 further converts the guest physical address into a system physical address of the memory subsystem 110.

[0087] In addition, each physical function 412 to 418 and 432 to 436 can be implemented in privileged mode or normal mode. When implemented in privileged mode, the physical function has a single management point that can control resource manipulation and storage provisioning of other functions implemented in normal mode. In addition, the physical function in privileged mode can perform management options, including, for example, enabling / disabling multiple physical functions, storage and quality of service (QoS) provisioning, firmware and controller updates, vendor unique statistics and events, diagnostics, secure erase / encryption, and others. Typically, the first physical function can implement privileged mode and the remaining physical functions can implement normal mode. However, in other embodiments, any of the physical functions can be configured to operate in privileged mode. Therefore, there can be one or more functions running in privileged mode.

[0088] The host system 220 can run multiple virtual machines VM(0) to VM(n) by executing a software layer (generally referred to as a hypervisor 424) above the hardware and below the virtual machines, such as Figure 4 Schematically shown in . In one illustrative example, the hypervisor 424 may be a component of a host operating system 422 executed by the host system 220. Alternatively, the hypervisor 424 may be provided by an application running under the host operating system 422 or may run directly on the host system 220 without an operating system under the hypervisor 424. The hypervisor 424 may abstract the physical layer including processors, memory, and I / O devices and present this abstraction to virtual machines VM(0) to VM(n) as virtual devices, including virtual processors, virtual memory, and virtual I / O devices. The virtual machines VM(0) to VM(n) may each execute a client operating system that may utilize a base virtual device, which may, for example, be mapped to a memory device 130 or a NIC 450 managed by one of the virtual NVMe controllers 402 to 408 in the memory subsystem. One or more applications may run on each VM under a client operating system.

[0089] In various embodiments, each virtual machine VM(0) to VM(n) may include one or more virtual processors and / or drivers. Processor virtualization may be implemented by hypervisor 424 scheduling time slots on one or more physical processors so that from the perspective of the client operating system, the time slots are scheduled on virtual processors. Memory virtualization may be implemented by page tables (PTs), which are memory structures that translate client memory addresses into physical memory addresses. Hypervisor 424 may run at a higher privilege level than the client operating system, and the latter may run at a higher privilege level than the client application.

[0090] In one embodiment, there may be multiple partitions on the host system 220 representing virtual machines VM(0) to VM(n). The parent partition corresponding to the virtual machine VM(0) is a root partition (i.e., root ring 0), which has additional privileges to control the life cycle of other child partitions (i.e., regular ring 0) corresponding to virtual machines VM(1) and VM(n), for example. Each partition has a corresponding virtual memory, and the child partition sees the physical device assigned to it rather than presenting a virtual device. When the host system 220 is initially started, the parent partition can directly see all physical devices. A pass-through mechanism (e.g., PCIe pass-through or direct device assignment) allows the parent partition to assign an NVMe device (e.g., one of the virtual NVMe controllers 402 to 408) to a child partition. The associated virtual NVMe controllers 402 to 408 can appear to each of the virtual machines VM(0), VM(1), VM(n) as a virtual storage resource accessible to the client operating system or the client application running therein. In one embodiment, for example, virtual machine VM(0) is associated with virtual NVMe controller 402, virtual machine VM(1) is associated with virtual NVMe controller 404, and virtual machine VM(n) is associated with virtual NVMe controller 408. In other embodiments, one virtual machine may be associated with two or more virtual NVMe controllers. Virtual machines VM(0) to VM(n) may use corresponding bus, device, and function (BDF) numbers to identify associated virtual NVMe controllers, as will be described in more detail below.

[0091] In some embodiments, the hypervisor 424 further includes a storage emulator 426 coupled to the NVMe drivers on the virtual machines VM(0) to VM(n) and coupled to the physical function NVMe driver 432 of the hypervisor 424. The physical function NVMe driver 432 can drive the physical functions 412 to 418 through the PCIe bus 411A with the help of the virtual NVMe controllers 402 to 408 and also drive the physical functions 432 available on the memory device 130 through the PCIe bus 411A and the second PCIe bus 411B. In addition, the hypervisor may include a NIC emulator 428 coupled to the NIC drivers on the virtual machines VM(0) to VM(n) and coupled to the physical function NIC driver 436 of the hypervisor 424. In various embodiments, the physical function NIC driver 434 controls the PF 436 of the NIC 450 through the PCIe bus 411A and the third PCIe bus 411C.

[0092] In at least some embodiments, the host system 220 submits memory commands (e.g., erase (or unmap), write, read) to a set of submission queues 442, input / output (I / O) commands to a set of I / O queues 446, and management ("admin") commands to a set of management queues 448, which are stored in the host memory 212 of the host system 220 or in the CMB 300 ( Figure 3 ) in one of the PCIe bus 411A. The controller 415 may retrieve these memory commands through the PCIe bus 411A and typically handle each memory command in turn according to priority (e.g., handling reads before writes). When the controller 415 has completed handling the memory commands residing in the set of submission queues 442, the controller 415 returns confirmation of the completion of the memory command by submitting a completion entry in a corresponding completion queue in a set of completion queues 444 also stored in the host memory 212. In some embodiments, the host memory 212 is composed of DRAM or other main memory type memory. In various embodiments, the queues 442, 444, 446, 448 may be numbered in the hundreds (or thousands) and are ordered sequentially (e.g., continuously) according to the virtual addresses of the memory commands. In other words, the memory commands are queued in these queues in a sequential order based on the virtual addresses in the memory commands within the virtual memory space.

[0093] In the disclosed embodiment, the controller 415 further includes an address translation circuit 413, which includes a similar reference Figures 1 to 3 The ATC 416 (or "cache") introduced. The ATC 416 may be a static random access memory (SRAM), a tightly coupled memory (TCM), or other fast access memory suitable for use as a cache. The address translation circuit 413 may be coupled to the NVMe control module 405 and typically to a PCIe protocol component of the controller 415 (e.g., a virtual NVMe controller). In this manner, the address translation circuit 413 provides a host interface circuit system that facilitates obtaining address translations and other handling of memory commands retrieved from the queues 442, 444, 448. The address translation service (ATS) of the address translation circuit 413 further enables direct connection between one or more memory devices 130 and virtual machines VM(0) to VM(n) to achieve data communication performance close to the PCIe line rate bypassing the hypervisor 424.

[0094] As explained, the address translation circuit 413 can receive (or retrieve) an address translation request from the HIF circuit that is handling the memory command, requesting the TA to perform the translation of the virtual address located within the memory command, and after receiving the physical address, store the mapping between the virtual address and the physical address (also known as the L2P mapping) in the ATC 416. After receiving a subsequent address translation request containing the same virtual address, the address translation circuit 413 can verify that the virtual address hits at the ATC 416 and copy the corresponding address translation from the ATC directly to the pipeline of the address translation circuit 413, which returns the corresponding address translation to the requesting HIF. Similar caching of address translations can also be performed for DMA operations, which will be discussed in more detail. Therefore, the functionality of the address translation circuit 413 can enable many address translation requests to bypass any need to interact with the hypervisor 424, which as software is a bottleneck and slows down the performance of obtaining address translations without caching such address translations. The resulting speed, latency, and throughput performance improvements through the ATS functionality can be significant.

[0095] Figure 5 5 is a system 500 according to some embodiments in which a memory subsystem controller, such as a PCIe controller 515, contains an address translation cache (ATC) 516. Within the system 500, a host system 520 (or any host system discussed herein) includes a translation agent (TA) 517 and an address translation and protection table (ATPT) 519, which together can be used by the host system 520 to provide address translations to the PCIe controller 515 according to the PCIe protocol. Specifically, the TA 517 can return a physical page address in response to a submitted virtual address in an address translation request or return "no translation" when the corresponding physical page has been swapped out and thus the virtual address currently has no translation.

[0096] These address translations may be associated with memory commands resident in the host memory 512 (and / or CMB 300) being handled by the PCIe controller 515 (or by other PCIe devices or virtual PCIe devices). To provide the translations, the TA 517 is configured to communicate with the address translation request / response through the PCIe root complex 523. In some systems, the TA 517 is also referred to as an input / output memory management unit (IOMMU) executed by the hypervisor 424 or virtual machine manager running on the host system 520. Thus, the TA 517 may be a hardware component or software (IOMMU) with a dedicated driver.

[0097] In various embodiments, to avoid this increased I / O traffic on the PCIe bus 511A between the host system 520 and the PCIe controller 515, address translations provided to the PCIe controller 515 may be cached in the ATC 516 and accessed to fulfill later (or subsequent) ATS-generated address translation requests without having to be returned to the TA 517 via an update request for each required translation. If an entry in the ATC 516 is invalid due to a change in assignment between virtual and physical addresses within the TA 517 and ATPT 519, the PCIe controller 515 (e.g., an ATS in the PCIe controller 515) may clear the corresponding entry in other ATS queues within the ATC 516 and in the host interface circuitry.

[0098] Figure 6 6 is a memory subsystem 610 for supporting page request interfaces when caching host memory address translation data of multiple host interface circuits according to some embodiments. In various embodiments, the memory subsystem 610 may be connected to Figure 1 The memory subsystem 110 or Figure 2 The memory subsystem 210 or Figure 5 In these embodiments, the memory subsystem 610 includes a controller memory buffer (CMB) 630 and a PRI handler 219 (see Figure 2 In some embodiments, controller 615 is controller 115, 215, or 415, local media controller 135, or a combination thereof (see Figure 1 ). In at least one embodiment, CMB 630 is a reference Figure 3 The CMB 300 discussed, and therefore, may also include the NVM of the memory device 130 .

[0099] In various embodiments, CMB 630 includes a host advanced extensible interface (AXI) interface 631 that supports the functionality of controller 615, CMB control / status registers 632, host read command SRAM / FIFO 634, host data SRAM 636, CMB host write buffer 638, controller write buffer 640, host read buffer 644, controller data SRAM 646, controller read buffer 648, and controller read command SRAM / FIFO 650, where FIFO stands for first-in-first-out buffer.

[0100] In some embodiments, the controller 615 includes a PCIe system on a chip (SoC) 601 and a host interface circuit system 603. The PCIe SoC 601 may include a PCIe IP that facilitates communication by the host interface circuit system 603 with a host system 620 (including a TA 617) using the PCIe protocol. Thus, the PCIe SoC 601 may include capabilities and control registers for each physical function and each virtual function presented to the memory subsystem 610. The PCIe SoC 601 may include an integrated development environment (IDE) link encryption circuit 605 that encrypts translation layer packets (TLPs) passed to the host system 620 over the PCIe bus 611A and decrypts TLPs received from the host system 620.

[0101] In various embodiments, the host interface circuit system 612 includes a local memory 619 (SRAM, TCM, and the like), a number of host interface circuits 608A, 608B, 608C, ... 608N (or collectively referred to as host interface circuits 608), an address translation circuit 613 (or ATS) including an ATC 616 and a translation logic 602, and an AXI matrix 623 that can be used to send interrupts to the host system 620 when interacting with reference to handling memory commands and DMA. The host interface circuits 608A-608N (also referred to herein as HIF circuits 608) include hardware components that facilitate the extraction and processing of commands (for host queues) and data (for DMA) to perform specific functions in command and DMA processing. Individual ones of the host interface circuits 608A-608N sometimes request address translations of virtual addresses or guest physical addresses (e.g., associated with commands or DMA). In an embodiment, for this purpose, individual HIF circuits 608 request address translation circuits 613 to provide translations. In this manner, address translation circuitry 613 interfaces with and supports both address translation generation and invalidation on behalf of respective host interface circuitry 608A- 608N, as will be discussed in greater detail below.

[0102] Furthermore, in at least one embodiment, the host interface circuit 608N may be a processing bridge circuit 608N that provides a hardware interface between the PRI handler 219 and other host interface circuits 608A, 608B, 608C, etc. The processing bridge circuit 608N may include a processor for processing page miss requests, page request responses, and reference requests. Fig.10 The various buffers and logic for other commands and messages discussed in more detail in the accompanying drawings. Additionally, in at least one embodiment, the host interface circuit 608B is a host queue interface circuit 608B that handles host command fetch automation (e.g., on the submission queue 306 of the host system 120). The host queue interface circuit 608B will refer to Fig. 9 and Fig.12In addition, in some embodiments, the host interface circuit 608C is a completion queue interface circuit 608C that handles command completion automation (e.g., completion queue 308 posting) to indicate to the host system 120 when the controller 115 has completed processing a particular command. The completion queue interface circuit 608C will refer to Figures 13 to 14 Discuss in more detail.

[0103] The address translation circuit 613 may employ a number of fields and parameters, such as the minimum translation unit (STU) of data (defined by a specific size that is typically smaller than a host tag) and an invalid queue depth, where queue entries are cleared when the invalid queue depth is exceeded. The translation logic 602 may use a PCIe memory read TLP to request a translation of a given untranslated address from the TA 617 (or IOMMU). For example, these address translation requests may carry a separate PCIe tag and identifier. The PCIe tag may include, for example, "PCIe memory read complete" indicating the translation address and translation size, an "S" bit that informs the translation size (e.g., the translation address may represent a continuous physical address space of 128KB or similar size), "R" and "W" bits that provide read / write permissions to pages in the physical address space, and a "U" bit that tells the device to use only untranslated addresses for DMA. The U bit may be helpful when the buffer is a one-time use and the TA 617 does not have to send an invalidate to the controller 615.

[0104] In an embodiment, when the translation is changed in the TA 617, the ATC 616 (one or more caches) in the memory subsystem 610 should purge the old entries corresponding to the virtual addresses that have been invalidated. The invalidation request may be sent by the TA 617 to the address translation circuit 613 using a PCIe message TLP that may be directed to the HIF circuit of a function that is handling the memory command processing. The address translation circuit 613 may direct the HIF circuit to purge all DMAs for the invalid address range, remove such entries from the ATC 616, and send an invalidation completion (e.g., another PCIe message TLP) to the host system 620 after the invalidation is confirmed by the HIF circuit.

[0105] In some embodiments, the address translation circuit 613 stores the address translation returned from the TA 617 in response to the address translation request in the ATC 616. In various embodiments, this address translation may include an I / O submission queue (SQ) base address, a PRP / SGL of outstanding commands in one or more HIF circuits, and / or an I / O completion queue (CQ) base address. The ATC 616 may be configured to handle finding an empty slot in the ATC 616 and storing a new translation in the empty slot, finding when a data TLP occurs, and clearing entries during function level reset (FLR), invalidation, and other resets. The ATC 616 may be further configured to age older entries to make room when the cache is running full.

[0106] In various embodiments, the queue portion 416A of the ATC 616 cache may be sized to include at least one or two entries per queue of the SQ 442, CQ 444, I / O queue 446, and management queue 448, but more are contemplated as cache memory device size and cost decrease. When there is enough space for two entries, the ATC 616 may store the address translation of the current page associated with the queue and the next page (e.g., the virtual address that sequentially follows the current page). This look-ahead buffering of address translations for a predetermined number of queues in the ATC 616 can significantly reduce the number of misses at the ATC while keeping the size of the ATC reasonable given the overhead of the cache memory. In addition, the DMA portion 416B of the ATC 616 (for storing DMA data) may be expanded to some integer multiple (e.g., 2 to 4 times) of the size of the queue portion of the ATC 616.

[0107] In at least some embodiments, the translation logic assigns a queue identifier to each of the SQ 442, CQ 444, I / O queue 446, and management queue 448 ("queues"). This may be as simple as sequentially incrementing the count number of each subsequent sequentially ordered queue in a set of queues. The translation logic 602 may then index the queue portion 416A of the ATC 616 cache according to the queue identifiers of the corresponding address translations stored therein and use such queue identifiers internally (within the address translation circuitry 613) to track address translation requests and responses. In addition, the translation logic 602 may index the DMA portion 416B of the cache according to the host tag value and use the hot tag value internally to track DMA-related address translation requests and responses.

[0108] In these embodiments, each DMA command has a host tag ("htag") containing a virtual address. If the data is greater than or equal to the pre-configured size of the data, the translation logic 602 may be configured to store a translation in the ATC 616 according to the htag of the data. The translation logic 602 may be further configured to store a translation in the ATC 616 according to the htag of the metadata (without a size limit indicating whether to cache). During the additional operation of the controller 615, as long as the subsequent TU (within the htag) uses the same translation physical address range, the address translation is used. When the DMA command begins to be transferred to or from different memory ranges, the cache translation is replaced. When the htag is assigned to different commands (such as commands using different memory locations), the cache address translation is also replaced.

[0109] Figure 7 is an example memory subsystem controller 715 including address translation circuitry 713 (or ATS) for implementing cache host memory address translation data according to some embodiments. The address translation circuitry 713 may be referenced to Figure 1 , Figure 2 , Figure 4 and Figure 6 A more detailed instantiation of address translation circuits 113, 213, 413 and / or 613 is discussed.

[0110] In various embodiments, the address translation circuitry 713 includes a pipeline of queues, buffers, and multiplexers that facilitate address translation request and response flow and interface to an address translation cache (ATC) 716, similarly as previously introduced and discussed. The multiplexers are generally included within the translation logic 602 and are therefore not always individually numbered or discussed separately from the translation logic 602.

[0111] In one embodiment, this pipeline receives address translation requests from the host interface (HIF) circuit 608 (see Figure 6 ) flows into a set of request holding queues 702. These address translation requests are typically received as a series of address translation requests from any given HIF circuit 608, so each queue in the set of request holding queues 702 contains multiple available entries for multiple virtual addresses associated with a series of incoming address translation requests, which may be received as a group, for example, from a given HIF circuit 608. The set of request holding queues 702 buffers the address translation requests each including a virtual address received from the host interface circuit. The multiplexer 703 then pushes each address translation request into both a set of reorder buffers 714 and the translation logic 602, which initially references Figure 6Discussion. Within the set of reorder buffers 714, an address translation request will provide an initial entry that will later be completed with an address translation that will be provided in response to a corresponding request HIF circuit 608. In an embodiment, the translation logic 602 is configured to store, for example, an incoming address translation from the set of reorder buffers 714 into the ATC 716.

[0112] In an embodiment, the translation logic 602 is coupled to the set of request staging queues 702, the set of reorder buffers 714, and the ATC 716, among other components. The translation logic 602 may determine the queue identifiers of the respective queues in the plurality of submission queues (SQ) and the plurality of completion queues (CQ) discussed previously and identify the queue identifier of the respective address translation request. The translation logic 602 may further index the queue portion 416A of the cache according to the queue identifier of the respective address translation request.

[0113] In at least some embodiments, the translation logic 602 is configured to store, for each address translation request in the set of request staging queues, a first address translation corresponding to a current page associated with the address translation request in the ATC 716 and a second address translation corresponding to a subsequent page that sequentially follows the current page according to the virtual address number in the ATC 716. This look-ahead buffering of address translations for a predetermined number of queues in the ATC 716 can significantly reduce the number of misses at the ATC 716 while keeping the size of the ATC reasonable in view of the overhead of the cache memory. As the cost of cache memory decreases, it is contemplated that one or more additional address translations that sequentially follow the first address translation (according to the virtual address) may also be stored in the ATC 716.

[0114] In some embodiments, for incoming DMA requests, the translation logic 602 may identify a first host tag associated with the data for DMA within at least some of the address translation requests. The translation logic may further identify a second host tag associated with metadata of the data within at least some of the address translation requests. The translation logic may then index the DMA portion 416B of the cache according to the host tag values ​​to include values ​​corresponding to each first host tag and each second host tag. In an embodiment, in response to a translation unit (TU) of a subsequent address translation request as a cached host tag within a translated memory range, the translation logic uses the address translation of the cached host tag in the ATC 716 to satisfy the subsequent address translation request. In addition, in response to a subsequent address translation request targeting a memory range different from the translated memory range, the translation logic may evict the address translation from the ATC 716. In response to the cached host tag being assigned to a different command, the translation logic 602 may also evict the address translation from the cache (ATC 716).

[0115] In some embodiments, the translation logic 602 further determines, for each incoming address translation request, whether the virtual address (or queue identifier) ​​within the address translation request is a hit or miss at the cache (ATC 716). If the virtual address has a hit within the cache, the translation logic 602 may reinsert the corresponding address translation (including the mapped physical address) into the reorder buffer 714 to be provided to the requesting HIF circuit 608. Thus, address translations that hit at the ATC 716 may be provided back to the requesting HIF circuit 608 at close to line rate without any further delay in requesting the TA / IOMMU at the host system 620 to perform the translation.

[0116] In an embodiment, the address translation circuitry 713 further includes a set of outbound request queues 704 for buffering one or more of the address translation requests that miss at the cache (ATC 716), e.g., as an additional pipeline stage for temporarily storing these address translation requests sent to the host system 620. As these address translation requests are forwarded to the host system 620, the translation logic 602 pushes each address translation request into a set of pending response queues 708. In an embodiment, the set of pending response queues 708 is configured to buffer the respective address translation requests that are waiting for address translation from the host system 620 while maintaining the order received within the set of request temporary storage queues 702. Maintaining this order within the set of pending response queues 708 helps the set of reorder buffers 714 to correctly reorder address translations coming back from the host system 620 (e.g., TA 617 or IOMMU) despite being pushed out of the set of request temporary storage queues 702 while waiting for the TA 617.

[0117] In these embodiments, the address translation request that misses at the ATC 716 is forwarded to the AXI converter 722, which issues the address translation request to the host interface 208 through a set of address channels for reading 732 and the destination is the TA 617. In this way, the address translation circuit 713 can request the translation agent of the host system (e.g., TA 617) to provide a physical address mapped to the first virtual address of the first address translation (for the current page) and the second virtual address of the second address translation (for the subsequent page that follows the current page sequentially according to the virtual address number). In response to the TA 617 providing the address translation, the address translation is received through a set of data channels 734 for reading and through the AXI converter 724.

[0118] In the disclosed embodiment, the translation logic 602 pops the incoming address translations into the set of pending response queues 708 while also storing the address translations (e.g., the first address translation and the second address translation) in the set of reorder buffers 714. The set of reorder buffers 714 also receives the correct order of the address translations from the set of pending response queues 708 with respect to the order in which the address translation requests in the set of request staging queues 702 were received. As mentioned, this may include a series of address translation requests from the same HIF circuit 608, and thus the set of reorder buffers 714 may be configured to reorder each pair of address translations for each of these address translation requests to match the order in which the address translation requests were buffered into the set of pending response queues 708. Thus, the set of reorder buffers 714 may reorder address translations in this manner and provide the first address translation (for each address translation request) to the request HIF circuit 608 in the same order as received, except for address translations that hit at the ATC 716 and have been supplied to the request HIF circuit 608. In these embodiments, the translation logic 602 detects incoming address translations and copies them from the set of reorder buffers into the ATC 716, where they may be accessed later for matching with subsequent address translation requests.

[0119] In some embodiments, the translation completion response may be returned from the host system 620 as a "miss" because the page corresponding to the virtual address has been unmapped. The translation logic 602 may then reply to the request HIF circuit 608 that the virtual address of the requested specific address translation did not hit. In response to receiving this miss TLP, the HIF circuit 608 may be triggered to generate a page request interface (PRI) message to the host system 620 to remap the virtual address to a physical page with a physical address.

[0120] In at least some embodiments, the address translation circuit 713 further includes a set of invalidation queues 718 to buffer invalidation requests received from the translation agent of the host system. More specifically, the invalidation request may be received through a set of data channels for writing 736 to the AXI converter 726, which receives the invalidation request from the host system 620. The translation logic 602 may then place each incoming invalidation request into the set of invalidation queues 718 for handling in the order received.

[0121] In these embodiments, the address translation circuit 713 further includes an invalidation handler logic 750 coupled to the set of invalidation queues 718, the host interface circuit 608, the set of request temporary queues 702, the set of pending response queues 708, and the set of reorder buffers 714. In an embodiment, the invalidation handler logic 750 detects an invalidation request within the set of invalidation queues 718, the invalidation request corresponding to a virtual address in the virtual address. The invalidation handler logic 750 may further cause the address translation associated with the virtual address to be marked as invalid within the set of request temporary queues 702, the set of pending response queues 704, the set of reorder buffers 714, and the cache (ATC 716). In an embodiment, the set of outbound request queues 704 triggers (e.g., via the translation logic 602) the set of reorder buffers 714 to mark entries therein as invalid in response to an invalidation signal received from the invalidation handler logic 750. The invalidate handler logic 750 may further send an associated invalidate request to the corresponding host interface circuit 608, also expecting that the corresponding host interface circuit 608 will clear the invalid entry.

[0122] In some embodiments, the address translation circuit 713 (e.g., corresponding queues and buffers) removes address translations marked as invalid within the set of request temporary queues 702, the set of pending response queues 704, the set of reorder buffers 714, and the cache (ATC 716). Each HIF circuit 608 may send an invalidation completion message to the address translation circuit 713, and the invalidation handler logic 750 replies to the invalidation completion message with an acknowledgment (ACK). In this way, the invalidation handler logic 750 may acknowledge that the host interface circuit 608 removed the address translation, and then send an invalidation completion response to the translation agent (TA 617). The ACK response may be used by the individual HIF circuit 608 to unfreeze any frozen function arbitration.

[0123] More specifically, when the invalidate handler logic 750 has acknowledged the address translation circuit 713 and the HIF circuit 608 has cleared the queues, buffers, and cache entries of the invalid address translation associated with the invalidate request, the invalidate handler logic 750 may send an invalidate response to the AXI translator 728. The AXI translator 728 may put the invalidate response out on a set of data lanes for writing 738 to the TA 617 at the host system 620.

[0124] Figure 8A flowchart illustrating an example method 800 of caching host memory address translation data in a memory subsystem according to some embodiments of the present disclosure. The method 800 may be performed by processing logic, which may include hardware (e.g., a processing device, a circuit system, dedicated logic, programmable logic, microcode, hardware of a device, an integrated circuit, etc.), software (e.g., instructions running or executing on a processing device), or a combination thereof. In some embodiments, the method 800 is performed by Figure 1 , Figure 2 , Figure 4 , Figure 6 and Figure 7 The corresponding address translation circuits 113, 213, 413, 613, 713 of the processor are executed. Although shown in a specific sequence or order, unless otherwise specified, the order of the process can be modified. Therefore, the illustrated embodiments should be understood as examples only, and the illustrated processes can be performed in different orders, and some processes can be performed in parallel. In addition, one or more processes can be omitted in various embodiments. Therefore, not every embodiment requires all processes. Other process flows are possible.

[0125] At operation 810 , processing logic buffers address translation requests received from a host interface circuit and each including a virtual address in a set of request temporary queues of an address translation circuit.

[0126] At operation 820, processing logic stores the respective address translation requests that are awaiting address translation from the host system in a set of pending response queues while maintaining the order received in the set of request stashing queues.

[0127] At operation 830, processing logic reorders address translations within a set of reorder buffers according to the order maintained within the set of pending response queues, wherein each address translation includes a physical address mapped to a respective virtual address.

[0128] At operation 840, processing logic sends the address translation from the reorder buffer to the corresponding host interface circuit of the host interface circuit that sent the corresponding address translation request.

[0129] At operation 850, processing logic stores a plurality of address translations associated with the address translation request in a cache coupled to the set of request staging queues and the set of reorder buffers for future access by the host interface circuitry. Also at operation 850, processing logic may further store a plurality of pointers (e.g., SGS / SGL pointers) of outstanding direct memory address (DMA) commands within one or more host interface circuits.

[0130] At operation 860, processing logic optionally reinserts the first address translation from the cache into the set of reorder buffers for subsequent requests for the first address translation by the host interface circuitry.

[0131] Fig. 9 608B, a host interface circuit that interacts with the PRI handler 219. In some embodiments, the system 900 includes an address translation circuit 613 (see Figure 6 ), host interface circuit 608 (for example, reference Figure 6 any of the host interface circuits 608A, 608B, 608C, ... 608N discussed above), processing bridge circuit 608N (see Fig.10 ), PRI handler 219 and controller memory 912. In an embodiment, the host interface circuit 608 is a host queue interface circuit 608B, which is configured to extract commands generated by the host system, and the commands are stored in one or more submission queues 306 for processing by the controller 215 or 615, as shown in FIG. Figure 3 Discuss in more detail.

[0132] In various embodiments, the host memory manager (e.g., TA 617) swaps out pages of host memory 212 for various reasons, which can be done without notifying any of the user processes or virtual machines VM(0) to VM(n). As discussed, a device including a controller 615 employing address translation circuitry 613 does not assume that there are pages required for DMA. Therefore, reference will be made to Fig.10 The page request interface discussed in more detail in conjunction with the PRI handler 219 may provide the host interface circuitry 603 with the ability to cause the translation agent 617 (eg, a host memory manager) to repin a memory page, return an acknowledgment that the page has been repinned, and provide a corresponding address translation.

[0133] In these embodiments, the host queue interface circuit 608B includes a translation request queue 904, a translation response queue 908, a hardware pipeline 907 having a number of hardware stages (HW_0 ... HW_N), a command packet queue 905 that buffers incoming command packets processed by the hardware pipeline 907, and a set of control registers 911. The set of control registers 911 may exist with respect to a physical function (PF), where associated or other virtual functions (VFs) may also use this set of control registers 911. The values ​​within the set of control registers 911 may define the allocation of each PF, how many PRI-related requests are outstanding for a given PF, and other settings or parameters of the hardware functionality of the PF.

[0134] In at least some embodiments, the host queue interface circuit 608B is configured to suspend command fetch arbitration on the submission queue 606 of the host system, which is targeted by an address translation request that misses at a cache (e.g., at the ATC 616). The host queue interface circuit 608B may further trigger the PRI handler 219 to send a page miss request to a translation agent (TA) of the host system, the page miss request including the virtual address of the address translation request. After the PRI handler receives the page miss response from the translation agent, the host queue interface circuit 608B may further receive a restart message from the PRI handler 219. The host queue interface circuit 608B may further restart command arbitration on the submission queue that has been suspended in response to the restart message.

[0135] In various embodiments, when a command packet is entering or is in one of the hardware stages of the hardware pipeline 907, the host queue interface circuit 608B submits the virtual address of the command packet to the translation request queue 904 to request the address translation circuit 613 to provide the address translation required by the hardware pipeline 907 to complete processing the command packet. If the address translation circuit 613 returns the address translation previously stored in the ATC 616, such as via the translation response queue 908, then the host queue interface circuit 608B continues to process the command packet normally without pausing.

[0136] In at least some embodiments, host queue interface circuitry 608B receives a message from address translation circuitry 613, e.g., via translation response queue 908, that an address translation request has missed at cache or ATC 616 and is therefore host interface circuitry 608 that was affected by the miss at cache. In these embodiments, the translation miss message includes the virtual address and the queue number associated with the host system's submission queue containing the command for which the address translation request was sent.

[0137] In an embodiment, this translation miss message triggers the host queue interface circuit 608B to remove the command packet from the hardware pipeline 907 and suspend command fetch arbitration on the submission queue 606, for example by referencing the values ​​within the set of control registers 911. The command packet may be associated with a memory read operation that relies on an address translation corresponding to an address translation request that missed at the cache. The host interface circuit may further send a translation miss message to the PRI handler 219 (e.g., via the controller memory 912), the translation miss message including the virtual address in the original address translation request and a queue number associated with the submission queue. For example, the queue number may be a slot in the SQ 306. This translation miss message may be queued within the controller memory 912 before reaching the PRI handler 219, such as by referencing the values ​​within the set of control registers 911. Fig.10 Discussion.

[0138] In various embodiments, the translation miss message includes information identifying where and / or how to reintroduce the command packet into the hardware pipeline. For example, an address translation request is triggered in hardware level HW_2, and the command packet is removed at this third hardware level of the hardware pipeline 907, the information included in the translation miss message to the PRI handler 219 may be an identifier of this third hardware level (HW_2). In an embodiment, the set of controller registers 911 stores hardware level identifiers mapped to the respective hardware levels of the hardware pipeline 907. In response to a restart message received for the PRI handler 219 corresponding to a translation miss message previously sent to the PRI handler 219, the host interface circuit 608 may reintroduce the command packet into the hardware pipeline 907, where the command packet was originally removed. In this way, DMA can continue (even if there is a slight delay during PRI-related processing) and enable the host interface circuit system to continue processing commands and other actions more efficiently when interacting with the host memory 212.

[0139] In additional embodiments, the PRI handler 219 is coupled to or included within the host queue interface circuit 608B. In these embodiments, the PRI handler 219 is configured to send a page miss request to a translation agent (TA) of the host system in response to receiving a translation miss message, the page miss request including the virtual address and the queue number of the SQ 306. The PRI handler 219 may be further configured to send a restart message to the host queue interface circuit 608B in response to receiving a page miss response, wherein the restart message also contains the queue number associated with the submission queue of the host system.

[0140] Fig.10 1 is a schematic block diagram of a system 1000 (e.g., a processing bridge system) including a processing bridge circuit 608N that implements page request interface support when caching host memory address translation data in a memory subsystem in accordance with some embodiments. In these embodiments, the system 1000 includes the processing bridge circuit 608N, a number of queues stored in a controller memory 1012 (e.g., of the controller 615), a PRI handler 219, and an AXI matrix 1023. In some embodiments, the controller memory 1012 is a DRAM, but may also be a combination of SRAM or volatile memory. The controller memory 1012 may be coupled to the controller memory 912 ( Fig. 9 ) are the same. In these embodiments, the processing bridge circuit 608N is coupled between the controller memory 1012 and the AXI matrix 1023, which provides data and address communications to the host system 620 and may also access the address translation circuit 613 previously discussed herein. Reference to the host system 620 may be understood to include reference to any other host system herein, including Figure 2 The host system 220.

[0141] In various embodiments, the PRI handler 219 is coupled to the controller memory 1012 and is optionally integrated with the processing bridge circuit 608N. As discussed, the PRI handler 219 may be processing logic that may include hardware (e.g., a processing device, circuitry, dedicated logic, programmable logic, microcode, hardware of a device, an integrated circuit, etc.), firmware (e.g., instructions running or executed on the processor 217), or a combination thereof. The controller memory 1012 may store a number of different queues to facilitate communication between the PRI handler 219 and the processing bridge circuit 608N.

[0142] In an embodiment, for example, the controller memory 1012 stores a set of translation miss queues 1070 coupled to respective HIF circuits 608. For example, host interface circuit 608A may be coupled to HIF1 circuit translation miss queue 1070A, host interface circuit 608B may be coupled to HIF2 circuit translation miss queue 1070B, host interface circuit 608C may be coupled to HIF3 circuit translation miss queue 1070C, and another host interface circuit may be coupled to HIF4 circuit translation miss queue 1070D, with additional or fewer miss request queues contemplated. Each queue in the set of translation miss queues 1070 receives a page miss message from the respective HIF circuit 608, wherein each page miss message indicates a miss at address translation cache (ATC) 616 for a host address translation. The controller memory 1012 may further store an outbound queue 1072, a host request queue 1074, and a bridge translation miss queue 1075, which will be discussed in greater detail.

[0143] In various embodiments, the PRI handler 219 tracks activities associated with each page miss message received from a respective one of the host interface circuits 608 at the set of translation miss queues 1070. In these embodiments, the PRI handler 219 tracks translation miss messages received from the host interface circuit 608, each translation miss message including a virtual address of a miss at the cache (or ATC 616). The PRI handler 219 further removes duplicate translation miss messages having the same virtual address to avoid, for example, unnecessarily sending multiple duplicate page miss requests to the host system 620. The PRI handler 219 further creates a set of page miss requests from non-duplicate translation miss messages categorized as page request groups. Thus, the page miss requests are associated with the page miss messages received from the respective host interface circuits. The PRI handler 219 further queues the page request groups sent to the translation agent of the host system, for example, within the host request queue 1074.

[0144] In these embodiments, for example, each page request group corresponds to a host interface circuit in host interface circuit 608. Each page miss request causes a translation agent (e.g., TA 617) to repin a physical memory page to the virtual address of the corresponding page miss request. In at least some embodiments, a page miss request is a PCIe message TLP containing the virtual address of the page and optionally a process address space identifier (PASID). The PASID may be included when the host system 620 executes a process (e.g., virtual machines VM(0) to VM(n)) that shares controller 615 or a memory device controlled by controller 615.

[0145] In some embodiments, each page requires its own page miss request, and potentially, for large I / O loads (such as may be required for interaction with a NIC), there may be thousands of page miss requests. Each page miss request may invoke page handler software at the host system 620, which batches multiple page requests. To simplify the description, TA 617 (which may be an IOMMU) may be understood to include this page handler software. In an embodiment, sending the same page address in multiple page miss requests may not trigger an error, but doing so may consume host CPU cycles.

[0146] In these embodiments, the PRI handler 219 further categorizes the page miss requests into groups of page requests, for example, based on an incoming queue in the set of translation miss queues 1070 or another identifier such as a hardware identifier of the source of the page miss request. The PRI handler 219 may further limit the outstanding page miss requests for a particular host interface circuit 608 to a threshold number of page miss requests. This may be permeated in firmware or as an actual limit based on the allocation of the controller memory 1012 to the respective translation miss queues assigned to the particular host interface circuit 608.

[0147] In various embodiments, the outbound queue 1072 can buffer memory commands (e.g., host memory reads or host memory writes) directed to the host memory 212 of the host system 620 and other messages directed to the host system 620. The processing bridge circuit 608N can include a multiplexer 1011 that variably transfers page miss requests from the host request queue 1074 (e.g., page response groups) and one or more memory commands or other messages from the outbound queue 1072 to the outbound buffer 1014 of the processing bridge circuit 608N. The outbound buffer 1014 is thus coupled to the controller memory 1012 and is configured to buffer page request groups and memory commands. In some embodiments, the multiplexer 1011 is part of the outbound queue handler logic 1002 of the processing bridge circuit 608N that decodes packets from the outbound queue 1072 and the host request queue 1074.

[0148] In an embodiment, the queue handling logic 1002 further includes a multiplexer 1016 that determines what the outbound buffer 1014 contains and directs the contents to the correct destination within the processing bridge circuit 608N. For example, if the outbound buffer 1014 contains a first page request group, the multiplexer 1016 causes page miss requests for the first page request group to be sent to a translation agent (e.g., TA 617) of the host system 620, which causes the translation agent to repin a physical memory page to each respective virtual address of the respective page miss request.

[0149] 1008 , the queue handling logic 1002 sends the memory command with the address translation to the host system 620. However, if the queue handling logic 1002 determines that the outbound buffer 1014 contains a memory command (e.g., a read or write to the host memory 212), the multiplexer 1016 directs the memory command to the additional outbound queue handling logic 1002. In various embodiments, the queue handling logic 1002 requests an address translation of the virtual address of the memory command from the address translation circuit 613 (or ATS). For example, the address translation request may be buffered in the translation request queue 1004. In response to receiving the address translation from the address translation circuit 613, for example, via the translation response queue 1008, the queue handling logic 1002 sends the memory command with the address translation to the host system 620 normally. However, in response to receiving a miss response from the address translation circuit 613, the queue handling logic 1002 discards the memory command and submits a page miss message to the bridge translation miss queue 1075. The PRI handler 219 may then process this page miss message as well as page miss messages received through the set of translation miss queues 1070 and submit a corresponding page miss request (or group of page requests) for the translation miss into the host request queue 1074. Once the address translation comes back from the TA 617 along with the page request response, e.g., via the inbound buffer 1054 (which will be discussed in more depth), the PRI handler 219 may resubmit the command packet (now including the address translation of the virtual address) to an outbound queue 1072, e.g., an outbound command queue.

[0150] In various embodiments, the queue translation logic 1002 buffers the read commands into a pending completion tracking queue 1060 that is also sent to the host system 620. Write commands and page miss requests may be sent to the host system 620 via the AXI translator 1028. The AXI translator 1028 may put the write commands and page miss requests out on a set of data lanes for writing 1038 to the TA 617 at the host system 620.

[0151] In these embodiments, the read command may be sent to the host system 620 via the AXI converter 1022, which issues the read command to the TA 617 of the host system 620 through a set of address channels for read 1032. Further, in these embodiments, a pending completion tracking queue 1060 tracks incoming data received through a set of data channels for read 1034 and through the AXI converter 1024, which passes the incoming data for the read command. In an embodiment, the pending completion tracking queue 1060 detects data received for the particular read command being tracked and once the data is complete, sends the data to the inbound completion buffer 1064 of the processing bridge circuit 608N. In at least one embodiment, the inbound completion buffer 1964 receives the data for the particular read command from the pending completion tracking queue 1060 and provides the data to the inbound completion queue 1080 stored in the controller memory 1012.

[0152] In various embodiments, the page request response returned from the TA 617 of the host system 620 is received via a set of data channels for writing 1036 into the AXI converter 1026. The AXI converter 1026 passes the page request response (and other memory commands and messages, including invalidation requests) received from the host system 620. The processing bridge circuit 608N further includes an inbound buffer 1054 for receiving the page request response and other memory commands and messages from the host system 620. In an embodiment, the inbound queue handling logic 1052 is coupled between the inbound buffer 1054 and the controller memory 1012.

[0153] The inbound queue handling logic 1052 may include a multiplexer 1056 for determining whether the inbound buffer 1054 includes a page request response, a memory command (for the controller memory 1012), a message, or an invalidation request. If an invalidation request, the multiplexer 1056 sends the invalidation request to the address translation circuit 613. The multiplexer 1056 sends a message, a memory command, or a page request response further to this content to a second multiplexer 1058. The multiplexer 1058 may send messages and memory commands to the inbound queue 1082 and page request responses to the host response queue 1082 of the memory controller 1012. The host response queue 1084 may buffer page request responses received from the inbound buffer 1054. The inbound queue 1082 may buffer incoming memory commands and other messages received from the inbound buffer 1054.

[0154] In at least some embodiments, the PRI handler 219 receives these page request responses corresponding to the page miss requests via the host response queue 1084. In some embodiments, each page request response indicates that the TA 617 has repinned the physical memory page to the virtual address of the corresponding page miss request. The page request response may further contain an address translation of the virtual address in the original page miss request (or page request group), which may be provided to the address translation circuit 613 for storage in the ATC 616.

[0155] In various embodiments, each page request response may be a TLP message sent by the host system 620 indicating a repin failure or success for a group of pages or for some individual pages. The TA 617 of the host system 620 does not fault a subset of pages, but not all pages, corresponding to a resident page miss request. Due to the functionality of the PRI handler 219 in conjunction with the processing bridge circuit 608N, the PRI handler 219 may not be allowed to send an "untranslated address" message even after a successful page request response.

[0156] In the disclosed embodiment, the controller memory 1012 includes a set of restart queues 1090 that buffer restart messages from the PRI controller 219 corresponding to the page request responses buffered in the host response queue 1084. In an embodiment, each restart queue 1090 is coupled to a respective host interface circuit 608 to provide a corresponding restart message to the respective host interface circuit 608. More specifically, the host interface circuit 608A may be coupled to the HIF1 circuit restart queue 1090A, the host interface circuit 608B may be coupled to the HIF2 circuit restart queue 1090B, the host interface circuit 608C may be coupled to the HIF3 circuit restart queue 1090C, and another host interface circuit may be coupled to the HIF4 circuit restart queue 1090D. Each restart message placed in one of the set of restart queues 1090 by the PRI handler 219 may be provided from the controller memory 1012 to the corresponding host interface circuit 608. As shown in FIG. Fig. 9 As discussed, the restart message causes the host interface circuit 608 to reinsert the command packet into the hardware pipeline 907 from which it had been initially removed.

[0157] Fig.11 1 is a flow chart of an example method 1100 for page request interface support when enabling cache of host memory address translation data in a memory subsystem according to some embodiments. The method 1100 may be performed by processing logic, which may include hardware (e.g., a processing device, a circuit system, a dedicated logic, a programmable logic, a microcode, hardware of a device, an integrated circuit, etc.), software (e.g., instructions running or executing on a processing device), or a combination thereof. In some embodiments, the method 1100 is performed by Figure 2 , Figure 6 and Fig. 9 PRI handler 219 of the embodiment. Although shown in a specific sequence or order, unless otherwise specified, the order of the processes may be modified. Therefore, the illustrated embodiments should be understood as examples only, and the illustrated processes may be performed in a different order, and some processes may be performed in parallel. In addition, one or more processes may be omitted in various embodiments. Therefore, not every embodiment requires all processes. Other process flows are possible.

[0158] At operation 1110, processing logic stores a plurality of address translations associated with address translation requests from a host interface circuit in a cache. Each address translation request includes a virtual address that needs to be translated.

[0159] At operation 1120, processing logic tracks translation miss messages received from the host interface circuitry, each translation miss message including a virtual address of a miss at the cache.

[0160] At operation 1130, processing logic removes duplicate translation miss messages having the same virtual address.

[0161] At operation 1140 , processing logic creates a plurality of page miss requests from non-duplicate translation miss messages categorized into page request groups, each page request group corresponding to a host interface circuit of the plurality of host interface circuits.

[0162] At operation 1150, processing logic queues the group of page requests to be sent to the translation agent of the host system. In an embodiment, the processing bridge circuit 608N then sends the queued group of page requests to the host system for handling.

[0163] Fig.12 1 is a flow chart of an example method 1200 for handling page request interface support in a host submission queue while enabling host memory address translation data to be cached in a memory subsystem in accordance with some embodiments. The method 1200 may be performed by processing logic, which may include hardware (e.g., a processing device, a circuit system, a dedicated logic, a programmable logic, a microcode, hardware of a device, an integrated circuit, etc.), software (e.g., instructions running or executing on a processing device), or a combination thereof. In some embodiments, the method 1200 is performed by Figure 6 and Fig. 9 The host queue interface circuit 608B of the embodiment of the present invention is executed. Although shown in a specific sequence or order, unless otherwise specified, the order of the processes may be modified. Therefore, the illustrated embodiments should be understood as examples only, and the illustrated processes may be performed in a different order, and some processes may be performed in parallel. In addition, one or more processes may be omitted in various embodiments. Therefore, not every embodiment requires all processes. Other process flows are possible.

[0164] At operation 1210, processing logic stores address translations with address translation requests from a host queue interface circuit in a cache (eg, ATC), each address translation including a virtual address.

[0165] At operation 1220, processing logic suspends command fetch arbitration on a submission queue of the host system that is targeted by an address translation request that missed at the cache.

[0166] At operation 1230, processing logic triggers a page request interface (PRI) handler to send a page miss request to a translation agent of the host system, the page miss request including the virtual address of the address translation request. In an embodiment, the PRI handler is the PRI handler 219 discussed throughout this disclosure.

[0167] At operation 1240 , processing logic receives a restart message from the PRI handler after the PRI handler receives the page miss response from the translation agent.

[0168] At operation 1250 , processing logic restarts command arbitration on the suspended submission queue in response to the restart message.

[0169] Fig.13 1 is a schematic block diagram of a system 1300 including a completion queue interface circuit 608C implementing page request interface support when enabling host memory address translation data to be cached in a memory subsystem in accordance with some embodiments. In various embodiments, the system 1300 includes an ATS 613, a completion queue interface circuit 608C, and a host command automation (HCA) circuit 1301. As shown in FIG. Figure 6 As discussed and illustrated, the ATS 613 may include an ATC 616 and translation logic 602 for determining address translations of host virtual addresses.

[0170] In various embodiments particularly relevant to NVMe handling by a host system, a completion queue (CQ) 308 ( Figure 3 ) is a circular buffer of fixed slot size with state for issuing completion commands. Completion commands may be uniquely identified by a combination of an associated SQ identifier and a command identifier assigned by host software. Multiple submission queues (SQs) may be associated with a single CQ. This feature may be used when a single worker thread processes all command completions via one CQ, even if the commands originate from multiple SQs. The CQ head pointer may be updated by host software after host software processes a completion entry indicating the last free CQ entry.

[0171] In some embodiments, a phase (P) bit may be defined in the completion entry of CQ 308 to indicate whether the entry has been recently issued without consulting the register. This enables host software to determine whether the new entry was issued as part of a previous or current round of completion notifications. Specifically, the controller 215 or 615 toggles the phase bit at each round through the CQ positions.

[0172] In at least some embodiments, the completion queue interface circuit 608C is configured to write a completion message (or notification) to the host system's completion queue (CQ) 308 after detecting that the controller 215 or 615 has completed handling the corresponding host command. For example, the completion queue interface circuit 608C may send these completion messages or notifications, such as that the controller has completed handling a write command or a read command, to the host system in response to a CQ doorbell memory request as part of the NVMe command completion. In these embodiments, even if the completion queue interface circuit 608C handles the report of command completion, even at this stage, the address translation request may not hit at the cache of the ATS 613 (e.g., at the ATC 616).

[0173] In these embodiments, HCA circuit 1301 arbitrates the completion of commands and coordinates the sending of CQ completion messages to the host system. To this end, HCA circuit 1301 may create a completion queue (CQ) 308 (see FIG. 10 ) placed in CMB 300. Figure 3 ) or host memory 212 (see Figure 2 ) in the CQ 308. The HCA circuit 1301 may generate a message signaling interrupt (e.g., MSI-X in NVMe) to alert the host system to process the entries in the CQ 308.

[0174] In various embodiments, the HCA circuit 1301 examines the operation code (opcode) of the incoming memory command to determine whether to complete the handling using firmware or hardware automation. If hardware automation, the HCA circuit 1301 can be configured to directly extract the command, perform the data transfer associated with the command, and insert the command identification metadata associated with the command into one of a set of completion request queues 1304 (individually labeled P0_Wr, P0_Rd, P1_Wr, P1_Rd) to initiate completion. However, if the HCA circuit 1301 uses firmware for handling, the HCA circuit 1301 can trigger the PRI handler 219 to provide (and / or generate) command identification metadata, which is inserted into a firmware completion request queue (labeled FW_Compl) in the set of completion request queues 1304 to initiate completion.

[0175] In at least some embodiments, the command metadata may include, for example, a command identifier (ID) and a host tag (htag). In these embodiments, the host system assigns a specific command identifier (ID) to an entry buffered in the submission queue (SQ306) and can also be used to identify the command of the corresponding entry in the completion queue 308. In addition, the HCA circuit 1301 (or other host command circuits operating similarly) can generate a host tag (or htag) to track completion requests within the completion queue interface circuit 308C. For ease of explanation, the command identification metadata will be referred to as a "completion request" once it is queued in the set of completion request queues 1304.

[0176] In these embodiments, multiplexer 1308 selects the completion request to be processed through the hardware and logic of the completion queue interface circuit 608C. At operation 1310, a command table read is performed to determine one or more parameters associated with the CQ entry (corresponding to the completion request) in order to generate a completion message or notification to the host system. At operation 1312, a pacing check is performed to see if pacing is enabled, e.g., meaning that entries into CQ 308 will be paced through the completion queue at a particular rate or in a particular manner. For example, if CQ 308 is full (which can be detected at operation 1316), pacing can be triggered. If CQ 308 is full, the completion request within the hardware pipeline can be reinserted back into the original completion request queue where the completion request was initiated. In some embodiments, an additional multiplexer can be employed that is triggered by a retry enable signal asserted when pacing is enabled.

[0177] After taking into account any pacing, at operation 1320, the completion queue interface circuit 608C may identify the current CQ tail (within the circular buffer) and increment the position associated with its completion of the memory command being arbitrated. At operation 1322, the translation request circuitry 1302 of the completion queue interface circuit 608C issues a translation request to the ATS 613 via the translation request queue 904. The ATS 613 may provide a response to the translation request via the translation response queue 908. This address translation process is referred to as Figure 6 Discuss in more detail.

[0178] In various embodiments, the translation request circuitry 1302 may then detect an address translation request for a completion request that missed at the cache at operation 1324. The completion request is buffered within a particular completion request queue in the set of completion request queues 1304 and is associated with an entry in a completion queue (CQ 308) of the host system. If the translation request misses at operation 1328, then at operation 1330, the translation request circuitry 1302 pauses the particular completion request queue in response to the detected miss, such as via controlling the multiplexer 1308. In addition, the translation request circuitry 1302 may further reinsert the completion request into the set of completion request queues 1304 in response to the detected miss at operation 1332, similarly as performed when the CQ 308 is full.

[0179] In an embodiment, the translation request circuit system 1302 also triggers a page request interface (PRI) handler (e.g., PRI handler 219) to send a page miss request to a translation agent (TA) of the host system. The page miss request includes the virtual address of the address translation request. In order to trigger the PRI handler 219, the translation request circuit system 1302 may send a translation miss message to the PRI handler 219, the translation miss message including the virtual address and the command identifier associated with the entry in the completion queue. In these embodiments, after the PRI handler receives the page miss response from the translation agent (TA), the translation request circuit system 1302 receives a restart message from the PRI handler 219. At operation 1330, the translation request circuit system 1302 may restart a specific completion request queue in response to the restart message, for example, via the control multiplexer 1308.

[0180] In some embodiments, the PRI handler 219 is a PRI handler coupled to or included in the completion queue interface circuit 608C. The PRI handler 219 may be configured to send a page miss request to a translation agent (TA) of the host system in response to receiving a translation miss message, the page miss request including a virtual address and a command identifier associated with an entry in the completion queue. The PRI handler 219 may further send a restart message to the completion queue interface circuit 608C in response to receiving a page miss response, the restart message containing a host tag associated with a particular completion queue.

[0181] In an embodiment, the translation request circuitry 1302 detects a completion request recycled from the set of completion request queues and requests the address translation circuitry 613 to obtain an address translation from the host system corresponding to the virtual address. The translation request circuitry 1302 may further receive the address translation from the address translation circuitry 613 and use the address translation to generate a completion queue message associated with the completion request and targeted to an entry in the completion queue, as will be explained in more detail.

[0182] In various embodiments, at operation 1350, the completion queue interface circuit 608C performs a tail pointer and phase update. For example, at operation 1340, the completion queue interface circuit 608C may read the incremented CQ tail buffer updated at operation 1320. In addition, at operation 1345, the completion queue interface circuit 608C may write the updated tail value to the tail pointer memory using the CQ command ID. The phase bit (discussed previously) may be updated at operation 1350 in order to update the phase. In some embodiments, the completion queue interface circuit 608C may update a latency first-in-first-out (FIFO) buffer 1352 based on tracking this phase information through a series of completion requests. Updating the FIFO buffer 1352 may be performed to assign each command whose completion is being requested into a time bucket depending on the latency required to complete the command (ringing from command fetch to submission of a completion message or notification). In an embodiment, the FIFO buffer 1352, tail pointer buffer and / or memory and associated pointers may be stored in a memory generally represented as control register 911 (see also Fig. 9 ) in or associated with local memory and / or registers.

[0183] In these embodiments, at operation 1360, the completion queue interface circuit 608C generates a completion queue message that is written into the completion queue (CQ 308). For example, the completion queue interface circuit 608C may determine a submission queue identifier that identifies a corresponding entry of the submission queue of the command and generate a completion queue message for the completion request that includes the command identifier, the submission queue identifier, and the host tag. Additionally, at operation 1365, the completion queue interface circuit 608C generates an MSI-X (or similar MSI-related) message to alert the host system to process the entry in CQ 308.

[0184] Fig.14 14 is a flow chart of an example method 1400 for automating page request interface support when handling host memory address translation data cached in a memory subsystem according to some embodiments. The method 1400 may be performed by processing logic, which may include hardware (e.g., a processing device, a circuit system, a dedicated logic, a programmable logic, a microcode, hardware of a device, an integrated circuit, etc.), software (e.g., instructions running or executing on a processing device), or a combination thereof. In some embodiments, the method 1400 is performed by Figure 6 and Fig.13 The completion queue interface circuit 608C of the embodiment of the present invention is executed. Although shown in a specific sequence or order, unless otherwise specified, the order of the process can be modified. Therefore, the illustrated embodiment should be understood as only as an example, and the illustrated processes can be performed in a different order, and some processes can be performed in parallel. In addition, one or more processes can be omitted in various embodiments. Therefore, not every embodiment requires all processes. Other process flows are possible.

[0185] At operation 1410, processing logic detects an address translation request for a completion request that missed at a cache, the completion request buffered within a particular completion request queue of the set of completion request queues and associated with an entry in a completion queue of the host system.

[0186] At operation 1420, processing logic pauses the particular completion request queue in response to the detected miss.

[0187] At operation 1430 , processing logic triggers a page request interface (PRI) handler to send a page miss request to a translation agent of the host system, the page miss request including a virtual address of the address translation request.

[0188] At operation 1440 , processing logic receives a restart message from the PRI handler after the PRI handler receives the page miss response from the translation agent.

[0189] At operation 1450, processing logic restarts the particular completion request queue in response to the restart message.

[0190] Fig.15 1 is a block diagram of a host interface circuit system 1500 including an HDPA circuit 232 and a pointer extraction interface circuit (PFIC) 1502 for handling data transfer operations according to some embodiments. In some embodiments, the host interface circuit system 1500 is the host interface circuit system 603 and the HDPA circuit 232 and the PFIC 1502 are reference Figure 6 Thus, the PFIC 1502 and the HDPA circuit 232 are configured to interact with the ATS 613 and thus the ATC 616 or 716 of the ATS 613, similar to that discussed with reference to the host interface circuit 608 in the embodiment of obtaining a translation of a virtual address (VA).

[0191] In some embodiments, one of the host interface circuits 608, which in at least one embodiment may be the HCA circuit 1301, optionally splits the memory command (whether it is a write command or a read command) into a number of chopped commands of a certain size, such as a maximum data transfer size (MDTS). By way of example only, each chopped command is a 2MB or 4MB (or similar MDTS) block of instruction information, depending on the implementation of the controller 215 of the system 200. In other embodiments, the host interface circuit system 608 does not split the memory command, which remains a single non-chopped command.

[0192] In an embodiment, the host interface circuit system 1500 further includes the HCA 1301 ( Fig.13 ), which is coupled between the HDPA circuit 232, the PFIC 1502, and the PRI handler 219. In some embodiments, the host interface circuit system 1500 further includes a pointer buffer 1510 coupled to the PFIC 1502, a replay queue 1515 coupled between the HCA 1301 and the PRI handler 219, and a PRI queue 1517 also coupled between the HCA 1301 and the PRI handler 219. In some embodiments, the PFIC 1502 buffers pointers retrieved from the host memory 212 in the pointer buffer 1510. In an embodiment, any of the pointer buffer 1510, the replay queue 1515, and the PRI queue 1517 are located in the DRAM 222, the SRAM 204 or 224, or the memory device 130 (see Figure 2 ). In some embodiments, HCA 1301 is integrated within HDPA circuit 232, and thus HDPA circuit 232 may also be understood to optionally perform the operations discussed herein with reference to HCA 1301, as highlighted by the dashed line between HDPA circuit 232 and PRI handler 219. In these embodiments, HDPA circuit 232 also accesses replay queue 1515 and PRI queue 1517 in the manner discussed with reference to HCA circuit 1301.

[0193] In at least some embodiments, host interface circuitry 1500 (including HCA 1301) coordinates pointer extraction by PFIC 1502 with data transfer via HDPA circuit 232 (and reference Figures 13 to 14The completion queue arbitration discussed above) is used to chop commands one at a time. In particular, the operations of the host interface circuitry 1500 are ordered so as to extract a pointer, which is then used to access (from the translation pointer) a physical memory (e.g., memory device 130) that knows the physical address to which data is written (for a write command) or from which data is read (for a read command). As will be discussed in detail, this coordination may include tracking specific commands (or chop commands) where a translation unit (TU) for a write command misses at a cache (e.g., ATC 616) so that the write command can be replayed later because the write command is executed atomically. Furthermore, this coordination may include tracking specific TUs with offsets within specific chop commands for read commands that miss at the cache so that the TUs can be replayed later because the TUs for a read command can be handled out of order.

[0194] In various embodiments, as discussed, HDPA circuit 232 controls data transfer while activating control path 240 for extracting (or retrieving) pointer 310 ( Figure 3 ), such as PRP / SGL, issuing completion and interrupts, and activating DMA 230 for actual data transfer between host system 220 and memory device 130 (see Figure 2 ). The HDPA circuit 232 may activate the DMA 230 by dispatching a TU-sized memory write or memory read request containing a translated physical address (PA) and other TU-based information for each TU to the host system for data transfer.

[0195] In at least some embodiments, the PFIC 1502 requests the ATC 616 to translate the virtual addresses of those pointers in order to ultimately determine the physical address destinations (e.g., data transfers) of the memory commands residing in the submission queue (SQ) 306. In various embodiments, these memory commands are distinguished as logical block address (LBA) read and write commands and non-LBA read and write commands. In these embodiments, the LBA read / write commands are with respect to specific LBA addresses (e.g., linear addresses) associated with virtual addresses of the host system 220, such as software-based addressing of the host memory 212. In some embodiments, in contrast, non-LBA read and write commands do not have explicit LBA-based addresses and may be generated by legacy I / O devices coupled to the host system 220. For example, non-LBA read and write commands may be addressed using other non-linear addressing schemes such as cylinder-head-sector (CHS), extended CHS, zone-recording, or the like.

[0196] To facilitate retrieval of pointers in the event of a cache miss, in some embodiments, PFIC 1502 interacts with PRI handler 219 to directly send a translation miss message and receive a translation miss response from PRI handler 219 for an address translation miss at ATC 616 (see dashed line, indicating an optional path). In these embodiments, PFIC 1502 triggers PRI handler 219 by sending a translation miss message to PRI handler 219. In an embodiment, the translation miss message includes the virtual address and the restart point of the start of a write command (LBA or non-LBA) or a non-LBA read command (see Figures 16A to 17 ) or the virtual address and the start of the chop command or the end of the LBA read command translation position (see Figures 17 to 18 ).

[0197] In some embodiments, PFIC 1502 interacts with HCA circuit 1301 with the support of HDPA circuit 232, as HCA circuit 1301 interfaces with PRI handler 219 to resolve address translation cache (ATC) 616 (see Figure 6 ) at the address translation miss. In an embodiment, as shown in reference Fig.13 As discussed, HCA circuit 1301 checks the operation code (opcode) of an incoming memory command to determine whether to complete the handling using firmware or hardware automation. In the disclosed embodiment, HCA circuit 1301 also arbitrates the completion of commands and coordinates sending CQ completion messages to the host system as well as other host command automation tasks.

[0198] For example, in various embodiments in an effort to complete a memory command, the HCA circuit 1301 sends a pointer fetch request to the PFIC 1502 to cause the PFIC 1502 to retrieve a pointer for the memory command being processed. If during the processing of a memory command and in accordance with some embodiments, the PFIC 1502 detects a translation miss at the ATC 616, the PFIC 1502 generates a translation miss message including the virtual address that missed at the cache and sends the translation miss message to the HCA circuit 1301. This translation miss message may trigger the HCA circuit 1301 to also generate a translation miss message to the PRI handler 219 and eventually receive an address translation message back from the PRI handler 219 that the page that missed at the ATC 616 has been re-pinned in the host memory 212. In response to the address translation message, in some embodiments, the HCA circuit 1301 commands or triggers the PFIC 1502 to reinsert the TU and move forward through the pointer fetch for the memory command.

[0199] As illustrated, according to at least some embodiments, the PFIC 1502 is configured to send an address translation request 1520 for a translation unit (TU) to the ATC 616. As will be discussed in more detail, a TU includes a subset of pointers and therefore the PFIC 1502 sends an address translation request with an LBA or non-LBA corresponding to a virtual address (VA) at the beginning of the TU. Because the HDPA circuit 232 operates at the TU level, this is sufficient in at least some embodiments.

[0200] According to some embodiments, the PFIC 1502 determines that the VA of a particular TU misses at the ATC 616. For example, the PFIC 1502 may determine the VA of the TU miss at the ATC 616 based on a previous address translation miss for the same virtual address or due to receiving a non-acknowledge status (NAK) from the HDPA circuit 232. In these embodiments, the PFIC 1502 sends a notification to the HCA circuit 1301 that the TU misses at the cache, thereby triggering the HCA circuit 1301 to generate a translation miss message to the PRI handler 219. In other embodiments, the PFIC 1502 sends the translation miss message (dashed line) directly to the PRI handler 219. In at least some embodiments, pointers used to write data to the host memory 212 are not translated by the PFIC 1502, and the PFIC 1502 may translate only those pointers required for the PFIC 1502 to traverse the PRP / SGL that constitutes the pointer list. For example, pointers contained within the SGL descriptor may not need to be translated by the PFIC 1502 but may be translated by the HDPA circuit 232 .

[0201] In some embodiments, the HCA circuit 1301 provides the HDPA circuit 232 with a TU-sized command descriptor, which is buffered in the descriptor queue 1540 and used to track the data transfer operation associated with the memory command whose pointer is being retrieved by the PFIC 1502. In some embodiments, the address translation request 1520 sent by the PFIC 1502 is passed through or otherwise visible to the HDPA circuit 232 so that the HDPA circuit 232 can track the state of the pointer required to implement the data transfer and obtain the pointer directly. For example, in an embodiment, the physical address (PA) returned from the ATC 616 in a successful address translation 1530 is received in response to the address translation request 1520. In some embodiments, after receiving an unsuccessful address translation from the ATC 616, the HDPA circuit 232 sends an address translation miss notification to the HCA circuit 1301, such as a non-acknowledgement (NAK) for the associated command descriptor.

[0202] In various embodiments, when the HDPA circuit 232 receives a translated address (e.g., PA) from the ATC 616, the HDPA circuit 232 also provides status in the form of an acknowledgement to the PFIC 1502. For example, a successful address translation may be reported as a zero ("0"), such as an acknowledgement (ACK), and an unsuccessful address translation may be reported as a one ("1"), such as a non-acknowledgement (NAK). The ACK status may trigger the PFIC 1502 to delete the pointers from the pointer buffer 1510 because those address translations were successfully delivered to the HDPA circuit 232. The NAK status of the write command may trigger the storage of the TU outside of the pointer buffer 1510 (e.g., in local memory), which will have to be at least partially reinserted after the PRI handler 219 resolves the address translation miss at the ATC 616, as will be referred to in the description of FIG. Figures 16A to 21 Discuss in more detail.

[0203] Fig.16A is a diagram of an exemplary memory command 1600 (eg, a write command or a non-logical block address (LBA) read command) in response to a translation miss during a pointer fetch or data transfer, according to some embodiments. Fig.16A In some embodiments, memory command 1600 is an LBA write command, a non-LBA write command, or a non-LBA read command. As illustrated, memory command 1600 is not split (e.g., truncated) and is therefore a series of consecutive translation units that each include a subset of pointers. In some embodiments, several such memory commands are executed sequentially, such as sequentially processing LBAs or bytes from a starting LBA to an ending LBA or a total number of bytes covering each memory command.

[0204] In some embodiments, in response to a miss in cache (e.g., at ATC 616) of a translation unit (TU) 1602 of a memory command 1600, the PFIC 1502 (or HDPA 232) triggers the PRI handler 219 to send a page miss request to a translation agent (TA) of the host system 220. In response to the translation miss, in an embodiment, the PFIC 1502 (or HDPA 232) sends a translation miss message to the HCA circuit 1301 or PRI handler 219 to trigger handling of the translation miss, which in some embodiments may be queued to the HCA PRI queue. The translation miss message may include the virtual address of the TU 1602 where the miss occurred and the restart point at the TU 1601, which is the start of the write command 1600. In some embodiments, the page miss request causes the translation agent to re-pin the physical memory page to the virtual address (VA). In these embodiments, the page miss response received back from the TA indicates that the physical page has been repinned. In some embodiments, in response to a cache miss for TU 1602, PFIC 1502 (or HDPA 232) discards a pointer to the entire memory command 1600, which will be reprocessed later from the first TU 1601 (e.g., the restart point at the beginning of the memory command 1600).

[0205] In at least some embodiments, to perform this discarding, each memory command that must be reprocessed may be flushed out of the command processing pipeline of the host interface circuitry 603 and / or 1500, some of which have been discussed elsewhere in this disclosure. This flushing may be performed by continuing to execute the command processing pipeline, but without performing any actual data transfers. The pointer buffer 1510 and queue of address translation requests 1520 between the PFIC 1502 and the HDPA 232 may thus be freed to handle further processing of subsequent memory commands.

[0206] In various embodiments, the PRI handler 219 collects page request addresses of translation units ordered after the translation unit that missed at the cache until the memory command has been cleared from the host interface circuitry by, for example, a command end indicator appearing at the end of the memory command 1600. The PRI handler 219 may further send a page request message for each respective page request address of the respective translation unit to the TA. The PRI handler 219 may further wait until detection of the command end indicator causes the translation unit to be reintroduced into the pointer extraction interface circuitry from a restart point (e.g., the first TU 1601), for example, by tracking PRI requests according to a particular VA.

[0207] In some embodiments, a single page miss response for the entire group of page requests associated with the corresponding TU of the memory command 1600 is received back from the TA. In other embodiments, individual page miss responses are received back, one for each corresponding TU of the memory command 1600. Thus, in these embodiments, in response to receiving a page miss response from the TA for the TU 1602 that missed at the cache, the PRI handler 219 causes the memory command to be reprocessed, such as by re-injecting the memory command into the HCA 1301, which sends a new pointer fetch request to the PFIC 1502 and sends the descriptor of the TU to the HDPA 232. In some embodiments, this reintroduction reintroduces the memory command 1600 to the host queue interface circuit 608B ( Figure 6 In these embodiments, the PRI handler 219 may also clear the HCA circuit 1301 (e.g., a descriptor of the HCA circuit 1301 or other tracking means associated with individual TUs) before reintroducing the memory command 1600 to the host queue interface circuit 608B, which may restart the processing of the memory command 1600.

[0208] Fig. 16B 1 is a diagram of an exemplary write command 1600 that has been split into a chop command and different actions by the PFIC 1502 (or other host interface circuitry) in response to a translation miss during a pointer fetch according to some embodiments. In some embodiments, the HCA circuit 1301 Fig.16A ) is split into multiple MDTS chop commands. By way of example only, each chop command is instruction information for 2MB or 4MB (or similar MDTS) blocks and each TU within the chop command is a pointer, such as a 2KB or 4KB (or similar) value, such as a virtual address of a pointer. The HCA circuit 1301 may provide an identifier for each chop command, such as "10" for the first chop command in a write command, "00" for any intermediate chop commands, and "01" for the last chop command. The memory command 1600 includes four chop commands, but any given memory command may include more or fewer chop commands.

[0209] In an exemplary embodiment, the PFIC 1502 buffers the chop commands in the pointer buffer 1510 one TU at a time as the pointers are retrieved from the system memory and provided to the HDPA circuit 232, which is illustrated from left to right. In this way, the PFIC 1502 keeps track of the virtual address of each pointer or TU so that data transfers are performed sequentially. Fig. 16B1600. The translation miss at TU 1607 is approximately halfway through the second chop command, so PFIC 1502 detects a translation miss at TU 1607 approximately 1 / 3 of the way through the entire second chop command. In response to the translation miss, in the illustrated embodiment, PFIC 1502 sends a translation miss message to HCA circuit 1301 or PRI handler 219 to trigger handling of the translation miss, which may be queued in PRI queue 1517 (e.g., FIFO queue) waiting to be transmitted to PRI handler 219. The translation miss message may include the virtual address of TU 1607 where the miss occurred and the restart point at TU 1602, which in this embodiment is the beginning of write command 1600.

[0210] In addition, in at least some embodiments, PFIC 1502 discards the pointers buffered so far from pointer buffer 1510 and triggers an abort to retrieve the pointers for memory command 1600 because the writes are handled sequentially, causing memory command 1600 to be completed concurrently or not completed, and therefore delayed. Specifically, a translation miss means that memory command 1600 will need to be restarted (or reprocessed) after obtaining a missed address translation. Therefore, in an embodiment, PFIC 1502 also sends an abort message for each chop command to other circuit systems of host interface circuit system 1500 (including at least HCA circuit 1301 and / or PRI handler 219) to convey the abort to retrieve the pointers for the remainder of the chop command that has not yet been processed. For example, in Fig. 16B In the example embodiment of , the remainder of the chop commands includes a third and final chop command.

[0211] In an embodiment, in response to the translation miss message, the PRI handler 219 sends a page miss request to a translation agent (TA) of the host system, the page miss request including the virtual address. In some embodiments, the PRI handler 219 sends the page miss request to the translation agent to cause the translation agent to repin the physical memory page to the virtual address. In an embodiment, the page miss response from the TA indicates that the physical page has been repinned.

[0212] In at least some embodiments, in response to receiving a page miss response from the TA, the PRI handler 219 causes the memory command located at the queue number in the submission queue to be reprocessed. In these embodiments, before the memory command is reprocessed, the PFIC 1502 sends an address translation request for the corresponding translation unit (TU) of the corresponding chop command of the subsequent memory command to the address translation circuit 613. In this way, the pointer buffer 1510 and other command processing queues remain active while waiting for the memory command 1600 to be reprocessed. In an embodiment, a firmware injection command for restarting the processing of the memory command 1600 (after PRI handling) is injected into the host queue interface circuit 608B, which has been discussed in more detail previously. However, the HCA circuit 1301 can receive instructions from the host queue interface circuit 608B to coordinate and control the completion of the memory command 1600.

[0213] In at least some embodiments, referring to the DMA memory command 1600, the HDPA circuit 232 detects a missed translation for the descriptor for which the data transfer is being prepared for the TU 1605 in the first chop command. In these embodiments, the HDPA circuit 232 sends a translation miss message to the HCA circuit 1301, which then notifies the PRI handler 219 (e.g., via the PRI queue 1517) to retrieve the correct physical address of the TU 1605. The translation miss message may include the virtual address that was missed at the ATC 616 and the buffer list address for de-allocating the data buffer (in the host memory 212) associated with the first chop command. The HDPA circuit 232 may then take further steps to send a NAK message for the htag containing the specific TU 1605 to the host interface circuit system 1500 (including the HCA circuit 1301). In these embodiments, a detected translation miss causes the HDPA circuit 232 to also discard the associated descriptor (chop command for a DMA write command) and restart the DMA memory command after receiving a translation response from the PRI handler or HCA circuit.

[0214] In some embodiments, the HCA circuit 1301 sends additional translation miss messages to the PRI handler 219, each translation miss message being associated with a subsequent chop command in a set of chop commands and including another buffer list address for de-allocating a data buffer associated with the subsequent chop command. In this way, the data buffer is freed for the entire DMA write command, which will be restarted after the PRI handler 219 resolves the page miss at the cache.

[0215] In some embodiments, the PRI handler 219 receives a page miss response from a translation agent, such as TA 517. In these embodiments, the PRI handler 219 causes the DMA memory command to be reprocessed by the host interface circuitry 603 or 1500 in response to the page miss response, for example to ensure that the DMA memory command is executed atomically.

[0216] Fig.17 1 is a flow chart of an example method 1700 for handling a translation miss during a pointer fetch of a memory command according to some embodiments. The method 1700 may be performed by processing logic, which may include hardware (e.g., a processing device, a circuit system, a dedicated logic, a programmable logic, a microcode, hardware of a device, an integrated circuit, etc.), software (e.g., instructions running or executing on a processing device), or a combination thereof. In some embodiments, the method 1700 is performed by Fig.15 The pointer extraction interface circuit 1502 of the embodiment of the present invention is executed. Although shown in a specific sequence or order, unless otherwise specified, the order of the process can be modified. Therefore, the illustrated embodiment should be understood as only an example, and the illustrated process can be performed in a different order, and some processes can be performed in parallel. In addition, one or more processes can be omitted in various embodiments. Therefore, not every embodiment requires all processes. Other process flows are possible.

[0217] At operation 1710, processing logic buffers a plurality of pointers associated with a memory command in pointer buffer 1510. In some embodiments, the memory command resides in a submission queue of the host system. In some embodiments, the memory command is a write command or a non-LBA read command.

[0218] At operation 1720 , processing logic sends an address translation request for a corresponding translation unit of the memory command to the address translation circuit (ATS) 613 , each translation unit including a subset of the plurality of pointers.

[0219] At operation 1730, processing logic triggers the page request interface (PRI) handler 219 to send a page miss request to the translation agent of the host system following the address translation request of the translation unit for the memory command that missed at the cache (e.g., at the ATC 616). In an embodiment, the page miss request includes a virtual address of the translation unit (or TU).

[0220] At operation 1740, processing logic discards the plurality of pointers from the pointer buffer 1510, for example because it is necessary to later cause the memory command to be reinserted into the host interface circuitry for reprocessing.

[0221] Fig.181 is a flow chart of an example method 1800 for handling a translation miss during a direct memory access (DMA) data transfer of a memory command according to some embodiments. The method 1800 may be performed by processing logic, which may include hardware (e.g., a processing device, a circuit system, a dedicated logic, a programmable logic, a microcode, hardware of a device, an integrated circuit, etc.), software (e.g., instructions running or executing on a processing device), or a combination thereof. In some embodiments, the method 1800 is performed by Fig.15 The HDPA circuit 232 and / or HCA 1301 of the present invention are executed. Although shown in a specific sequence or order, unless otherwise specified, the order of the processes may be modified. Therefore, the illustrated embodiments should be understood as examples only, and the illustrated processes may be performed in a different order, and some processes may be performed in parallel. In addition, one or more processes may be omitted in various embodiments. Therefore, not every embodiment requires all processes. Other process flows are possible.

[0222] At operation 1810, processing logic buffers a plurality of descriptors associated with a translation unit of a direct memory access (DMA) memory command of a host system in a descriptor queue 1540. In some embodiments, the DMA memory command is a DMA write command or a non-LBA DMA read command. In some embodiments, each descriptor is associated with a pointer to a destination in a memory device containing data associated with the corresponding translation unit. In some embodiments, the memory device is Figure 1 Memory device 130 or other physical memory.

[0223] At operation 1820, processing logic sends an address translation request for pointers of corresponding translation units of the DMA memory command to the address translation circuit. In some embodiments, processing logic segments the DMA memory command into a group of chop commands, detects that a plurality of pointers of a first chop command in the group of chop commands have been completely fetched from the host memory, and initiates an address translation request for a plurality of pointers corresponding to a plurality of translation units in response to the detection.

[0224] At operation 1830, the processing logic triggers the page request interface (PRI) handler 219 to send a page miss request to the translation agent of the host system after the address translation request for the first pointer of the first translation unit that missed at the cache. In an embodiment, the page miss request includes the virtual address of the first translation unit. In some embodiments, the page miss request causes the translation agent to re-pin the physical memory page to the virtual address and the page miss response indicates that the physical page has been re-pinned.

[0225] In some embodiments, PRI handler 219 sends a page miss request to a translation agent of the host system in response to receiving a translation miss message, the page miss request including the virtual address. PRI handler 219 may further cause a DMA memory command to be reprocessed by the host interface circuitry in response to receiving a page miss response.

[0226] At operation 1840, processing logic discards the plurality of descriptors to abort execution of the DMA memory command.

[0227] Fig.19 is a diagram of an exemplary LBA read command 1900 that has been split into a chop command and different actions for the PFIC (or other host interface circuitry) in response to a translation miss during a pointer fetch, according to some embodiments. This example is similar to Fig. 16B The same example is shown, only four chop commands are included, because the HCA circuit 1301 can split the read command 1900 into fewer or more than four chop commands. In addition, in some embodiments, the pointer that retrieves the read command 1900 proceeds from the first or starting chop command (the "10" chop command) to the end or last chop command (the "01" chop command).

[0228] exist Fig.19 In the example of , while the first chop command (Chop_0) starting with TU 1903 is processed without a translation miss, PFIC 1502 detects a translation miss at TU 1905 midway through the second chop command (Chop_1). In some embodiments, in response to detecting this miss at ATC 616, PFIC 1502 causes a translation miss message to be sent to page request interface (PRI) handler 219, such as optionally via HCA circuit 1301. In some embodiments, the translation miss message contains the virtual address of the translation unit and the restart point of the chop command. In some embodiments, the translation miss message triggers PRI handler 219 to send a page miss request to a translation agent (TA) of host system 220. In at least some embodiments, the translation miss message is triggered by PRI handler 219 to send a page miss request to a translation agent (TA) of host system 220 for a write command. Fig. 16B Unlike the embodiment of the present invention, the restart point is the logical block address (LBA) at the beginning of the second chop command (e.g., Chop_1) or at the translation unit (TU) within the pointer buffer 1510. This restart point does not need to return to the beginning of the LBA read command 1900 within the pointer buffer 1510 because the read operation is handled out of order and can continue from wherever the read command 1900 stopped processing.

[0229] Additional references Fig.19In some embodiments, PFIC 1502 further causes one or more subsequent messages for subsequent chop commands following the chop command (e.g., for the third chop command (Chop_2) and the fourth chop command (Chop_3)) to be sent to PRI handler 219. In some embodiments, PFIC 1502 causes subsequent messages to be sent from HCA circuit 1301, as shown in FIG. Fig.15 In some embodiments, one or more subsequent messages each include the LBA of a corresponding subsequent chop command within the pointer buffer 1510, such as the second (2 nd ) News and the third (3 rd As described, in some embodiments, these messages contain a value indicating that no PRI is required, but do not include the starting LBA of each corresponding subsequent chop command.

[0230] In some embodiments, the PRI handler 219 sends a page miss request to the translation agent (TA) to cause the translation agent to re-pin the physical memory page to the virtual address. In an embodiment, the page miss response indicates that the physical page has been re-pinned by the TA. In some embodiments, therefore, the PRI handler 219 receives the page miss response from the translation agent. In an embodiment, the PRI handler 219 causes the pointer whose address translation request hits at the cache to be reinserted into the pointer buffer 1510 in response to the page miss response. In some embodiments, the PRI handler 219 waits until the end of the last command (e.g., Chop_3) is received before starting to reinsert the pointer, as just described. In some embodiments, when causing the pointer to be reinserted into the pointer buffer 1510, the PFIC 1502 maintains the chop boundaries of multiple pointers corresponding to the chop commands starting from the restart point.

[0231] In an embodiment, the PRI handler 219, in response to the page miss response, also causes the insertion into the pointer buffer 1510 of a pointer corresponding to a chop command starting from the restart point of the pointer buffer 1510 identified by the page miss response. In some embodiments, the PFIC 1502 further stores in local memory (e.g., SRAM or DRAM) corresponding pointers of a plurality of pointers that its address translation request hits at a cache (e.g., at the ATC 616) while waiting for the page miss response and in response to the pointer buffer 1510 becoming full. In this way, these address translations will be readily available for reinsertion into the pointer buffer 1510 when necessary because the pointer buffer 1510 is too full to continue to hold that information.

[0232] Fig. 202000 is a flow chart of an example method 2000 for handling a translation miss during pointer fetch of an LBA read command according to some embodiments. The method 2000 may be performed by processing logic, which may include hardware (e.g., a processing device, a circuit system, a dedicated logic, a programmable logic, a microcode, hardware of a device, an integrated circuit, etc.), software (e.g., instructions running or executing on a processing device), or a combination thereof. In some embodiments, the method 2000 is performed by Fig.15 The pointer extraction interface circuit 1502 of the embodiment of the present invention is executed. Although shown in a specific sequence or order, unless otherwise specified, the order of the process can be modified. Therefore, the illustrated embodiment should be understood as only an example, and the illustrated process can be performed in a different order, and some processes can be performed in parallel. In addition, one or more processes can be omitted in various embodiments. Therefore, not every embodiment requires all processes. Other process flows are possible.

[0233] At operation 2010 , processing logic buffers a plurality of pointers associated with a plurality of chop commands of LBA read commands residing in a submission queue of a host system in a pointer buffer 1510 of host interface circuitry of a processing device.

[0234] At operation 2020, processing logic sends an address translation request for a corresponding translation unit of a corresponding axe command to an address translation circuit (eg, ATS 613), each translation unit including a subset of the plurality of pointers.

[0235] At operation 2030 , processing logic detects, at a cache (eg, ATC 616 ) of the address translation circuitry, an address translation request miss of a translation unit for a chop command of the plurality of chop commands.

[0236] At operation 2040, the processing logic sends a translation miss message to the page request interface (PRI) handler 219. The translation miss message contains the virtual address of the translation unit and the restart point of the axe command. The translation miss message is used to trigger the PRI handler to send a page miss request to the translation agent of the host system.

[0237] Additional references Fig.15, the HDPA circuit 232 performs slightly different operations in response to a TU of a DMA read command (which is associated with a logical block address) missing at the cache (e.g., at the ATC 616) compared to a miss of a DMA write command. For example, in some embodiments, the HDPA circuit 232 sends a NAK to the HCA circuit 1301 only for descriptors with TUs that missed at the cache. As a result, subsequent descriptors within the corresponding chopped command remain in the descriptor queue 1540 and are able to continue ahead of the TU that missed at the cache, where the latter can be reprocessed after the PRI handler 219 handles the cache miss.

[0238] Thus, in these embodiments, the HDPA circuit 232 causes a translation miss message to be sent (e.g., via the HCA circuit 1301) to the PRI handler 219 for the VA of the pointer that missed at the cache. In some embodiments, the translation miss message contains the virtual address of the first pointer and triggers the PRI handler 219 to send a page miss request to the translation agent (e.g., TA517) of the host system. In some embodiments, in response to receiving the page miss response from the translation agent, the PRI handler 219 causes the first descriptor of the first translation unit to be reinserted into the data path interface circuit (e.g., the HDPA circuit 232). In some embodiments, the first descriptor is inserted in the correct order relative to at least one second descriptor whose pointer also missed at the cache, but maintaining the order is optional because the TUs of the DMA read command can be handled out of order.

[0239] In some embodiments, the HDPA circuit 232 sends a non-acknowledgement (NAK) to the PFIC 1502 for the first TU having a pointer that missed at the ATC 616. In response to the NAK, in some embodiments, the PFIC 1502 stores the order in which the pointers (e.g., a first pointer associated with the first TU) and at least one second pointer relative to the second TU that also missed at the cache flowed through the pointer buffer 1510. In some embodiments, the PRI handler 219 corresponding to the page miss request receives a page miss response from the translation agent. In these embodiments, in response to the page miss response, the PRI handler 219 sends a replay indicator for the first TU to the PFIC 1502. In some embodiments, the replay indicator triggers the PFIC 1502 to reinsert the first pointer into the pointer buffer 1510 for access by the HDPA circuit 232. In some embodiments, the reinsertion is optionally in the correct order relative to the second pointer.

[0240] In at least some embodiments, to trigger the PRI handler 219 to send a page miss request to the TA, the data path interface circuit 232 further sends a non-acknowledge (NAK) message for a translation unit (e.g., a first TU) to the HCA circuit 1301. In an embodiment, the NAK triggers the HCA circuit 1301 to send a translation miss message to the PRI handler 219. In an embodiment, the NAK message includes a virtual address and a value identifying the first TU. In these embodiments, the HCA circuit 1301 also inserts a first descriptor for the first TU into the replay queue 1515. In this way, the HCA 1301 (and the HDPA circuit 232) is able to track TU-specific descriptors and associated information (including the VA of the TU) that will be used to replay the descriptors into the HDPA circuit 232 to, for example, be able to complete a DMA read command associated with a TU that missed at the cache.

[0241] In some embodiments, for example, PRI handler 219 receives a page miss response from a translation agent (TA) associated with the first TU and sends a replay message in response to the page miss response to HCA circuit 1301. In these embodiments, the replay message triggers HCA circuit 1301 to reinsert the first descriptor from replay queue 1515 into HDPA circuit 232.

[0242] Fig.21 2 is a flow chart of an example method 2100 for handling a translation miss during a DMA data transfer of an LBA-based read command according to some embodiments. The method 2100 may be performed by processing logic, which may include hardware (e.g., a processing device, a circuit system, a dedicated logic, a programmable logic, a microcode, hardware of a device, an integrated circuit, etc.), software (e.g., instructions running or executing on a processing device), or a combination thereof. In some embodiments, the method 2100 is performed by Fig.15 The HDPA circuit 232 and / or HCA 1301 of the present invention are executed. Although shown in a specific sequence or order, unless otherwise specified, the order of the processes may be modified. Therefore, the illustrated embodiments should be understood as examples only, and the illustrated processes may be performed in a different order, and some processes may be performed in parallel. In addition, one or more processes may be omitted in various embodiments. Therefore, not every embodiment requires all processes. Other process flows are possible.

[0243] At operation 2110, processing logic buffers a plurality of descriptors associated with the translation units of the host system's DMA read command in a descriptor queue 1540 of the HDPA 232 of a processing device (e.g., controller 215). In an embodiment, each descriptor is associated with a pointer to a destination in a memory device containing data associated with the corresponding translation unit. In some embodiments, the memory device is Figure 1Memory device 130 or other physical memory.

[0244] At operation 2120, processing logic sends an address translation request for pointers of corresponding translation units of the DMA read command to the address translation circuit. In some embodiments, the DMA read command is associated with a logical block address. In some embodiments, processing logic segments the DMA read command into a group of chop commands, detects that multiple pointers of a first chop command in the group of chop commands have been completely fetched from the host memory, and initiates address translation requests for multiple pointers corresponding to multiple translation units in response to the detection.

[0245] At operation 2130, processing logic detects, at a cache of the address translation circuitry, an address translation request miss for a first pointer of a first translation unit linked to a first descriptor of the plurality of descriptors. In an embodiment, the page miss request causes the translation agent to repin the physical memory page to the virtual address and the page miss response indicates that the physical page has been repinned.

[0246] In some embodiments, PRI handler 219 sends a page miss request to a translation agent of the host system in response to receiving a translation miss message, the page miss request including the virtual address. PRI handler 219 may further cause a DMA read command to be reprocessed by the host interface circuitry in response to receiving a page miss response.

[0247] At operation 2140, processing logic causes a translation miss message to be sent to a page request interface (PRI) handler 219. In an embodiment, the translation miss message contains the virtual address of the first pointer and triggers the PRI handler 219 to send a page miss request to the translation agent of the host system.

[0248] Fig. 22 An example machine illustrating a computer system 2200 in which a set of instructions for causing the machine to perform any one or more of the methodologies discussed herein may be executed. In some embodiments, the computer system 2200 may correspond to a host system (e.g., Figure 1 ) that includes, is coupled to, or utilizes a memory subsystem (e.g., Figure 1 In some embodiments, the machine may be connected (e.g., using a network) to other machines in a LAN, an intranet, an extranet, and / or the Internet. The machine may operate in the capacity of a server or a client user machine in a client-server network environment, as a peer machine in a peer-to-peer (or distributed) network environment, or as a server or a client user machine in a cloud computing infrastructure or environment.

[0249] The machine may be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular phone, a network appliance, a server, a network router, a switch or a bridge, or any machine capable of executing (sequentially or otherwise) a set of instructions that specify actions to be taken by the machine. Furthermore, while a single machine is described, the term "machine" shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.

[0250] The example computer system 2200 includes a processing device 2202, a main memory 2204 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) (e.g., synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM)), etc.), a static memory 2206 (e.g., flash memory, static random access memory (SRAM), etc.) and a data storage system 2218, which communicate with each other via a bus 2230.

[0251] The processing device 2202 represents one or more general processing devices, such as a microprocessor, a central processing unit, or the like. More specifically, the processing device may be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor implementing other instruction sets or multiple processors implementing a combination of instruction sets. The processing device 2202 may also be one or more special processing devices, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a network processor, or the like. The processing device 2202 is configured to execute instructions 2226 for performing the operations and steps discussed herein. The computer system 2200 may further include a network interface device 2208 to communicate through a network 2220.

[0252] The data storage system 2218 may include a machine-readable storage medium 2224 (also referred to as a computer-readable medium) on which is stored one or more sets of instructions 2226 or software embodying any one or more of the methodologies or functions described herein. The instructions 2226 may also reside completely or at least partially within the main memory 2204 and / or within the processing device 2202 during execution thereof by the computer system 2200, the main memory 2204 and the processing device 2202 also constituting machine-readable storage media. The machine-readable storage medium 2224, the data storage system 2218, and / or the main memory 2204 may correspond to Figure 1 Memory subsystem 110.

[0253] In one embodiment, instructions 2226 include instructions for implementing Figure 1The functional instructions of the controller 115 of the present invention. Although the machine-readable storage medium 2224 is shown as a single medium in the example embodiment, the term "machine-readable storage medium" should be taken to include a single medium or multiple media storing one or more sets of instructions. The term "machine-readable storage medium" should also be taken to include any medium capable of storing or encoding a set of instructions for machine execution and causing the machine to perform any one or more of the methodologies of the present disclosure. Therefore, the term "machine-readable storage medium" should be taken to include (but not limited to) solid-state memory, optical media, and magnetic media.

[0254] Some portions of the foregoing detailed description have been presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is generally conceived here to be a self-consistent sequence of operations leading to a desired result. Operations are those requiring physical manipulation of physical quantities. Typically, but not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, combined, compared, and otherwise manipulated. It has proven convenient at times, primarily for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

[0255] It should be remembered, however, that all of these and similar terms should be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. The present disclosure may involve actions and processes of computer systems or similar electronic computing devices that manipulate or transform data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system's memories or registers or other such information storage systems.

[0256] The present disclosure also relates to an apparatus for performing the operations herein. This apparatus may be specially constructed for the intended purpose, or it may comprise a general purpose computer selectively activated or reconfigured by a computer program stored in the computer. This computer program may be stored in a computer-readable storage medium, such as (but not limited to) any type of disk (including floppy disks, optical disks, CD-ROMs, and magneto-optical disks), read-only memory (ROM), random access memory (RAM), EPROM, EEPROM, magnetic or optical cards, or any type of medium suitable for storing electronic instructions, each coupled to a computer system bus.

[0257] The algorithms and displays presented herein are not inherently related to any particular computer or other device. Various general purpose systems may be used in conjunction with the programs according to the teachings herein, or it may prove convenient to construct more specialized devices to perform the methods. The structures of various of these systems will appear as set forth in the description below. In addition, the present disclosure is not described with reference to any particular programming language. It should be appreciated that various programming languages ​​may be used to implement the teachings of the present disclosure described herein.

[0258] The present disclosure may be provided as a computer program product or software, which may include a machine-readable medium having instructions stored thereon, which can be used to program a computer system (or other electronic device) to perform a process according to the present disclosure. A machine-readable medium includes any mechanism for storing information in a form that can be read by a machine (e.g., a computer). In some embodiments, a machine-readable (e.g., computer-readable) medium includes a machine (e.g., computer) readable storage medium, such as a read-only memory ("ROM"), a random access memory ("RAM"), a magnetic disk storage medium, an optical storage medium, a flash memory component, etc.

[0259] In the foregoing description, embodiments of the present disclosure have been described with reference to specific example embodiments of the present disclosure. It should be understood that various modifications may be made to the present disclosure without departing from the broader spirit and scope of the embodiments of the present disclosure as set forth in the appended claims. Therefore, the description and drawings should be viewed in an illustrative rather than a restrictive sense.

Claims

1. A memory subsystem controller, comprising: A host interface circuit system, which is used to interact with a host system and a memory device, the host interface circuit system comprising: a data path interface circuit comprising a descriptor queue; and address translation circuitry coupled to the data path interface circuitry, the address translation circuitry for handling address translation requests of the host system and including a cache for storing address translations; and The data path interface circuit is used for: buffering a plurality of descriptors associated with a translation unit of a direct memory access (DMA) memory command of the host system in the descriptor queue, each descriptor being used to relate to a pointer to a destination in the memory device of data associated with the corresponding translation unit, wherein the DMA memory command is one of a DMA write command or a non-logical block address (non-LBA) DMA read command; sending an address translation request for the pointer of the corresponding translation unit of the DMA memory command to the address translation circuit; triggering a page request interface PRI handler to send a page miss request to a translation agent of the host system after an address translation request for a first pointer of a first translation unit that misses at the cache, wherein the page miss request includes a virtual address of the first translation unit; and The plurality of descriptors are discarded to abort execution of the DMA memory command.

2. The memory subsystem controller of claim 1 , further comprising the PRI handler coupled to the data path interface circuit and configured to perform operations comprising: receiving a page miss response from the translation agent, wherein the page miss request is used to cause the translation agent to repin a physical memory page to the virtual address, and wherein the page miss response indicates that the physical page has been repinned; and Responsive to the page miss response, causing the DMA memory command to be reprocessed by the host interface circuitry.

3. The memory subsystem controller of claim 1 , wherein the DMA memory command is segmented into a set of chopped commands, and wherein the data path interface circuit is further configured to: detecting that a plurality of pointers to a first chop command in the set of chop commands have been completely fetched from the host memory; and In response to the detecting, an address translation request is initiated for the plurality of pointers corresponding to a plurality of translation units.

4. The memory subsystem controller of claim 3, wherein the first translation unit is one of the plurality of translation units, the host interface circuitry further comprising a host command automation (HCA) circuit coupled to the data path interface circuit, the data path interface circuit further for providing the virtual address to the HCA circuit, and wherein the HCA circuit is for: sending a translation miss message to the PRI handler triggering the PRI handler to send the page miss request, wherein the translation miss message includes the virtual address and a buffer list address for deallocating a data buffer associated with the first chop command; and Additional translation miss messages are sent to the PRI handler, each translation miss message being associated with a subsequent chop command in the set of chop commands and including another buffer list address for de-allocating a data buffer associated with the subsequent chop command.

5. The memory subsystem controller of claim 1 , wherein the first translation unit is one of a plurality of translation units associated with a host tag of the DMA memory command, the host interface circuitry further comprising a pointer extraction interface circuit coupled to the data path interface circuit, and wherein the data path interface circuit is further configured to send an acknowledgment ACK for each of the plurality of translation units to the pointer extraction interface circuit, wherein the ACK causes the pointer extraction interface circuit to discard the pointers of the plurality of translation units.

6. The memory subsystem controller of claim 1 , wherein the first translation unit is one of a plurality of translation units associated with a host tag of the DMA memory command, and wherein the page miss request further includes a buffer list address for deallocating a data buffer associated with the plurality of translation units.

7. The memory subsystem controller of claim 1 , wherein to trigger the PRI handler, the data path interface circuit is configured to send a translation miss message to the PRI handler, the translation miss message including the virtual address and a restart point at which the memory command begins, the memory subsystem further comprising the PRI handler coupled to the host interface circuitry and configured to perform operations including: In response to receiving the translation miss message, sending the page miss request to the translation agent of the host system, the page miss request including the virtual address; and In response to receiving the page miss response, causing the DMA memory command to be reprocessed through the host interface circuitry from the restart point.

8. The memory subsystem controller of claim 7, wherein the operations further comprise: collecting page request addresses of translation units ordered after the translation unit that missed at the cache until the memory command has been cleared from the host interface circuitry via an end-of-command indicator; sending a page request message for each corresponding page request address of the corresponding translation unit to the translation agent; and Waiting until the end-of-command indicator is detected causes the translation unit to be reintroduced into the pointer extraction interface circuit from the restart point.

9. A memory subsystem controller, comprising: A host interface circuit system, which is used to interact with a host system and a memory device, the host interface circuit system comprising: a data path interface circuit comprising a descriptor queue; and address translation circuitry coupled to the data path interface circuitry, the address translation circuitry for handling address translation requests of the host system and including a cache for storing address translations; and The data path interface circuit is used for: buffering in the descriptor queue a plurality of descriptors associated with a translation unit of a direct memory access (DMA) read command of the host system, each descriptor being associated with a pointer to a destination in the memory device of data associated with a corresponding translation unit, wherein the DMA read command is associated with a logical block address; Sending an address translation request for the pointer of the corresponding translation unit of the DMA read command to the address translation circuit; detecting, at the cache, an address translation request miss for a first pointer of a first translation unit linked to a first descriptor of the plurality of descriptors; and A translation miss message is caused to be sent to a page request interface PRI handler, wherein the translation miss message contains the virtual address of the first pointer and is used to trigger the PRI handler to send a page miss request to the translation agent of the host system.

10. The memory subsystem controller of claim 9, wherein the host interface circuitry further comprises a pointer extraction interface circuit coupled to the data path interface circuit, and wherein the data path interface circuit is further operable to send a non-acknowledgement NAK for the first translation unit to the pointer extraction interface circuit, in response to which the pointer extraction interface circuit is operable to store the first pointer.

11. The memory subsystem controller of claim 10, further comprising the PRI handler coupled to the data path interface circuit and configured to perform operations comprising: receiving a page miss response from the translation proxy; and In response to the page miss response, a replay indicator for the first translation unit is sent to the pointer extraction interface circuit, the replay indicator is used to trigger the pointer extraction interface circuit to reinsert the first pointer into a pointer buffer for access by the data path interface circuit.

12. The memory subsystem controller of claim 9, further comprising the PRI handler coupled to the data path interface circuit and configured to perform operations comprising: receiving a page miss response from the translation proxy; and In response to the page miss response, causing the first descriptor of the first translation unit to be reinserted into the data path interface circuit.

13. The memory subsystem controller of claim 9, wherein the host interface circuitry further comprises: Replay queue; and a host command automation HCA circuit coupled to the replay queue, the data path interface circuit, and the PRI handler, wherein to trigger the PRI handler to send the page miss request, the data path interface circuit is further configured to send to the HCA circuit a non-acknowledged NAK message for the first translation unit that triggers the HCA circuit to: send the translation miss message to the PRI handler, wherein the NAK message includes the virtual address and a value identifying the first translation unit; and inserting the first descriptor of the first translation unit into the playback queue.

14. The memory subsystem controller of claim 13, further comprising the PRI handler coupled to the data path interface circuit and configured to perform operations comprising: receiving a page miss response from the translation proxy; and In response to the page miss response, a replay message is sent to the HCA circuitry for triggering the HCA circuitry to reinsert the first descriptor from the replay queue into the data path interface circuitry.

15. The memory subsystem controller of claim 9, wherein the DMA read command is segmented into a set of chopped commands, and wherein the data path interface circuit is further configured to: detecting that a plurality of pointers including the first pointer of a first chop command in the set of chop commands have been completely fetched from the host memory; and In response to the detecting, an address translation request is initiated for the plurality of pointers corresponding to a plurality of translation units.

16. The memory subsystem controller of claim 9, wherein the page miss request is used to cause the translation agent to repin a physical memory page to the virtual address, and wherein the page miss response indicates that the physical page has been repinned.

17. A method comprising: buffering a plurality of descriptors associated with a translation unit of a direct memory access (DMA) read command of a host system in a descriptor queue of a data path interface circuit of the processing device, each descriptor being associated with a pointer to a destination in a memory device of data associated with a corresponding translation unit, wherein the DMA read command is associated with a logical block address; Sending an address translation request for the pointer of the corresponding translation unit of the DMA read command to an address translation circuit; detecting, at a cache of the address translation circuitry, an address translation request miss for a first pointer of a first translation unit linked to a first descriptor of the plurality of descriptors; and A translation miss message is caused to be sent to a page request interface PRI handler, wherein the translation miss message contains the virtual address of the first pointer and is used to trigger the PRI handler to send a page miss request to the translation agent of the host system.

18. The method according to claim 17, further comprising: Sending a non-acknowledgement NAK for the first translation unit to the pointer extraction interface circuit; and The first pointer is stored by the pointer extraction interface circuit in response to the NAK.

19. The method according to claim 18, further comprising: Receiving, by the PRI handler, a page miss response from the translation agent; and In response to the page miss response, the PRI handler sends a replay indicator for the first translation unit to the pointer extraction interface circuit, wherein the replay indicator is used to trigger the pointer extraction interface circuit to reinsert the first pointer into a pointer buffer for access by the data path interface circuit.

20. The method of claim 17, further comprising: Receiving, by the PRI handler, a page miss response from the translation agent; and In response to the page miss response, causing, by the PRI handler, the first descriptor of the first translation unit to be reinserted into the data path interface circuit.

21. The method of claim 17, further comprising: sending a non-acknowledgement NAK message for the first translation unit to a host command automation (HCA) circuit, wherein the NAK message includes the virtual address and a value identifying the first translation unit; Sending the translation miss message to the PRI handler by the HCA circuit; and The first descriptor of the first translation unit is inserted into a playback queue by the HCA circuit.

22. The method of claim 21, further comprising: Receiving, by the PRI handler, a page miss response from the translation agent; and In response to the page miss response, a replay message is sent by the PRI handler to the HCA circuit to trigger the HCA circuit to reinsert the first descriptor from the replay queue into the data path interface circuit.

23. The method of claim 17, wherein the DMA read command is segmented into a group of chopped commands, the method further comprising: detecting that a plurality of pointers including the first pointer of a first chop command in the set of chop commands have been completely fetched from a host memory; and In response to the detecting, an address translation request is initiated for the plurality of pointers corresponding to a plurality of translation units.

Images