Three-layer hierarchical memory system

Abstract

Systems, devices, and methods related to a three-tier hierarchical memory system are described herein. A three-tier hierarchical memory system can utilize persistent memory to store data that is typically stored in non-persistent memory, thereby increasing the amount of storage space allocated to a computing system at a lower cost than methods that rely solely on non-persistent memory. An example device may include persistent memory and one or more non-persistent memories, the non-persistent memories being configured to map addresses associated with an input / output (I / O) device to addresses in a logic circuit system before the device receives a request from the I / O device to access data stored in the persistent memory, and to map the addresses associated with the I / O device to addresses in the non-persistent memory after the device receives the request and accesses the data.

Description

Technical Field

[0001] The present disclosure relates generally to semiconductor memories and methods, and more particularly to a three-level hierarchy type memory system. Background Art

[0002] Memory devices are typically provided as internal semiconductor integrated circuits in computers or other electronic systems. There are many different types of memory, including volatile and non-volatile memory. Volatile memory may require power to maintain its data (e.g., host data, error data, etc.), and includes random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), and synchronous dynamic random access memory (SDRAM), etc. Non-volatile memory can provide persistent data by retaining stored data when power is off, and can include NAND flash memory, NOR flash memory, and resistance variable memory, such as phase change random access memory (PCRAM), resistive random access memory (RRAM), and magnetoresistive random access memory (MRAM), such as spin torque transfer random access memory (STT RAM), etc.

[0003] A memory device may be coupled to a host (e.g., a host computing device) to store data, commands, and / or instructions for use by the host while a computer or electronic system is running. For example, during operation of a computing or another electronic system, data, commands, and / or instructions may be transferred between the host and the memory device. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Figure 1 is a functional block diagram of a device including hierarchical memory components according to several embodiments of the present disclosure.

[0005] Figure 2 is a functional block diagram of a computing system including hierarchical memory components according to several embodiments of the present disclosure.

[0006] Figure 3 is another functional block diagram of a computing system including hierarchical memory components in accordance with several embodiments of the present disclosure.

[0007] Figure 4 is a flow chart showing data access operations according to several embodiments of the present disclosure.

[0008] Figure 5 is a flow chart representing a data programming operation according to several embodiments of the present disclosure.

[0009] Figure 6 is a flowchart representing an example method of a three-level hierarchical memory system according to several embodiments of the present disclosure.

[0010] Figure 7 is another flow chart representing an example method of a three-level hierarchical memory system according to several embodiments of the present disclosure. DETAILED DESCRIPTION

[0011] A three-tier hierarchical memory system is described herein. A three-tier hierarchical memory system according to the present disclosure can utilize persistent memory to store data that is typically stored in non-persistent memory, thereby increasing the amount of storage space allocated to a computing system at a lower cost than methods that rely solely on non-persistent memory. An example device includes a persistent memory, a first non-persistent memory, and a second non-persistent memory, wherein the second non-persistent memory is configured to map an address associated with an input / output (I / O) device to an address in a logic circuit system before the device receives a request from the I / O device to access data stored in the persistent memory, and to map the address associated with the I / O device to an address in the first non-persistent memory after the device receives the request and accesses the data.

[0012] Computing systems use various types of memory resources during operation. For example, a computing system may use a combination of volatile (e.g., random access memory) memory resources and nonvolatile (e.g., storage) memory resources during operation. Generally speaking, volatile memory resources can operate much faster than nonvolatile memory resources and can have longer lifespans than nonvolatile memory resources; however, volatile memory resources are typically more expensive than nonvolatile memory resources. As used herein, volatile memory resources may alternatively be referred to as "non-persistent memory devices," and nonvolatile memory resources may alternatively be referred to as "persistent memory devices."

[0013] However, a persistent memory device may more broadly refer to the ability to access data in a persistent manner. As an example, in the context of persistent memory, a memory device may store a plurality of logical-to-physical mappings or translation data and / or lookup tables in a memory array to track the location of data in the memory device, regardless of whether the memory is non-volatile. Furthermore, a persistent memory device may refer to the non-volatility of the memory in addition to using non-volatility by including the ability to service commands for a continuous process (e.g., by using logical-to-physical mappings, lookup tables, etc.).

[0014] These characteristics may need to be weighed in computing systems in order to provide computing systems with sufficient resources to meet the growing demands of consumers and computing resource providers. For example, in a multi-user computing network (e.g., cloud-based computing system deployments, software-defined data centers, etc.), relatively large amounts of volatile memory may be provided to provide virtual machines running in the multi-user network. However, by relying on volatile memory to provide memory resources to the multi-user network (as is common in some approaches), the costs associated with providing memory resources to the network may increase, especially as users of the network need to use larger and larger pools of computing resources.

[0015] Furthermore, in methods that rely on volatile memory to provide memory resources for provisioning virtual machines in a multi-user network, once the volatile memory resources are exhausted (e.g., once the volatile memory resources are allocated to users of the multi-user network), additional users cannot be added to the multi-user network until additional volatile memory resources are available or added. This may result in potential users being turned away, which may result in a loss of revenue that could otherwise be incurred if the multi-user network could use additional memory resources.

[0016] Volatile memory resources, such as dynamic random access memory (DRAM), tend to operate in a deterministic manner, while non-volatile memory resources, such as storage class memory (e.g., NAND flash memory devices, solid-state drives, resistance variable memory devices, etc.) tend to operate in a non-deterministic manner. For example, due to error correction operations, encryption operations, RAID operations, etc. performed on data retrieved from the storage class memory device, the amount of time between requesting data from the storage class memory device and the data being available may vary from read to read, making the retrieval of data from the storage class memory device non-deterministic. In contrast, the amount of time between requesting data from a DRAM device and the data being available may remain fixed from read to read, making the retrieval of data from the DRAM device deterministic.

[0017] Additionally, due to the distinction between the deterministic behavior of volatile memory resources and the non-deterministic behavior of non-volatile memory resources, data transferred to and from memory resources typically traverses a specific interface (e.g., a bus) associated with the type of memory used. For example, data transferred to and from DRAM devices is typically passed via a double data rate (DDR) bus, while data transferred to and from NAND devices is typically passed via a peripheral component interconnect express (PCI-e) bus. However, as will be appreciated, examples of interfaces that may be used to transfer data to and from volatile memory resources and non-volatile memory resources are not limited to these specific listed examples.

[0018] Due to the different behaviors of non-volatile memory devices and volatile memory devices, some methods choose to store specific types of data in volatile or non-volatile memory. This can alleviate problems that may arise due to, for example, the deterministic behavior of volatile memory devices compared to the non-deterministic behavior of non-volatile memory devices. For example, in some methods, a computing system stores a small amount of data that is regularly accessed during operation of the computing system in a volatile memory device, while larger or less frequently accessed data is stored in a non-volatile memory device. However, in a multi-user network deployment, the vast majority of data may be stored in a volatile memory device. In contrast, embodiments of the present invention may allow data to be stored and retrieved from a non-volatile memory device deployed in a multi-user network.

[0019] As described herein, some embodiments of the present disclosure are directed to computing systems in which data from non-volatile and therefore non-deterministic memory resources is transferred via an interface that is otherwise limited to use with volatile and deterministic memory resources. For example, in some embodiments, data may be transferred to and from non-volatile, non-deterministic memory resources, such as NAND flash devices, resistance variable memory devices, such as phase change memory devices and / or resistive memory devices (e.g., three-dimensional cross point (3D XP) memory devices), solid state drives (SSDs), self-selected memory (SSM) devices, etc., via an interface such as a DDR interface, which in some methods is reserved for transferring data to and from volatile, deterministic memory resources. Therefore, compared to methods in which volatile, deterministic memory devices are used to provide main memory to a computing system, embodiments of the present invention may allow non-volatile, non-deterministic memory devices to be used as at least a portion of the main memory of a computing system.

[0020] In some embodiments, data may be transferred from a non-volatile memory resource to a cache (e.g., a small static random access memory (SRAM) cache) or buffer and then made available to an application requesting the data. By storing data that is normally provided in a deterministic manner in a non-deterministic memory resource and allowing access to the data as described herein, computing system performance may be improved: allowing more memory resources to be provided to a multi-user network at a significantly reduced cost compared to methods using volatile memory resources.

[0021] To facilitate embodiments of the present disclosure, visibility of non-volatile memory resources may be obfuscated for various devices of a computing system that deploys a hierarchical memory system. For example, a host, a network interface card, a virtual machine, etc. deployed in a computing system or a multi-user network may not be able to distinguish whether data is stored by a volatile memory resource of the computing system or by a non-volatile memory resource. For example, a hardware circuit system may be deployed in a computing system that can register addresses corresponding to data in a manner that a host, a network interface card, a virtual machine, etc. cannot distinguish whether the data is stored by a volatile memory resource or a non-volatile memory resource.

[0022] As described in more detail herein, a hierarchical memory system may include hardware circuitry (e.g., logic circuitry) that may intercept redirected data requests, register addresses in the logic circuitry associated with the requested data (even if the hardware circuitry is not backed up by its own memory resources to store the data), and use the logic circuitry to map the addresses registered in the logic circuitry to physical addresses corresponding to the data in a non-volatile memory device.

[0023] In the following detailed description of the present disclosure, reference is made to the accompanying drawings which form a part hereof, and in which is shown by way of illustration how one or more embodiments of the present disclosure may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the embodiments of the present disclosure, and it is understood that other embodiments may be used and process, electrical and structural changes may be made without departing from the scope of the present disclosure.

[0024] As used herein, designators such as "N," "M," and the like, particularly with respect to reference numerals in the accompanying drawings, indicate that the number of particular features so designated may be included. It should also be understood that the terminology used herein is intended only to describe specific embodiments and is not intended to be limiting. As used herein, the singular forms "a," "an," and "the" may include both single and plural references unless the context clearly dictates otherwise. Additionally, "several," "at least one," and "one or more" (e.g., several memory groups) may refer to one or more memory groups, while "plurality" is intended to refer to more than one such thing.

[0025] In addition, the words "may" and "may" are used in this application in a permissive sense (i.e., it is possible to, can), rather than in a mandatory sense (i.e., must). The term "including" and its derivatives mean "including but not limited to". The term "coupled" or "coupling" means a direct or indirect physical connection, or for accessing and moving (transferring) commands and / or data, depending on the context. The terms "data" and "data value" are used interchangeably in this document and can have the same meaning depending on the context.

[0026] The figures herein follow a numbering convention in which the first digit or digits correspond to the figure number and the remaining digits identify the element or component in the figure. Similar elements or components between different figures may be identified by using similar digits. For example, 104 may refer to Figure 1 Component "04" in , similar components in Figure 2 106-1, 106-2, ..., 106-N (e.g., 106-1 to 106-N) may be collectively referred to as 106. As will be appreciated, the elements shown in the various embodiments herein may be added, exchanged, and / or removed to provide several additional embodiments of the present disclosure. In addition, the proportions and / or relative dimensions of the elements provided in the figures are intended to illustrate certain embodiments of the present disclosure and should not be considered to be limiting.

[0027] Figure 1 is a functional block diagram of a computing system 100 in the form of an apparatus including a hierarchical memory component 104 according to several embodiments of the present disclosure. As used herein, "apparatus" may refer to, but is not limited to, any of a variety of structures or combinations of structures, such as a circuit or circuit system, one or more dies, one or more modules, one or more devices, or one or more systems. In some embodiments, the hierarchical memory component 104 may be provided as a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a number of discrete circuit components, etc., and may alternatively be referred to herein as a "logic circuit system."

[0028] like Figure 1 As shown in , the hierarchical memory component 104 may include a memory resource 102, which may include a read buffer 103, a write buffer 105, and / or an input / output I / O device access component 107. In some embodiments, the memory resource 102 may be a random access memory resource, such as a block RAM, which may allow data to be stored within the hierarchical memory component 104 in embodiments where the hierarchical memory component 104 is an FPGA. However, embodiments are not limited thereto, and the memory resource 102 may include various registers, caches, memory arrays, latches, and SRAM, DRAM, EPROM, or other suitable memory technology that may store data, such as a string of bits, including a registered address corresponding to a physical location where the data is stored external to the logic circuit system 104. The memory resource 102 is internal to the hierarchical memory component 104 and is typically smaller than memory external to the hierarchical memory component 104, such as persistent and / or non-persistent memory resources that may be external to the hierarchical memory component 104.

[0029] Read buffer 103 may include a portion of memory resource 102 reserved for storing data that has been received by hierarchical memory component 104 but has not yet been processed by hierarchical memory component 104. In some embodiments, read buffer 103 may be approximately 4 kilobytes (KB) in size, although embodiments are not limited to this particular size. Read buffer 103 may buffer data to be registered in one of address registers 106-1 through 106-N.

[0030] Write buffer 105 may include a portion of memory resources 102 reserved for storing data awaiting transfer to a location external to hierarchical memory component 104. In some embodiments, write buffer 105 may be approximately 4 kilobytes (KB) in size, although embodiments are not limited to this particular size. Write buffer 105 may buffer data registered in one of address registers 106-1 through 106-N.

[0031] The I / O access component 107 may include a reservation of the memory resource 102 for storing components corresponding to components external to the access hierarchy memory component 104 (e.g., Figure 2 and 3 The I / O access component 107 may store data corresponding to addresses of the I / O devices, which may be used to read data from the I / O devices and / or write data to the I / O devices. Additionally, in some embodiments, the I / O access component 107 may receive, store, and / or transmit data corresponding to hypervisors (e.g., Figure 3 The data of the state of the hypervisor 312 shown in FIG. Figure 3 Describe in more detail.

[0032] The hierarchical memory component 104 may further include a memory access multiplexer (MUX) 109, a state machine 111, and / or a hierarchical memory controller 113 (or, for simplicity, a "controller"). Figure 1 As shown, the hierarchical memory controller 113 may include a plurality of address registers 106-1 through 106-N and / or an interrupt component 115. The memory access MUX 109 may include circuitry that may include one or more logic gates and may be configured to control data and / or address busing of the logic circuitry 104. For example, the memory access MUX 109 may transfer messages to and from the memory resources 102 and communicate with the hierarchical memory controller 113 and / or the state machine 111, as described in more detail below.

[0033] In some embodiments, MUX 109 may redirect incoming messages and / or commands received from a host (e.g., a host computing device, a virtual machine, etc.) to hierarchical memory component 104. For example, MUX 109 may redirect incoming messages and / or commands received from an input / output (I / O) device (e.g., a virtual machine, etc.) to a hierarchical memory component 104. Figure 2 and 3 Incoming messages corresponding to access (e.g., read) or programming (e.g., write) requests from one of the I / O devices 210 / 310 shown to one of the address registers (e.g., address registers 106-N, which may be the BAR4 region of the hierarchical memory controller 113, as described below) are redirected to the read buffer 103 and / or write buffer 105.

[0034] In addition, MUX 109 may redirect requests (eg, read requests, write requests) received by hierarchical memory component 104. In some embodiments, requests may be received by hierarchical memory component 104 from a hypervisor (eg, Figure 3 The request may be received by a hypervisor 312 shown), a bare metal server, or a host computing device communicatively coupled to the hierarchical memory component 104. Such a request may be redirected by the MUX 109 from the read buffer 103, the write buffer 105, and / or the I / O access component 107 to an address register (e.g., the address register 106-2, which may be the BAR2 region of the hierarchical memory controller 113, as described below).

[0035] MUX 109 may redirect such requests as part of determining an address to access in address register 106. In some embodiments, MUX 109 may redirect such requests as part of determining an address to access in address register 106 in response to assertion of a hypervisor interrupt (e.g., an interrupt asserted to a hypervisor coupled to hierarchical memory component 104 generated by interrupt component 115).

[0036] In response to determining that the request corresponds to data (e.g., data to be written to a location external to the hierarchical storage component 104 (e.g., a persistent storage device, as described herein in Figure 2 and 3In response to determining that the request corresponds to data to be read from a location external to the hierarchical memory component 104 (e.g., a persistent memory device), MUX 109 may facilitate data retrieval, transfer of data to read buffer 103, and / or transfer of data or address information associated with the data to a location internal to the hierarchical memory component 104 (e.g., address register 106).

[0037] As a non-limiting example, if the hierarchical memory component 104 receives a read request from an I / O device, the MUX 109 may facilitate the retrieval of data from the persistent memory device via the hypervisor by selecting an appropriate message to be sent from the hierarchical memory component 104. For example, the MUX 109 may facilitate the generation of an interrupt using the interrupt component 115, causing the interrupt to be asserted on the hypervisor, buffering the data received from the persistent memory device into the read buffer 103, and / or responding to the I / O device with an indication that the read request has been fulfilled. In a non-limiting example where the hierarchical memory component 104 receives a write request from the I / O device, the MUX 109 may facilitate the transfer of data to the persistent memory device via the hypervisor by selecting an appropriate message to be sent from the hierarchical memory component 104. For example, the MUX 109 may facilitate the generation of an interrupt using the interrupt component 115, causing the interrupt to be asserted on the hypervisor, buffering the data to be transferred to the persistent memory device into the write buffer 105, and / or responding to the I / O device with an indication that the write request has been fulfilled.

[0038] The state machine 111 may include one or more processing devices, circuit components, and / or logic configured to perform operations on inputs and generate outputs. In some embodiments, the state machine 111 may be a finite state machine (FSM) or a hardware state machine that may be configured to receive continuously changing inputs and generate resulting outputs based on the received inputs. For example, the state machine 111 may transfer access information (e.g., "I / O ACCESS INFO") to and from the memory access multiplexer 109, and transfer interrupt configuration information (e.g., "INTERRUPT CONFIG") and / or interrupt request messages (e.g., "INTERRUPT REQUEST") to and from the hierarchical memory controller 113. In some embodiments, the state machine 111 may further transfer control messages (e.g., "MUX CTRL") to and from the memory access multiplexer 109.

[0039] The ACCESS INFO message may include information corresponding to a data access request received from an I / O device external to the hierarchical memory component 104. In some embodiments, the ACCESS INFO may include logical addressing information corresponding to data to be stored in a persistent memory device or addressing information corresponding to data to be retrieved from a persistent memory device.

[0040] The INTERRUPT CONFIG message may be asserted by the state machine 111 on the hierarchical memory controller 113 to configure the appropriate interrupt message to be asserted external to the hierarchical memory component 104. For example, when the hierarchical memory component 104 asserts an interrupt at a hypervisor coupled to the hierarchical memory component 104 as part of implementing a redirected read or write request, the INTERRUPT CONFIG message may be generated by the state machine 111 to generate the appropriate interrupt message based on whether the operation is an operation to retrieve data from a persistent memory device or an operation to write data to a persistent memory device.

[0041] The INTERRUPT REQUEST message may be generated by the state machine 111 and asserted on the interrupt component 115, causing the interrupt message to be asserted on the hypervisor (or bare metal server or other computing device). As described in more detail herein, the interrupt 115 may be asserted on the hypervisor, causing the hypervisor to prioritize the retrieval of data or the writing of data to the persistent storage device as part of the operation of the hierarchical memory system.

[0042] MUX CTRL messages may be generated by state machine 111 and asserted on MUX 109 to control the operation of MUX 109. In some embodiments, MUX CTRL messages may be asserted on MUX 109 by state machine 111 (or by MUX 109 on state machine 111) as part of performing the MUX 109 operations described above.

[0043] The hierarchical memory controller 113 may include a core, such as an integrated circuit, a chip, a system on a chip, or a combination thereof. In some embodiments, the hierarchical memory controller 113 may be a peripheral component interconnect express (PCIe) core. As used herein, a "core" refers to a reusable unit of logic, a processor, and / or a co-processor that receives instructions and performs tasks or actions based on the received instructions.

[0044] Hierarchical memory controller 113 may include address registers 106-1 to 106-N and / or interrupt component 115. Address registers 106-1 to 106-N may be base address registers (BARs) that may store addresses for logic circuitry 104 or a computing system (e.g., as referred to herein in Figure 2 and 3At least one of the address registers (e.g., address register 106-1) may store a memory address provided from an external location (e.g., Figure 3 The hypervisor 312 shown is used to access the memory address of the internal register of the logic circuit system 104.

[0045] Another address register (e.g., address register 106-2) may be used to store addresses corresponding to interrupt controls, as described in more detail herein. In some embodiments, address register 106-2 may map direct memory access (DMA) read and DMA write control and / or status registers. For example, address register 106-2 may include addresses corresponding to descriptors and / or control bits for a DMA command chain that may include the generation of one or more interrupt messages that may be asserted to a hypervisor as part of the operation of a hierarchical memory system, as described herein in conjunction with Figure 3 described.

[0046] Yet another address register (e.g., address register 106-3) may store addresses corresponding to addresses to and from a hypervisor (e.g., Figure 3 In some embodiments, the address register may map addresses corresponding to data transferred via a DMA (e.g., AXI DMA) of the hierarchical memory component 104 to locations external to the hierarchical memory component 104.

[0047] In some embodiments, at least one address register (eg, address register 106-N) may store a corresponding address to an I / O device (eg, Figure 2 104). The address registers 106-N may store addresses that are bypassed by DMA components associated with the hierarchical memory component 104. The address registers 106-N may be provided so that addresses mapped thereto are not "backed up" by physical memory locations of the logical circuitry 104. That is, in some embodiments, the hierarchical memory component 104 may be configured to store addresses corresponding to memory locations stored in a persistent memory device (e.g., Figure 2 216) but does not correspond to addresses of data stored by the hierarchical memory component 104. For example, the address registers 106-N may be configured as a virtual address space that may store logical addresses corresponding to physical memory locations (e.g., in a memory device) where data is stored.

[0048] In some embodiments, address register 106-N may include a number of addresses corresponding to memory devices (eg, Figure 2 and 3 For example, if the memory device contains one terabyte of storage, then the address registers 106-N may be configured to have an address space that may contain one terabyte of address space. However, as described above, the address registers 106-N do not actually contain one terabyte of storage, but are configured to appear to have one terabyte of storage space.

[0049] although Figure 1 104 may be coupled to a host computing system. The host computing system may include a system motherboard and / or a backplane, and may include a number of processing resources (e.g., one or more processors, microprocessors, or some other type of control circuitry). For example, the host and device 100 may be a server system and / or a high performance computing (HPC) system and / or portions thereof. In some embodiments, the computing system may have a von Neumann architecture, however, embodiments of the present disclosure may be implemented in a non-von Neumann architecture, which may not include one or more components (e.g., CPU, ALU, etc.) typically associated with a von Neumann architecture.

[0050] Figure 2 is a functional block diagram of a computing system 201 including a hierarchical memory component 204 (eg, logic circuitry) according to several embodiments of the present disclosure. Figure 2 As shown, computing system 201 may include a hierarchical memory component 204, which may be similar to Figure 1 hierarchical memory component 104 is shown. In addition, computing system 201 may include input / output (I / O) devices 210, persistent memory (e.g., persistent memory device) 216, first non-persistent memory (e.g., non-persistent memory device) 230, intermediate memory component 220, and memory management circuitry (e.g., memory management component) 214 having second non-persistent memory 219. Thus, computing system 201 may be referred to as a three-tier (e.g., three-level) hierarchical memory system, wherein persistent memory device 216, first non-persistent memory device 230, and second non-persistent memory 219 each comprise a different layer (e.g., level) of the hierarchical memory system. Communication between logic circuitry 204, I / O devices 210 and persistent memory device 216, non-persistent memory device 230, and memory management component 214 (e.g., second non-persistent memory 219) may be facilitated via interface 208.

[0051] The I / O device 210 may be a device configured to provide direct memory access via a physical address and / or a virtual machine physical address. In some embodiments, the I / O device 210 may be a network interface card (NIC) or a network interface controller, a storage device, a graphics rendering device, or other I / O device. The I / O device 210 may be a physical I / O device, or the I / O device 210 may be a virtualized I / O device 210. For example, in some embodiments, the I / O device 210 may be a physical card physically coupled to a computing system via a bus or an interface such as a PCIe interface or other suitable interface. In embodiments where the I / O device 210 is a virtualized I / O device 210, the virtualized I / O device 210 may provide I / O functions in a distributed manner.

[0052] The persistent memory device 216 may include several arrays of memory cells. For example, the array may be a flash array having a NAND architecture. However, embodiments are not limited to a particular type of memory array or array architecture. For example, the memory cells may be grouped into several blocks that include several physical pages. Several blocks may be included in one plane of the memory cells, and the array may include several planes.

[0053] The persistent memory device 216 may include volatile memory and / or non-volatile memory. In several embodiments, the persistent memory device 216 may include a multi-chip device. The multi-chip device may include several different memory types and / or memory modules. For example, the memory system may include non-volatile or volatile memory on any type of module. In embodiments where the persistent memory device 216 includes non-volatile memory, the persistent memory device 216 may be a flash memory device, such as a NAND or NOR flash memory device.

[0054] However, embodiments are not limited thereto, and the persistent memory device 216 may include other nonvolatile memory devices, such as nonvolatile random access memory devices (e.g., NVRAM, ReRAM, FeRAM, MRAM, PCM), "emerging" memory devices such as resistance variable memory devices (e.g., resistance and / or phase change memory devices, such as 3D cross point (3D XP) memory devices), memory devices including self-select memory (SSM) cell arrays, etc., or combinations thereof. The resistance and / or phase change nonvolatile memory array may perform bit storage based on changes in body resistance in conjunction with a stackable cross-grid data access array. Additionally, in contrast to many flash-based memories, the resistance and / or phase change memory devices may perform write-in-situ operations, where nonvolatile memory cells may be programmed without previously erasing the nonvolatile memory cells. In contrast to flash-based memories, the self-select memory cells may include memory cells having a single chalcogenide material acting as both a switch and a storage element for the memory cell.

[0055] Persistent memory device 216 may provide storage capacity for computing system 201 and may therefore be used as additional memory or storage for computing system 201, main memory for computing system 201, or a combination thereof. However, embodiments are not limited to a particular type of memory device, and persistent memory device 216 may include RAM, ROM, SRAM DRAM, SDRAM, PCRAM, RRAM, and flash memory, among others. Furthermore, although in Figure 2 2 , but embodiments are not limited thereto, and computing system 201 may include one or more persistent memory devices 216, each of which may or may not have the same architecture associated therewith. As a non-limiting example, in some embodiments, persistent memory device 216 may include two discrete memory devices having different architectures, such as a NAND memory device and a resistance variable memory device.

[0056] The non-persistent memory device 230 and the non-persistent memory 219 may each include a volatile memory, such as an array of volatile memory cells. In several embodiments, the non-persistent memory device 230 may include a multi-chip device. The multi-chip device may include several different memory types and / or memory modules. In some embodiments, the non-persistent memory device 230 may be used as the main memory of the computing system 201. For example, the non-persistent memory device 230 may be a dynamic random access (DRAM) memory device for providing main memory to the computing system 230. In some embodiments, the non-persistent memory device 219 may also include DRAM (e.g., a DRAM cell array). However, the embodiments are not limited to the non-persistent memory device 230 and the non-persistent memory 219 including DRAM. In some embodiments, the non-persistent memory device 230 and / or the non-persistent memory device 219 may include other non-persistent memories, such as RAM, SRAM DRAM, SDRAM, PCRAM and / or RRAM, etc.

[0057] As part of the operation of computing system 201, non-persistent memory device 230 may store data (e.g., user data) that may be requested (e.g., accessed) by, for example, a host computing device. For example, when computing system 201 is part of a multi-user network, non-persistent memory device 230 may store data that may be transferred between host computing devices (e.g., virtual machines deployed in the multi-user network) during operation of computing system 201.

[0058] In some approaches, non-persistent storage, such as non-persistent storage device 230, can store all user data accessed by a host (e.g., a virtual machine deployed in a multi-user network). For example, due to the speed of non-persistent storage, some approaches rely on non-persistent storage to provide memory resources for virtual machines deployed in a multi-user network. However, in such approaches, cost may become an issue because non-persistent storage is generally more expensive than persistent storage (e.g., persistent storage device 216).

[0059] In contrast, as described in more detail below, embodiments herein may allow at least some of the data stored in non-persistent memory device 230 to be stored in persistent memory device 216. This may allow additional memory resources to be provided to a multi-user network or the like computing system 201 at a lower cost than approaches that rely on non-persistent memory for user data storage.

[0060] Computing system 201 may include a memory management component 214 that may be communicatively coupled to non-persistent memory device 230 and / or interface 208. In some embodiments, memory management component 214 may be an input / output memory management unit (IO MMU) that may communicatively couple, for example, a direct memory access bus of interface 208 to non-persistent memory device 230. However, embodiments are not limited in this regard, and memory management component 214 may be other types of memory management hardware for facilitating communication between interface 208 and non-persistent memory device 230.

[0061] The non-persistent memory 219 of the memory management component 214 may map device-visible virtual addresses to physical addresses. For example, the non-persistent memory 219 may map virtual addresses associated with the I / O device 210 to physical addresses in the non-persistent memory device 230 and / or the persistent memory device 216. For example, the non-persistent memory 219 may support a partial virtual address space (e.g., compared to a cache that replicates an existing virtual address space). In some embodiments, mapping virtual entries associated with the I / O device 210 may be performed by mapping the virtual addresses associated with the I / O device 210 to physical addresses in the non-persistent memory device 230 and / or the persistent memory device 216. Figure 1 This is facilitated by the read buffers, write buffers and / or I / O access buffers shown.

[0062] In some embodiments, the memory management component 214 may read a virtual address associated with the I / O device 210, and the non-persistent memory 219 may map the virtual address to a physical address in the non-persistent memory device 230 or an address in the hierarchical memory component 204. In embodiments where the non-persistent memory 219 maps the virtual I / O device 210 address to an address in the hierarchical memory component 204, the memory management component 214 may redirect a read request (or a write request) received from the I / O device 210 to the hierarchical memory component 204, which may store the virtual address information associated with the I / O device 210 read or write request in an address register (e.g., address register 206-N) of the hierarchical memory component 204. That is, the address in the hierarchical memory component 204 to which the address associated with the I / O device 210 is mapped may be an address in the address register 206-N. In some embodiments, the address register 206-N may be a specific base address register of the hierarchical memory component 204, such as the BAR4 address register.

[0063] The redirected read (or write) request may be transferred from the memory management component 214 to the hierarchical memory component 204 via the interface 208. In some embodiments, the interface 208 may be a PCIe interface, and thus information may be transferred between the memory management component 214 and the hierarchical memory component 204 according to the PCIe protocol. However, the embodiments are not limited thereto, and in some embodiments, the interface 208 may be an interface or bus operating according to another suitable protocol.

[0064] After the virtual NIC address is stored in the hierarchical memory component 204, data corresponding to the virtual NIC address can be written to the persistent memory device 216. For example, data corresponding to the virtual NIC address stored in the hierarchical memory component 204 can be stored in a physical address location of the persistent memory device 216. In some embodiments, the transfer of data to and / or from the persistent memory device 216 can be facilitated by a hypervisor, as described herein in conjunction with Figure 3-5 described.

[0065] When data is requested by, for example, a host computing device (e.g., a virtual machine deployed in computing system 201), the request may be redirected from I / O device 210 to hierarchical memory component 204 via memory management component 214. Because the virtual NIC address corresponding to the physical location of the data in persistent memory device 216 is stored in address register 206-N of hierarchical memory component 204, hierarchical memory component 204 may facilitate retrieval of data from persistent memory device 216 in conjunction with a hypervisor, as described herein in conjunction with Figure 3-5 Describe in more detail.

[0066] In some embodiments, when data stored in persistent memory device 216 is transferred out of persistent memory device 216 (e.g., when data stored in persistent memory device 216 is requested by a host computing device), the data may be transferred to intermediate memory component 220 and / or non-persistent memory device 230 before being provided to the host computing device. For example, because data transferred to the host computing device may be transferred in a deterministic manner (e.g., via a DDR interface), the data may be temporarily transferred to a memory operating using a DDR bus, such as intermediate memory component 220 and / or non-persistent memory device 230, before the data request is fulfilled.

[0067] Figure 3 is another functional block diagram of a computing system including a hierarchical memory component according to several embodiments of the present disclosure. Figure 3 As shown in FIG. 1 , computing system 301 may include a hierarchical memory component 304, which may be similar to Figure 1 and 2 In addition, computing system 301 may include I / O device 310, persistent memory device 316, non-persistent memory device 330, intermediate memory component 320, memory management component 314 including non-persistent memory 319, and hypervisor 312.

[0068] In some embodiments, computing system 301 may be a multi-user network, such as a software-defined data center, a cloud computing environment, etc. In such embodiments, the computing system may be configured to have one or more virtual machines 317 running thereon. For example, in some embodiments, one or more virtual machines 317 may be deployed on hypervisor 312 and may be accessed by users of the multi-user network.

[0069] I / O device 310, persistent memory device 316, non-persistent memory device 330, intermediate memory component 320, memory management component 314, and non-persistent memory 319 may be similar to Figure 2 2, 304. The I / O device 210, persistent memory device 216, non-persistent memory device 230, intermediate memory component 220, memory management component 214, and non-persistent memory 219 are shown. For example, persistent memory device 316, non-persistent memory device 330, and non-persistent memory 319 may each comprise a different layer of a hierarchical memory system, as previously described herein. Communication between logic circuitry 304, I / O device 310 and persistent memory device 316, non-persistent memory device 330, hypervisor 312, and memory management component 314 (e.g., non-persistent memory 319) may be facilitated via interface 308, which may be similar to Figure 2 Interface 208 is shown.

[0070] As above combined Figure 2 As described above, the memory management component 314 can redirect a read request or a write request associated with the I / O device 310 to the hierarchical memory component 304. The hierarchical memory component 304 can generate and / or store a logical address corresponding to the requested data. As described above, the hierarchical memory component 304 can store the logical address corresponding to the requested data in a base address register, such as the address register 306-N of the hierarchical memory component 304.

[0071] like Figure 3 As shown in FIG. 1 , the hypervisor 312 can communicate with the hierarchical storage component 304 and / or the I / O device 310 via the interface 308. The hypervisor 312 can access the components (e.g., Figure 1 NIC access component 107) is shown to transfer data between hierarchical memory components 304. In addition, hypervisor 312 can communicate with persistent memory device 316, non-persistent memory device 330, intermediate memory component 320, and memory management component 314. The hypervisor can be configured to execute specific instructions to perform the operations and / or tasks described herein.

[0072] For example, the hypervisor 312 may execute instructions to monitor data traffic and data traffic patterns to determine whether data should be programmed into (e.g., stored in) the non-persistent memory device 330 or whether data should be programmed into (e.g., transferred to) the persistent memory device 316. That is, in some embodiments, the hypervisor 312 may execute instructions to learn user data request patterns over time and selectively store portions of data in the non-persistent memory device 330 or the persistent memory device 316 based on the patterns. This may allow data that is accessed more frequently to be stored in the non-persistent memory device 330, while data that is accessed less frequently is stored in the persistent memory device 316.

[0073] Because users may access recently used or viewed data more frequently than data that is used or viewed less frequently, the hypervisor may execute specialized instructions to cause data that is used or viewed less frequently to be stored in persistent memory device 316, and / or cause data that is accessed or viewed more frequently to be stored in non-persistent memory device 330. In a non-limiting example, users may view photos taken recently (e.g., within a week, etc.) on social media more frequently than photos taken more recently (e.g., a month ago, a year ago, etc.). Based on this information, hypervisor 312 may execute specialized instructions to cause photos that are viewed or taken less frequently to be stored in persistent memory device 316, thereby reducing the amount of data stored in non-persistent memory device 330. This may reduce the total amount of non-persistent memory that needs to be provided to computing system 301, thereby reducing costs and allowing more users to access non-persistent memory device 330.

[0074] In operation, computing system 301 may be configured to intercept data requests from I / O device 310 and redirect the requests to hierarchical memory component 304. In some embodiments, hypervisor 312 may control whether data corresponding to a data request is stored in non-persistent memory device 330 or stored in persistent memory device 316 (or retrieved from non-persistent memory device 330 or from persistent memory device 316). For example, hypervisor 312 may execute instructions to selectively control whether data is stored in persistent memory device 316 or stored in non-persistent memory device 330 (or retrieved from persistent memory device 316 or from non-persistent memory device 330).

[0075] As part of controlling whether data is stored in the persistent memory device 316 or stored in the non-persistent memory device 330 (or retrieved from the persistent memory device 316 or retrieved from the non-persistent memory device 330), the hypervisor 312 may cause the non-persistent memory 319 to map logical addresses associated with data to be redirected to the hierarchical memory component 304 and stored in the address registers 306 of the hierarchical memory component 304. For example, the hypervisor 312 may execute instructions to control read and write requests involving data to be selectively redirected to the hierarchical memory component 304 via the non-persistent memory 319.

[0076] The non-persistent memory 319 may map contiguous virtual addresses to underlying segmented physical addresses. Thus, in some embodiments, the non-persistent memory 319 may allow virtual addresses to be mapped to physical addresses without requiring the physical addresses to be contiguous. For example, a particular address associated with the I / O device 310 mapped by the non-persistent memory 319 may be contiguous with other addresses associated with the I / O device 310 mapped by the non-persistent memory 319, but the address to which the particular address associated with the I / O device 310 in the non-persistent memory device 330 is mapped may not be contiguous with other addresses to which the non-persistent memory 319 is mapped in the non-persistent memory device 330. Furthermore, in some embodiments, the non-persistent memory 319 may allow devices that do not support memory addresses of sufficient length to address their corresponding physical memory space to address in the non-persistent memory 319.

[0077] Due to the non-deterministic nature of data transfer associated with persistent memory device 316, in some embodiments, hierarchical memory component 304 may be configured to notify computing system 301 that a delay may occur in transferring data to or from persistent memory device 316. As part of the initialization delay, hierarchical memory component 304 may provide page fault handling for computing system 301 when a data request is redirected to hierarchical memory component 304. In some embodiments, hierarchical memory component 304 may generate and assert an interrupt to hypervisor 312 to initiate an operation to transfer data to or from persistent memory device 316. For example, due to the non-deterministic nature of data retrieval and storage associated with persistent memory device 316, hierarchical memory component 304 may generate hypervisor interrupt 315 when a request is made to transfer data stored in persistent memory device 316.

[0078] In response to the page fault interrupt generated by the hierarchical memory component 304, the hypervisor 312 may retrieve information corresponding to the data from the hierarchical memory component 304. For example, the hypervisor 312 may receive NIC access data from the hierarchical memory component, which may include a logical to physical address mapping corresponding to the data stored in the address register 306 of the hierarchical memory component 304.

[0079] Once data is stored in the persistent memory device 316, a portion (e.g., page, block, etc.) of the non-persistent memory device 330 may be marked as inaccessible to the hierarchical memory component 304 so that the computing system 301 does not attempt to access the data from the non-persistent memory device 330. This may allow data requests to be intercepted due to page faults, which may be generated by the hierarchical memory component 304 and asserted to the hypervisor 312 when data already stored in the persistent memory device 316 is requested by the I / O device 310.

[0080] Compared to the method in which a page fault exception occurs in response to an application request to access a memory page that is not mapped by a memory management unit (e.g., the non-persistent memory 319 of the memory management component 314), in an embodiment of the present disclosure, the page fault described above may be generated by the hierarchical memory component 304 in response to data being mapped in the non-persistent memory 319 to the hierarchical memory component 304, which in turn maps the data to the persistent memory device 316.

[0081] In some embodiments, the intermediate memory component 320 can be used to buffer data stored in the persistent memory device 316 in response to a data request initiated by the I / O device 310. Compared to the persistent memory device 316, which can transfer data via a PCIe interface, the intermediate memory component 320 can use a DDR interface to transfer data. Therefore, in some embodiments, the intermediate memory component 320 can operate in a deterministic manner. For example, in some embodiments, the requested data stored in the persistent memory device 316 can be temporarily transferred from the persistent memory device 316 to the intermediate memory component 320, and then transferred to the host computing device via the DDR interface that couples the intermediate memory component 320 to the I / O device 310.

[0082] In some embodiments, the intermediate memory component may include a discrete memory component (e.g., an SRAM cache) deployed in the computing system 301. However, the embodiments are not limited thereto, and in some embodiments, the intermediate memory component 320 may be a portion of the non-persistent memory device 330 that may be allocated for transferring data from the persistent memory device 316 in response to a data request.

[0083] In a non-limiting example, memory management circuitry (e.g., memory management component 314) may be coupled to logic circuitry (e.g., hierarchical memory component 304). The memory management circuitry may be configured to receive a request to write data having a corresponding virtual network interface controller address associated therewith to a non-persistent memory device (e.g., non-persistent memory device 330). The memory management circuitry may be further configured to redirect the request to write the data to the logic circuitry based at least in part on characteristics of the data. The characteristics of the data may include a frequency of request or access of the data, an amount of time that has elapsed since the data was last accessed or requested, a type of the data (e.g., whether the data corresponds to a particular file type, such as a photo, document, audio file, application file, etc.), and the like.

[0084] In some embodiments, the memory management circuitry may be configured to redirect requests to write data to the logic circuitry based on commands generated by and / or instructions executed by hypervisor 312. For example, as described above, hypervisor 312 may execute instructions that control whether data corresponding to a data request (e.g., a data request generated by I / O device 310) is stored in persistent memory device 316 or non-persistent memory device 330.

[0085] In some embodiments, hypervisor 312 may facilitate redirection of requests by writing addresses (e.g., logical addresses) to non-persistent memory 319 of memory management circuitry. For example, if hypervisor 312 determines that data corresponding to a particular data request is to be stored in (or retrieved from) persistent memory device 316, hypervisor 312 may cause an address corresponding to the redirection of the request to be stored by non-persistent memory 319 such that the data request is redirected to the logical circuitry.

[0086] Upon receiving the redirected request, the logic circuitry may be configured to generate an address corresponding to the data in response to receiving the redirected request and / or store the address in address register 306 within the logic circuitry. In some embodiments, the logic circuitry may be configured to associate an indication that the data is not accessible to non-persistent memory device 330 with the data based on receiving the redirected request.

[0087] The logic circuitry may be configured to cause data to be written to a persistent memory device (e.g., persistent memory device 316) based at least in part on receiving the redirected request (e.g., after memory management component 314 receives the write request and redirects the write request to the logic circuitry). In some embodiments, the logic circuitry may be configured to generate an interrupt signal and assert the interrupt signal to a hypervisor (e.g., hypervisor 312) coupled to the logic circuitry as part of causing the data to be written to persistent memory device 316. As described above, persistent memory device 316 may include a 3D XP memory device, a self-selected memory cell array, a NAND memory device, or other suitable persistent memory, or a combination thereof.

[0088] In some embodiments, the logic circuitry may be configured to receive a redirected request from the memory management circuitry to access (e.g., retrieve) data from the persistent memory device 316, transfer the request to retrieve the data from the persistent memory device 316 to the hypervisor 312 coupled to the logic circuitry, and / or assert an interrupt signal to the hypervisor 312 as part of the request to retrieve the data from the persistent memory device 316. The hypervisor 312 may be configured to retrieve the data from the persistent memory device 316 (e.g., after the memory management circuitry receives the request and redirects the request to the logic circuitry) and / or transfer the data to the non-persistent memory device 330 after the data is retrieved. Once the data is retrieved from the persistent memory device 316, the hypervisor 312 may be configured to cause an updated address associated with the data to be transferred to the non-persistent memory 319 and stored therein. The updated address may be a virtual address corresponding to the data. For example, the updated address may map the data to the non-persistent memory device 330 (e.g., the location in the non-persistent memory device 330 to which the data is transferred).

[0089] In another non-limiting example, computing system 301 may be a multi-user network, such as a software-defined data center, a cloud computing environment, etc. The multi-user network may include a computing resource pool including a non-persistent memory device 330 and a persistent memory device 316. The multi-user network may further include an interface 308 coupled to a logic circuit system (e.g., a hierarchical memory component 304) including a plurality of address registers 306. In some embodiments, the multi-user network may further include a hypervisor 312 coupled to the interface 308.

[0090] The hypervisor 312 may be configured to receive a request to access data corresponding to the non-persistent memory component 330, determine that the data is stored in a persistent memory device, and redirect the request to access the data to the logic circuit system. The request to access the data may be a request to read data from a persistent memory device or a non-persistent memory device or a request to write data to a persistent memory device or a non-persistent memory device.

[0091] In some embodiments, the logic circuitry may be configured to transfer a request to access data from persistent storage device 316 to hypervisor 312 in response to determining that the data is stored in persistent storage device 316. The logic circuitry may be configured to assert an interrupt to the hypervisor as part of the request to hypervisor 312 to access data corresponding to persistent storage device 316.

[0092] The hypervisor 312 may be configured to cause data to be accessed using the persistent memory device 316 based on a request received from the logic circuit system. As described above, the persistent memory device 316 may include a resistance variable memory device, such as a resistive memory, a phase change memory, an optional memory cell array, or a combination thereof. In some embodiments, the hypervisor 312 may be configured to cause data to be transferred to the non-persistent memory device 330 as part of causing data to be accessed using the persistent memory device 316.

[0093] The hypervisor 312 may be further configured to update information stored in the non-persistent memory 319 of the memory management component 314 associated with the multi-user network in response to causing the data to be accessed using the persistent memory device 316. For example, the hypervisor 312 may be configured to cause an updated virtual address corresponding to the data to be stored in the non-persistent memory 319.

[0094] In some embodiments, the multi-user network may include an I / O device 310 coupled to the logic circuitry. In such embodiments, the logic circuitry may be configured to send a notification to the I / O device 310 in response to the hypervisor 312 causing data to be accessed using the persistent storage device 316.

[0095] Figure 4 4 is a flow chart 440 showing a data access (eg, read) operation according to several embodiments of the present disclosure. At block 441, for example Figure 2 and 3 The I / O device 210 / 310 shown in FIG. 1 may initiate a read operation using an address corresponding to a data request. In some embodiments, the address may be a physical address, such as a virtual machine physical address. The data request may include a request to read data associated with a specific address corresponding to a logical address where the data is stored. The physical address may correspond to a persistent memory device (e.g., a memory device described herein). Figure 2 and 3 216 / 316) or a non-persistent memory device (e.g., Figure 2 and 3 The location in the non-persistent memory device 230 / 330) shown in FIG.

[0096] If the data is stored in a non-persistent memory device, then the data can be retrieved and the data request can be fulfilled. However, if the data is stored in a persistent memory device (e.g., if the physical address of the data corresponds to a location in the persistent memory device), then at block 442, a memory management component (e.g., as described herein) Figure 2 and 3 The memory management component 214 / 314 shown in FIG. 2 may redirect data requests to a hierarchical memory component (e.g., Figure 1-3 As described above, data requests may be based on a hypervisor (e.g., Figure 3 312) shown in the hypervisor (e.g., commands or instructions executed by it) to redirect.

[0097] At block 443, the hierarchical memory component may receive address register access information corresponding to the data request. In some embodiments, the address register access information may correspond to an address register (e.g., Figure 1-3 For example, the address register access information may correspond to a location in an address register in a hierarchical memory component, where the logical address corresponds to a physical address in a persistent memory device where data is stored.

[0098] At block 444, the hierarchical memory component may generate a hypervisor interrupt. Figure 3 As described above, once the hierarchical memory component receives a redirected data request from the memory management component, the hierarchical memory component may generate an interrupt and, in a hypervisor (e.g., Figure 3 445 ). In some embodiments, an interrupt may be a signal that is asserted on the hypervisor to notify the hypervisor that an event requires immediate attention. For example, an interrupt signal may be asserted on the hypervisor, causing the hypervisor to interrupt the currently executing instruction and instead execute instructions associated with collecting address register access information at block 445 .

[0099] At block 445, the hypervisor may collect address register access information from the hierarchical memory component. For example, the hypervisor may receive logical address information corresponding to the physical address of the requested data from the hierarchical memory component. The logical address information may be stored in the hierarchical memory component in an address register (e.g., a base address register) of the hierarchical memory component, such as described herein. Figure 1-3 The address registers 106 / 206 / 306 shown in FIG.

[0100] At block 446, the hypervisor may determine the physical location of the requested data. For example, based on the address register access information and thus the logical address associated with the data collected from the hierarchical memory component, the hypervisor may determine the physical location of the data stored in the persistent memory device.

[0101] At block 447, the hypervisor may read data corresponding to the address register access information. That is, in some embodiments, the hypervisor may read (eg, retrieve) the requested data from the persistent memory device.

[0102] At block 448, the hypervisor may cause the data to be transferred to a non-persistent memory device. In some embodiments, the non-persistent memory device may be a Figure 2 and 3 The non-persistent memory device 230 / 330 shown in FIG. 1 is shown in FIG. 1 , but the embodiments are not limited thereto, and in some embodiments, the hypervisor may cause the data to be transferred to an intermediate memory component, such as the one shown in FIG. 1 . Figure 2 and 3 The intermediate storage component 220 / 320 shown in FIG.

[0103] At block 449, the hypervisor may write I / O device data corresponding to the requested data to the hierarchical memory component. The I / O device data may be stored in address registers of the hierarchical memory component, as described above.

[0104] At block 450, the hierarchical memory component may complete the data read transaction. For example, the hierarchical memory component may transfer a command to the I / O device to notify the I / O device that the data read request has been fulfilled, and data will be transferred via a deterministic interface to fulfill the data read request.

[0105] At block 451, the hypervisor may update the non-persistent memory of the memory management component (eg, Figure 2 and 3 219 / 319 shown in FIG. 4) to map (e.g., redirect) the I / O device address to the non-persistent memory device (e.g., an address in the non-persistent memory device). For example, because the data has been transferred from the persistent memory device to the non-persistent memory device (e.g., the non-persistent memory device and / or the intermediate memory component) at block 450, the hypervisor can update the non-persistent memory of the memory management component so that the address corresponding to the requested data is mapped to the non-persistent memory device. In some embodiments, the address can be a physical address, such as a virtual machine physical address.

[0106] At block 452, the hypervisor may record which memory is used to satisfy the data request. For example, the hypervisor may record that the data was already stored in the persistent memory device when the data request was received from the I / O device. In some embodiments, the hypervisor may use the time-varying information to selectively direct data writes to the persistent memory device or the non-persistent memory device.

[0107] Figure 5 5 is a flow chart 560 showing a data programming (eg, writing) operation according to several embodiments of the present disclosure. At block 561, for example Figure 2 and 3The I / O device 210 / 310 shown in FIG. 1 may initiate a write operation using an address corresponding to a data write request. The address may be a physical address, such as a virtual machine physical address. The data write request may include a request to write data associated with a specific virtual address corresponding to a logical address where the data is to be stored. The physical address may correspond to a persistent memory device (e.g., a virtual machine physical address). Figure 2 and 3 216 / 316) or a non-persistent memory device (e.g., Figure 2 and 3 The location in the non-persistent memory device 230 / 330) shown in FIG.

[0108] If the data is to be stored in a non-persistent memory device, then the data may be written to the non-persistent memory device and the data write request may be implemented. However, if the data is to be stored in a persistent memory device, then at block 562, a memory management component (e.g., as described herein) may write the data to the non-persistent memory device. Figure 2 and 3 The memory management component 214 / 314 shown in FIG. 2 may redirect data write requests to a hierarchical memory component (e.g., Figure 1-3 As described above, data requests may be based on a hypervisor (e.g., Figure 3 312) shown in the hypervisor (e.g., commands or instructions executed by it) to redirect.

[0109] At block 563, the hierarchical memory component may receive address register access information corresponding to the data write request. In some embodiments, the address register access information may correspond to an address register (e.g., Figure 1-3 For example, the address register access information may correspond to a location in an address register in a hierarchical memory component, where the logical address corresponds to a physical address in a persistent memory device where data is to be stored.

[0110] At block 564, the hierarchical memory component may generate a hypervisor interrupt. Figure 3 As described, once the hierarchical memory component receives a redirected data write request from the memory management component, the hierarchical memory component may generate an interrupt and initiate an interrupt in a hypervisor (e.g., Figure 3 The interrupt is confirmed on the hypervisor 312 shown in FIG.

[0111] At block 565, the hypervisor may gather address register access information from the hierarchical memory component. For example, the hypervisor may receive logical address information from the hierarchical memory component corresponding to the physical address where the data is to be stored.

[0112] At block 566, the hypervisor may optionally write (or cause the data to be written) the data to the persistent memory device. For example, based on the redirected data write request, the hypervisor may determine that the data is to be written to the persistent memory device, and cause the data to be written to the persistent memory device. In embodiments where block 566 is optional, the data may be intermediately written to a non-persistent memory device. Additionally, I / O device data corresponding to the data may optionally be written to the non-persistent memory device as part of writing the data to the non-persistent memory device.

[0113] Optionally, the hypervisor may write (or cause the data to be written) the data to the non-persistent memory device at block 567. In some embodiments, the hypervisor may write the data to the non-persistent memory device so that the data may be retrieved via a deterministic interface or bus if a read request corresponding to the data is received.

[0114] At block 568, the hypervisor may update the non-persistent memory of the memory management component (e.g., Figure 2 and 3 219 / 319 shown in the non-persistent memory 219 / 319) to map (e.g., direct) the I / O device virtual address to the non-persistent memory device (e.g., an address in the non-persistent memory device). For example, if data is written to the non-persistent memory device at block 567, then at block 568, the hypervisor may update the virtual address stored by the non-persistent memory of the memory management component so that the virtual address associated with the data and stored by the non-persistent memory of the memory management component is mapped to the physical address of the non-persistent memory device storing the data.

[0115] Figure 6 6 is a flow chart of an example method 670 of a hierarchical memory system according to several embodiments of the present disclosure. At block 672, the method 670 may include mapping addresses associated with I / O devices in a non-persistent memory of the memory management circuitry to addresses in the logic circuitry. For example, the non-persistent memory may be the memory device described herein. Figure 2 and 3 The non-persistent memory 219 / 319 of the memory management component 214 / 314 shown in FIG. 1 , for example, the I / O device may be the non-persistent memory 219 / 319 of the memory management component 214 / 314 shown in FIG. Figure 2 and 3 , and for example, the logic circuit system may be the I / O device 210 / 310 shown in Figure 1 , 2and the hierarchical memory components 104 / 204 / 304 shown in 3. For example, virtual addresses associated with I / O devices may be mapped to physical addresses in non-persistent memory, as previously described herein.

[0116] At block 674, method 670 may include receiving a request from an I / O device to access data stored in a persistent memory device. For example, the persistent memory device may be Figure 2 and 3 , and for example, the request may be received from an I / O device via the persistent memory device 216 / 316 shown in FIG. Figure 2 and 3 The request may be received by the interface 208 / 308 shown in . For example, the request may be a read request, as previously described herein.

[0117] At block 676, method 670 may include accessing data stored in the persistent memory device after receiving the request. For example, the data may be accessed after the request is redirected to the logic circuitry, as previously described herein.

[0118] At block 678, method 670 may include, after accessing the data, mapping, in the non-persistent memory of the memory management circuitry, an address associated with the I / O device to an address in the non-persistent memory device. For example, the non-persistent memory device may be a memory device described herein. Figure 2 and 3 The non-persistent memory device 230 / 330 shown in . For example, the address can be mapped by updating the non-persistent memory 219 / 319, as previously described herein.

[0119] Figure 7 780 is another flow chart of an example method 780 of a three-tier hierarchical memory system according to several embodiments of the present disclosure. At block 782, the method 780 may include mapping addresses associated with the I / O device to addresses in the logic circuit system in a first non-persistent memory. For example, the first non-persistent memory may be the one described herein. Figure 2 and 3 The non-persistent memory 219 / 319 shown in FIG. 1 , for example, the I / O device may be Figure 2 and 3 , and for example, the logic circuit system may be the I / O device 210 / 310 shown in Figure 1 , 2 and the hierarchical memory components 104 / 204 / 304 shown in 3. For example, virtual addresses associated with I / O devices may be mapped to physical addresses in non-persistent memory, as previously described herein.

[0120] At block 784, method 780 may include receiving a request from an I / O device to program data to a persistent memory. For example, the persistent memory may be a memory device described herein. Figure 2 and 3 , and for example, the request may be made via the persistent memory device 216 / 316 shown in Figure 2 and 3 The interface 208 / 308 shown in FIG. 1 receives from the I / O device. For example, the request may be a write request, as previously described herein.

[0121] At block 786, method 780 may include programming the data to the persistent memory or the second non-persistent memory after receiving the request. For example, the data may be programmed after the request is redirected to the logic circuit system, as previously described herein. For example, the second non-persistent memory may be the second non-persistent memory described herein. Figure 2 and 3 For example, whether data is programmed to the persistent memory or to the second non-persistent memory may depend on a traffic pattern associated with the data, as previously described herein.

[0122] At block 788, method 780 may include, after programming the data, mapping in the first non-persistent memory an address associated with the I / O device to an address in the second non-persistent memory. For example, the address may be mapped by updating the first non-persistent memory, as previously described herein.

[0123] Although specific embodiments have been illustrated and described herein, it will be appreciated by those skilled in the art that arrangements calculated to achieve the same results may replace the specific embodiments shown. The present disclosure is intended to cover modifications or variations of one or more embodiments of the present disclosure. It should be understood that the above description is by way of illustration and not by way of limitation. Upon reviewing the above description, the combination of the above embodiments and other embodiments not specifically described herein will be apparent to those skilled in the art. The scope of one or more embodiments of the present disclosure is included in other applications in which the above structures and processes are used. Therefore, the scope of one or more embodiments of the present disclosure should be determined with reference to the appended claims and the full scope of equivalents to which such claims are assigned.

[0124] In the foregoing Detailed Description, some features are grouped together in a single embodiment for the purpose of simplifying the disclosure. This approach to the disclosure should not be interpreted as reflecting an intention that the disclosed embodiments of the disclosure must use more features than those explicitly recited in each claim. On the contrary, as reflected in the appended claims, the subject matter of the present invention lies in less than all of the features of a single disclosed embodiment. Therefore, the appended claims are hereby incorporated into the Detailed Description, with each claim standing on its own as a separate embodiment.

Claims

1. A three-layer hierarchical memory device (100, 201, 301), comprising: Persistent storage (216, 316); a first non-persistent memory (230, 330); as well as A second non-persistent memory (219, 319) configured to: Before the apparatus (100, 201, 301) receives a request from an input / output (I / O) device (210, 310) to access data stored in the persistent memory (216, 316), mapping an address associated with the I / O device (210, 310) to an address in the logic circuitry (104, 204, 304); and After the apparatus (100, 201, 301) receives the request and accesses the data, the address associated with the I / O device (210, 310) is mapped to an address in the first non-persistent memory (230, 330), wherein: The address associated with the I / O device (210, 310) is contiguous with other addresses associated with the I / O device (210, 310) mapped by the second non-persistent memory (219, 319); and The address in the first non-persistent memory (230, 330) to which the address associated with the I / O device (210, 310) is mapped by the second non-persistent memory (219, 319) is not consecutive with other addresses in the first non-persistent memory (230, 330) to which the second non-persistent memory (219, 319) is mapped.

2. The device of claim 1, wherein the second non-persistent memory is included in memory management circuitry (214, 314) of the device.

3. The apparatus according to any one of claims 1 to 2, wherein: The persistent memory comprises NAND memory or 3D cross point memory or both; and The first non-persistent memory and the second non-persistent memory each include dynamic random access memory (DRAM).

4. The apparatus according to any one of claims 1 to 2, wherein: The persistent memory is configured to provide storage capacity to the computing system; and The first non-persistent memory is configured to store data requested by a host computing device of the computing system during operation of the computing system.

5. The apparatus of any one of claims 1-2, wherein the address in the logic circuitry to which the address associated with the I / O device is mapped is an address in a register (106, 206, 306) of the logic circuitry.

6. A three-layer hierarchical memory device (100, 201, 301), comprising: Persistent storage device (216, 316); a first non-persistent memory device (230, 330); as well as A memory management circuit system (214, 314) having a second non-persistent memory (219, 319), the second non-persistent memory being configured to: Before the apparatus (100, 201, 301) receives a request from an input / output (I / O) device (210, 310) to program data into the persistent memory (216, 316), mapping an address associated with the I / O device (210, 310) to an address in the logic circuitry (104, 204, 304); as well as After the apparatus (100, 201, 301) receives the request and programs the data, mapping the address associated with the I / O device (210, 310) to an address in the first non-persistent memory device (230, 330), wherein: The address associated with the I / O device (210, 310) is contiguous with other addresses associated with the I / O device (210, 310) mapped by the second non-persistent memory (219, 319); and The address in the first non-persistent memory (230, 330) to which the address associated with the I / O device (210, 310) is mapped by the second non-persistent memory (219, 319) is not consecutive with other addresses in the first non-persistent memory (230, 330) to which the second non-persistent memory (219, 319) is mapped.

7. The apparatus of claim 6, wherein the second non-persistent memory is configured to map additional addresses associated with the I / O device to addresses in the persistent memory device.

8. The apparatus of claim 6, wherein the second non-persistent memory is configured to map addresses associated with the data to the logic circuitry.

9. A method of operating a three-layer hierarchical memory, comprising: mapping addresses associated with input / output (I / O) devices (210, 310) in non-persistent memory (219, 319) of memory management circuitry (214, 314) to addresses in logic circuitry (104, 204, 304); receiving a request from the I / O device (210, 310) to access data stored in a persistent memory device (216, 316); After receiving the request, accessing the data stored in the persistent memory device (216, 316); as well as After accessing the data, mapping the address associated with the I / O device (210, 310) to an address in a non-persistent memory device (230, 330) in the non-persistent memory (219, 319) of the memory management circuitry (214, 314), wherein: The address associated with the I / O device (210, 310) is contiguous with other addresses associated with the I / O device (210, 310) mapped in the non-persistent memory (219, 319) of the memory management circuitry (214, 314); and The address in the non-persistent memory device (230, 330) to which the address associated with the I / O device (210, 310) is mapped in the non-persistent memory (219, 319) of the memory management circuit system (214, 314) is not continuous with other addresses in the non-persistent memory device (230, 330) mapped in the non-persistent memory (219, 319) of the memory management circuit system (214, 314).

10. The method according to claim 9, wherein: The address associated with the I / O device is a virtual address; and The address in the non-persistent memory device is a physical address.

11. The method of claim 9, wherein the method comprises: redirecting the request to access the data to the logic circuitry via an interface (208, 308); and The data is accessed after redirecting the request.

12. The method of claim 11, wherein the method includes storing in the non-persistent memory of the memory management circuitry an address corresponding to the redirection of the request.

13. The method of claim 9, wherein the method includes storing an updated address associated with the data in the non-persistent memory of the memory management circuitry after accessing the data.

14. The method of claim 13, wherein the updated address is a virtual address that maps the data to the non-persistent memory device.

15. The method of any one of claims 9 to 14, wherein the method comprises transferring the data to the non-persistent memory device after accessing the data.

16. A method of operating a three-layer hierarchical memory, comprising: mapping addresses associated with an input / output (I / O) device (210, 310) in a first non-persistent memory (219, 319) to addresses in the logic circuitry (104, 204, 304); receiving a request from the I / O device (210, 310) to program data to persistent memory (216, 316); After receiving the request, programming the data to the persistent memory (216, 316) or a second non-persistent memory (230, 330); and After programming the data, mapping the address associated with the I / O device (210, 310) in the first non-persistent memory (219, 319) to an address in the second non-persistent memory (230, 330), wherein: The address associated with the I / O device (210, 310) is contiguous with other addresses associated with the I / O device (210, 310) mapped in the first non-persistent memory (219, 319); and The address in the second non-persistent memory (230, 330) to which the address associated with the I / O device (210, 310) is mapped in the first non-persistent memory (219, 319) is not continuous with other addresses in the second non-persistent memory (219, 319) mapped in the first non-persistent memory (219, 319).

17. The method of claim 16, wherein the method comprises: redirecting the request to program the data to the logic circuitry via an interface (208, 308); and The data is programmed after redirecting the request.

18. The method of any of claims 16 to 17, wherein the method comprises determining which of the persistent memory and the second non-persistent memory to program the data to based on a traffic pattern associated with the data.

19. The method of any one of claims 16 to 17, wherein the method comprises storing an updated address associated with the data in the first non-persistent memory after programming the data, wherein The updated address maps the data to a physical address in the second non-persistent memory.

Images