Hierarchical Memory System

Abstract

Apparatus, systems, and methods for hierarchical memory systems are described. A hierarchical memory system can utilize persistent memory to store data that is typically stored in non-persistent memory, thereby increasing the amount of storage space allocated to a computing system at a lower cost than methods that rely solely on non-persistent memory. In one exemplary apparatus, an input / output (I / O) device can receive signaling including a command to write data to an address corresponding to a non-persistent memory device or to read data from it, and can determine where to redirect the request. For example, the I / O device can determine to write the data to the non-persistent memory device or the persistent memory device and / or to read the data from it based at least in part on one or more characteristics of the data.

Description

Technical Field

[0001] The present disclosure relates generally to semiconductor memories and methods, and more particularly to devices, systems, and methods related to hierarchical memory systems. Background Art

[0002] Memory devices are typically provided as internal, semiconductor, integrated circuits in computers or other electronic systems. There are many different types of memory, including volatile and non-volatile memory. Volatile memory may require power to maintain its data (e.g., host data, error data, etc.), and includes random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), and synchronous dynamic random access memory (SDRAM), etc. Non-volatile memory can provide persistent data by saving stored data when not powered, and can include NAND flash memory, NOR flash memory, and resistance variable memory, such as phase change random access memory (PCRAM), resistance random access memory (RRAM), and magnetoresistive random access memory (MRAM), such as spin torque transfer random access memory (STT RAM), etc.

[0003] The memory device can be coupled to a host (e.g., a host computing device) to store data, commands, and / or instructions for use by the host when the computer or electronic system is operating. For example, data, commands, and / or instructions can be transferred between the host and the memory device during operation of the computing or other electronic system. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Figure 1 is a functional block diagram of a device in the form of a device including logic circuitry according to various embodiments of the present disclosure.

[0005] Figure 2 is a functional block diagram of a computing system including logic circuitry and input / output (I / O) devices according to various embodiments of the present disclosure.

[0006] Figure 3 is another functional block diagram of a computing system including logic circuitry and input / output (I / O) devices in accordance with various embodiments of the present disclosure.

[0007] Figure 4 is a flow chart showing a data reading operation according to various embodiments of the present disclosure.

[0008] Figure 5 is a flow chart showing a data writing operation according to various embodiments of the present disclosure.

[0009] Figure 6is a flow chart representing an exemplary method for a hierarchical memory system according to various embodiments of the present disclosure. DETAILED DESCRIPTION

[0010] Apparatus, systems, and methods for hierarchical memory systems are described. A hierarchical memory system can utilize persistent memory to store data that is typically stored in non-persistent memory, thereby increasing the amount of storage space allocated to a computing system at a lower cost than methods that rely solely on non-persistent memory. In one exemplary apparatus, an input / output (I / O) device can receive signaling including a command to write data to an address corresponding to a non-persistent memory device or to read data from it, and can determine where to redirect the request. For example, the I / O device can determine to write the data to the non-persistent memory device or the persistent memory device and / or to read the data from it based at least in part on one or more characteristics of the data.

[0011] Computing systems utilize various types of memory resources during operation. For example, a computing system may utilize a combination of volatile (e.g., random access memory) memory resources and nonvolatile (e.g., storage) memory resources during operation. Generally speaking, volatile memory resources can operate at much faster speeds than nonvolatile memory resources and can have a longer lifespan than nonvolatile memory resources; however, volatile memory resources are typically more expensive than nonvolatile memory resources. As used herein, volatile memory resources may be referred to as "non-persistent memory devices" in the alternative, and nonvolatile memory resources may be referred to as "persistent memory devices" in the alternative.

[0012] However, persistent memory devices may more broadly refer to the ability to access data in a persistent manner. As an example, in the context of persistent memory, a memory device may store a plurality of logical-to-physical mapping or conversion data and / or lookup tables in a memory array to track the location of data in the memory device, whether or not the memory is non-volatile. Furthermore, persistent memory devices may refer to the non-volatility of the memory, as well as the ability to use the non-volatility by including the ability to serve commands for a continuous process (e.g., by using logical-to-physical mappings, lookup tables, etc.).

[0013] These characteristics may need to be weighed in computing systems in order to provide computing systems with sufficient resources to operate in accordance with the growing demands of consumers and computing resource providers. For example, in a multi-user computing network (e.g., cloud-based computing system deployments, software-defined data centers, etc.), relatively large amounts of volatile memory may be provided to provide virtual machines operating in the multi-user network. However, by relying on volatile memory to provide memory resources for a multi-user network, as is common in some approaches, the costs associated with providing memory resources for the network may increase, especially as users of the network demand that larger and larger pools of computing resources be available for use.

[0014] Furthermore, in methods that rely on volatile memory to provide memory resources for provisioning virtual machines in a multi-user network, once the volatile memory resources are exhausted (e.g., once the volatile memory resources are allocated to users of the multi-user network), additional users may not be added to the multi-user network until additional volatile memory resources are available or added. This may result in potential users being turned away, thereby potentially resulting in lost revenue where the multi-user network could have used additional memory resources.

[0015] Volatile memory resources, such as dynamic random access memory (DRAM), tend to operate in a deterministic manner, while non-volatile memory resources, such as storage class memory (e.g., NAND flash memory devices, solid-state drives, resistance variable memory devices, etc.) tend to operate in a non-deterministic manner. For example, due to error correction operations, encryption operations, RAID operations, etc. performed on data retrieved from the storage class memory device, the amount of time between requesting data from the storage class memory device and the data becoming available may vary on each read, making the retrieval of data from the storage class memory device non-deterministic. In contrast, the amount of time between requesting data from a DRAM device and the data becoming available may remain fixed on each read, making the retrieval of data from the DRAM device deterministic.

[0016] Furthermore, due to the distinction between the deterministic behavior of volatile memory resources and the non-deterministic behavior of non-volatile memory resources, data transferred to and from memory resources typically passes through a specific interface (e.g., bus) associated with the type of memory used. For example, data transferred to and from DRAM devices is typically passed via a double data rate (DDR) bus, while data transferred to and from NAND devices is typically passed via a peripheral component interconnect express (PCI-e) bus. However, as will be appreciated, the examples of interfaces through which data may be transferred to and from volatile memory resources and non-volatile memory resources are not limited to these specifically enumerated examples.

[0017] Due to the different behaviors of non-volatile memory devices and volatile memory devices, some methods choose to store certain types of data in volatile or non-volatile memory. This can alleviate problems that may arise due to, for example, the deterministic behavior of volatile memory devices compared to the non-deterministic behavior of non-volatile memory devices. For example, in some methods, a computing system stores a small amount of data that is regularly accessed during operation of the computing system in a volatile memory device, while larger or less frequently accessed data is stored in a non-volatile memory device. However, in a multi-user network deployment, the vast majority of data may be stored in a volatile memory device. In contrast, embodiments of the present invention can allow data storage and retrieval from non-volatile memory devices deployed in a multi-user network.

[0018] As described herein, in other methods, some embodiments of the present disclosure relate to computing systems in which data from non-volatile and therefore non-deterministic memory resources is transferred via an interface that is restricted to be used by volatile and deterministic memory resources. For example, in some embodiments, data can be transferred to and from non-volatile, non-deterministic memory resources via an interface such as a DDR interface, the memory resources such as NAND flash memory devices, resistance variable memory devices, such as phase change memory devices and / or resistive memory devices (e.g., three-dimensional cross point (3DXP) memory devices), solid state drives (SSDs), self-selected memory (SSM) devices, etc., and in some methods, the interface is reserved for transferring data to and from volatile, deterministic memory resources. Therefore, compared to methods that use volatile, deterministic memory devices to provide main memory to a computing system, embodiments of the present invention can allow non-volatile, non-deterministic memory devices to be used as at least part of the main memory of a computing system.

[0019] In some embodiments, data may be intermediately transferred from the nonvolatile memory resource to a cache (e.g., a small static random access memory (SRAM) cache) or buffer and subsequently made available to applications requesting the data. By storing data that is normally provided in a deterministic manner in a non-deterministic memory resource and allowing access to the data as described herein, computing system performance may be improved by, for example, allowing larger amounts of memory resources to be available to a multi-user network at a significantly reduced cost compared to methods operating using volatile memory resources.

[0020] For the convenience of embodiments of the present disclosure, the visibility of non-volatile memory resources may be obscured for various devices of a computing system that deploys a hierarchical memory system. For example, a host, a network interface card, a virtual machine, etc. deployed in a computing system or a multi-user network may not be able to distinguish whether data is stored by a volatile memory resource of the computing system or by a non-volatile memory resource. For example, a hardware circuit system may be deployed in a computing system that may register addresses corresponding to data in a manner such that a host, a network interface card, a virtual machine, etc. cannot distinguish whether data is stored by a volatile memory resource or by a non-volatile memory resource.

[0021] As described in more detail herein, a hierarchical memory system may include hardware circuitry (e.g., logic circuitry) that may intercept redirected data requests, register addresses in the logic circuitry associated with the requested data (even though the hardware circuitry is not backed up by its own memory resources to store the data), and use the logic circuitry to map the addresses registered in the logic circuitry to physical addresses corresponding to the data in the non-volatile memory device.

[0022] In the following detailed description of the present disclosure, reference is made to the accompanying drawings which form a part of the present disclosure and in which are shown by way of illustration the manner in which one or more embodiments of the present disclosure may be practiced. These embodiments are described in sufficient detail to enable one of ordinary skill in the art to practice the embodiments of the present disclosure, and it is understood that other embodiments may be utilized and process, electrical and structural changes may be made without departing from the scope of the present disclosure.

[0023] As used herein, designators such as "N," "M," etc., specifically with respect to a figure numeral in the accompanying drawings indicate that a plurality of the particular feature so designated may be included. It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used herein, the singular forms "a," "an," and "the" may include singular and plural referents unless the context clearly dictates otherwise. Additionally, "plurality," "at least one," and "one or more" (e.g., a plurality of memory banks) may refer to one or more memory banks, while "plurality" is intended to refer to more than one such thing.

[0024] In addition, in this application, the words "may" and "might" are used in a permissive sense (i.e., it is possible to, able to), rather than in a mandatory sense (i.e., must). The term "including" and its derivatives mean "including but not limited to". Depending on the context, the terms "coupled" and "coupling" refer to physically connecting or accessing and moving (transmitting) commands and / or data directly or indirectly. Depending on the context, the terms "data" and "data value" are used interchangeably herein and may have the same meaning.

[0025] The figures herein follow a numbering convention in which the first digit or digits correspond to the figure number and the remaining digits identify an element or component in the figure. Similar elements or components between different figures may be identified by using similar numerals. For example, 104 may refer to Figure 1 04 in the figure, and similar elements may be referred to as Figure 2 204 in. A group or plurality of similar elements or components may generally be referred to herein by a single element number. For example, a plurality of reference elements 106-1, 106-2, ..., 106-N (e.g., 106-1 to 106-N) may be collectively referred to as 106. As will be appreciated, the elements shown in the various embodiments herein may be added, exchanged, and / or eliminated to provide a plurality of additional embodiments of the present disclosure. In addition, the proportions and / or relative dimensions of the elements provided in the accompanying drawings are intended to illustrate certain embodiments of the present disclosure and should not be considered in a limiting sense.

[0026] Figure 1 is a functional block diagram of a computing system 100 in the form of a plurality of embodiments of the present disclosure, the computing system comprising an apparatus comprising a logic circuit system 104. As used herein, a "device" may refer to, but is not limited to, any of a variety of structures or combinations of structures, such as a circuit or circuit system, one or more dies, one or more modules, one or more devices, or one or more systems. In some embodiments, the logic circuit system 104 may be provided as a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a plurality of discrete circuit components, etc., and may be alternatively referred to herein as a "hierarchical memory component".

[0027] like Figure 1As shown in , logic circuitry 104 may include memory resources 102, which may include read buffers 103, write buffers 105, and / or input / output I / O device access components 107. In some embodiments, memory resources 102 may be random access memory resources, such as block RAM, which may allow data to be stored within logic circuitry 104 in embodiments where logic circuitry 104 is an FPGA. However, embodiments are not limited thereto, and memory resources 102 may include various registers, caches, memory arrays, latches, and SRAM, DRAM, EPROM, or other suitable memory technologies that may store data such as bit strings containing a registered address corresponding to a physical location where the data is stored external to logic circuitry 104. Memory resources 102 are internal to logic circuitry 104 and are typically smaller than memory external to logic circuitry 104, such as persistent and / or non-persistent memory resources that may be external to logic circuitry 104.

[0028] Read buffer 103 may include a portion of memory resources 102 that is reserved for storing data that has been received by logic circuitry 104 but has not yet been processed by logic circuitry 104. In some embodiments, the size of read buffer 103 may be around 4 kilobytes (KB), but embodiments are not limited to this particular size. Read buffer 103 may buffer data to be registered in one of address registers 106-1 through 106-N.

[0029] Write buffer 105 may comprise a portion of memory resources 102 that is reserved for storing data awaiting transfer to a location external to logic circuitry 104. In some embodiments, write buffer 105 may be approximately 4 kilobytes (KB) in size, but embodiments are not limited to this particular size. Write buffer 105 may buffer data registered in one of address registers 106-1 through 106-N.

[0030] I / O access component 107 may include a portion of memory resources 102 that is reserved for storing and accessing components external to logic circuitry 104 (e.g., Figure 2 and 3 100). The I / O access component 107 may store data corresponding to addresses of the I / O devices, which may be used to read and / or write data to and from the I / O devices. In addition, in some embodiments, the I / O access component 107 may receive, store, and / or transmit data corresponding to the management program (e.g., Figure 3 312) shown in the state of the hypervisor, as described herein Figure 3 Described in more detail.

[0031] Logic circuitry 104 may further include a memory access multiplexer (MUX) 109, a state machine 111, and / or a hierarchical memory controller 113 (or simply “controller”). Figure 1 , hierarchical memory controller 113 may include a plurality of address registers 106-3 through 106-N and / or an interrupt component 115. Memory access MUX 109 may include circuitry that may include one or more logic gates and may be configured to control data and / or address bus connections of logic circuitry 104. For example, memory access MUX 109 may transfer messages to and from memory resources 102, as well as communicate with hierarchical memory controller 113 and / or state machine 111, as described in more detail below.

[0032] In some embodiments, MUX 109 may redirect incoming messages and / or commands received from a host (e.g., a host computing device, a virtual machine, etc.) to logic circuitry 104. For example, MUX 109 may redirect incoming messages and / or commands received from a host (e.g., a host computing device, a virtual machine, etc.) to logic circuitry 104. Figure 2 and 3 Incoming messages corresponding to access requests of one of the address registers (e.g., address registers 106-N, which may be the BAR4 region of the hierarchical memory controller 113, as described below) of the I / O device 210 / 310 shown in the figure are redirected to the read buffer 103 and / or the write buffer 105.

[0033] In addition, MUX 109 may redirect requests (eg, read requests, write requests) received by logic circuitry 104. In some embodiments, requests may be received by logic circuitry 104 from a hypervisor (eg, Figure 3 106-2, which may be the BAR2 region of the hierarchical memory controller 113, as described below).

[0034] MUX 109 may redirect such requests as part of an operation to determine an address in address register 106 to be accessed. In some embodiments, MUX 109 may redirect such requests as part of an operation to determine an address in an address register to be accessed in response to assertion of a hypervisor interrupt (e.g., an interrupt asserted to a hypervisor coupled to logic circuitry 104, which is generated by interrupt component 115).

[0035] In response to determining that the request corresponds to data (e.g., data associated with the address being written to a location external to logic circuitry 104 (e.g., written to a persistent memory device, such as the one described herein) Figure 2 and 3 In response to determining that the request corresponds to data being read from a location external to logic circuitry 104 (e.g., from a persistent memory device 216 / 316 shown in FIG), MUX 109 may facilitate data retrieval, transfer of data to write buffer 105, and / or transfer of data to a location external to logic circuitry 104. In response to determining that the request corresponds to data being read from a location external to logic circuitry 104 (e.g., from a persistent memory device), MUX 109 may facilitate data retrieval, transfer of data to read buffer 103, and / or transfer of data or address information associated with the data to a location internal to logic circuitry 104, such as address register 106.

[0036] As a non-limiting example, if logic circuitry 104 receives a read request from an I / O device, MUX 109 may facilitate retrieval of data from a persistent memory device via the hypervisor by selecting an appropriate message to be sent from logic circuitry 104. For example, MUX 109 may facilitate generation of an interrupt using interrupt component 115, causing the interrupt to be asserted on the hypervisor, buffering data received from the persistent memory device into read buffer 103, and / or responding to the I / O device with an indication that the read request has been satisfied. In a non-limiting example where logic circuitry 104 receives a write request from the I / O device, MUX 109 may facilitate transfer of data to the persistent memory device via the hypervisor by selecting an appropriate message to be sent from logic circuitry 104. For example, MUX 109 may facilitate generation of an interrupt using interrupt component 115, causing the interrupt to be asserted on the hypervisor, buffering data to be transferred to the persistent memory device into write buffer 105, and / or responding to the I / O device with an indication that the write request has been satisfied.

[0037] The state machine 111 may include one or more processing devices, circuit components, and / or logic configured to operate on inputs and generate outputs. In some embodiments, the state machine 111 may be a finite state machine (FSM) or a hardware state machine that may be configured to receive varying inputs and generate resulting outputs based on the received inputs. For example, the state machine 111 may transfer access information (e.g., “I / O ACCESS INFO”) to and from the memory access multiplexer 109, and transfer interrupt configuration information (e.g., “INTERRUPT CONFIG”) and / or interrupt request messages (e.g., “INTERRUPT REQUEST”) to and from the hierarchical memory controller 113. In some embodiments, the state machine 111 may further transfer control messages (e.g., “MUX CTRL”) to and from the memory access multiplexer 109.

[0038] The ACCESS INFO message may include information corresponding to a data access request received from an I / O device external to logic circuitry 104. In some embodiments, the ACCESS INFO may include logical addressing information corresponding to data to be stored in a persistent memory device or addressing information corresponding to data to be retrieved from a persistent memory device.

[0039] The INTERRUPT CONFIG message may be asserted by state machine 111 on hierarchical memory controller 113 to configure the appropriate interrupt message to be asserted external to logic circuitry 104. For example, when logic circuitry 104 asserts an interrupt as part of satisfying a redirected read or write request on a hypervisor coupled to logic circuitry 104, the INTERRUPT CONFIG message may be generated by state machine 111 to generate the appropriate interrupt message based on whether the operation is an operation to retrieve data from a persistent memory device or an operation to write data to a persistent memory device.

[0040] The INTERRUPT REQUEST message may be generated by the state machine 111 and asserted on the interrupt component 115 to cause the interrupt message to be asserted on the hypervisor (or bare metal server or other computing device). As described in more detail herein, the interrupt 115 may be asserted on the hypervisor to cause the hypervisor to prioritize data retrieval or writing of data to a persistent memory device as part of the operation of the hierarchical memory system.

[0041] MUX CTRL messages may be generated by state machine 111 and asserted on MUX 109 to control the operation of MUX 109. In some embodiments, MUX CTRL messages may be asserted on MUX 109 by state machine 111 (or vice versa) as part of the conduct of the MUX 109 operation described above.

[0042] The hierarchical memory controller 113 may include a core, such as an integrated circuit, a chip, a system on a chip, or a combination thereof. In some embodiments, the hierarchical memory controller 113 may be a peripheral component interconnect express (PCIe) core. As used herein, a "core" refers to a reusable unit of logic, a processor, and / or a coprocessor that receives instructions and performs tasks or actions based on the received instructions.

[0043] Hierarchical memory controller 113 may include address registers 106-1 to 106-N and / or interrupt component 115. Address registers 106-1 to 106-N may be registers that may store information received from logic circuitry 104 or a computing system (e.g., Figure 2 and 3 At least one of the address registers (e.g., address register 106-1) may store a base address register (BAR) for a memory address used by the computing system 201 / 301 shown in FIG. 106-1. Figure 3 Memory addresses of accesses to internal registers of logic circuit system 104 from an external location of hypervisor 312 are shown in FIG.

[0044] A different address register (e.g., address register 106-2) may be used to store addresses corresponding to interrupt controls, as described in more detail herein. In some embodiments, address register 106-2 may map direct memory access (DMA) read and DMA write control and / or status registers. For example, address register 106-2 may contain addresses corresponding to descriptors and / or control bits for a DMA command chain, which may include the generation of one or more interrupt messages that may be asserted to a hypervisor as part of the operation of a hierarchical memory system, as described in conjunction with the present disclosure. Figure 3 described.

[0045] Another one of the address registers (e.g., address register 106-3) may store a message associated with a supervisor program (e.g., Figure 3In some embodiments, the address register may map addresses corresponding to data transferred via a DMA (e.g., AXI DMA) of logic circuitry 104 to locations external to logic circuitry 104.

[0046] In some embodiments, at least one address register (eg, address register 106-N) may store a value associated with an I / O device (eg, Figure 2 104). The address registers 106-N may store addresses that are bypassed by DMA components associated with the logic circuit system 104. The address registers 106-N may be provided so that addresses mapped thereto are not “backed up” by physical memory locations of the logic circuit system 104. That is, in some embodiments, the logic circuit system 104 may be configured with an address space that stores addresses associated with a persistent memory device (e.g., Figure 2 106-N) rather than corresponding to data stored by logic circuitry 104. For example, address registers 106-N may be configured as a virtual address space that may store logical addresses corresponding to physical memory locations (e.g., in a memory device) where data is stored.

[0047] In some embodiments, address register 106-N may include a memory device (e.g., Figure 2 and 3 106-N may be configured to have an address space that corresponds to the size of the persistent memory device 216 / 316 shown in FIG. 106-N. For example, if the memory device contains 1 terabyte of storage, then the address register 106-N may be configured to have an address space that may contain 1 terabyte of address space. However, as described above, the address register 106-N does not actually contain 1 terabyte of storage, but is configured to appear to have 1 terabyte of storage space.

[0048] Although not in Figure 1Although not explicitly shown in the figure, the logic circuit system 104 can be coupled to a host computing system. The host computing system can include a system motherboard and / or a backplane, and can include multiple processing resources (e.g., one or more processors, microprocessors, or some other type of control circuit system). The host and device 100 can be, for example, a server system and / or a high performance computing (HPC) system and / or a portion thereof. In some embodiments, the computing system can have a von Neumann architecture, however, embodiments of the present disclosure can be implemented in a non-von Neumann architecture, which generally may not include one or more components (e.g., CPU, ALU, etc.) that are typically associated with a von Neumann architecture.

[0049] As described herein, logic circuitry 104 may be connected to the network via I / O devices (e.g., Figure 2 and 3 100) receives incoming messages and / or commands from a host (e.g., a host computing device, a virtual machine, etc.), which redirects the messages and / or commands to logic circuitry 104 and / or non-persistent memory devices (e.g., Figure 2 and 3 230 / 330 shown in FIG. 230). Incoming messages and / or commands redirected from an I / O device to a logic circuit system may include treating the non-persistent memory device 230 / 330 of this document as being from / to a persistent memory device (e.g., Figure 2 and 3 Requests for data to be read and / or written to the persistent memory device 216 / 316) shown in FIG. 1 and data to be written to the persistent memory device.

[0050] Since data transferred to / from a persistent memory device is usually of considerable size, it is desirable to reduce the amount of time (in order to read and / or write data from / to the persistent memory device) required by a computing system (e.g., Figure 2 and 3 The number of data transfers between components of the computing system 201 / 301 shown in FIG. 2 can significantly improve the overall processing speed of the computing system. Therefore, in various embodiments, a particular device of the computing system that first receives the request from the host, such as an I / O device, can be configured as a device for determining where to redirect the request. This can provide fewer transfers than using a different device (e.g., different from the I / O device that first receives the request from the host) to determine where to redirect the request. Figure 2-6 Describes additional details of redirecting requests and data using I / O devices.

[0051] Figure 22 is a functional block diagram of a computing system 201 including logic circuitry (eg, logic circuitry 204) and input / output (I / O) devices according to various embodiments of the present disclosure. Figure 2 As shown in FIG. 1 , computing system 201 may include logic circuitry 204, which may be similar to Figure 1 2. In addition, computing system 201 may include I / O devices 210, persistent memory devices 216, non-persistent memory devices 230, and intermediate memory components 220. Communication between logic circuitry 204, I / O devices 210, and persistent memory devices 216 and non-persistent memory devices 230 may be facilitated via interface 208.

[0052] The persistent memory device 216 may include multiple arrays of memory cells. For example, the array may be a flash array having a NAND architecture. However, embodiments are not limited to a particular type of memory array or array architecture. The memory cells may, for example, be grouped into multiple blocks that include multiple physical pages. Multiple blocks may be included in a plane of memory cells, and the array may include multiple planes.

[0053] The persistent memory device 216 may include volatile memory and / or non-volatile memory. In various embodiments, the persistent memory device 216 may include a multi-chip device. The multi-chip device may include multiple different memory types and / or memory modules. For example, the memory system may include non-volatile or volatile memory on any type of module. In embodiments where the persistent memory device 216 includes non-volatile memory, the persistent memory device 216 may be a flash memory device, such as a NAND or NOR flash memory device.

[0054] However, embodiments are not limited thereto, and the persistent memory device 216 may include other non-volatile memory devices, such as non-volatile random access memory devices (e.g., NVRAM, ReRAM, FeRAM, MRAM, PCM), "emerging" memory devices, such as resistance variable memory devices (e.g., resistive and / or phase change memory devices, such as 3D cross point (3D XP) memory devices), memory devices including self-select memory (SSM) cell arrays, etc., or the like or a combination thereof. The resistive and / or phase change array of the non-volatile memory may be combined with a stackable cross-grid data access array to perform bit storage based on changes in body resistance. In addition, compared to many flash-based memories, the resistive and / or phase change memory devices may perform in-situ write operations, where the non-volatile memory cells may be programmed without pre-erasing the non-volatile memory cells. Compared to flash-based memories, the self-select memory cells may include memory cells having a single chalcogenide material that serves as a switch and storage element for the memory cell.

[0055] Persistent memory device 216 may provide a storage volume for computing system 201 and may therefore be used as additional memory or storage for the entire computing system 201, main memory for computing system 201, or a combination thereof. However, embodiments are not limited to a particular type of memory device, and persistent memory device 216 may include RAM, ROM, SRAM DRAM, SDRAM, PCRAM, RRAM, and flash memory, among others. In addition, although in Figure 2 216, but embodiments are not limited thereto, and computing system 201 may include one or more persistent memory devices 216, each of which may or may not have the same architecture associated therewith. As a non-limiting example, in some embodiments, persistent memory device 216 may include two discrete memory devices of different architectures, such as a NAND memory device and a resistance variable memory device.

[0056] The non-persistent memory device 230 may include a volatile memory, such as an array of volatile memory cells. In multiple embodiments, the non-persistent memory device 230 may include a multi-chip device. The multi-chip device may include multiple different memory types and / or memory modules. In some embodiments, the non-persistent memory device 230 may be used as the main memory of the computing system 201. For example, the non-persistent memory device 230 may be a dynamic random access (DRAM) memory device, which is used to provide main memory for the computing system 230. However, the embodiments are not limited to the non-persistent memory device 230 including a DRAM memory device, and in some embodiments, the non-persistent memory device 230 may include other non-persistent memory devices, such as RAM, SRAM DRAM, SDRAM, PCRAM and / or RRAM, etc.

[0057] Non-persistent memory device 230 may store data that may be requested by, for example, a host computing device as part of the operation of computing system 201. For example, when computing system 201 is part of a multi-user network, non-persistent memory device 230 may store data that may be transferred between host computing devices (e.g., virtual machines deployed in the multi-user network) during operation of computing system 201.

[0058] In some approaches, non-persistent memory, such as non-persistent memory device 230, can store all user data accessed by a host (e.g., a virtual machine deployed in a multi-user network). For example, due to the speed of non-persistent memory, some approaches rely on non-persistent memory to provide memory resources for virtual machines deployed in a multi-user network. However, in such approaches, cost may become an issue because non-persistent memory is generally more expensive than persistent memory (e.g., persistent memory device 216).

[0059] In contrast, as described in more detail below, embodiments herein may allow at least some of the data stored in non-persistent memory device 230 to be stored in persistent memory device 216. This may allow additional memory resources to be provided to computing system 201, such as a multi-user network, at a lower cost than approaches that rely on non-persistent memory for user data storage.

[0060] The computing system 201 may include an I / O device 210, which may be communicatively coupled to the logic circuit system 204 and the non-persistent memory device via the interface 208. The I / O device 210 may be a device configured to provide direct memory access via a physical address and / or a virtual machine physical address. In some embodiments, the I / O device 210 may be a network interface card (NIC) or a network interface controller, a storage device, a graphics rendering device, or other I / O device. The I / O device 210 may be a physical I / O device, or the I / O device 210 may be a virtualized I / O device 210. For example, in some embodiments, the I / O device 210 may be a physical card that is physically coupled to the computing system via a bus or interface such as a PCIe interface or other suitable interface. In embodiments where the I / O device 210 is a virtualized I / O device 210, the virtualized I / O device 210 may provide I / O functions in a distributed manner.

[0061] The I / O device 210 may include a plurality of mapping tables 214 (e.g., mapping tables 214-1, ..., 214-N), which may be used to map device-visible virtual addresses to physical addresses. For example, when receiving a read / write request to the I / O device 210 from a host computing device, the I / O device may map a virtual address associated with the requested data to a physical address in the non-persistent memory device 230 and / or an address in the logic circuit system 204, which may be further mapped to a corresponding physical address in the persistent memory device 216. By utilizing the mapping tables 214, the I / O device 210 may determine where to redirect a request received from the host computing device. For example, if the mapping table 214 indicates that the address associated with the requested data is mapped to a physical address of the non-persistent memory device 230, the I / O device 210 may redirect the request to the non-persistent memory 230. For example, if mapping table 214 indicates that an address associated with the requested data is mapped to an address in logic circuitry 204 , I / O device 210 may redirect the request to logic circuitry 204 , which may transfer the redirected request from I / O device 210 to persistent memory device 216 .

[0062] In some embodiments, mapping a virtual entry associated with an I / O device 210 may be performed by Figure 12. The read buffer, write buffer, and / or I / O access buffer shown in FIG. 2 facilitates. Logic circuit system 204 may store virtual address information associated with a read / write request received at I / O device 210 in an address register (e.g., address register 206-1, 206-2, ..., and / or 206-N) of logic circuit system 204. In some embodiments, address register 206-N may be a specific base address register of logic circuit system 204, such as a BAR4 address register.

[0063] In some embodiments, interface 208 may be a PCIe interface and thus may transfer information between I / O device 210 and logic circuitry 204 according to the PCIe protocol. However, embodiments are not limited thereto, and in some embodiments, interface 208 may be an interface or bus that functions according to another suitable protocol.

[0064] After the virtual NIC address is stored in logic circuitry 204, data corresponding to the virtual NIC address may be written to persistent memory device 216. For example, data corresponding to the virtual NIC address stored in logic circuitry 204 may be stored in a physical address location of persistent memory device 216. In some embodiments, the transfer of data to and / or from persistent memory device 216 may be facilitated by a hypervisor, such as described herein. Figure 3-5 described.

[0065] For example, when data is requested by a host computing device such as a virtual machine deployed in computing system 201, the request may be redirected from I / O device 210 to logic circuitry 204. Because the virtual NIC address corresponding to the physical location of the data in persistent memory device 216 is stored in one of address registers 206 (e.g., address register 206-N) of logic circuitry 204, logic circuitry 204 may facilitate retrieval of data from persistent memory device 216 in conjunction with the hypervisor, as described herein. Figure 3-5 Described in more detail.

[0066] In some embodiments, when data already stored in persistent memory device 216 is transferred out of persistent memory device 216 (e.g., when data already stored in persistent memory device 216 is requested by a host computing device), the data may be transferred to intermediate memory component 220 and / or non-persistent memory device 230 before being provided to the host computing device. For example, because data transferred to a host computing device may be transferred in a deterministic manner (e.g., via a DDR interface), the data may be temporarily transferred to a memory operating using a DDR bus, such as intermediate memory component 220 and / or non-persistent memory device 230, before the data request is satisfied.

[0067] Figure 3 is another functional block diagram of a computing system including logic circuitry and I / O devices according to various embodiments of the present disclosure. Figure 3 As shown in FIG. 1 , computing system 301 may include logic circuitry 304, which may be similar to Figure 1 and 2 1. In addition, computing system 301 may include I / O devices 310, persistent memory devices 316, non-persistent memory devices 330, intermediate memory components 320, and hypervisor 312.

[0068] In some embodiments, computing system 301 may be a multi-user network, such as a software-defined data center, a cloud computing environment, etc. In such embodiments, the computing system may be configured to run thereon one or more virtual machines 317. For example, in some embodiments, one or more virtual machines 317 may be deployed on hypervisor 312 and may be accessed by users of the multi-user network.

[0069] I / O device 310, persistent memory device 316, non-persistent memory device 330, and intermediate memory component 320 may be similar to Figure 2 304, I / O devices 310, persistent memory devices 216, non-persistent memory devices 230, and intermediate memory components 220 are shown in FIG. Communication between logic circuitry 304, I / O devices 310, and persistent memory devices 316, non-persistent memory devices 330, and hypervisor 312 may be facilitated via interface 308, which may be similar to Figure 2 Interface 208 shown in FIG.

[0070] As above combined Figure 2 As described, I / O device 310 may redirect a read request or a write request received from a host (e.g., a host computing device, a virtual machine, etc.) to logic circuit system 304. Logic circuit system 304 may generate and / or store a logical address corresponding to the request data. As described above, logic circuit system 304 may store the logical address corresponding to the request data in a base address register, such as address register 306-N of logic circuit system 304.

[0071] like Figure 3 As shown in FIG. 1 , hypervisor 312 may communicate with logic circuit system 304 and / or I / O device 310 via interface 308. Hypervisor 312 may access components (e.g., Figure 1NIC access component 107 shown in FIG. 10A ) transfers data between logical circuitry 304. In addition, hypervisor 312 can communicate with persistent memory device 316, non-persistent memory device 330, and intermediate memory component 320. The hypervisor can be configured to execute specialized instructions to perform the operations and / or tasks described herein.

[0072] For example, hypervisor 312 may execute instructions to monitor data traffic and data traffic patterns to determine whether data should be stored in non-persistent memory device 330 or whether data should be transferred to persistent memory device 316. That is, in some embodiments, hypervisor 312 may execute instructions to learn user data request patterns over time and selectively store portions of data in non-persistent memory device 330 or persistent memory device 316 based on the patterns. This may allow more frequently accessed data to be stored in non-persistent memory device 330, while less frequently accessed data is stored in persistent memory device 316.

[0073] Because a user may access recently used or viewed data more frequently than less recently used or less recently viewed data, the hypervisor may execute specialized instructions to cause less recently used or viewed data to be stored in persistent memory device 316, and / or cause recently accessed or viewed data to be stored in non-persistent memory device 330. In one non-limiting example, a user may view photos on social media taken recently (e.g., within a week, etc.) more frequently than photos taken less recently (e.g., a month ago, a year ago, etc.). Based on this information, hypervisor 312 may execute specialized instructions to cause photos not recently viewed or taken to be stored in persistent memory device 316, thereby reducing the amount of data stored in non-persistent memory device 330. This may reduce the total amount of non-persistent memory required to provide computing system 301, thereby reducing costs and allowing more users to access non-persistent memory device 330.

[0074] In operation, computing system 301 may be configured to intercept data requests from I / O device 310 and redirect the requests to logic circuitry 304. In some embodiments, hypervisor 312 may control whether data corresponding to a data request is stored in (or retrieved from) non-persistent memory device 330 or stored in persistent memory device 316. For example, hypervisor 312 may execute instructions to selectively control whether data is stored in (or retrieved from) persistent memory device 316 or stored in non-persistent memory device 330.

[0075] As part of controlling whether data is stored in (or retrieved from) persistent memory device 316 and / or non-persistent memory device 330, hypervisor 312 may cause I / O device 310 to map addresses associated with the data for redirection to logic circuitry 304 and storage in address registers 306 of logic circuitry 304. For example, hypervisor 312 may execute instructions to control read and write requests involving data to be selectively redirected to logic circuitry 304 via I / O device 310.

[0076] I / O device 310 may map continuous virtual addresses to underlying segmented physical addresses. Therefore, in some embodiments, I / O device 310 may allow virtual addresses to be mapped to physical addresses without requiring the physical addresses to be continuous. In addition, in some embodiments, I / O device 310 may allow devices that do not support memory addresses long enough to address their corresponding physical memory space to be addressed in I / O device 310.

[0077] Due to the non-deterministic nature of data transfers associated with persistent memory device 316, in some embodiments, logic circuitry 304 may be configured to notify computing system 301 that a delay may occur in transferring data to or from persistent memory device 316. As part of the initialization delay, logic circuitry 304 may provide page fault handling for computing system 301 when a data request is redirected to logic circuitry 304. In some embodiments, logic circuitry 304 may generate an interrupt and assert an interrupt to hypervisor 312 to initiate the transfer of data to or from persistent memory device 316. For example, due to the non-deterministic nature of data retrieval and storage associated with persistent memory device 316, logic circuitry 304 may generate hypervisor interrupt 315 when requesting a transfer of data stored in persistent memory device 316.

[0078] In response to the page fault interrupt generated by logic circuitry 304, hypervisor 312 may retrieve information corresponding to the data from logic circuitry 304. For example, hypervisor 312 may receive NIC access data from the logic circuitry, which may include a logical to physical address mapping corresponding to the data stored in address register 306 of logic circuitry 304.

[0079] Once data has been stored in persistent memory device 316, a portion (e.g., a page, block, etc.) of non-persistent memory device 330 may be marked by logic circuitry 304 as inaccessible so that computing system 301 does not attempt to access data from non-persistent memory device 330. This may allow data requests to be intercepted by page faults, which may be generated by logic circuitry 304 and asserted to hypervisor 312 when data already stored in persistent memory device 316 is requested by I / O device 310.

[0080] Compared to the method of causing a page fault exception in response to an application request to access a memory page that is not mapped by the I / O device 310, in an embodiment of the present disclosure, the above-mentioned page fault can be generated by the logic circuit system 304 in response to data being mapped to the logic circuit system 304 in the I / O device 310, and the logic circuit system then maps the data to the persistent memory device 316.

[0081] In some embodiments, the intermediate memory component 320 can be used to buffer data stored in the persistent memory device 316 in response to a data request initiated by the I / O device 310. Compared to the persistent memory device 316 that can transfer data via a PCIe interface, the intermediate memory component 320 can use a DDR interface to transfer data. Therefore, in some embodiments, the intermediate memory component 320 can operate in a deterministic manner. For example, in some embodiments, the requested data stored in the persistent memory device 316 can be temporarily transferred from the persistent memory device 316 to the intermediate memory component 320 and then transferred to the host computing device via the DDR interface that couples the intermediate memory component 320 to the I / O device 310.

[0082] In some embodiments, the intermediate memory component may include a discrete memory component (e.g., an SRAM cache) deployed in the computing system 301. However, the embodiments are not limited thereto, and in some embodiments, the intermediate memory component 320 may be a portion of a non-persistent memory device 330 that may be allocated to transfer data from the persistent memory device 316 in response to a data request.

[0083] In one non-limiting example, an I / O device (e.g., I / O device 310) may be coupled to a logic circuit system (e.g., logic circuit system 304) and a non-persistent memory device (e.g., non-persistent memory device 330). The I / O device may be configured to receive signaling including a command to write data to an address corresponding to the non-persistent memory device. The I / O device may be further configured to map the received address to an address in the logic circuit system or a physical address in the non-persistent memory device based at least in part on one or more characteristics of the data. The one or more characteristics of the data may include a frequency with which the data is requested or accessed, an amount of time that has passed since the data was last accessed or requested, a type of data (e.g., whether the data corresponds to a particular file type, such as a photo, document, audio file, application file, etc.), and the like. The I / O device may be further configured to write the data to the logic circuit system based at least in part on the received address being mapped to an address in the logic circuit system so that the logic circuit system writes the data to the persistent memory device. In some embodiments, the I / O device may be configured to write data to a location of the non-persistent memory device corresponding to the physical address based at least in part on the received address being mapped to a physical address in the non-persistent memory.

[0084] In some embodiments, the I / O device may include a mapping table (e.g., mapping table 314). The I / O device may be configured to store the received address and the corresponding address to which the received address is mapped in the mapping table. Thus, the I / O device may determine that the received address is mapped to an address in the logic circuitry based on the mapping table in response to receiving a request to access data previously stored in a persistent memory device (e.g., persistent memory device 316), and redirect the request to access the data to the logic circuitry, for example, to cause the logic circuitry to access the data from the persistent memory device.

[0085] In another non-limiting example, an I / O device (e.g., I / O device 310) may be coupled to a logic circuit system (e.g., logic circuit system 304) that is coupled to a persistent memory device (e.g., persistent memory device 316) and a non-persistent memory device (e.g., non-persistent memory device 330). The I / O device may include a first address that is mapped to a corresponding physical address in the non-persistent memory device and a second address that is mapped to a corresponding address in the logic circuit system. The I / O device may be configured to receive signaling including a command to access data from an address corresponding to the non-persistent memory device and determine that the received address is one of the second addresses. The I / O device may be further configured to redirect the command to the logic circuit system based at least in part on the received address being determined to be one of the second addresses to cause the logic circuit system to access data from the persistent memory device.

[0086] In some embodiments, the I / O device may be configured to determine that the received address is one of the first addresses and redirect the command to the non-persistent memory device. The I / O device may be further configured to receive data from the non-persistent memory device in response to redirecting the command to the non-persistent memory device.

[0087] In some embodiments, the first address and the second address may be corresponding virtual machine physical addresses. In some embodiments, the first address and the second address may be stored in a mapping table (eg, mapping table 314).

[0088] In another non-limiting example, a system may include a logic circuit system (e.g., logic circuit system 304), a non-persistent memory device (e.g., non-persistent memory device 330) coupled to the logic circuit system via an interface (e.g., interface 308), and an I / O device (e.g., I / O device 310) coupled to the logic circuit system and coupled to the non-persistent memory device via the interface. The I / O device may be configured to receive signaling from a host including a command to access data from an address corresponding to the non-persistent memory device, and determine whether the received address corresponds to an address in the logic circuit system or a physical address in the non-persistent memory device based at least in part on a mapping table (e.g., mapping table 314) stored in the I / O device. The I / O device may be further configured to redirect the command to the logic circuit system or the non-persistent memory device or both based on the determination. For example, the I / O device may be configured to redirect the command to the logic circuit system in response to the received address corresponding to an address in the logic circuit system. For example, the I / O device may be further configured to redirect the command to the non-persistent memory device in response to the received address corresponding to a physical address in the non-persistent memory device.

[0089] In some embodiments, in response to the command being redirected from the I / O device to the logic circuitry, the logic circuitry may be configured to retrieve data from the persistent memory device via the interface and transfer the retrieved data to the I / O device. In some embodiments, the logic circuitry may be configured to receive the command redirected from the I / O device. In response to the command being redirected from the I / O device to the logic circuitry, the logic circuitry may be configured to transfer the command to a hypervisor (e.g., hypervisor 312) communicatively coupled to the logic circuitry via the interface. The logic circuitry may further be configured to communicatively assert an interrupt signal to the hypervisor as part of the transferred command. The interface may include a PCIe interface.

[0090] In some embodiments, the logic circuitry may be configured to receive signaling including a command to write data to an address corresponding to a non-persistent memory device. In response to the command received from the host, the hypervisor may be configured to cause the I / O device to map an address associated with the command to write the data to a physical address in the non-persistent memory device or an address in the logic circuitry based at least in part on one or more characteristics of the data. As described herein, the characteristics of the data may include the frequency with which the data is requested or accessed, the amount of time that has passed since the data was last accessed or requested, the type of data (e.g., whether the data corresponds to a particular file type, such as a photo, document, audio file, application file, etc.), and the like.

[0091] In some embodiments, the logic circuitry may be configured to associate information indicating that the data is not accessible by the non-persistent memory device with the data in response to the received address corresponding to an address in the logic circuitry. The logic circuitry may include a buffer configured to store data to be written to the persistent memory device or transferred to the I / O device.

[0092] In some embodiments, the non-persistent memory device may include dynamic random access memory (DRAM) cells, and the persistent memory device may include a resistive memory cell array, a phase change memory device, a selectable memory cell array, or a combination thereof.

[0093] Figure 4 is a flow chart 440 showing a data read operation according to various embodiments of the present disclosure. At block 441, an I / O device, such as Figure 2 and 3 The I / O device 210 / 310 shown in FIG. 1 may use an address corresponding to a data request to initiate a read operation. In some embodiments, the address may be a physical address, such as a virtual machine physical address. The data request may include a request to read data and a corresponding address (e.g., a physical address), which may correspond to a persistent memory device (e.g., the physical address of the present invention). Figure 2 and 3 216 / 316) shown in the persistent memory device 216 / 316) or a non-persistent memory device (e.g., Figure 2 and 3 As described herein, information regarding whether a physical address corresponds to a location in a persistent memory device or a location in a non-persistent memory device may be stored in a plurality of mapping tables (e.g., Figure 2 and 3 In the mapping table 214 / 314 shown in .

[0094] By utilizing the mapping table, the I / O device can determine whether the requested data is stored in the non-persistent memory device or in the persistent memory device. If the data is stored in the non-persistent memory device, the data can be retrieved and the data request can be satisfied. However, if the data is stored in the persistent memory device (e.g., if the physical address of the data corresponds to a location in the persistent memory device), then at block 442, the I / O device can redirect the data request to the logic circuitry (e.g., the logic circuitry described herein). Figure 1-3 As described above, data requests may be based on a hypervisor (e.g., the logic circuit system 104 / 204 / 304 shown in FIG. 1 ). Figure 3 312) shown in FIG.

[0095] At block 443, the logic circuit system may receive address register access information corresponding to the data request. In some embodiments, the address register access information may correspond to an address register (e.g., Figure 1-3 For example, the address register access information may correspond to a location in an address register in a logic circuit system, where the logical address corresponds to a physical address in a persistent memory device storing data.

[0096] At block 444, the logic circuitry may generate a hypervisor interrupt. For example, as described above in conjunction with Figure 3 As described above, once the logic circuitry has received a redirected data request from an I / O device, the logic circuitry may generate an interrupt and in a hypervisor (e.g., Figure 3 In some embodiments, an interrupt may be asserted on the hypervisor to notify the hypervisor that an event requires immediate attention. For example, an interrupt signal may be asserted on the hypervisor to cause the hypervisor to interrupt the currently executing instruction but execute instructions associated with collecting address register access information at block 445.

[0097] At block 445, the hypervisor may collect address register access information from the logical circuit system. For example, the hypervisor may receive logical address information corresponding to the physical address of the requested data from the logical circuit system. The logical address information may be stored in an address register (e.g., a base address register) of the logical circuit system, such as the base address register of the present invention. Figure 1-3 The address registers 106 / 206 / 306 are shown in FIG.

[0098] The hypervisor may determine the physical location of the requested data at block 446. For example, based on the address register access information and thus the logical address associated with the data collected from the logic circuitry, the hypervisor may determine the physical location of the data stored in the persistent memory device.

[0099] At block 447, the hypervisor may read data corresponding to the address register access information. That is, in some embodiments, the hypervisor may cause the requested data to be read (eg, retrieved) from the persistent memory device.

[0100] At block 448, the hypervisor may cause the data to be transferred to the non-persistent memory device. In some embodiments, the non-persistent memory device may be the Figure 2 and 3 The non-persistent memory device 230 / 330 shown in FIG. 1 is a non-persistent memory device 230 / 330, however, the embodiments are not limited thereto, and in some embodiments, the hypervisor may cause the data to be transferred to an intermediate memory component, such as the one described herein. Figure 2 and 3 The intermediate storage component 220 / 320 shown in FIG.

[0101] The hypervisor may write I / O device data corresponding to the request data to the logic circuitry at block 449. As described above, the I / O device data may be stored in address registers of the logic circuitry.

[0102] The logic circuitry may complete the data read transaction at block 450. For example, the logic circuitry may transfer a command to the I / O device to notify the I / O device that the data read request has been satisfied and that data will be transferred via a deterministic interface to satisfy the data read request.

[0103] At block 451, the hypervisor may update the I / O device (e.g., Figure 2 and 3 4 (shown in the mapping table 214 / 314) to direct the I / O device address to the non-persistent memory device. For example, as data is transferred from the persistent memory device to the non-persistent memory device (e.g., the non-persistent memory device and / or the intermediate memory component) at block 450, the hypervisor may update the mapping table of the I / O device so that the address corresponding to the requested data is mapped to the non-persistent memory device. In some embodiments, the address may be a physical address, such as a virtual machine physical address.

[0104] At block 452, the hypervisor may record which memory is used to satisfy the data request. For example, the hypervisor may record that the data was already stored in the persistent memory device when the data request was received from the I / O device. In some embodiments, the hypervisor may use the information over time to selectively direct data writes to the persistent memory device or the non-persistent memory device.

[0105] Figure 5 5 is a flowchart 560 showing a data write operation according to various embodiments of the present disclosure. In block 561, an I / O device, such as Figure 2 and 3 The I / O device 210 / 310 shown in FIG. 1 may use an address received along with a data write request to initiate a write operation. The address may be a physical address, such as a virtual machine physical address. The received address (e.g., physical address) may correspond to a persistent memory device (e.g., Figure 2 and 3 216 / 316) shown in the persistent memory device 216 / 316) or a non-persistent memory device (e.g., Figure 2 and 3 As described herein, information regarding whether a received address corresponds to a location in a persistent memory device or a location in a non-persistent memory device may be stored in a plurality of mapping tables of the I / O device (e.g., Figure 2 and 3 In the mapping table 214 / 314 shown in .

[0106] If the data should be stored in the non-persistent memory device, the data may be written to the non-persistent memory device and the data write request may be satisfied. However, if the data should be stored in the persistent memory device, at block 562 the I / O device may redirect the data write request to the logic circuitry (e.g., the I / O device of the present invention). Figure 1-3 As described above, data requests may be based on a hypervisor (e.g., the logic circuit system 104 / 204 / 304 shown in FIG. 1 ). Figure 3 The hypervisor 312 shown in FIG. 10 may be redirected using information provided by the hypervisor 312 (eg, a command or instruction executed by it) which may be used to determine whether the received address should be mapped to a location in a non-persistent memory device or a location in a persistent memory device.

[0107] At block 563, the logic circuit system may receive address register access information corresponding to the data write request. In some embodiments, the address register access information may correspond to an address register (e.g., Figure 1-3For example, the address register access information may correspond to a location in an address register in the logic circuit system, where the logical address corresponds to a physical address in the persistent memory device where the data should be stored.

[0108] At block 564, the logic circuitry may generate a hypervisor interrupt. For example, as described above in conjunction with Figure 3 As described above, once the logic circuitry has received a redirected data write request from an I / O device, the logic circuitry may generate an interrupt and in a hypervisor (e.g., Figure 3 An interrupt is asserted on the hypervisor 312 shown in FIG.

[0109] The hypervisor may collect address register access information from the logical circuitry at block 565. For example, the hypervisor may receive logical address information from the logical circuitry corresponding to a physical address where data should be stored.

[0110] At block 566, the hypervisor may optionally write (or cause the data to be written) the data to the persistent memory device. For example, based on the redirected data write request, the hypervisor may determine that the data should be written to the persistent memory device and cause the data to be written to the persistent memory device. In embodiments where block 566 is optionally performed, the data may be intermediately written to a non-persistent memory device. Additionally, I / O device data corresponding to the data may optionally be written to the non-persistent memory device as part of writing the data to the non-persistent memory device.

[0111] Optionally, the hypervisor may write (or cause the data to be written) the data to the non-persistent memory device at block 567. In some embodiments, the hypervisor may write the data to the non-persistent memory device such that the data may be retrieved via a deterministic interface or bus upon receiving a read request corresponding to the data.

[0112] At block 568, the hypervisor may update the I / O device (e.g., Figure 2 and 3 214 / 314 shown in the mapping table 214 / 314) to direct the I / O device virtual address to the non-persistent memory device. For example, if data is written to the non-persistent memory device in box 567, the hypervisor can update the mapping table of the I / O device in box 568 so that the virtual address associated with the data written to the non-persistent memory device is mapped to the corresponding physical address in the non-persistent memory device. Therefore, when the I / O device subsequently receives a request to access the same data previously stored in the non-persistent memory device, the I / O device can determine that the data is stored in the non-persistent memory device based on the information stored in the mapping table, and redirect the request to access the data to the non-persistent memory device.

[0113] Figure 6 6 is a flowchart showing an exemplary method 670 for a hierarchical memory system according to various embodiments of the present disclosure. At block 672, the method 670 may include receiving a message including writing to a non-persistent memory device (e.g., Figure 2 and 3 At block 674, method 670 may include determining whether to write the data to a non-persistent memory device or to a persistent memory device (e.g., the non-persistent memory device 230 / 330 shown in FIG. 1 ). Figure 2 and 3 ).

[0114] At block 676, method 670 may include, in response to the data being determined to be written to the persistent memory device, mapping the received address corresponding to the non-persistent memory device to a logic circuit system (eg, Figure 2 and 3 204 / 304) to update a mapping table stored in an input / output (I / O) device (e.g., Figure 2 and 3 At block 678, method 670 may include writing data to the logic circuitry based at least in part on mapping addresses corresponding to the non-persistent memory device to addresses in the logic circuitry.

[0115] In some embodiments, method 670 may further include determining (based at least in part on characteristics of the data) that the data should be written to a non-persistent memory device and updating a mapping table stored in the I / O device by mapping the received address to a physical address in the non-persistent memory device. In response to mapping the received address to a physical address in the non-persistent memory device, method 670 may further include writing the data to the non-persistent memory device. In this example, the data may be written to the non-persistent memory device via an interface to which the non-persistent memory device and the I / O device are coupled (e.g., the interface described herein). Figure 2 and 3 interface 208 / 308 shown in FIG. 20A ) is transferred to a non-persistent memory device.

[0116] Although specific embodiments have been shown and described herein, it will be appreciated by those skilled in the art that arrangements calculated to achieve the same results may replace the specific embodiments shown. The present disclosure is intended to cover modifications or variations of one or more embodiments of the present disclosure. It should be understood that the above description is provided in an illustrative manner and not in a limiting manner. Upon reviewing the above description, the combination of the above embodiments and other embodiments not specifically described herein will be apparent to those skilled in the art. The scope of one or more embodiments of the present disclosure includes other applications in which the above structures and processes are used. Therefore, the scope of one or more embodiments of the present disclosure should be determined with reference to the attached claims together with the full range of equivalents given by such claims.

[0117] In the foregoing detailed description, some features are grouped together in a single embodiment for the purpose of simplifying the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the disclosed embodiments of the disclosure must use more features than are explicitly recited in each claim. In fact, as reflected in the following claims, the subject matter of the present invention lies in less than all of the features of a single disclosed embodiment. Therefore, the following claims are hereby incorporated into the detailed description, with each claim standing on its own as a separate embodiment.

Claims

1. A device for a hierarchical memory system, comprising: An input / output (I / O) device (210; 310) coupled to the logic circuitry (104; 204; 304) and a non-persistent memory device (230; 330), wherein the I / O device is configured to: receiving signaling including a command to write data to an address corresponding to the non-persistent memory device; mapping the received address to an address in the logic circuitry or a physical address in the non-persistent memory device based at least in part on one or more characteristics of the data; and In response to the received address being mapped to the address in the logic circuit system, the command to write the data is redirected to the logic circuit system to assert an interrupt signal to a hypervisor (312) as part of the redirected command to write the data to the persistent memory device (216; 316).

2. An apparatus according to claim 1, wherein the I / O device is configured to write the data to a location of the non-persistent memory device corresponding to the physical address based at least in part on the received address being mapped to the physical address in the non-persistent memory device.

3. An apparatus according to any one of claims 1 to 2, wherein the I / O device includes a mapping table (214-1, ..., 214-N; 314-1, ..., 314-N), and the I / O device is configured to store the received address and the corresponding address to which the received address is mapped in the mapping table.

4. The apparatus of claim 3, wherein the I / O device is configured to: receiving signaling including a command to access said data previously stored in said persistent memory device; determining, based on the mapping table, that an address associated with the data to be accessed is mapped to the address in the logic circuitry; and The command to access the data is redirected to the logic circuitry.

5. A device for a hierarchical memory system, comprising: An input / output (I / O) device (210; 310) coupled to the logic circuitry (104; 204； 304), the logic circuit system is coupled to a persistent memory device (216; 316) and a non-persistent memory device (230; 330), the I / O device comprises: a first address that is mapped to a corresponding physical address in the non-persistent memory device; and a second address mapped to a corresponding address in the logic circuitry; Wherein the I / O device is configured to: receiving signaling including a command to access data from an address corresponding to the non-persistent memory device; determining that the received address is one of the second addresses; and At least in part based on the received address being determined to be one of the second addresses, redirecting the command to access the data to the logic circuit system so that the logic circuit system asserts an interrupt signal to a hypervisor (312) as part of the redirected command, thereby causing the logic circuit system to access the data from the persistent memory device. The apparatus of claim 5 , wherein the first address and the second address are respective virtual machine physical addresses.

7. The apparatus of any one of claims 5 to 6, wherein the I / O device is configured to: determining that the received address is one of the first addresses; redirecting the command to the non-persistent memory device; and In response to redirecting the command to the non-persistent memory device, the data is received from the non-persistent memory device.

8. A method for a hierarchical memory system, comprising: Receiving includes writing data to a non-persistent memory device (230; 330) signaling of a command corresponding to the address; determining whether to write the data to the non-persistent memory device or to a persistent memory device based at least in part on one or more characteristics of the data (216; 316); In response to the data being determined to be written to the persistent memory device, updating a mapping table (214-1, ..., 214-N; 314-1, ..., 314-N) stored in an input / output (I / O) device (210; 310) by mapping a received address corresponding to the non-persistent memory device to an address in the logic circuitry; and Based at least in part on mapping a received address corresponding to the non-persistent memory device to the address in the logic circuitry, redirecting the command to write the data to the logic circuitry so that the logic circuitry asserts an interrupt signal as part of the redirected command to write the data to the persistent memory device.

9. The method according to claim 8, further comprising: determining, based at least in part on one or more characteristics of the data, that the data should be written to the non-persistent memory device; updating the mapping table stored in the I / O device by mapping the received address to a physical address in the non-persistent memory device; and In response to mapping the received address to the physical address in the non-persistent memory device, the data is written to the non-persistent memory device.

10. The method of claim 9, wherein writing the data to the logic circuitry comprises transferring the data to the logic circuitry via an interface (208; 308) to which the non-persistent memory device and the I / O device are coupled.

11. A system for hierarchical memory operations, comprising: Logic circuit system (104; 204; 304)； a non-persistent memory device coupled to the logic circuitry and configured to operate according to a first interface protocol; and an input / output (I / O) device (210; 310) coupled to the logic circuitry and to the non-persistent memory device, the non-persistent memory device being configured to operate according to a second interface protocol; Wherein the I / O device is configured to: receiving signaling from a host including a command to access data from an address corresponding to the non-persistent memory device; determining, based at least in part on a mapping table (214-1, ..., 214-N; 314-1, ..., 314-N) stored in the I / O device, whether the received address corresponds to an address in the logical circuitry or to a physical address in the non-persistent memory device; and Based on the determination, redirecting the command to the logic circuitry or the non-persistent memory device, or both; and wherein the logic circuitry is configured to: transferring the command to a hypervisor, the hypervisor being communicatively coupled to the logic circuitry; as well as asserting an interrupt signal to said hypervisor as part of said command to transfer; Wherein the hypervisor is configured to transfer the data from the persistent memory device to the non-persistent memory device in response to the interrupt signal so that the data is accessed by the I / O device from the non-persistent memory device and via the second interface protocol, regardless of whether the data is stored in the persistent memory device or the non-persistent memory device.

12. The system of claim 11, wherein the I / O device is configured to: responsive to the received address corresponding to the address in the logic circuitry, redirecting the command to the logic circuitry; and In response to the received address corresponding to the physical address in the non-persistent memory device, the command is redirected to the non-persistent memory device.

13. The system according to any one of claims 11 to 12, wherein The logic circuitry is configured to receive signaling including a command to write data to an address corresponding to the non-persistent memory device; and The hypervisor is configured to, in response to the command received from the host: Based at least in part on one or more characteristics of the data, cause the I / O device to map the address associated with the command to write the data to a physical address in the non-persistent memory device or an address in the logical circuitry.

14. The system of any one of claims 11 to 12, wherein the logic circuitry is configured to In response to the received address corresponding to the address in the logic circuitry, information indicating that the data is not accessible by the non-persistent memory device is associated with the data.

15. The system of any of claims 11-12, wherein the logic circuitry comprises a buffer (105) configured to store the data to be written to the persistent memory device or transferred to the I / O device.

16. The system of any one of claims 11-12, wherein the non-persistent memory device comprises a dynamic random access memory (DRAM) cell.

17. The system of any one of claims 11-12, wherein the persistent memory device comprises an array of resistive memory cells, a phase change memory device, an array of self-selected memory cells, or a combination thereof.

18. The system of any of claims 11-12, wherein the first interface protocol comprises a Peripheral Component Interconnect Express (PCIe) interface protocol.

Images