Electrically buffered NV-DIMM and method of use therewith

Abstract

Storage system (1500, 2300), comprising: a plurality of non-volatile memory devices (1540, 2340); a controller (1530, 2330) in communication with the plurality of non-volatile memory devices (1540, 2340), the controller (1530, 2330) being configured to: Receiving a read command from a host (100); in response to receiving the read command from the host (100), reading data from the plurality of non-volatile memory devices (1540, 2340); Performing an operation with an indefinite duration from the host's perspective; Sending a ready signal to the host (100) after performing the operation; Receiving a send command from the host (100); and in response to receiving the send command from the host (100), sending the data to the host (100); a plurality of data buffers (1550, 2100, 2200, 2350) in communication with the controller (1530, 2330) and configured to store data sent between the controller (1530, 2330) and the host (100); and a command and address buffer (1520, 2320) configured to store commands and addresses sent from the host (100), wherein the command and address buffer (1520, 2320) is further configured to synchronize the flow of data into and out of the plurality of data buffers (1550, 2100, 2200, 2350).

Description

background

[0001] Many computer systems use one or more dual in-line memory modules (DIMMs) connected to a central processing unit (CPU) to store data. Some DIMMs contain dynamic random-access memory (DRAM) chips. However, DRAM is relatively expensive, requires a relatively large amount of power, and fails to scale capacity with rate-adapting processing power, which can be undesirable for use in servers, such as enterprise and hyperscale systems in data centers where large amounts of data are stored. To address these problems, non-volatile DIMMs (NV-DIMMs) have been developed to replace volatile DRAM chips with non-volatile memory devices.Compared to DRAM-based DIMMs, NV-DIMMs can provide lower cost per gigabyte, lower power consumption, and longer data retention, especially in the event of a power failure or system crash. Like some DRAM-based DIMMs, some NV-DIMMs are designed to communicate over a clock data parallel interface, such as a double data rate (DDR) interface.

[0002] EP 3 449 378 B1 relates to devices and methods in connection with a non-deterministic memory protocol, wherein operations on a memory device can be performed based on commands received from a host according to a protocol comprising non-deterministic timing of the operations. DE 11 2017 003 334 T5 relates to a storage circuit having a buffer coupled between a memory controller and a non-volatile memory device. US 2014 / 0 025 876 ​​A1 relates to a device having a memory channel control circuit and associated methods for forwarding commands to logical units. US 7 802 061 A1 relates to a command-based control of NAND flash memories. US 2014 / 0 250 260 A1 relates to an asynchronous FIFO buffer for memory accesses.The document US 2014 / 0 122 822 A1 relates to devices and methods for memory operations having variable latencies. Short description of the drawings Fig. 1 is a block diagram of a host and storage systems of one embodiment. Fig. 2A is a block diagram of a memory system of one embodiment in which the memory system takes the form of a non-volatile dual in-line memory module (NV-DIMM). Fig. 2B is a block diagram of a memory system of one embodiment with a distributed controller. Fig. 3 is a block diagram showing signals between a host and storage systems of one embodiment. Fig. 4 is a flowchart of a method for reading data from a DRAM DIMM. Fig. 5 is a timing diagram of a method for reading data from a DRAM DIMM. Fig. 6 is a flowchart of a method of one embodiment for a host to send a read command. Fig. 7 is a flowchart of a method of one embodiment for a host to request a return of read data using a send command and to process received data. Fig. 8A and Fig. 8B are timing diagrams of a non-deterministic method for reading data from a memory system of one embodiment. Fig. 8C is a timing diagram of a non-deterministic method for writing data to a memory system of one embodiment. Fig. 9 is a block diagram of a controller of a memory system of one embodiment. Fig. 10 is a flowchart of a method for reading data from a memory system of one embodiment. Fig. 11 is a flowchart of a method for writing data to a memory system of one embodiment. Fig. 12 and Fig. 13 are diagrams showing read and write flows of a DRAM-based DIMM, respectively. Fig. Figure 14 is an illustration of the internal states of data flow in a DRAM-based DIMM. Fig. 15 is a block diagram of a memory system of one embodiment in which the memory system takes the form of a non-volatile dual in-line memory module (NV-DIMM). Fig. 16 is a block diagram illustrating a read operation of a memory system of one embodiment. Fig. 17 is a block diagram illustrating a write operation of a memory system of one embodiment. Fig. 18A and Fig. 18B are flow diagrams of a read operation of one embodiment. Fig. 19A and Fig. 19B are flow diagrams of a write operation of one embodiment. Fig. 20 is a diagram showing a change in clock speed of an embodiment. Fig. Figure 21 is a block diagram of a data buffer. Fig. 22 is a block diagram of a data buffer of one embodiment. Fig. 23A is a block diagram of a memory system of one embodiment in which non-volatile memory devices are connected to data buffers without passing through an NVM controller. Fig. 23B is a block diagram of a registered clock driver (RCD) of one embodiment. Detailed description

[0003] According to the invention, storage systems having the features of the independent claims are provided; dependent claims relate to preferred embodiments. overview

[0004] By way of introduction, the following embodiments relate to an electrically buffered NV-DIMM and a method of use therewith. In one embodiment, a memory system is provided comprising a plurality of non-volatile memory devices; a controller in communication with the plurality of non-volatile memory devices; a plurality of data buffers in communication with the controller and configured to store data sent between the controller and an input / output bus; and a command and address buffer configured to store commands and addresses sent by a host, wherein the command and address buffer is further configured to synchronize the flow of data into and out of the plurality of data buffers.

[0005] In some embodiments, the controller is configured to associate read and / or write commands with identifiers so that the read and / or write commands can be processed in a different order than the order in which they are received from the host.

[0006] In some embodiments, the command and address buffer includes a registered clock driver.

[0007] In some embodiments, the plurality of data buffers comprise random access memories.

[0008] In some embodiments, the command and address buffer is further configured to reduce a frequency of a clock received from the host.

[0009] In some embodiments, the command and address buffer is further configured to perform bandwidth conversion.

[0010] In some embodiments, physical layers and command layers of the memory system are designed to be compatible with a DRAM DIMM communication protocol.

[0011] In some embodiments, physical layers and command layers of the memory system are configured to be compatible with one or more of the following: unbuffered DIMM (UDnVIM), registered DIMM (RDIMM), and load reduced DIMM (LRDIMM).

[0012] In some embodiments, the controller is further configured to: send a ready signal after data requested by the host is ready to be sent to the host; receive a send command from the host; and in response to receiving the send command from the host, send the data to the host.

[0013] In some embodiments, the data is sent to the host after a time delay, where the time delay is selected based on a communication protocol used with the host.

[0014] In some embodiments, the controller is configured to communicate with the host using a clock data parallel interface.

[0015] In some embodiments, the clock data parallel interface comprises a double data rate (DDR) interface.

[0016] In some embodiments, at least one of the plurality of non-volatile memory devices comprises a three-dimensional memory.

[0017] Other embodiments are possible, and each of the embodiments may be used alone or together in combination. General introduction to an implementation of an embodiment

[0018] As explained in the background section above, dual in-line memory modules (DIMMs) can be connected to a host's central processing unit (CPU) to store data. Non-volatile dual in-line memory modules (NV-DIMMs) were designed to replace volatile DRAM chips on standard DIMMs with non-volatile memory devices, such as NAND. Compared to DRAM-based DIMMs, NV-DIMMs can provide lower cost per gigabyte, lower power consumption, and longer data retention, especially in the event of a power failure or system crash. Like some DRAM-based DIMMs, some NV-DIMMs are designed to communicate over a clock data parallel interface, such as a double data rate (DDR) interface.

[0019] However, existing standards suitable for DRAM-based DIMMs may not be suitable for NVDIMMs. For example, some existing standards require that read and write operations complete within a specified ("deterministic") time period. While completing read and write operations within the specified time period is typically not a problem for DRAM memory, the mechanism of reading and writing to non-volatile memory can result in delays that exceed the specified time period. This means that DRAM-based DIMM protocols expect consistent, predictable, and fast responses, which non-volatile memory may not be able to provide. To address this, some emerging standards (e.g., JEDEC's NVDIMM-P standard) allow "non-deterministic" read and write operations to introduce "slack" into the communication between the storage system and the host.According to such standards, read and write operations to the NV-DIMM do not have to be completed within a specific time period. In the case of a read operation, the NV-DIMM instead informs the host when the requested data is ready, so the host can then retrieve it. In the case of a write operation, the host can be restricted from having more than a certain number of outstanding write commands, ensuring that the non-volatile memory device does not receive more write commands than it can handle.

[0020] The approach of allowing non-deterministically timed operations at a protocol level is only one possible approach for dealing with the unpredictable nature of non-volatile memory. Other approaches do not utilize non-deterministic modifications of the DDR standard. Instead, they rely on software approaches to construct composite read and write procedures from conventional DDR primitives. Each DDR primitive can either correspond to a direct access to the non-volatile memory itself or it can correspond to indirect operations performed through the use of intermediary elements such as control registers or buffers.Although the read or write algorithms themselves may require an unspecified number of iterations or DRR instructions to complete—and thus may not complete within a certain timeframe—each individual primitive DDR operation completes within the well-defined time bounds imposed by the usual (deterministically timed) DDR standards.

[0021] Some of the following embodiments benefit from the non-deterministic aspect of the emerging standard to enable the NV-DIMM to perform time-consuming actions that it may not have time to perform under conventional DRAM-based DIMM standards. These actions are sometimes referred to herein as indefinite-duration operations from the host's perspective and may include memory and data management operations. These memory and data management operations may be important to the operation of the NV-DIMM. For example, compared to DRAM, a non-volatile memory device may have lower endurance (i.e., number of writes before an error) and may store data less reliably (e.g., due to internal memory errors that cause bits to be stored incorrectly).These problems may be even more pronounced with emerging non-volatile memory technologies, which would likely be used as DRAM replacements in an NV-DIMM. As such, in one embodiment, the NV-DIMM takes advantage of not being "pressured" to perform operations with an indefinite duration from the host's perspective, such as memory and data management operations (e.g., wear leveling and error correction operations), which it may not be able to perform in the allotted time according to conventional DRAM-based DIMM standards.

[0022] It should be noted that this introduction discusses only one particular implementation of an embodiment, and that other implementations and embodiments may be used, as discussed in the following paragraphs. Furthermore, although some of these embodiments are discussed with respect to an NV-DIMM connected to a host CPU, it should be understood that any type of memory system may be used in any suitable type of environment. Accordingly, specific architectures and protocols discussed herein should not be read into the claims unless expressly stated therein. General discussion of clock data parallel interfaces and new protocols

[0023] Clock data parallel interfaces provide a simple way to transfer digitized data and commands between any two devices. Each transmission line carrying data or commands from one device to another is accompanied by a separate "clock" transmission line, which provides a timing reference for sample changes in the data and command buses. In some embodiments, the clock can be disabled when the interface is idle and not transmitting data or commands. This provides a convenient way to reduce power dissipation during inactivity.In some embodiments of clock data parallel interfaces, the clock is a single-ended transmission line, meaning that the clock consists of an additional transmission line whose voltage is compared to a common voltage reference shared by many transmission lines between the CPU and the memory devices. In other embodiments, the timing reference may be a differential clock with both a positive clock reference and a clock complement that switches to a low voltage simultaneously with each low-to-high voltage transition of the positive clock—an event known as the "rising edge" of the clock—and conversely, the clock complement switches to the high voltage state with each high-to-low voltage transition of the positive clock reference—an event known as the "falling edge" of the clock.Clock data parallel interfaces are often classified by how many data beats are sent along with the clock. In single-data rate, or SDR, interfaces, the command or data buses toggle once per clock cycle, often on the rising edge of the reference clock. In double-data rate, or DDR, interfaces, the command and data buses send twice as much data per clock period by allowing the command and data buses to toggle twice per period: once on the rising edge of the clock and once on the falling edge of the clock. Quad-data rate (QDR) protocols also exist, allowing four data or command transitions per clock. Typically, clock data parallel interfaces are efficient and low-latency due to their simplicity, and the receiver circuitry can be as simple as a single bank of logic flip-flops.However, additional complexity may exist, caused by the need to synchronize the newly latched data with the internal clock of the devices themselves, one of the many jobs handled by a collection of signal conditioning circuits known as the "physical communication layer" or simply "phy layer."

[0024] Serial interfaces, on the other hand, typically rely on clock data recovery processes to extract the timing reference from a single electrical transmission line that switches voltage at regular time intervals, but in a pattern that also communicates commands and / or data (in some embodiments, many different lines are executed in parallel for increased bandwidth, and thus each line can encode data for an entire command and data sequence, or only a portion of a command or data sequence). Encoding the clock and data on the same physical transmission line reduces timing uncertainties caused by mismatched delays between the clock and data or command lines, thus enabling clock frequencies of 25 GHz or higher for very high-bandwidth communication.However, such interfaces also have some disadvantages. Due to the nature of clock data recovery, the transmission line must remain continuously active to maintain synchronization of the inferred clock reference between the communicating partners. Power-saving modes are possible, but returning to active mode requires significant retraining delays. Furthermore, this type of clock data recovery requires slightly more time to decode each message, and one-way communication delays are common even on a well-trained serial link. This adds additional latency to each data request.

[0025] The interface between computer CPUs and their corresponding memory devices is an example of an interface where optimization of both performance and latency is desired. Thus, while high-bandwidth serial CPU memory interfaces exist, such as Hybrid Memory Cube, the majority of today's interfaces between CPUs and memory devices still use clock-data parallel interfaces. For example, synchronous dynamic random-access memory (SDRAM) uses a single clock to synchronize instructions on an instruction bus consisting of a plurality of transmission lines, each encoding one bit of instruction sequence information.Depending on the embodiment, instructions in an SDRAM instruction sequence may include, but are not limited to: enabling a row of cells in a two-dimensional data array for future reading or writing; reading some columns in a currently active row; writing some columns in a currently active row; selecting a different bank of cells for reading or writing; writing some bits to the memory mode registers to change aspects of the memory device's behavior; and reading back values ​​from the mode registers to identify the memory device's status.

[0026] Data associated with these commands is sent or received along a separate data bus consisting of a separate and parallel plurality of data transmission lines referred to as the DQ bus. In some embodiments, the DQ bus may be half-duplex and bidirectional, meaning the same lines are used for receiving and transmitting data, and data cannot be sent from the memory device to the CPU simultaneously while data is flowing in the opposite direction, or vice versa. In other embodiments, the DQ bus may be full-duplex, with separate lines for receiving or transmitting data. It can be safely assumed that the data on the DQ bus is synchronized with the device command clock. However, for longer transmission lines or faster operating frequencies, this may result in poor synchronization.Thus, other embodiments exist in which the entire DQ bus is divided into a plurality of smaller DQ groups, each with its own "DQ strobe" signal, DQS, which serves as a separate timing frequency for the wires in that DQ group. For example, in one embodiment, a 64-bit DQ bus may be divided into 8 groups (or "byte lanes") of 8 DQ lines each, each synchronized by its own DQS strobe. The DQS strobes may be differential or single-ended, depending on the embodiment. In some embodiments, some DQ lines may provide encoding not only for data stored by the host, but also additional parity or other signaling data for the purpose of recording additional error correction codes.Depending on the implementation, many DDR protocols include a range of other control signal transmission lines driven by the CPU to the memory device, which in some embodiments may command, for example, but is not limited to, the following functions: Command Suppression lines (CS_N), Clock Enable (CKE), or On-Die Termination (ODT) activation.

[0027] An electronic system may consist of one or more data processing elements—where the processing may include computing, analyzing, storing data, or transmitting data over a network or peripheral bus—connected to a plurality of memory devices. Examples of data processing elements include, but are not limited to, CPUs, CPU caches, application-specific integrated circuits, peripheral buses, direct memory access (DMA) engines, or network interface devices. In the many DRAM configurations, a plurality of memory circuits are bundled together into modules; for example, modules described by the dual inline memory module (DIMM) standard.Within a module, some devices may transfer data in parallel along separate DQ groups, while others may all be connected in parallel to the same transmission lines within a DQ group. To reiterate, in many typical DRAM configurations, a plurality of modules may then be connected in parallel to form a channel. In addition to the memory modules, each channel is connected to exactly one data processing element, which is subsequently referred to as the host. Each memory device may be connected to the host via part of a half-duplex DQ bus (as opposed to a full-duplex DQ bus) or may further be connected to the same DQ transmission lines as several other memory devices—either on the same module or on other neighboring modules in the same channel.Therefore, there is a risk that a memory device could choose to assert data on the DQ bus at the same time as other memory devices on the same bus, and thus a need for arbitration on the bus. Therefore, SDRAM protocols rely on a centralized, time-windowed bus arbitration scheme: by default, the host is the only device allowed to transfer data on the DQ bus, and by default, all memory devices leave their DQ lines high-impedance most of the time. When a command requiring a response is sent to a specific memory device, the device is allowed to transfer data on the DQ bus, but only within a specific time window after the first pulse of the command.The window starts a fixed number of clock cycles after the instruction and typically lasts only one or two clock cycles longer than the time required to transfer the data. Storage devices that transfer data outside this window will either fail to successfully send their data to the host or will corrupt data returned from neighboring storage devices.

[0028] The DQ bus arbitration scheme used by these clock-parallel SDRAM protocols works well for DRAM. The technology behind DRAM devices has advanced to the point where their data access times are extremely uniform and predictable. However, DRAM is a relatively power-hungry technology, requiring frequent refreshes thousands of times per second.

[0029] Non-volatile memories, such as phase-change random access memory (PCM), oxidative resistive random access memory (OxRAM or ReRAM), conductive-bridge random access memory (CBRAM), NAND flash (NAND), magnetic tunnel junction-based magnetic random access memory (MRAM), memristor, NOR flash (NOR), spin torque-transfer magnetic memory (STT-MRAM), and ferroelectric random access memory (FeRAM), all promise low-latency data access, can be optimized for lower power consumption for many data-heavy workloads, and may soon provide higher-density random access memory than DRAM. However, they require somewhat looser data access protocols than DRAM. All of these non-volatile memories exhibit non-deterministic read and write latencies.It is impossible to know exactly how long it would take to access or commit data to or from a non-volatile memory cell at the time a read or write command is written, for all NVM options and for all NVM device architectures. However, it is possible to mimic deterministic latencies. Deterministic latencies can be mimicked by assuming worst-case timing conditions or by abandoning a read operation that might take too long. Modifications to DDR SDRAM protocols could be specified based on pessimistic read or write latency specifications.For example, a memory that commits most writes within 100 ns but occasionally takes 10 us to commit data for unpredictable reasons could use a DDR protocol that does not allow writes a full 10 us after the previous write, nor does it allow reads during that time period (since, in some memory technologies, writes mean that reads must also be delayed). However, this would place a drastic limit on the maximum bandwidth achievable by such a device and could further limit the performance of other devices on the same channel. Conversely, one can imagine a modification of the standard DDR or SDR or QDR SDRAM protocols that allows flexibility for non-deterministic read latencies and non-deterministic write latencies.In one embodiment, this protocol is referred to as a synchronous non-volatile RAM (SNVRAM) protocol.

[0030] For example, in some embodiments of SNVRAM protocols, the read command may be split into three smaller commands. Where previously a read command sequence consisted of two parts: an enable command followed by a read operation to specify the row and column of requested data, the command would now consist of an enable command, a read command, and finally—after an indefinite delay—a send command. The enable / read combination would specify the two-part request to read a specific region. However, no response would be sent after the read command; instead, the memory device would assert a signal, denoted, for example, as "READ READY" (sometimes referred to herein as "R_RDY"), back to the host at some unspecified time after the read command.This assertion would then prompt the host to issue the SEND command, allowing other SDRAM activity to transfer the fully extracted data from the storage device back to the host. The response from the SEND command would go out over the shared DQ bus within a predetermined window following the SEND command. In this way, the typical read command would support non-deterministic read latencies; however, performance characteristics, such as the average minimum latency or overall system bandwidth, are not limited by the slowest possible read operation. The average performance of the protocol is consistent with the typical device performance, while still allowing some flexibility for outliers, which are clearly expected as a physical consequence of the media choice.

[0031] In one embodiment, the SNVRAM includes the following features: * Similar to existing SDRAM or DDR protocols, it supports communication between a single host and multiple memory devices on the same memory channel. Hosts can be connected to separate memory channels, although each channel operates independently, and thus the protocol does not specify the behavior of devices on other channels. Transmission lines for operating one channel can be used exclusively by that channel. In other embodiments, the host can be connected to a single memory device, and that memory device can forward the commands and data to a second device in a daisy-chained delivery style. * As with existing SDRAM or DDR protocols, any signal or bus from the host to the channel can be synchronous to a clock signal following a parallel transmission line. * As with existing SDRAM or DDR protocols, there are logical commands such as "Activate Address Block", "Read Element Within Active Address Block", or "Write to Element Within Active Address Block" that can be sent along a command bus. * As with existing SDRAM or DDR protocols, the command bus can be synchronized with a master clock or master command strobe for the channel. * As with existing SDRAM or DDR protocols, data returning from the memory device may be sent along a separate data bus consisting of a plurality of transmission lines called the DQ bus. * As with existing SDRAM or DDR protocols, in some embodiments, each line on the DQ bus may be synchronized with the master clock. In other embodiments, the DQ bus is synchronized with a separate DQ strobe signal (generated either by the host or by the memory device), referred to below as DQS. In some embodiments, multiple DQS lines may be present, each corresponding to a subset of the DQ bus lines. * As with existing SDRAM or DDR protocols, some embodiments exist where the DQ bus can be bidirectional, carrying storable data from the host to the memory device. Other embodiments may include a separate write DQ bus. * As with existing SDRAM or DDR protocols, data can be transferred from the host to the memory device on a DQ bus synchronously with either the master clock or the corresponding DQS lines, depending on the embodiment considered. * As with existing SDRAM or DDR protocols, the DQ buses can be connected to multiple memory devices in addition to the single host. Arbitration on this bus is based on time slots. When a memory device receives a command from the host that requires a response, it has a narrow time window in which it owns the DQ bus and can assert data. * As with existing SDRAM or DDR protocols, memory devices within a channel can be grouped as a plurality to form coordinated modules. * SNVRAM protocols typically differ from SDRAM protocols in that there are additional control lines that send signals from the memory system to the host. (Typical SDRAM interfaces only include control signals sent from the host to the memory system.) These additional control lines are referred to below as the "response bus" (or RSP). The response bus may, in some embodiments, be synchronous with the master clock or, in other embodiments, may have its own strobe signal generated by the memory module. The response bus includes, but is not limited to, signals identified for our purposes herein as "READ READY" (R_RDY) and "CREDIT INCREMENT WRITE" (WC_INC). It should be noted, however, that different embodiments of SNVRAM protocols may include electrical signals with similar functions, although the protocol may refer to them by a different name.Accordingly, it is understood that specific signal names used herein are merely examples. * In some embodiments of NVRAM protocols, the response bus may be shared by all modules in a channel and arbitrated by the host, or in other embodiments, the response bus may consist of different transmission lines - not shared by any modules - which are only forwarded from each module to the host without making electrical contact with other modules. * Just as various embodiments of the SDRAM or DDR protocols transfer data at protocol-specified rates, data on any command bus can be specified for transfer at SDR, DDR, or QDR rates by the particular protocol embodiment. * Data on any command bus, clocks, or strobes may be sent single-ended or differentially depending on the specifications contained in the SNVRAM protocol implementation. * SNVRAM protocols provide a simple way to accommodate the erratic behavior of non-deterministic, non-volatile media without unnecessarily limiting their bandwidth. However, there are many other possibilities that can be realized through such protocols. In addition to compensating for the non-deterministic behavior of memory, these protocols can also be used to provide time for various maintenance tasks and data quality improvements, such as error correction, I / O scheduling, memory wear leveling, in-situ media characterization, and logging of controller-specific events and functions. As the hardware implementing these functions becomes more complex, contention for hardware resources performing these functions becomes another potential source of delay.Any such delays can cause significant performance or reliability issues when using a standard SDRAM communication protocol. However, using a non-deterministically timed SNVRAM protocol allows for flexible operation and freedom from hardware complexity. Furthermore, non-deterministic read timing allows for the possibility of occasionally faster read responses through caching. Discussion of the drawings

[0032] If we now turn to the drawings, Fig. 1 is a block diagram of a host 100 in communication with storage systems of one embodiment. As used herein, the term "in communication with" could mean directly in communication with or indirectly in communication with via one or more components that may or may not be described or shown herein. In this illustration, two storage systems are shown (Storage System A and Storage System B); however, it is understood that more than two storage systems may be used, or only one storage system may be used. In this embodiment, the host 100 includes one or more central processing units (CPUs) 110 and a storage controller 120. In this illustration, there are two CPUs (CPU A and CPU B); however, it is understood that more than two CPUs may be used, or only a single CPU may be used.The storage controller may also be connected to devices other than CPUs and may be configured to forward storage requests on behalf of other devices, such as, but not limited to, network cards or other storage systems (e.g., hard disks or solid-state drives (SSDs)). Furthermore, the storage controller may forward storage requests on behalf of one or more software applications executing on the CPU that send requests to the storage controller 120 for access to the connected storage systems.

[0033] In this embodiment, host 100 also includes a memory controller 120 in communication with CPUs 110 (although in other embodiments, no memory controller is used), which communicates with the memory systems using a communication interface, such as a clock data parallel interface (e.g., DDR), and operates according to a particular protocol (e.g., one set forth by the Joint Electron Device Engineering Council (JEDEC)). In one embodiment, memory controller 120 correlates access requests to the memory systems from CPUs 110 and sorts responses from the memory systems and delivers them to the appropriate CPUs 110.

[0034] As in Fig. 1, storage system A includes a media (non-volatile memory) controller 130 in communication with a plurality of non-volatile memory devices 140. In this embodiment, storage systems A and B include the same components; thus, storage system A also includes a media (non-volatile memory) controller 150 in communication with a plurality of non-volatile memory devices 160. It should be noted that in other embodiments, the storage systems may include different components.

[0035] The media controller 130 (sometimes referred to as a "non-volatile memory (NVM) controller" or just "controller") may take the form of processing circuitry, a microprocessor or processor, and a computer-readable medium storing computer-readable program code (e.g., firmware) executable by, for example, the (micro)processor, logic gates, switches, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. The controller 130 may be configured with hardware and / or firmware to perform the various functions described below and shown in the flowcharts.

[0036] Generally, controller 130 receives requests for access to the memory system from memory controller 120 in host 100, processes and sends the requests to non-volatile memory 140, and provides responses back to memory controller 120. In one embodiment, controller 130 may take the form of a non-volatile (e.g., flash) memory controller, which may format the non-volatile memory to ensure that the memory operates properly, identify bad non-volatile memory cells, and allocate spare cells to replace future faulty cells. A portion of the spare cells may be used to contain firmware for operating the non-volatile memory controller and implementing other features.During operation, when the host 100 needs to read data from or write data to non-volatile memory, it communicates with the non-volatile memory controller. If the host 100 provides a logical address at which data is to be read / written, the flash memory controller can convert the logical address received from the host 100 into a physical address in the non-volatile memory. (Alternatively, the host 100 can provide the physical address.)) The non-volatile memory controller may also perform various operations with an indefinite duration from the host's perspective, such as, but not limited to, wear leveling (distributing writes to avoid wearing out specific memory blocks that would otherwise be repeatedly written to) and garbage collection (after a block is full, only the valid pages of data are moved to a new block so that the full block can be erased and reused). Further information regarding a specific embodiment of the controller 130 is provided below in connection with . Fig. 6.

[0037] A non-volatile memory device 140 may also take any suitable form. For example, a non-volatile memory device 140 may include a single memory die or multiple memory dies, and may be equipped with or without an internal controller. As used herein, the term "die" refers to a collection of non-volatile memory cells and associated circuitry for managing the physical operation of these non-volatile memory cells formed on a single semiconductor substrate. A non-volatile memory die 104 may comprise any non-volatile storage medium, including NAND flash memory cells, NOR flash memory cells, PCM, RRAM, OxRAM, CBRAM, MRAM, STT-RAM, FeRAM, or any other non-volatile technology.Volatile memory that mimics non-volatility may also be used, such as volatile memory that is battery-backed or otherwise protected by an auxiliary power source. The memory cells may take the form of solid-state (e.g., flash) memory cells and may be one-time programmable, few-time programmable, or many-time programmable. The memory cells may also be single-level cells (SLC), multiple-level cells (MLC), triple-level cells (TLC), or use other memory cell-level technologies now known or later developed. Also, the memory cells may be fabricated in a two-dimensional or three-dimensional manner. Some other memory technologies were discussed above, and further discussion of possible memory technologies that may be used is provided below.Also, different memory technologies may have different algorithms (e.g. existing program and wear leveling) that apply to that technology.

[0038] For simplification, Fig. 1 illustrates a single line connecting the controller 130 and the non-volatile memory device 140; it should be understood that this connection may include a single channel or multiple channels. For example, in some architectures, 2, 4, 8, or more channels may exist between the controller 130 and a memory device 140. Accordingly, in any of the embodiments described herein, more than a single channel may exist between the controller 130 and the memory device 140, even if a single channel is shown in the drawings.

[0039] The host 100 and the storage systems may take any suitable form. For example, in one embodiment (in Fig. 2A) takes the form of a non-volatile dual in-line memory module (NV-DIMM) 200, and the host 100 takes the form of a computer with a motherboard that accepts one or more DIMMs. In Fig. In the NV-DIMM 200 shown in Figure 2A, there are nine non-volatile memory devices 40, and the NV-DIMM 200 has an interface 210 that includes nine data input / output DQ groups (DQ0-DQ8), a command bus, and a response bus. Of course, these are merely examples, and other implementations may be used. For example, Fig. 2B illustrates an alternative embodiment in which the storage system includes a distributed controller 31 and a master controller 212 (which, although not shown, is connected to all distributed controllers 31). Compared to the storage system in Fig. 2A, each NVM device 41 communicates with its own NVM controller 31, rather than all NVM devices communicating with a single NVM controller. In one embodiment, the master controller 212 performs any necessary synchronization activity, including determining when all distributed controllers 31 are read to send the RD_RDY signal, which is discussed in more detail below.

[0040] As mentioned above, multiple memory systems can be used, where signals can be passed through one memory system to reach another. This is Fig. 3 shown. In Fig. 3, memory system A is closer to host 100 than memory system B. Arrow 300 represents shared memory input signals sent from host 100 to the command pin in both the first and second memory systems. Examples of shared memory input signals that may be used include, but are not limited to, an address signal, a read chip select signal, a bank group signal, a command signal, an enable signal, a clock enable signal, a schedule control signal, and a command identifier (ID) signal. Arrow 310 represents a memory channel clock, which may also be sent on the command pin. Arrow 320 represents shared memory output signals that may be sent on the DQ0-DQ8 groups. Examples of shared memory output signals include, but are not limited to, data signals, parity signals, and data strobe signals.Arrow 330 represents dedicated memory input signals to memory system B, and arrow 350 represents dedicated memory input signals to memory system A. Examples of dedicated memory input signals that may be sent on the command pin include, but are not limited to, clock enable signals, data strobe signals, chip select signals, and termination control signals. Arrow 340 represents a device-dedicated response line to memory system B, and arrow 360 represents a device-dedicated response line to memory system A. Examples of signals sent on the device-dedicated response lines that may be sent on the command pin include, but are not limited to, read data ready (R_RDY) signals, a read identifier (ID) signal, and a write flow control signal. These signals are discussed in more detail below.

[0041] One aspect of these embodiments is how the NVM controller 130 in the storage system handles read and write commands. Before discussing this aspect of these embodiments, the flowchart 400 in Fig. 4 to illustrate how a conventional host reads data from a conventional DDR-based DRAM DIMM. Flowchart 400 is used in conjunction with timing diagram 500 in Fig. 5. As discussed in Fig. 4 when the host reads data from the DIMM (called “device” in Fig. 4) (operation 410), the memory controller in the host sends an activation command with the upper address (operation 420). The memory controller in the host then sends a read command with the lower address (operation 430). This is referred to as the "Act." and "Rd" fields on the command / address line in Fig. 5. The memory controller in the host then waits a predetermined period of time (sometimes referred to as the “preamble time”) (operation 440). This is referred to as the “predefined delay” in Fig. 5. After the predetermined (“deterministic”) time period has elapsed, the memory controller in the host accepts the data (with data strobes for fine-grained time synchronization) (operation 450) (fields D1-DN on the data line in Fig. 5), and the data is provided to the host (operation 460).

[0042] As mentioned above, while this interaction between a host and the memory system is adequate when the memory system is a DRAM DIMM, complications can arise when using a deterministic protocol with an NV-DIMM, as the mechanism behind reading and writing to the non-volatile memory can result in delays that exceed the specified time for a read or write operation according to the protocol. To address this, some emerging standards allow for "non-deterministic" read and write operations. According to such standards, read and write operations to the NV-DIMM do not have to be completed within a specific time period.

[0043] In the case of a read operation, the NV-DIMM informs the host 100 when the requested data is ready so that the host can then retrieve it. This is shown in flowcharts 600, 700 in Fig. 6 and Fig. 7 and the timing diagram 800 in Fig. 8A. As shown in Fig. 6, when the host 100 requests data from the memory system (operation 610), the host 100 generates a double data rate identifier (DDR ID) for the request (operation 620). The host 100 then associates the DDR ID with a host request ID (e.g., an ID from the CPU or other entity in the host 100 that requested the data) (operation 630). Next, the host 100 sends the activate command and the upper address (operation 640) and then sends the read command, the lower address, and the DDR ID (operation 650). This is indicated by the "Act." and "Rd +ID" fields on the command / address line in Fig. 8A shown. ( Fig. Figure 8B is another timing diagram 810 for the read process discussed above, but here there are two read commands, and the later received read command (read command B) completes before the first received read command (read command B). As such, data B is returned to host 100 before data A.

[0044] In response to receiving the read command, the controller 130 takes an indefinite amount of time to read the data from the non-volatile memory 140. After the data is read, the controller 130 notifies the host 100 that the data is ready by sending an R_RDY signal on the response bus (operation 710 in Fig. 7). In response, the host 100 sends a "Send" command on the command / address line (operation 720), and after a predefined delay, the controller 130 returns the data to the host 100 (operation 730) (as indicated by the "D1"-"DN" fields on the data line and the "ID" field on the ID line in Fig. 8B). The memory controller 120 in the host 100 then accepts the data and the DDR ID (operation 740). Next, the memory controller 120 determines whether the DDR ID is associated with a particular host ID of one of the CPUs 110 in the host 100 (operation 750). If so, the memory controller 120 returns the data to the correct CPU 110 (operation 760); otherwise, the memory controller 120 ignores the data or throws an exception (operation 770).

[0045] In the case of a write operation, the host 100 may be restricted to having more than a certain number of outstanding write commands to ensure that the non-volatile memory device does not receive more write commands than it can handle. This is illustrated in the write timing diagram 820 in Fig. 8C. As shown in Fig. As shown in Figure 8C, each time host 100 issues a write command, it decrements its write flow control credits (labeled "WC" in the drawing). When a write operation is complete, media controller 130 sends a response to host 100 to increment its write flow control credits.

[0046] The protocol discussed above is one embodiment of an NVRAM protocol that supports read and write operations of unpredictable duration. As previously discussed, in some embodiments, the controller 130 may exploit the non-deterministic aspect of read and write operations to perform time-consuming actions (which may be referred to herein as operations with an indefinite duration from the perspective of the host) for which it may not have the time according to conventional DRAM-based DIMM standards. These operations, which have an indefinite duration from the perspective of the host, such as storage and data management operations, may be important for the operation of the NV-DIMM. For example, a non-volatile memory device 140 may have lower endurance (i.e., number of writes before an error) compared to DRAM and may store data less reliably (e.g.,These problems may be even more pronounced with emerging non-volatile memory technologies, which would likely be used as DRAM replacements in an NV-DIMM. As such, in one embodiment, the NV-DIMM takes advantage of not being "pressured" to perform operations with an indefinite duration from the host's perspective (e.g., wear-leveling and error-correction operations) that it may not be able to perform in the allotted time according to conventional DRAM-based DIMM standards.

[0047] In general, an operation that has an indefinite duration from the host's perspective refers to an operation that (1) inherently does not have a predetermined duration (e.g., because the duration of the operation depends on one or more variables) or (2) has a predetermined duration, but this duration is unknown to the host (e.g., a decryption operation may have a predetermined duration, but this duration is indefinite from the host's perspective because the host does not know whether the storage system will perform a decryption operation or not). An "operation that has an indefinite duration from the host's perspective" may take any suitable form. For example, such an operation may be a "storage and data management function," which is an action by the controller 130 to manage the health and integrity of the NVM device.Examples of memory and data management functionality include, but are not limited to, wear leveling, data movement, metadata write / read (e.g., logging, controller state and health tracking, wear leveling tracking updates), data decode variations (ECC engine variations (syndromes, BCH vs. LDPC, soft bit decodes), soft reads or rereads, layered ECC requiring increased transfers and reads, RAID or parity reads with their compounded decode and component latencies), resource contention (ECC engine, channels, NVM property (die, block, level, I / O circuits, buffers), DRAM access, scramblers, other hardware engines, other RAM contention), controller exceptions (bugs, peripherals (temperature, NOR), media characterization activities (determining the effective age of memory cells, determining the Bit Error Rate (BER) or probing memory defects).Furthermore, the media controller may introduce elements such as caches that have the inverse effect (fast programs, temporary writes with reduced retention, or other properties) and serve to accelerate read or write operations in a way that would be difficult to predict deterministically.

[0048] Furthermore, operations with indefinite duration from the host's perspective may include, but are not limited to, program refreshes, verification steps (e.g., skip verification, regular settings, tight settings), data movement from one medium / state to another memory location or state (e.g., SLC to TLC, ReRam to NAND, STT-MRAM to ReRam, burst settings to hardened settings, low ECC to high ECC), and longer media settings (e.g., simpler voltage transients). Such operations may be performed, for example, for endurance extension, retention improvement or mitigation, and performance acceleration (e.g., writing this data burst quickly or programming this data more strongly in the preferred direction so that future reads stabilize more quickly).

[0049] The media / NVM controller 130 may be equipped with various hardware and / or software modules to perform these storage and data management operations. As used herein, a "module" may, for example, take the form of a packaged functional hardware unit designed for use with other components, a piece of program code (e.g., software or firmware) executable by a (micro)processor or processing circuitry that typically performs a specific function of related functions, or a standalone hardware or software component connected to a larger system via an interface.

[0050] Fig. Figure 9 is a block diagram of an NVM controller 130 of one embodiment, showing various modules that can be used to perform storage and data management functions. In this particular embodiment, controller 130 is configured to perform encryption, error correction, wear leveling, instruction scheduling, and data aggregation. However, it should be noted that controller 130 can be configured to perform other types and numbers of storage and data management functions.

[0051] As in Fig. 9, this NVM controller 900 includes a physical layer 900 and a non-volatile RAM ("SNVRAM") protocol logic interface (including instruction and memory location decoding) 905, which is used to communicate with the host 100 (via the memory controller 120). The physical layer 900 is responsible for latching in the data and instructions, and the interface 905 separates the instructions and memory locations and handles additional signaling pins between the host 100 and the controller 130. The controller 130 also includes N number of memory finite state machines (MemFSMs) 910 and an NVM physical layer (Phy) 910, which communicate with M number of non-volatile memory devices 140.

[0052] Between these input and output portions, the controller 130 has a write path on the right, a command path in the middle, and a read path on the left. Although not shown, the controller 130 may include a processor (e.g., a CPU executing firmware) that executes the various Fig. 9 and may be connected to them via an interface. Turning first to a write operation, after a command and memory location have been decoded by interface 905, the address is sent to a wear-leveling address translation module 955. In this embodiment, host 100 sends a logical address with a command to write data, and wear-leveling address translation module 955 translates the logical address into a physical address in memory 140. During this translation, wear-leveling address translation module 955 shuffles the data to be placed at a physical address that has not been heavily worn. Wear-leveling data movement module 960 is responsible for reordering the data if a sufficiently unworn memory region cannot be found within the address translation scheme.The resulting physical address, along with the associated command and address where the data can be found in local buffers within controller 130, is input to NVM I / O scheduling module 940, which schedules read and write operations to memory 140. NVM I / O scheduling module 940 may include other scheduling functions, such as, but not limited to, erase operations, setting changes, and defect management.

[0053] In this embodiment, in parallel with address translation, the data for a write operation is first encrypted by the encryption engine 925. Next, the media error correction code (ECC) encoder 930 generates ECC protection for the data at rest in the NVM memory 140. Protecting data at rest may be preferable because non-volatile memory is much more susceptible to errors than DRAM when retrieving previously stored data. However, decoding data with error correction is not always a constant-time operation, so it would be difficult to perform such operations according to deterministic protocols.Although ECC is used in this example, it should be understood that any suitable data protection scheme may be used, such as, but not limited to, cyclic redundancy check (CRC), redundant array of independent disks (RAID), scrambling, data weighting / modulation, or other modifications to protect against degradation due to physical events such as temperature, time, and voltage stress (DRAM is also error-prone, but NVM is susceptible to other errors. Thus, each NVM at rest likely requires a different protection scheme. It is often a trade-off between latency and cost).Although not shown to simplify the drawing, it should also be noted that other data protection systems may be used by the controller 130 to protect data as it is "in transit" between the host 100 and the controller 130 and as it moves within the controller 130 (e.g., using CRC, ECC, or RAID).

[0054] As mentioned above, other data protection schemes than ECC can be used. The following paragraphs provide some additional information about different data protection schemes.

[0055] With regard to ECC, some embodiments of error-checking codes, such as BCH or other Hamming codes, allow the decoding engine, which may use a near-instantaneous syndrome, to perform a check to validate the correctness of the data. However, a syndrome-checking error may involve solving complex algebraic equations, which can result in significant delay. Furthermore, if multiple syndrome-checking errors occur simultaneously, hardware-resource-generated backlogs may exist due to the unavailability of hardware resources for decoding. However, these occasional delays can be managed by delaying the read-ready notification to the host.Other coding schemes, such as LDPC or additional CRC checks, may also be included for more efficient use of space or higher reliability, and although these other schemes are likely to have additional timing variations to process the data coming from the storage medium, these variations can also be handled by simply delaying the read ready signal.

[0056] Another form of data protection can take the form of soft-bit decoding, where the binary value of the data stored in the medium is measured with higher confidence by measuring the analog values ​​of the data stored in the physical storage medium multiple times, relative to multiple thresholds. Such techniques will take longer to implement and may introduce additional variability to the combined data reading and decoding process. However, these additional delays can be easily managed if necessary by delaying the read-ready signal back to the host.

[0057] Further reliability can be enhanced using nested or layered error correction schemes. For example, the data can be encoded in the medium so that the data can survive N errors out of every A bytes read, and M (where M>N) errors out of every B (where B>A) bytes read. A small read of size A may thus be optimal for fast operation, but suboptimal for data reliability in the face of a very bad data block with more than N errors. Occasional problems in this scheme can be corrected by first reading and validating A bytes. If errors persist, the controller has the opportunity to read the much larger block, with the penalty of a delay but with the successful decoding of the data.This is another emergency decoding option enabled by the non-deterministic read timings provided by the SNVRAM-assisted media controller.

[0058] Major malfunctions of a specific storage device could also be encoded using RAID techniques. Data could be distributed across multiple storage devices to accommodate the complete failure of a specific number of storage devices within that set. Spare storage devices could be included in a storage module as fail-in-place replacements to maintain redundant data whenever a faulty storage device is encountered.

[0059] Returning to Fig. 9. After the media error correction code (ECC) encoder 930 generates ECC protection for the data, the data is sent to the write cache management module 935, which determines whether or not there is space in the write data cache buffers 945 and where the data should be placed within those buffers 945. The data is stored in the write data cache buffers 945, where it is stored until read. Thus, if there is a delay in the scheduling of the write command, the data may be stored in the write data cache buffer 945 indefinitely until the memory 140 is ready to receive the data.

[0060] Once the write command associated with this write data cache buffer entry advances in the queue, the data entry is forwarded to the NVM write I / O queue 950. When indicated by the NVM I / O scheduler 940, the command is forwarded from the NVM I / O scheduler 940 to the NVM data routing, command routing, and data aggregation module 920, and the data is forwarded from the NVM write I / O queue 950 to the NVM data routing, command routing, and data aggregation module 920. The command and data are then forwarded to the appropriate channel. The memory finite state machine (MemFSM) 910 is responsible for parsing the instructions into finer-grained, NVM-specific instructions and controlling the timing when these instructions are dispatched to the NVM devices 140.The NVM Phy 915 controls timing to an even finer level, ensuring that the data and command pulses are spaced at well-synchronized intervals with respect to the NVM clock.

[0061] Turning now to the read path, when data from read commands returns from NVM devices 140, NVM data routing, command routing, and data aggregation module 920 places the read data into NVM read I / O queue 965. In this embodiment, the read data can take one of three forms: data requested by a user, NVM register data (for internal use by controller 130), and write validation data. In other embodiments, one or more of these data classes may be held in other queues. If the data was read for internal purposes, it is processed by internal read processing module 960 (e.g., to verify that previously written data was written correctly before sending an acknowledgment back to host 100 or sending a rewrite request to scheduler 940).If the data was requested by the user, metadata indicating the command ID associated with the read data is appended to the data. This command ID metadata is associated with the read data as it is transferred through the read pipeline (as indicated by the double-headed arrow). The data is then sent to the media ECC decoder 975, which decodes the data, and then to the decryption module 980, which decrypts the data before sending it to the read data cache 955. The data remains in the read data cache 955 until the host 100 requests it by identifying the command ID block. At this time, the data is sent to the interface 905 and physical layer 900 for transmission to the host 100.

[0062] Fig. 10 is a flowchart 1000 of a method for reading data using the controller 130 of Fig. 6. As in Fig. As shown in Figure 10, host 100 first sends a read request to the storage system (operation 1050). The NVM controller 130, in this embodiment, then extracts the following elements from the request: address, read request ID, and request length (operation 1010). The NVM controller 130 then converts the logical address from the request into a physical address for wear leveling (operation 1015).

[0063] The NVM controller 130 then determines whether the physical address corresponds to a portion of the memory array that is busy or unavailable for read operations (operation 1020). If the memory portion is busy or unavailable, the NVM controller 130 schedules the read from the non-volatile memory devices 140 for a later time (operation 1022). At that later time, if the physical address becomes available (operation 1024), the NVM controller 130 determines whether other higher-priority operations are pending that prevent the read (operation 1026). If so, the NVM controller 130 waits (operation 1028).

[0064] If / when the storage portion becomes available, the NVM controller 130 sends read commands to the NVM devices 140 to read the requested data (operation 1030). The NVM devices 140 then return the requested data (operation 1035). Depending on the type of devices used, the NVM devices 140 may return the data after a fixed, predetermined period of time. The NVM controller 130 may then process the returned data. For example, after aggregating the data returned from the various NVM devices 140 (operation 1040), the NVM controller 130 may determine whether the data passes an error correction code (ECC) check (operation 1045). If the data fails the ECC check, the NVM controller 130 may initiate an error recovery process (operation 1046).After the error recovery process is complete (operation 1048), or if the aggregated data passes the ECC check, the NVM controller 130 determines whether the data is encrypted (operation 1050). If the data is encrypted, the NVM controller 130 initiates a decryption process (operation 1052).

[0065] After the decryption process is complete (operation 1054), or if the data was not encrypted, the NVM controller 130 optionally determines whether the host 100 has previously agreed to use non-deterministic reads (operation 1055). (Operation 1055 allows the NVM controller 130 to be used for both deterministic and non-deterministic reads, but may not be used in certain embodiments.) If the host 100 has a prior agreement, the NVM controller 130 holds (or sets aside) the read data for a future send command (as discussed below) (operation 1060). The NVM controller 130 also sends a signal on the "READY READ" line to the host 100 (operation 1065). If ready, the storage controller 120 in the host 100 sends a send command (operation 1070).In response to receiving the send command from host 100, NVM controller 130 transmits the processed read data along with the command ID to host 100 (e.g., after a predefined delay (there may be global timeouts from the storage controller in the host)) (operation 1075).

[0066] If the host 100 has not previously agreed to use non-deterministic reads (operation 1055), the NVM controller 130 will handle the read operation as in the conventional system discussed above. That is, the NVM controller 130 will determine whether the elapsed time exceeds the previously agreed transfer time (operation 1080). If the elapsed time has not exceeded the previously agreed transfer time, the NVM controller 130 will transfer the data to the host 100 (operation 1075). However, if the elapsed time has exceeded the previously agreed transfer time, the read operation will fail (operation 1085).

[0067] If we now turn to a write operation, Fig. 11 shows a flowchart 1100 beginning when host 100 has data to write (operation 1105). Next, host 1110 checks to see if there is available flow control credit for the write operation (operations 1110 and 1115). If flow control credit is available, host 100 issues the write request (operation 1130), and media controller 130 receives the write request from host 10 (operation 1125). Controller 130 then extracts the destination address and user data from the request (operation 1130). Because a non-deterministic protocol is used in this embodiment, controller 130 can now spend time performing storage and data management operations. For example, if the data requires encryption (operation 1135), controller 130 encrypts the data (operation 1140). Otherwise, the controller 130 encodes the data for error correction (operation 1145).As mentioned above, any suitable error correction scheme may be used, such as, but not limited to, ECC, cyclic redundancy check (CRC), redundant array of independent disks (RAID), scrambling, or data weighting / modulation. Next, the controller 130 uses wear-leveling hardware (or software) to convert the logical address to a physical (NVM) address (operation 1150). The controller 130 then determines whether the write cache is full (operation 1155). If so, the controller 130 signals an error (operation 1160). An error may be signaled in any suitable manner, including, but not limited to, using a range of voltages on a dedicated pin or pins on the response bus, writing the error to a log (e.g.,in the NVM controller) or incrementing or annotating the error in the Serial Presence Detect (SPD) data. If not, the controller 130 associates a write cache entry with the current request (operation 1165) and writes the data to the write cache (operation 1170).

[0068] The controller 130 then determines whether the physical media at the required physical address is busy (operation 1175). If so, the controller 130 schedules the write operation for future processing (operation 1180). If not, the controller 130 waits until the current operation completes (operation 1182) and then determines whether a higher priority request is still pending (operation 1184). If not, the controller 130 distributes the data to the NVM devices 140 using write commands (operation 1186). The controller 130 then waits due to typical delays when writing to NVM devices (operation 1188). Next, the controller 140 optionally ensures that the write commit was successful (operation 1190) by determining whether the write operation was successful (operation 1192).If the write operation was unsuccessful, controller 130 determines whether further attempts are warranted (operation 1193). If not, controller 130 may optionally apply error correction techniques (operation 1194). If and when the write operation is successful, controller 130 releases the write cache entry (operation 1195) and notifies host 100 of additional write buffer space (operation 1196), and the write operation then completes (operation 1197).

[0069] The flowcharts in Fig. 10 and Fig. 11 both describe the process for performing a single read or write operation. However, in many media controller embodiments, multiple read or write operations can occur in parallel, thus creating a continuous pipeline of read or write processes. Many of these steps will, in turn, support out-of-order processing. The flowcharts serve as an example of the steps that may be required to process a single read or write request.

[0070] In summary, some of the above embodiments provide a media controller that interfaces with a host using a particular embodiment of the SNVRAM protocol and also interfaces with a plurality of storage devices. In addition to utilizing the non-deterministic read and write timing features of the SNVRAM protocol, the media controller is specifically designed to improve non-volatile memory (NVM), optimally correct errors in the media, and schedule requests through the media to optimize throughput, all while presenting a low-latency, high-bandwidth storage interface to the host. In this way, the media controller can manage the health and integrity of the storage medium by "massaging" memory idiosyncrasies.In addition, the media controller can collect and aggregate data from NVM chips for more efficient data processing and error handling.

[0071] There are many alternatives that can be used in these embodiments. For example, while a clock data parallel interface was present in the above examples, other types of interfaces may be used in various embodiments, such as, but not limited to, SATA (Serial Advanced Technology Attachment), PCIe (Peripheral Component Interface Express), NVMe (Non-Volatile Memory Express), RapidIO, ISA (Industry Standard Architecture), Lightning, Infiniband, or FCoE (Fiber Channel over Ethernet). Accordingly, while a DDR parallel interface was used in the above example, other interfaces, including serial interfaces, may be used in alternative embodiments. However, current serial interfaces may encounter long latency and I / O delays (whereas a DDR interface provides fast access times).Also, as mentioned above, although the storage system in the above examples takes the form of an NV-DIMM, other types of storage systems may be used, including, but not limited to, embedded and removable devices such as a solid-state drive (SSD) or a memory card (e.g., Secure Digital (SD), Micro Secure Digital (Micro SD) card, or Universal Serial Bus (USB) drives).

[0072] As a further alternative, NVM chips can be built that can speak either standard DDR or newer SNVRAM protocols without using a media controller. However, the use of a media controller is currently preferred because existing NVM devices have much greater features than more advanced DRAM devices; thus, NVM chips cannot be relied upon to speak at current DDR frequencies. The memory controller may slow down DDR signals to communicate with the NVM chips. Furthermore, the functions performed by the media controller can be relatively complex and expensive to integrate into the memory chips themselves. Furthermore, as media controller technology is likely to evolve, it may be desirable to allow the media controller to be upgraded separately to better handle a specific type of memory chip.This means that sufficient isolation of the NVM and NVM controller enables the incubation of new memory while also providing DRAM speed throughput for mature NVMs. Additionally, the media controller enables error-checking codes and wear-leveling schemes that distribute data across all chips and handle defects, and there is an advantage to aggregating data together through one device.

[0073] As discussed above, in some embodiments, the controller 130 may exploit the non-deterministic aspect in read and write operations to perform time-consuming actions that have an indeterminate duration from the host's perspective.Although storage and data management operations were mentioned above as examples of such actions, it should be understood that there are many other examples of such actions, such as monitoring the health of individual non-volatile media cells to protect them from wear, identifying failures in the circuitry used to access the cells, ensuring that user data is transferred to or removed from the cells in a timely manner and in accordance with the operational requirements of the NVM device, and ensuring that user data is reliably stored and not lost or corrupted due to bad cells or media circuitry failures.Furthermore, in cases where sensitive data may be stored on such a device, operations with an indefinite duration from the host's perspective may involve encryption as a management service to prevent the theft of non-volatile data by malicious entities.

[0074] More generally, an operation with an indefinite duration from the host's perspective may include, but is not limited to, one or more of the following: (1) NVM activity, (2) protection of data stored in the NVM, and (3) data movement efficiencies in the controller.

[0075] Examples of NVM activity include, but are not limited to, user data handling, non-user media activity, and scheduling decisions. Examples of user data handling include, but are not limited to, NVM endurance improvement and mitigation (e.g., wear-leveling data movement, where wear-leveling is the distribution of localized user activity across a larger physical space to extend the endurance of the device, and writing or reading the NVM in a way to influence the endurance characteristics of that storage location), NVM retention improvement or mitigation (e.g., program refreshes, data movement, and retention verifications), differential media latency handling to better manage the wear effect on the media during media activity (writes, reads, erases, verifications, or other interactions) (e.g.,Use of longer or shorter latency techniques, such as those required for NVM handling, to improve a desired property (endurance, retention, future read latency, BER, etc.), and folding data from temporary storage (SLC or STT-MRAM) to more permanent storage (TLC or ReRam). Examples of non-user media activities include, but are not limited to, device logs (e.g., errors, debug information, host usage information, warranty support information, settings, activity trace information, and device history information), controller status, and state tracking (e.g.,Algorithm and state tracking updates for enhanced or continuous behavior during power failure or power-on handling and intermediate verification state conditions for media write acknowledgements, defect identifications and data protection updates for ECC (updating parity or layered ECC values), media characterization activities (e.g., characterizations of NVM age or BER and probing the NVM for defects), and remapping of defective areas.

[0076] Examples of protecting data stored in the NVM include, but are not limited to, various ECC engine implementations (e.g., BCH or Hamming (hardware implementation options of size, parallelization of implementation, syndromes, and coding implementation options such as generator polynomial, protection level, or special case orderings), LDPC (e.g., hardware implementation options of size, parallelization of implementation, array size, and clock rate; and coding implementation options such as protection level and polynomial selection to favor media BER properties), parity (e.g., user data CRC before ECC and RAID), layered protection of any of the above in any order (e.g.,CRC on the user data, ECC over the user data and CRC, two ECC blocks together receive another ECC, calculating the RAID over multiple blocks with ECC for a full stripe of RAID), decode retry paths (e.g., options when initiating and utilizing the other protection layers (e.g., speculative soft read, waiting until miss failure before reading the entire RAID stripe, low-performance vs. high-performance ECC engine modes)), ECC retries with or without any of the following: speculative bit flips, soft-bit decodes, soft reads, new reads (e.g., rereads and soft reads (rereading the same data with different settings) and decode misses), and data shaping for improved memory behavior (e.g., reduced inter-cell interference (e.g., use of a scrambler or weighted scrambler for improved sense circuit performance)).

[0077] Examples of data movement efficiencies in the controller include, but are not limited to, scheduling architecture and scheduling decisions. The scheduling architecture can relate to the availability of single versus multiple paths for each of the following: prioritization, speculative early starts, parallelization, component acceleration, resource arbitration, and implementation options specific to that component. The quantity, throughput, latencies, and connections of each device resource will implicitly impact scheduling. The scheduling architecture can also address internal bus contention during transfers (e.g., AXI bus contention), ECC engines, NVM communication channels (e.g.,Bandwidth, speeds, latencies, idle times, congestion of traffic to the other NVM, ordering or prioritization options and efficiencies of use for instruction, data, state and other NVM interactions), NVM access contention often occurs due to the layout and internal circuit access of each particular NVM (e.g., die, block, level, I / O circuits, buffers, bays, arrays, wordlines, strings, cells, combs, layers and bitlines), memory access (e.g., external DRAM, SRAM, eDRAM, internal NVMs and ECC on these memories), scramblers, internal data transfers, interrupt delays, polling delays, processors and firmware delays (e.g., processor code execution speed, code efficiency and functionality, thread or interrupt swapping) and cache engines (e.g.,Efficiency of cache lookups, cache insertion costs, cache filling strategies, cache hits, and efficient de-concurrent NVM and controller activity and cache ejection strategies. Scheduling decisions may include instruction overlap detection and ordering, memory location decoding and storage schemes (e.g., cached lookup tables, hardware-controlled tables, and layered tables), controller exceptions (e.g., firmware hangs, component timeouts, and unexpected component states), peripheral handling (e.g., alternative NVM handling such as NOR or EEPROM, temperature, SPD (Serial Presence Detect) interactions on the NVDIMM-P, and alternative device access paths (e.g., low-power modes and out-of-band commands), power circuit states), and reduced power modes (e.g.,Off, reduced power states, idle, idle active, and higher power states that may be used for acceleration or bursts).

[0078] The memory system discussed above can benefit from the use of a command and address buffer and data buffers (DBs). An example of a command and address buffer is a register clock driver (RCD). Although an RCD is used in the following examples, it should be understood that other types of command and address buffers can be used. In addition, a command and address buffer can have other functionality. For example, a command and address buffer, such as an RCD, can also have data-parallel decode synchronization capabilities to synchronize the flow of data into and out of the DBs.

[0079] RCDs and DBs were used with DRAM-based DIMMs to improve signal integrity. For example, if long, stray electrical wires within the DIMM cause poor electrical properties on the command and address sets of signals, the RCD 1220 receives and repeats the command and address to the DRAM chips 1210 to ensure they receive them. RDIMM (Registered DIMM) is an example of a DIMM with an RCD, and LRDIMM (Load Reduced DIMM) (or FBDIMM (Fully Buffered DIMM)) is an example of a DIMM that has both an RCD and DBs (a UDIMM (Unbuffered DIMM) enforces electrical routing rules that affect the bus). Signal integrity and other problems can arise when using an NV-DIMM, especially one with a media controller like the one discussed above.The following paragraphs discuss the general use of RCDs and DBs in this context before turning to their use in an NV-DIMM.

[0080] Returning to the drawings, Fig. 12 and Fig. 13 illustrations of a DRAM-DIMM 1200, which includes a plurality of DRAM chips 1210, an RCD 1220 and a plurality of DBs 1230. Although not shown in order to simplify the drawings, Fig. 12 and Fig. As shown in Figure 13, the RCD 1220 is in communication with all of the DRAM chips 1210 and the DBs 1230. Generally, the DBs 1230 store data sent to or read from the DIMM 1200, and the RCD 1220 acts as a repeater to repeat the command and address received on the DIMM's CMD / Addr line to the DRAM chips 1210. The RCD 1220 also controls when the DBs 1230 release the data they store.

[0081] Fig. 12 shows the read flow in DIMM 1200, and Fig. 13 shows the write flow in the DIMM. As in Fig. As shown in Figure 12, a read command is received from the RCD 1220 on the CMD / Addr line (arrow 1). Next, the RCD 1220 communicates a "read" command to the address in each DRAM block 1210, since each DRAM block is addressed the same here (arrow 2). The data is then read from each of the DRAMs 1210 and moved to the corresponding DB 1230 (arrow 3). In the DRAM-based DIMM protocol, the DIMM has a certain amount of time after receiving the read command to deliver the data back to the host. After this time, the RCD 1220 then signals the DBs 1230 to release the data to the host (arrow 4). Between each of these steps, a variation in this scheme is allowed. With this architecture, the RCD 1220 simply assumes that the data is in the DBs 1230 after the timeout, and usually this is a safe assumption due to the reliable DRAM latency when reading data.

[0082] If you now turn Fig. 13, a write command is received from the RCD 1220 on the CMD / Addr line in a write operation (arrow 1). Almost immediately thereafter, the RCD 1220 communicates the write process to the DRAM blocks 1210 (arrow 2). Next, after a fixed time delay tWL, the DBs 1230 receive the data to be written (arrow 3) and then transfer the data to the DRAM blocks 1210 (arrow 4).

[0083] Fig. Figure 14 is a diagram of the internal states of data flow in a DRAM-based DIMM. The earlier layer of decoding and forwarding allows us to assume that each sub-block in this diagram is correctly decoded and understood as a group. Theoretically, each of the sub-groups can be advanced to a larger set of data moving together. The dashed boxes in this diagram represent four of the groups that can be treated together. Although there are times when CMD / ADDR may arrive before the DQ data, the relationships are well-established, so this time delay can be ignored. In any case, a maximum of DQ and CMD / ADDR can describe the state of the physical layer.

[0084] Now that the general background of RCDs and DBs has been provided, the following paragraphs discuss the use of RCDs and DBs in an NV-DIMM. Returning to the drawings, Fig. 15 is a block diagram of a memory system 1500 which corresponds to the memory system 200 discussed above in Fig. 2A. Like storage system 200, this storage system 100 includes an interface 1510, which includes nine data input / output pins (DQ0-DQ8), command pins, and response pins, an NVM controller 1530, and nine non-volatile memory devices 1240. New to this embodiment are the RCD 1520 and the DBs 1550.

[0085] An advantage of this embodiment is that the RCD 1520 and the DBs 1550 act to electrically buffer the NV-DIMM. For example, as in the memory system 200 in Fig. 2A, the DQ traces can be long and difficult to route, which can compromise the signal integrity (SI) quality of the buses. In contrast, traces 1560 between the DRAM bus pins and the RCD 1520 and the DBs 1560 are relatively short, ensuring signal integrity from the DRAM bus. These traces 1560 can be tightly specified for maximum SI and NV-DIMM-P operability in any of UDIMM, RDIMM, LRDIMM, and any other DIMM configurations (existing now or developed later) without degrading bus integrity (this can increase supplier competition and reduce system integration challenges). That is, the speed of the lines 1560 can have sufficient signal integrity and speed to match other physical DRAM communications.In contrast, lines 1570 between the RCD 1520 and the DBs 1550 and the NVM controller 1530, as well as lines 1580 between the NVM controller 1530 and the NVM devices 1540, can be specified with looser specifications, as the communication on these lines 1570 and 1580 can be absorbed into the existing JEDEC specification latency responses (i.e., the latency can be isolated behind the RCD 1520 and the DBs 1550), or the electrical routing, which is entirely contained within the DIMM, can ensure sufficient SI for the transmission. This enables multi-vendor development of DB and RCD chips and the "agnostic" placement of the NVM devices and the NVM controller. Furthermore, this enables sufficient isolation of the NVM devices and the NVM controller to enable the incubation of new memories, while also providing DRAM speed throughput for mature NVMs.In addition, the RAM buffers in the DBs 1550 and the RCD 1520 with non-deterministic protocol may be sufficient to separate and align behaviors of NV-DIMM-P internals and DRAM bus externals.

[0086] In one embodiment, each DQx infers a grouping of data, strobe, and clock signals coming from memory controller 120 in host 100. The number of sets of DQs could have a maximum of DQ7 or DQ8 in a deployment, but there are other maximums, such as DQ9. (Some specifications refer to these as CBs (check bits).) Accordingly, these embodiments may apply to any number of data group signals, and the maximum DQ group number is referred to herein as N. DQ and RCD signal timings and constraints within each group (e.g., message content lines, strobes, and clocks) can be very strict. For example, the "message lines" can be either data in the case of DQ or command and address in the case of RCD. This ensures that eight bytes of data, commands, and addresses are received together and correctly decoded by group.Each message can be received and correctly interpreted by the DBs 1550 or the RCD 1530 (depending on the respective group), so the overall timing constraints between each DQ and the RCD 1530 can be more lenient. The delay structure of the entire DRAM bus can be much looser than a single edge of the DRAM bus clock rate. Thus, the DQ and the RCD 1530 are able to correctly decode and encode to the corresponding and related buffers. In one embodiment, the memory controller 1530 sends the message groups all at once, ensuring the correct placements and signal integrity rules so that the data reaches each component and is correctly decoded.

[0087] The basic operation of the RCD 1520 and the DBs 1550 is similar to the operation of the RCD 1220 and the DBs 1230 in the DRAM-based DIMM example above, with some differences to account for the use of NVM devices 1540 and the NVM controller 1530. That is, in general, the DBs 1550 store data that is sent to or read from the NVM devices 1540, and the RCD 1520 serves as a repeater to repeat the command and address received on the CMD / Addr line of the storage system 1500 to the NVM devices 1540. However, the DRAM-based DIMM uses a deterministic protocol, with the RCD 1220 instructing the DBs 1230 to release their data to the host after a predetermined period of time. As mentioned above, due to the mechanism of reading data from a non-volatile memory, the requested data may not be ready to be sent to the host in the predetermined time period.Examples of this mechanism include, but are not limited to, media choice (e.g., MRAM, PRAM, RRAM, etc.) and material for the media, process nodes, I / O circuit behavior, I / O circuit protocol, intermittent logic dies, controller delays, data errors (BER, defects) requiring higher or lower ECC, which means a higher or lower number of NVM dies, placements of NVM devices and controllers, NVM communication channel delays (e.g., instruction vs. data groups of instructions, shared data and instruction, serializer / deserializer (SerDes) vs. parallel), and NVM channel interconnect options (e.g., through silicon via (TSV), through silicon sidewall (TSW), direct, interleaved).

[0088] Accordingly, the RCD 1520 is in the Fig. 15 (e.g., by programming a processor in the RCD 1520 with firmware / software or by providing a pure hardware implementation) to receive and respond to the new read command discussed above. Specifically, in this embodiment, the RCD 1520 is configured to provide a ready signal on the CMD / Addr line when the DBs 1550 contain the data in response to a read command and is further configured to instruct the DBs 1550 to release their data (after a predefined delay) to the host in response to the RCD 1520 receiving a transmit command.

[0089] Fig. 16 is a block diagram illustrating a read operation. As shown in Fig. As shown in Figure 16, a read command is received by the RCD 1520 from the storage controller in the host (arrow 1). The address and read command are then transmitted from the RCD 1520 to the NVM controller 1530 (arrow 2). The read command is processed and transmitted to the relevant NVM devices 1540 (arrow 3), and the read data returns to the NVM controller and then on to the DBs 1550 (arrow 4). When the RCD 1520 knows that the DBs 1550 contain the data (e.g., by polling or otherwise communicating with the DBs 1550, or upon instruction from the NVM controller 1530), the RCD 1520 sends the RD_RDY signal to the storage controller in the host (arrow 5). In response, the memory controller in the host issues a SEND command on the command bus (arrow 6), and in response, the RCD 1520 instructs the DBs 1550 to transfer the data (after an optional specified delay (tsend)) to the host (arrow 7).

[0090] If we now turn to the write operation (see Fig. 17), the storage controller in the host first checks the write count to ensure that there is remaining credit for the write operation. If so, the storage controller in the host transmits a write command and address to the RCD 1520 (arrow 2), and the storage controller decrements its write credit. Next, after a specified JEDEC delay, the storage controller in the host transmits data to the DBs 1550 (arrow 3). Then, the command and data are transferred from the RCD 1520 and DBs 1550 to the NVM controller 1530 (arrow 4), although the RCD 1520 may forward the address and command before the data arrives from the DBs 1550. Next, the write data is committed to the NVM devices 1540 (arrow 5), and the write credit is passed back to the host storage controller on the bus (arrow 6). Note that actions 5 and 6 can be interchanged.However, if persistence is required before a write credit acknowledgment, then it may be preferable to perform action 5 before 6. If persistence is not required before a write credit acknowledgment, then it may be preferable to perform action 6 before 5. In either case, the memory controller in the host increments the write credit count (the write credit response back to host 100 can be either single credits or multiple credits per message to host 100).

[0091] Due to the mechanism of reading and writing in NVM storage devices, read and write commands may not complete in the order in which they were received. As discussed above, a second received read command (Read B) may complete before a first received read command (Read A), for example, if Read B has a higher priority or if the physical address of Read A is unavailable for read operations and Read A is scheduled for a later time. This is not a problem for DRAM-based DIMMs, as read and write commands are processed in the order in which they are received. However, this can be a problem with NV-DIMMs, as the data released from the NV-DIMM to the host may not be the data the host expects (e.g., the host expects data from Read A but instead receives data from Read B).To address this problem, an identifier (ID) is associated with different instructions to track which data belongs to which instructions. This is defined in . Fig. 18 and Fig. 19 illustrates.

[0092] Fig. 18A is a flowchart of a read operation of one embodiment using the memory system 1500 in Fig. 15. As in Fig. As shown in Figure 18A, the host commands a read from an address (and optionally specifies an read ID) (operation 1880). The RCD then forwards the command, address, and ID (operation 1882). Note that this ID (which can be used to enable out-of-order operations) may or may not be the same as the ID received from the host. Next, the data from the NVM is ready (operation 1884), and the RCD notifies the host that the read data is ready (optionally including the ID of the ready read) (operation 1886). The host then issues the transmit signal (operation 1888), and the RCD tells the NVM controller to transmit (operation 1890). The data (1892) is then transmitted (operation 1894), along with a response including the ID (operation 1896).

[0093] Fig. 18B is a flowchart of a read operation of another embodiment. As in Fig. 18B, host 100 commands a read from an address and includes an optional read identifier (ID) (operation 1805). RCD 1520 receives the command, address, and ID to NVM controller 1520 (operation 1810). RCD 1520 also forwards the command and ID (but not the address) to DBs 1550 (operation 1815). In response, DBs 1550 allocate space for the read data and reference the allocated space by the ID (operation 1820). (In another embodiment, the DBs always have some space available, and the ID is correlated in a delayed manner with the ID contained within the RCD.) After the NVM controller 1530 reads the requested data from the NVM devices (operation 1825), the NVM controller 1520 sends the data and ID to the DBs 1550, thereby placing the data in the allocated space identified by the ID (operation 1835).The NVM controller 1520 also sends a completion signal and the ID to the RCD 1520 (operation 1840), which can either wait until the DBs 1550 confirm that the data is placed or wait a predefined time (operation 1845). After either the DBs 1550 confirm the storage of the data or after the predefined time has elapsed, the RCD 1520 notifies the host 100 that the read operation is ready (and may also include the ID) (operation 1850). The host 100 later sends a transmit command (with the ID) to request the read data (operation 1855). The RCD then notifies the NVM controller that it should transmit (operation 1859). In response, the NVM controller tells the DBs 1550 to transmit the data associated with the ID after an optional predetermined delay specified by a standard (operation 1860).The DBs 1550 then transmit the data associated with the ID (operation 1865), and the RCD transmits its corresponding information (operation 1870).

[0094] If you now turn Fig. 19A is Fig. 19A is a flowchart of a write operation of one embodiment. As in Fig. 19A, host 100 first determines whether it can send a write command by checking whether any credits remain in the write counter and / or checking whether the persistence level is greater than 0 (operation 1904). It should be noted that the write counter and the persistence counter are optional, and that an implementation may have one, both, or neither counter. This particular example uses both write and persistence counters, and if the write operation is permitted, host 100 decrements the count in both counters (operation 1908). When RCD 1520 receives the write command from host 100, it sends the command and address to NVM controller 1530 (operation 1912) and sends the data to be written to DBs 1550 (operation 1922). The RCD 1520 may also include the optional ID in embodiments where the NVM controller 1530 retrieves the data from the DBs 1550 (operation 1925). The data is then repeated (operation 1926).The NVM controller 1530 then accepts the data from the DBs 1550 into its write buffers (operation 1932). The NVM controller 1530 then moves the data through its buffers and may eventually be in an optional state that protects against power loss and ensures that writing occurs (operation 1934). The NVM controller 1530 then writes the data to the NVM devices 1540 (operation 1936).

[0095] In this embodiment, there are three points at which the storage system 100 can communicate back to the host 100 that the write operation is complete. The protocol may or may not distinguish between them, and it may or may not track them separately. Also, customers or manufacturers may implement different behaviors. As shown in Fig. 19, in one embodiment, the write persistence indicator and counter are incremented (operations 1944 and 1948). In another embodiment, the write persistence indicator and counter are incremented (operations 1952 and 1956). In yet another embodiment, the write completion indicator and counter are incremented (operations 1964 and 1968).

[0096] Fig. 19B is a flowchart of a write operation of another embodiment. As in Fig. 19B, the host 100 first determines whether it can send a write command by checking whether any credits remain in the write counter and / or checking whether the persistence level is greater than 0 (operation 1905). It should be noted that the write counter and the persistence counter are optional, and that an implementation may have one, both, or neither counter. This particular example uses both write and persistence counters, and if the write operation is permitted, the host 100 decrements the count in both counters (operation 1910). When the RCD 1520 receives the write command from the host 100, it sends the command and address to the NVM controller 1530 (operation 1915) and sends the data to be written to the DBs 1550 (operation 1920). The RCD 1520 may also include the write ID in embodiments where the NVM controller 1530 fetches the data from the DBs 1550 (operation 1925).If the NVM controller 1530 does not fetch the data from the DBs 1550, the DBs 1550 push the write data to the NVM controller 1520, as coordinated by the RCD 1520, to request data for the ID (operation 1930). The data is then moved to the NVM controller 1530 (operation 1932). The NVM controller 1530 then accepts the data from the DBs 1550 into its write buffers (operation 1935). The NVM controller 1530 then moves the data through its buffers and may finally be in an optional state that protects against power loss and ensures that writing occurs (operation 1940). The NVM controller 1530 then writes the data to the NVM devices 1540 (operation 1945).

[0097] In this embodiment, there are three points at which the storage system 100 can communicate back to the host 100 that the write operation is complete. The protocol may or may not distinguish between them, and it may or may not track them separately. Also, customers or manufacturers may implement different behaviors. As shown in Fig. 19, in one embodiment, the write persistence indicator and counter are incremented (operations 1955 and 1960). In another embodiment, the write persistence indicator and counter are incremented (operations 1970 and 1975). In yet another embodiment, the write completion indicator and counter are incremented (operations 1985 and 1990).

[0098] Another issue that may need to be addressed due to the use of an NVM controller 1520 is the clock speed, as the NVM controller 1520 may require a slower clock than that generated by the host 100 on the SDRAM bus. High-speed bus lines of conventional DIMMs may require complex circuitry in the input / output connections on the NVM controller 1520, as well as careful routing in the memory system 1500. To address this, in one embodiment, the RCD 1520 may change the clock speed to transfer data on the internal lines in the memory system 100 at a slower frequency. (Alternatively, instead of the RCD 1520 performing this functionality, the NVM controller 1520 or another component in the memory system 100 may change the clock speed.) This is illustrated diagrammatically in Fig. 20 for incoming data (the same conversion can be done in reverse to send data back to the host 100). Fig. Figure 20 shows the clock, DQ and DQ strobe signal from the host 100 side (left part of Fig. 20) and from the side of the NVM controller 1530 (right part of Fig. 20). As shown in this drawing, the clock signal from host 100 is at a frequency Thost, which, due to the DDR protocol, causes data and data strobes to occur at a relatively high frequency, which may be too much to be handled by the NVM controller 1530 without significant changes to its circuitry. In contrast, data and data strobes, as shown by the right part of Fig. 20, by slowing down the clock on Tnvsdimm to a relatively low frequency, which is easier for the NVM controller 1530.

[0099] The RCD 1520 may be configured to slow down the clock using any suitable method. For example, the RCD 1520 may include clock dividers to generate slower clocks from the source clock (e.g., by dividing the frequency by an integer to produce a slower frequency). The RCD 1520 may also include a phase-locked loop (PLL) to increase the clock frequency, which may be important for dividing the clock frequency by a non-integral fraction. For example, to divide the clock frequency by 3 / 2 (or in other words, multiply by 2 / 3), a PLL may be used to first double the clock frequency before dividing it by three. As another example, the RCD 1520 may include delay compensation circuits (e.g.,A phase-locked loop may include the delay to compensate for it in its feedback loop, and thus the delay would be automatically subtracted from the clock output; or explicit delay-locked loops can be added to explicitly adjust the delays. As yet another example, the RCD 1520 may include data synchronizers that slow down the data, not just the clock. This can be done using a first-in, first-out memory, which has the advantage of safely moving the data from one clock domain to another.

[0100] As mentioned above, instead of implementing these clock reclocking components in the RCD 1520, they can be implemented in the NVM controller 1520. Furthermore, the RCD 1520 can include the clock and data reclocking functions to relax the signal integrity and routing requirements on the DIMM's internal wiring. Furthermore, three clocks can be used (one for talking to the host (very fast), one for sending data to the media controller (less fast), and one for talking to the NVM (even less fast)), in which case both the NVM controller 1520 and the RCD 1520 could perform clock conversion.

[0101] In embodiments where the data clock rate decreases as it is forwarded through the RCD, the clock is preferably distributed among all DBs. This allows the DBs to receive a copy of the host clock and the media controller side clock. The RCD also preferably knows how slow the media controller side clock is, so it can continue to fulfill its task of synchronizing DB data transfers.

[0102] Furthermore, in addition to clock conversion, bandwidth considerations may be present. For example, the bandwidth in the left part of Fig. 20 defined as: N bits * (1 ns) / (Thost) * 1 GHz or N / (Thost / 1 ns) [Gbits / s]. In the right part of Fig. 20, the bandwidth would be defined as: N / (nvdimm / 1 ns) [Gbits / s]. There are different approaches that can be used to account for the bandwidth difference. For example, one approach uses serializers and deserializers to achieve the same bandwidth as a DDR over the DIMM. The deserializer can take a narrow bus of N bits with a frequency of f cycles / second and a transfer rate of f*N bits / second and transform it to a wider bus of N*a bits with a frequency of f / b cycles / second and a transfer rate of f*N*a / b bits / second (for a=b, the bandwidth is the same for the wider, slower bus). Using the serializer can transform the width back to N bits with a frequency of f cycles / second.

[0103] In another approach, queues can be used to compensate for bandwidth mismatches. The bus width is the same for DB input and output. With this approach, incoming data (from host 100 to NVM controller 1330) is held in a buffer, which may or may not be a first-in, first-out (FIFO) buffer. Using a buffer may increase the transfer time to NVM controller 1520, but the buffer provides a temporary holding location during the transfer. Outgoing data (from NVM controller 1530 to the DBs) can be accumulated in a buffer (such as, but not limited to, a FIFO) when it trickles in at a low bandwidth. The data can only be retransmitted to the host when a complete packet is received.

[0104] Changes to the DBs 1550 can also be made to accommodate the use of non-volatile memory and the NVM controller 1530. To understand these changes, a Fig. 21. This DB 2100 includes a set of components for the DQ signals and for the DQ strobe signals. As shown in Fig. As shown in Figure 21, the components for the DQ signals include I / O buffers 2110, 2120, input and output FIFOs 2130, 2140, and synchronization / phase adjustment logic 2115. The components for the DQ strobe signals include I / O buffers 2150, 2160, and strobe generators 2170, 2180. The DB 2100 also includes command parsing logic 2190, which has the clock and command bus signals as inputs. In this embodiment, the FIFOs 2130, 2140 are used to buffer data and are synchronized by the RCD and DQ strobe generators. In another implementation, the FIFOs are not used, and the DB 2100 is configured in pass-through mode.

[0105] If a DB is designed to down-convert data to a lower frequency, additional components can be used, as described in Fig. 22. As the DB 2100 in Fig. 21, the components for the DQ strobe signals include I / O buffers 2250, 2260 and strobe generators 2270, 2280, and the components for the DQ signals include I / O buffers 2210, 2220 and synchronization / phase adjustment logic 2215. However, the DB 22 in Fig. 22 I / O buffers 2230 and 2240 are used instead of input and output FIFOs, and the command parsing logic 2290 includes the following inputs: Clock A (host side), Clock B (NV-DIMM side), and command bus signals from the RCD. Additionally, the DB 2200 includes dual-port, dual-clock random access memory 2235 to enable out-of-order processing, as the input and output buffers 2230 and 2240 serve as both data storage and a staging area for synchronization (a second FIFO can be used for further synchronization).

[0106] Returning to the drawings. Fig. 23 is an illustration of an alternative architecture to that shown in Fig. 15 shown.

[0107] As in Fig. As shown in Figure 23A, the NVM devices 2340 are connected to the DBs 2350 without passing through the NVM controller 2330. This embodiment may be useful when NVM devices operating at DRAM speed are capable of matching data rates with the DBs 2350 and the bus 2310. Write and read operations that conflict with media storage locations, causing unforeseen latencies, can be absorbed by the DBs 2350 without impacting the bus 2310. The NVM controller 2330 can coordinate the DBs 2350, the RCD 2320, and the NVM activity while allowing data to be passed directly between the DBs 2350 and the NVM devices 2340.

[0108] Furthermore, as mentioned above, the memory system can be added with an RCD and DBs in various DIMM variations (e.g., UDIMM, RDIMM, and LRDIMM). There are variations within each of these DIMM formats. For example, UDIMMs have straight, short lines in terms of electrical routing rules. UDIMMs generally have a small number of DIMMs, DRAM banks / ranks per package, and closest physical layout in the server motherboard. The DRAM packages and command routing lines are all specified for repeatable system integration and system electrical interactions. This makes UDIMMs the most cost-effective to produce. RDIMMs have an RCD and generally have a larger number of DIMMs. DRAM banks / ranks per package are possible. DRAM packages, terminations, routing for data, and RCD details are specified. RCD-to-DRAM connections are loosely specified. Compared to UDIMM, there is an incremental cost for RCD.LRDIMMs have insulators for all electrical communication groups, and DB and RCD connections to the memory controller are tightly specified. LRDIMMs have the highest cost of these three formats, but allow the highest number of DIMMs, BGAs, and banks / ranks per memory controller.

[0109] For each DRAM bus (UDIMM, RDIMM, LRDIMM), the memory system may use specifications for the external interacting components. These specifications may include physical and electrical characteristics for maximum interoperability. This may include changes to both the physical signaling layer (e.g., to adapt electrical specifications) and the command layer (e.g., to provide appropriate command decoding). Changes to the physical signaling layer may include introducing additional transmission lines in the control set or changes to the geometry, impedance, and / or termination of any of the clock, command, data, or control set lines (including both standard SDRAM / DDR control set lines and the response bus).At the instruction layer, these changes may also include choosing between different Tsends depending on the delay to which these different formats are exposed, or adding new interpretation to new instructions (e.g., associating certain row decode bits not with addresses within a rank, but inferred selection of additional ranks within a DIMM).

[0110] Parameterized specifications can also be defined for the internal connections from an NVM controller to the RCD and the DBs. The internal connections can be optional to enable vendor-specific optimizations, package integrations, or ASIC integration. The specifications can be sufficiently robust to handle diverse NVM controller placements, diverse data communication rates, and signal integrity characteristics. The specifications for RAM buffer sizing and RCD timing behavior can also be used for successful vendor-agnostic interoperability.

[0111] Returning to the drawings, Fig. 23B is an illustration of an RCD 2360 of one embodiment. As in Fig. 23B, the RCD 2360 in this embodiment includes input buffers 2363, latches / FFs 2363, control registers 2364, output buffers 2365, CS, CKE, decode logic 2366, control logic 2367, clock buffers 2368, a PLL 2369, and a PLL feedback delay compensation module 2370. Many of the circuit elements in this RCD 2360 may be similar to those found in the RCD discussed above. However, the configuration of the control logic 2367 may be changed to accommodate the nature of the non-deterministically timed SNVRAM command sequences to support SNVRAMs. The control logic 2367 is responsible for the behavioral response of the RCD, and changes may be made so that the DRAM DIMM RCDs implement the functions described in the flowcharts in Fig. 18 and Fig.19. The RCD also has the differentiated ability to understand more commands, controls, and addresses. Additional outputs and inputs can be provided to synchronize new parts, such as the NVM controller.

[0112] Finally, as mentioned above, any suitable type of memory may be used. Semiconductor memory devices include volatile memory devices, such as dynamic random access memory (DRAM) or static random access memory (SRAM), non-volatile memory devices, such as resistive random access memory (ReRAM), electrically erasable programmable read-only memory (EEPROM), flash memory (which can also be considered a subset of EEPROM), ferroelectric random access memory (FRAM), and magnetoresistive random access memory (MRAM), and other semiconductor devices capable of storing information.Each type of memory device can have different configurations. For example, flash memory devices can be designed in a NAND or NOR configuration.

[0113] The memory devices can be formed from passive and / or active elements in any combination. As a non-limiting example, passive semiconductor memory elements include ReRAM device elements, which in some embodiments include a resistive switching memory element, such as an anti-fuse phase-change material, etc., and optionally a steering element, such as a diode, etc. Further, as a non-limiting example, active semiconductor memory elements include EEPROM and Flash memory device elements, which in some embodiments include elements containing a charge storage region, such as a floating gate, conductive nanoparticles, or a charge storage dielectric material.

[0114] Multiple storage elements may be configured so that they are connected in series or so that each element can be accessed individually. As a non-limiting example, flash memory devices in a NAND (NAND memory) configuration typically include series-connected storage elements. A NAND memory array may be configured so that the array consists of multiple strings of memory, where a string consists of multiple storage elements that share a single bitline and are accessed as a group. Alternatively, storage elements may be configured so that each element can be accessed individually, such as a NOR memory array. NAND and NOR memory configurations are exemplary, and storage elements may be configured differently.

[0115] The semiconductor memory elements arranged within and / or above a substrate may be arranged in two or three dimensions, such as in a two-dimensional memory structure or in a three-dimensional memory structure.

[0116] In a two-dimensional memory structure, the semiconductor memory elements are arranged in a single plane or a single memory device level. Typically, memory elements in a two-dimensional memory structure are arranged in a plane (e.g., a plane in the xz direction) that extends substantially parallel to a major surface of a substrate that supports the memory elements. The substrate may be a wafer over or in which the layer of memory elements is formed, or it may be a carrier substrate that is attached to the memory elements after they have been formed. As a non-limiting example, the substrate may comprise a semiconductor such as silicon.

[0117] The memory elements may be arranged in an ordered array at the individual memory device level, such as in a plurality of rows and / or columns. However, the memory elements may be arranged in non-regular or non-orthogonal configurations. The memory elements may each have two or more electrodes or contact lines, such as bit lines and word lines.

[0118] A three-dimensional memory array is arranged so that memory elements occupy multiple planes or multiple memory device levels, thereby forming a structure in three dimensions (i.e., in the x, y, and z directions, with the y direction being substantially perpendicular and the x and z directions being substantially parallel to the main surface of the substrate).

[0119] As a non-limiting example, a three-dimensional memory structure may be arranged vertically as a stack of multiple two-dimensional memory device levels. As a further non-limiting example, a three-dimensional memory array may be arranged as multiple vertical columns (e.g., columns extending substantially perpendicular to the main surface of the substrate, i.e., in the y-direction), with each column having multiple memory elements in each column. The columns may be arranged in a two-dimensional configuration, e.g., in an xz-plane, resulting in a three-dimensional array of memory elements with elements on multiple vertically stacked memory levels. Other configurations of memory elements in three dimensions may also constitute a three-dimensional memory array.

[0120] As a non-limiting example, the memory elements in a three-dimensional NAND memory array may be coupled together to form a NAND string within a single horizontal (e.g., xz) memory device level. Alternatively, the memory elements may be coupled together to form a vertical NAND string traversing multiple horizontal memory device levels. Other three-dimensional configurations are conceivable, in which some NAND strings contain memory elements within a single memory level, while other strings contain memory elements spanning multiple memory levels. Three-dimensional memory arrays may also be designed in a NOR configuration and in a ReRAM configuration.

[0121] Typically, in a monolithic three-dimensional memory array, one or more memory device levels are formed over a single substrate. Optionally, the monolithic three-dimensional memory array may also include one or more memory layers at least partially within the single substrate. As a non-limiting example, the substrate may comprise a semiconductor such as silicon. In a monolithic three-dimensional array, the layers forming each memory device level of the array are typically formed on the layers of the underlying memory device level of the array. However, layers of adjacent memory device levels of a monolithic three-dimensional memory array may be shared or may have intermediate layers between memory device levels.

[0122] Then, in turn, two-dimensional arrays can be formed separately and then packaged together to form a non-monolithic memory device with multiple memory layers. For example, non-monolithic stacked memories can be constructed by forming memory levels on separate substrates and then stacking the memory levels on top of each other. The substrates can be thinned or removed from the memory device levels before stacking, but because the memory device levels are initially formed over separate substrates, the resulting memory arrays are not monolithic three-dimensional memory arrays. Furthermore, multiple two-dimensional memory arrays or three-dimensional memory arrays (monolithic or non-monolithic) can be formed on separate chips and then packaged together to form a stacked-chip memory device.

[0123] Associated circuitry is typically required for the operation of the memory elements and for communication with the memory elements. As non-limiting examples, memory devices may include circuitry used to control and drive memory elements to perform functions such as programming and reading. This associated circuitry may be located on the same substrate as the memory elements and / or on a separate substrate. For example, a controller for memory read / write operations may be located on a separate controller chip and / or on the same substrate as the memory elements.

[0124] Those skilled in the art will recognize that this invention is not limited to the described two-dimensional and three-dimensional exemplary structures, but covers all relevant memory structures within the spirit and scope of the invention as described herein and as understood by those skilled in the art.

[0125] It is intended that the foregoing detailed description be understood as an illustration of selected forms the invention may take, rather than as a definition of the invention. Only the following claims, including all equivalents, are intended to define the scope of the claimed invention. Finally, it should be noted that any aspect of any of the preferred embodiments described herein may be used alone or in combination with one another.

Claims

[1] Storage system (1500, 2300), comprising: a plurality of non-volatile memory devices (1540, 2340); a controller (1530, 2330) in communication with the plurality of non-volatile memory devices (1540, 2340), the controller (1530, 2330) being configured to: Receiving a read command from a host (100); in response to receiving the read command from the host (100), reading data from the plurality of non-volatile memory devices (1540, 2340); Performing an operation with an indefinite duration from the host's perspective; Sending a ready signal to the host (100) after performing the operation; Receiving a send command from the host (100); and in response to receiving the send command from the host (100), sending the data to the host (100); a plurality of data buffers (1550, 2100, 2200, 2350) in communication with the controller (1530, 2330) and configured to store data sent between the controller (1530, 2330) and the host (100); and a command and address buffer (1520, 2320) configured to store commands and addresses sent from the host (100), wherein the command and address buffer (1520, 2320) is further configured to synchronize the flow of data into and out of the plurality of data buffers (1550, 2100, 2200, 2350). [2] The storage system of claim 1, wherein read and / or write commands are associated with identifiers so that the read and / or write commands can be processed in a different order than an order in which they are received from the host (100). [3] The memory system of claim 1, wherein the command and address buffer (1520, 2320) comprises a registered clock driver. [4] The memory system of claim 1, wherein the plurality of data buffers (1550, 2200, 2350) comprise random access memories (2235). [5] The memory system of claim 1, wherein the command and address buffer (1520, 2320) is further configured to change a frequency of a clock received from the host (100). [6] The memory system of claim 1, wherein the command and address buffer (1520, 2320) is further configured to perform bandwidth conversion. [7] The memory system of claim 1, wherein physical layers and command layers of the memory system are configured to be compatible with a DRAM-DIMM communication protocol. [8] The memory system of claim 7, wherein physical layers and command layers of the memory system are configured to be compatible with one or more of the following: unbuffered DIMM (UDIMM), registered DIMM (RDIMM), and load reduced DIMM (LRDIMM). [9] The storage system of claim 1, wherein the data is sent to the host (100) after a time delay, and wherein the time delay is selected based on a communication protocol used with the host (100). [10] The memory system of claim 1, wherein the controller (1530, 2330) is configured to communicate with the host (100) using a clock data parallel interface. [11] The memory system of claim 10, wherein the clock data parallel interface comprises a double data rate (DDR) interface. [12] The memory system of claim 1, wherein at least one of the plurality of non-volatile memory devices (1540, 2340) comprises a three-dimensional memory. [13] Storage system (1500, 2300), comprising: a plurality of non-volatile memory devices (1540, 2340); a controller (1530, 2330) in communication with the plurality of non-volatile memory devices (1540, 2340), the controller (1530, 2330) being configured to: Receiving a write command from a host (100), wherein the host is only allowed a certain number of outstanding write commands as tracked by a write counter in the host; Performing an operation with an indefinite duration from the perspective of the host (100); Writing data to the plurality of non-volatile storage devices; and after writing the data, sending a write counter increment signal to the host (100); a plurality of data buffers (1550, 2100, 2200, 2350) in communication with the controller (1530, 2330) and configured to store data sent between the controller (1530, 2330) and the host (100); and a command and address buffer (1520, 2320) configured to store commands and addresses sent from the host (100), wherein the command and address buffer (1520, 2320) is further configured to synchronize the flow of data into and out of the plurality of data buffers (1550, 2100, 2200, 2350). [14] The storage system of claim 13, wherein read and / or write commands are associated with identifiers such that the read and / or write commands can be processed in a different order than the order in which they are received from the host (100). [15] The memory system of claim 13, wherein the command and address buffer (1520, 2320) comprises a registered clock driver. [16] The memory system of claim 13, wherein the plurality of data buffers (1550, 2200, 2350) comprise random access memories (2235). [17] The memory system of claim 13, wherein the command and address buffer (1520, 2320) is further configured to change a frequency of a clock received from the host (100). [18] The memory system of claim 13, wherein the command and address buffer (1520, 2320) is further configured to perform bandwidth conversion. [19] The memory system of claim 13, wherein physical layers and command layers of the memory system are configured to be compatible with a DRAM-DIMM communication protocol. [20] The memory system of claim 19, wherein physical layers and command layers of the memory system are configured to be compatible with one or more of the following: unbuffered DIMM (UDIMM), registered DIMM (RDIMM), and load reduced DIMM (LRDIMM). [21] The storage system of claim 13, wherein the data is sent to the host (100) after a time delay, and wherein the time delay is selected based on a communication protocol used with the host (100). [22] The memory system of claim 13, wherein the controller (1530, 2330) is configured to communicate with the host (100) using a clock data parallel interface. [23] The memory system of claim 22, wherein the clock data parallel interface comprises a double data rate (DDR) interface. [24] The memory system of claim 13, wherein at least one of the plurality of non-volatile memory devices (1540, 2340) comprises a three-dimensional memory. [25] Storage system (1500, 2300), comprising: a plurality of non-volatile memory devices (1540, 2340); means for receiving a read command from a host (100); Means for reading data from the plurality of non-volatile memory devices (1540, 2340) in response to receiving the read command from the host (100); Means for performing an operation of indefinite duration from the perspective of the host (100); means for sending a ready signal to the host (100) after performing the operation; means for receiving a transmit command from the host (100); means for sending the data to the host (100) in response to receiving the send command from the host (100); Means for storing data sent between the controller (1530, 2330) and the host (100); and Means for storing commands and addresses sent by the host (100), wherein the command and address buffer (1520, 2320) is further configured to synchronize the flow of data into and out of the plurality of data buffers (1550, 2100, 2200, 2350). [26] The storage system of claim 25, further comprising: Means for receiving a write command from the host, wherein the host is only allowed a certain number of outstanding write commands as tracked by a write counter in the host (100); Means for performing an operation of indefinite duration from the perspective of the host (100); means for writing data to the plurality of non-volatile memory devices (1540, 2340); and Means for sending a write counter increment signal to the host after writing the data.

Images