Storage system and method for storing logical-to-physical address table entries in codewords in volatile memory

Abstract

A storage system caches logical-to-physical address table entries read in volatile memory. The logical-to-physical address table entries are stored in codewords. The storage system can change the number or size of entries in a codeword. Additionally or alternatively, each codeword can store both a full logical-to-physical address table entry and a partial logical-to-physical address table entry. In one example, a codeword with 62 bytes of data and two bytes of error correction code stores 15 full logical-to-physical address table entries and one partial logical-to-physical address table entry, with the remainder of the partial entry stored in another codeword. This configuration achieves a good balance between storage space efficiency and random access write performance.

Description

Background Technology

[0001] In some storage environments, the host provides the storage system with a logical address for read or write commands, and the storage system queries a logical-to-physical address translation table to identify the physical address in non-volatile memory associated with the logical address. The table is stored in non-volatile memory, but some or all of it can be cached in volatile memory for faster access. Error correction codes can be generated based on the cached table and stored in volatile memory. Attached Figure Description

[0002] Figure 1A This is a block diagram of a non-volatile storage system implementation plan.

[0003] Figure 1B This is a block diagram illustrating a storage module in one implementation scheme.

[0004] Figure 1C This is a block diagram illustrating a hierarchical storage system implementation.

[0005] Figure 2A This illustrates an implementation scheme. Figure 1A The diagram shows a block diagram of the components of the controller for a non-volatile memory system.

[0006] Figure 2B This illustrates an implementation scheme. Figure 1A The diagram shows the components of a non-volatile memory system.

[0007] Figure 3 This is a block diagram of the host and storage system of an implementation scheme.

[0008] Figure 4 This is a diagram of the codeword in one implementation where four bytes of data are protected by a one-byte error correction code.

[0009] Figure 5 This is a diagram of the codeword of one implementation where 128 bytes of data are protected by two bytes of error correction code.

[0010] Figure 6 This is a diagram of the codeword of one implementation where 64 bytes of data are protected by two bytes of error correction code.

[0011] Figure 7 This is a table illustrating one implementation of DRAM usage in multiple scenarios.

[0012] Figure 8 This is a diagram of two codewords in one implementation where each codeword includes 62 bytes of data protected by two bytes of error correction code. Detailed Implementation

[0013] The following embodiments relate throughout to a storage system and method for storing logic-to-physical address table entries in codewords in volatile memory. In one embodiment, a storage system is provided, comprising: a non-volatile memory; a volatile memory; means for reading a codeword from the volatile memory, wherein the codeword includes a data portion and error correction codes generated from the data portion, and wherein the data portion includes at least one logic-to-physical address table entry read from the non-volatile memory; and means for changing the number or size of at least one logic-to-physical address table entry in the codeword. In another embodiment, a storage system is provided, comprising a non-volatile memory, a volatile memory, and a controller. The controller is configured to store a first codeword and a second codeword in the volatile memory. The data payload of the first codeword includes a portion of a first logic-to-physical address table entry read from the non-volatile memory and a portion of a second logic-to-physical address table entry read from the non-volatile memory, and the data payload of the second codeword includes the remainder of a second logic-to-physical address table entry. In yet another embodiment, a method is provided comprising: storing a codeword in volatile memory, wherein the codeword includes a data portion and error correction codes generated from the data portion, wherein the data portion includes a complete logic-to-physical address table entry read from non-volatile memory and a partial logic-to-physical address table entry read from non-volatile memory; and storing at least one additional codeword in volatile memory, wherein the at least one additional codeword includes the remainder of the partial logic-to-physical address table entry. Other embodiments are provided, and each of these embodiments may be used individually or in combination.

[0014] Now turn to the attached image. Figures 1A to 1C Storage systems suitable for implementing these implementation schemes are shown. Figure 1A This is a block diagram illustrating a non-volatile storage system 100 (sometimes referred to herein as a storage device or simply a device) according to one embodiment of the subject matter described herein. Reference Figure 1A The nonvolatile memory system 100 includes a controller 102 and nonvolatile memory that may be composed of one or more nonvolatile memory dies 104-1. As described herein, the term die refers to a collection of nonvolatile memory cells formed on a single semiconductor substrate, and the associated circuitry for managing the physical operations of those nonvolatile memory cells. The controller 102 interacts with a host system and transmits sequences of commands for read, program, and erase operations to the nonvolatile memory die 104-1.

[0015] In one embodiment, controller 102 is a NAND memory controller (e.g., for an SSD). In other embodiments, controller 102 may be another type of non-volatile memory controller (e.g., flash memory, resistive random access memory (ReRAM), phase-change memory (PCM), or magnetoresistive random access memory (MRAM) controller). (It should be noted that ReRAM and MRAM are direct random access write media, and therefore they may not have a logic-to-physical address table stored in DRAM, which is relevant to the discussion below.) Additionally, controller 102 may take the form of processing circuitry, a microprocessor or processor, and a computer-readable medium storing computer-readable program code (e.g., firmware) executable by a (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Controller 102 may be configured with hardware and / or firmware to perform the various functions described below and shown in the flowcharts. Furthermore, some components shown as internal to the controller may also be stored externally, and other components may be used. Furthermore, the phrase “operationally communicating with…” may mean communicating directly or indirectly (wired or wirelessly) with one or more components, or communicating through one or more components, which may or may not be shown herein.

[0016] As used herein, a nonvolatile memory controller is a device that manages data stored on nonvolatile memory and communicates with a host device such as a computer or electronic device. In addition to the specific functions described herein, a nonvolatile memory controller can have various functions. For example, a nonvolatile memory controller can format the nonvolatile memory to ensure that the memory is operating correctly, identify bad nonvolatile memory cells, and allocate spare cells to replace future failed cells. Some portions of the spare cells can be used to house firmware to operate the nonvolatile memory controller and implement other features. In operation, when the host needs to read data from or write data to the nonvolatile memory, it can communicate with the nonvolatile memory controller. If the host provides a logical address to read / write data, the nonvolatile memory controller can translate the logical address received from the host into a physical address in the nonvolatile memory. (Alternatively, the host may provide a physical address.) The non-volatile memory controller may also perform various memory management functions, such as, but not limited to, wear leveling (allocating writes to specific memory blocks to avoid wear that would otherwise be repeatedly written to) and garbage collection (moving only valid data pages to a new block after a block is full, thus allowing the entire block to be erased and reused). Additionally, the structure used for the “means” described in the claims may include some or all of the structures of the controller described herein, such that these structures are appropriately programmed or manufactured such that the controller operates to perform the functions described herein.

[0017] The non-volatile memory die 104-1 may include any suitable non-volatile memory medium, including resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), phase-change memory (PCM), NAND flash memory cells, and / or NOR flash memory cells. The memory cells may be in the form of solid-state (e.g., flash memory) memory cells and may be programmable once, programmable multiple times, or programmable many times. The memory cells may also be single-level cells (SLC), multi-level cells (MLC), three-level cells (TLC), four-level cells (QLC), or other memory cell-level technologies now known or developed thereafter. Furthermore, the memory cells may be manufactured in a two-dimensional or three-dimensional manner.

[0018] The interface located between controller 102 and non-volatile memory die 104-1 can be any suitable flash memory interface, such as switching modes 200', 400, or 800. In one embodiment, storage system 100 can be a card-based system, such as a Secure Digital Card (SD) or a Micro Secure Digital (Micro SD) card (or USB, SSD, etc.). In an alternative embodiment, storage system 100 can be part of an embedded storage system.

[0019] Although Figure 1A In the example shown, the non-volatile memory system 100 (sometimes referred to herein as a memory module) includes a single channel between the controller 102 and the non-volatile memory die 104-1; however, the subject matter described herein is not limited to having a single memory channel. For example, in some memory system architectures (such as...) Figure 1B and Figure 1C In the architecture shown, there can be two, four, eight, or more memory channels between the controller and the memory device, depending on the controller's capabilities. In any of the embodiments described herein, even if a single channel is shown in the figures, more than one single channel can exist between the controller and the memory die.

[0020] Figure 1B A storage module 200 comprising multiple non-volatile storage systems 100 is illustrated. Similarly, the storage module 200 may include a storage controller 202 that interacts with a host and a storage system 204 comprising the multiple non-volatile storage systems 100. The interface between the storage controller 202 and the non-volatile storage systems 100 may be a bus interface, such as a Serial Advanced Technology Attachment (SATA), a Peripheral Component Rapid Interconnect (PCIe) interface, or a Double Data Rate (DDR) interface. In one embodiment, the storage module 200 may be a solid-state drive (SSD) or a non-volatile dual in-line memory module (NVDIMM), such as those found in server PCs or portable computing devices such as laptops and tablets.

[0021] Figure 1C This is a block diagram illustrating a hierarchical storage system. The hierarchical storage system 250 includes a plurality of storage controllers 202, each of which controls a corresponding storage system 204. A host system 252 can access memory within the storage system via a bus interface. In one embodiment, the bus interface may be a Non-Volatile Memory Fast (NVMe) or Fibre Channel Ethernet (FCoE) interface. In one embodiment, Figure 1C The system shown can be a rack-mountable mass storage system that can be accessed by multiple host computers, such as in a data center or other locations where mass storage is required.

[0022] Figure 2A This is a block diagram illustrating exemplary components of controller 102 in more detail. Controller 102 includes a front-end module 108 that interacts with a host, a back-end module 110 that interacts with one or more non-volatile memory dies 104-1, and various other modules that perform the functions now described in detail. Modules may take the form of, for example, packaged functional hardware units designed for use in conjunction with other components, portions of program code (e.g., software or firmware) executable by a (micro)processor or processing circuitry that typically performs a specific function of the associated functionality, or stand-alone hardware or software components that interact with a larger system. Controller 102 may sometimes be referred to herein as a NAND controller or flash memory controller, but it should be understood that controller 102 can be used with any suitable memory technology, some of which are exemplified below.

[0023] Referring again to the module of controller 102, the buffer management / bus controller 114 manages the buffer in random access memory (RAM) 116 and controls the internal bus arbitration of controller 102. Read-only memory (ROM) 118 stores the system boot code. Although Figure 2A The RAM 116 and ROM 118 are shown positioned separately from the controller 102, but in other embodiments, one or both of them may be located within the controller. In yet another embodiment, portions of the RAM and ROM may be located both inside and outside the controller 102.

[0024] Front-end module 108 includes a host interface 120 and a physical layer interface (PHY) 122 that provides an electrical interface to the host or next-level storage controller. The type of host interface 120 can be selected depending on the type of memory used. Examples of host interfaces 120 include, but are not limited to, SATA, SATA Express, Serial Attached Small Computer System Interface (SAS), Fibre Channel, Universal Serial Bus (USB), PCIe, and NVMe. Host interface 120 is typically used for transmitting data, control signals, and timing signals.

[0025] Back-end module 110 includes an error correction code (ECC) engine 124 that encodes data bytes received from the host and decodes and error-corrects data bytes read from non-volatile memory. Command sequencer 126 generates command sequences, such as programming and erasing command sequences, to be transmitted to non-volatile memory die 104-1. RAID (Redundant Array of Independent Drives) module 128 manages the generation of RAID parity and the recovery of failed data. RAID parity can be used as an additional level of integrity protection for data written to the memory device. In some cases, RAID module 128 may be part of ECC engine 124. Memory interface 130 provides command sequences to and receives status information from non-volatile memory die 104-1. In one embodiment, memory interface 130 may be a double data rate (DDR) interface, such as a switching mode 200', 400, or 800 interface. Flash control layer 132 controls the overall operation of back-end module 110.

[0026] Storage system 100 also includes other discrete components 140, such as external electrical interfaces, external RAM, resistors, capacitors, or other components that can interact with controller 102. In alternative embodiments, one or more of the physical layer interface 122, RAID module 128, media management layer 138, and buffer management / bus controller 114 are optional components that are not required in controller 102.

[0027] Figure 2BThis is a block diagram showing the components of the non-volatile memory die 104-1 in more detail. The non-volatile memory die 104-1 includes peripheral circuitry 141 and a non-volatile memory array 142. The non-volatile memory array 142 includes non-volatile memory cells for storing data. The non-volatile memory cells can be any suitable non-volatile memory cells, including ReRAM, MRAM, PCM, NAND flash memory cells, and / or NOR flash memory cells in a two-dimensional and / or three-dimensional configuration. The non-volatile memory die 104-1 also includes a data cache 156 for caching data. The peripheral circuitry 141 includes a state machine 152 that provides status information to the controller 102.

[0028] Return again Figure 2A The flash control layer 132 (which will be referred to herein as the Flash Translation Layer (FTL), or more generally as the “media management layer,” since the memory may not be flash) handles flash errors and interacts with the host. Specifically, the FTL (which may be an algorithm in the firmware) is responsible for the internals of memory management and translates writes from the host into writes to memory 104. An FTL may be necessary because memory 104 may have limited endurance, may only have a few pages written to it, and / or may not have any writes at all (unless it is erased as a block). The FTL understands these potential limitations of memory 104, which may not be visible to the host. Therefore, the FTL attempts to translate writes from the host into writes to memory 104.

[0029] The FTL may include a logical-to-physical address (L2P) mapping (sometimes referred to herein as a table or data structure) and allocated cache memory. In this way, the FTL translates logical block addresses (“LBAs”) from the host into physical addresses in memory 104. The FTL may include other features such as, but not limited to, power-off recovery (enabling the FTL’s data structures to recover in the event of a sudden power loss) and wear leveling (ensuring wear is evenly distributed across memory blocks to prevent some blocks from becoming excessively worn, which would lead to a greater chance of failure).

[0030] Please refer to the attached diagram again. Figure 3 This is a block diagram of a host 300 and a storage system (sometimes referred to herein as a device) 100 according to one embodiment. The host 300 can take any suitable form, including but not limited to a computer, mobile phone, digital camera, tablet computer, wearable device, digital video recorder, surveillance system, etc. The host 300 includes a processor 330 configured to send data (e.g., initially stored in the host's memory 340) to the storage system 100 for storage in the storage system's memory 104.

[0031] As described above, storage system 100 can store logical data blocks in non-volatile memory 104 (e.g., NAND). In operation, host 300 provides data that addresses each logical data block using logical block addresses. Because memory 104 can be lossy, controller 102 of storage system 100 can store logical data blocks at various physical locations (physical addresses) within memory 104. The mapping from logical block addresses to physical addresses is stored in a table referred to herein as a logical-to-physical address table (“L2P table”) or mapping graph. When host 300 reads a specific logical block address, controller 102 looks up the physical address in the L2P table by reading the entry corresponding to the logical block address in the table. Controller 102 then reads the logical block data from memory 104 and sends it to host 300.

[0032] The L2P table can be large, and reading the L2P table from non-volatile memory 104 each time an address translation is required can take a relatively long time and degrade the performance of storage system 100. Therefore, controller 102 can read all or some of the L2P table from non-volatile memory 104 and cache it in volatile memory (e.g., RAM 116) because reading from volatile memory is faster than reading from non-volatile memory 104.

[0033] Because volatile memories such as DRAM are relatively expensive, it may be preferable to store L2P tables as space-efficiently as possible to minimize the number of DRAM devices required in storage system 100. However, when using error correction codes (ECC) to protect data stored in volatile memory, a space-performance tradeoff may be encountered. More specifically, because volatile memory can be destructive, data stored in volatile memory may be unintentionally corrupted (e.g., by stray radiation). To prevent such data loss, the controller 102 in storage system 100 or a separate ECC engine can generate ECC bits for the data stored in volatile memory. For example, a single error-corrected double error-detected error correction code (SECDED ECC) can be used. However, the amount of data stored and the size of the ECC required to protect the data can vary depending on the configuration of storage system 100, and different configurations can lead to different space-performance tradeoffs.

[0034] The following examples illustrate different configurations and their trade-offs in storage space and performance. It should be noted that, for the sake of brevity, the term "performance" discussed herein is simplified to some extent, and performance complexities introduced, for example, by various DRAM device characteristics, are not described. It should also be noted that the following configurations are merely examples, and other configurations may be used. Therefore, unless expressly stated herein, the details of these configurations should not be considered in the claims.

[0035] These examples will be discussed in terms of “codewords.” As used herein, a “codeword” refers to a data payload and the error correction code (ECC) generated from that data payload. In these examples, the data payload includes entries in the L2P table (referred to herein as “L2P table entries” or “L2P entries”), and the ECC is generated based on the data payload (e.g., by controller 102 or a separate ECC engine). When a codeword is read from volatile memory, the ECC is used to check the integrity of the data to detect and, if possible, correct errors. Generally, higher space efficiency (i.e., a larger percentage of volatile memory used for data and ECC) typically reduces the overall cost of volatile memory. In some storage systems (e.g., enterprise SSDs), most volatile memory accesses consist of completely random L2P entry reads or writes (i.e., the volatile memory workload is primarily small random accesses).

[0036] In these examples, 32 bits of data are used for each L2P table entry, and multiple L2P table entries or multiple portions of an L2P table entry can be protected using a single ECC. Furthermore, the volatile memory has a minimum “burst size” that defines the minimum read or write transaction. In the illustration below, the burst size is 16 “bups” (e.g., the minimum read / write transaction is 16 times the width of the bus between the volatile memory and controller 102). Therefore, for example, a single burst is 80 bytes for a five-byte wide bus and 64 bytes for a four-byte wide bus.

[0037] A common configuration ( Figure 4 The codeword shown is a "4+1" codeword with a five-byte (40-bit) wide bus (i.e., a codeword with four bytes of data and one byte of ECC protecting that data). This codeword is relatively inefficient (80%) because only four of the five bytes in the codeword are used to store the data. This means that due to the invalidation of this ECC, an additional 20% of volatile memory capacity will need to be added to the storage system 100.

[0038] In this configuration, an L2P entry is encapsulated in a single codeword (because in this example, 32 bits (four bytes) of data are used for each L2P table entry). Therefore, five bytes need to be read from volatile memory to read an L2P entry, and five bytes need to be written to write an L2P entry. Thus, no read-modify-write (RMW) operation is needed because each codeword is completely overwritten when updating an L2P entry. Because the burst size is 80 bytes (16 times the width of a five-byte bus), the five-byte codeword is a single burst transaction, meaning that DRAM random access performance is relatively high.

[0039] Figure 5 Here is an illustration of another codeword for one implementation: a "128+2" codeword with a four-byte (32-bit) wide bus (i.e., a codeword with 128 bytes of data and two bytes of ECC protecting that data). In this configuration, 32 L2P entries are encapsulated in a single codeword. This codeword is relatively efficient (98.5%) because 128 of the 130 bytes in the codeword are used to store data. In contrast to the 4+1 codeword, this codeword avoids the need to add an additional 20% of volatile memory capacity to the storage system 100 due to ECC invalidation. For example, in an eight-terabyte (TB) drive, the storage system 100 could use eight gigabytes (GiB) of raw DRAM for the 128+2 codeword and 10 GiB of raw DRAM for the 4+1 codeword.

[0040] In this configuration, 130 bytes need to be read from volatile memory to read an L2P entry, and 130 bytes need to be written to write an L2P entry. However, since only four bytes change in the codeword when writing an L2P entry, each L2P entry update requires a 130-byte read for the read-modify-write operation. Given a minimum burst size of 64 bytes for volatile memory, this 130-byte read requires three burst transactions for reading and six burst transactions for writing. This means that the random access performance of volatile memory for 128+2 codewords is relatively low compared to 4+1 codewords; thus, it implies a trade-off between storage space and performance.

[0041] The following paragraphs describe a specific embodiment in which an application-specific integrated circuit (ASIC) (e.g., controller 102) is used to provide hardware assistance to modify L2P indicator table data stored in the data payload of a codeword. These embodiments can be used alone or in combination with the embodiments described below (e.g., to extend performance-enhancing solutions for high-capacity drives). It should be understood that the details presented below are merely illustrative and should not be considered in the claims unless expressly stated therein.

[0042] Currently, in high-capacity enterprise SSDs, the absolute size of the SSD drive leads to an expansion of the L2P table to address a certain amount of physical NAND. By default, enterprise SSDs use a 4-kilobyte (K) indirection size for the L2P table. Therefore, each 4-kilobyte cell in the NAND has one entry in the L2P table. For example, an 8TB SSD includes more than 8 GiB of physical NAND (due to over-provisioning in the NAND dies) and requires a 32-bit entry in the L2P table to locate a single 4-kilobyte logical page within that physical NAND address space (i.e., physical NAND greater than 8 GiB but less than 16 GiB). However, a 16TB drive would require 33-bit L2P entries, a 32TB drive would require 34-bit L2P entries, and a 64TB drive would require 35-bit L2P entries. Therefore, this requires more DRAM to maintain the L2P table, which consumes a significant portion of the DRAM capacity. The challenge is to avoid increasing the amount of DRAM required in high-capacity drives, as this increases cost and power consumption.

[0043] In some storage systems, an additional 20% DRAM is required to maintain ECC and / or extra L2P tablespace. In these systems, variable L2P entry sizes are used, where L2P entries can be 32-bit or 33-36-bit to accommodate high-capacity SSDs, and the DRAM codewords have reduced ECC size to compensate. Specifically, 32-bit entries will use 4+1 codewords with 20% ECC overhead, but 33-36-bit entries will use 9+1 codewords (each nine-byte codeword will store two 36-bit L2P entries). Therefore, even with only 10% ECC overhead, there is still a 10% overhead to maintain the extended L2P entries; hence, an overall 20% overhead. This results in 20% additional DRAM cost and power in these high-capacity drives, and also leads to the use of a 40-bit DQ (data pins) for Double Data Rate (DDR) buses, which can also add some ASIC cost (compared to a 32-bit DQ scheme with 12 fewer ASIC pins).

[0044] As another option, it features 128+2 codewords with 32-bit L2P entries and a 32-bit DQ DDR bus. Figure 6This can remove a 20% cost / power adder for drives up to eight TB. However, if such codewords do not use 32-bit L2P entries, they may not be usable for high-capacity drives such as 16-64TB drives. Another issue with 128+2 codewords is in enterprise SSD use cases, where there is a true random workload, and access to write / update L2P entries triggers additional read-modify-write operations from volatile memory. For example, when an enterprise SSD updates an L2P entry, it only reads that L2P entry to get the previous physical address (e.g., a 32-bit DRAM read, meaning the ECC engine must read the entire 128+2 codeword to get the four-byte entry from within it). It then only writes that L2P entry (e.g., a 32-bit write). However, since the DRAM codeword payload is 128 bytes, the ECC engine must read the entire 128+2 codeword, modify the four bytes within it, regenerate the two-byte ECC, and then write the entire modified 128+2 codeword. This means that a single update to an L2P entry involves two codeword reads and one codeword write. This additional read for the read-modify-write operation has a significant impact on the DRAM input-output operations per second (IOPS) limit for a 100% random write workload.

[0045] To address these issues, the following paragraphs provide implementation schemes to enhance the existing DDR 32-bit DQ inline ECC (IECC) scheme, enabling it to support enterprise high-capacity drives. The current 32-bit DQ IECC scheme uses 128+2 codewords, and the following implementation scheme (e.g., via controller 102) for adding ASIC hardware assistance modifies the L2P indirection table data stored within the 128-byte payload of the codeword. As mentioned above, as the size of each L2P entry (mapping the user's 4,000-byte LBA range to the physical NAND location) increases (e.g., for 16-64TB drives, 33 / 34 / 35-bit L2P entries are required respectively),...

[0046] It should be noted that the examples described below are for external aids to the IECC engine, but they can also be implemented within the IECC engine. Some controller ASICs currently have a hardware aid called an L2PM engine, which manages all access to the L2P table in DRAM. Therefore, whenever the firmware requests an update to a specific L2P table entry from the L2PM engine, it automatically reads the current L2P entry from DRAM and then writes the new L2P entry to DRAM (and is currently unaware of the internal DRAM codeword structure).

[0047] The following implementation schemes can be used to enhance the L2PM engine to know the 128+2 codeword structure and change the expected number of L2P entries in each 128-byte payload. For example, the changes could be as follows: 32 entries per codeword for 32-bit L2P entries, 31 entries per codeword for 33-bit L2P entries (for a 16TB drive with 4Kbyte indirection), 30 entries per codeword for 34-bit L2P entries for a 32TB drive with 4Kbyte indirection, and 29 entries per codeword for 35-bit L2P entries for a 64TB drive with 4Kbyte indirection.

[0048] To enable it, the L2PM (e.g., controller 102) can calculate which DRAM codeword includes the specified L2P entry, read the entire 128-byte codeword payload from the IECC engine, extract a single 32-35 bit entry to obtain the previous address, modify the 32-35 bit entry using the address, and write the entire 128-byte payload back to the IECC engine for writing to DRAM. Because it is writing the entire alignment codeword payload, there is no read-modify-write operation within the IECC engine, thus improving DRAM performance for enterprise workloads.

[0049] The above scheme refers to using some spare (unused) bits in DRAM codewords for 33-35 bit mode, which uses a small amount of DRAM capacity. Modeling is performed on various objects stored in DRAM to confirm that the scheme will physically "fit" within the reduced DRAM size. Figure 7 The results of this modeling are shown, and it's demonstrated that this scheme won't fit a single enterprise TB drive, but it does fit all other capacities, all the way up to a 64TB drive with 64 GiB of DRAM. Note that for the 64TB drive case, in order to make this fit, some DRAM overhead is reduced by doubling the size of the L2P table "blocks" used for logging, which halves the size of the DRAM table used to track the contents of the log on the NAND. This means a small decrease in random write performance on that drive to allow this cost-effective DRAM scheme to fit, but it's an acceptable trade-off.

[0050] Utilizing high-capacity drivers (i.e., 33-35 bit L2P entries), hardware assistance can also manage the retrieval of 32 (or 64, 128, etc.) segments of the L2P table used for L2P logging into NAND. The L2P NAND logging system stores portions of the L2P table based on the modulo 32 number of L2P entries (e.g., 32 (or 64, etc.) entries per log entry). This means that, using high-capacity drivers, a 32-entry segment storing the L2P table can each comprise two DRAM 128+2 codewords. Therefore, L2PM hardware assistance can automatically read the required DRAM codewords (i.e., read a 256-byte payload if necessary) and return only 32 L2P entries for logging. It will also perform hardware assistance in the reverse case, restoring the stored L2P table to DRAM in 32 (or 64, etc.) entry blocks at startup. L2PM hardware assistance could take, for example, a 32-entry block of the L2P table read from NAND and then write it into two DRAM codewords as needed (performing a read-modify-write operation), since it will only update 32 of the 66 (or 68 or 70) L2P entries included in the two codewords. An alternative to this hardware assistance for L2P logging or rebuilding the L2P table at boot time is for firmware to change the spacing of the L2P table blocks so they match the entries in each DRAM codeword. For example, using 33-bit entries, the L2P table block could be 31 entries; however, this could significantly increase the firmware complexity of the implementation (compared to a hardware assistance solution that could be almost transparent to the firmware).

[0051] These implementations offer numerous advantages. For example, they can save up to 20% on DRAM costs and power consumption on high-capacity drives (e.g., 16-64TB drives). This translates to significant cost savings per drive. For instance, a 32TB drive would now have 32 (instead of 40) GiB of DRAM, and at current pricing, removing 8 GiB of DRAM saves approximately $32 per drive. Furthermore, the inherent elimination of the need for additional read-modify-write operations in these implementations, along with the improved read operations for L2P entry updates, means increased available DRAM performance and the ability to achieve user random write performance targets via a single 32-bit DQ DDR bus. Figure 7 This is a table illustrating one implementation of DRAM usage in multiple scenarios.

[0052] Other implementation schemes can be used. For example, the following implementation scheme provides codewords that better optimize the balance between storage space and random access write performance. Figure 8As shown, in this implementation, a 62+2 codeword with a four-byte wide data bus is provided. In this codeword, 62 bytes of data are protected by a two-byte ECC. Here, a single codeword encapsulates 15 complete L2P entries plus half of the additional L2P entries. This results in an L2P entry being split between two consecutive codewords. Figure 8 (Entry 15 in the table). (It should be noted that in other implementations, some entries may be split between more than two codewords.) The space efficiency is relatively high (96.9%) because 62 out of 64 bytes in the codeword are used to store data. Although this is slightly less efficient than 128+2 codewords, the codewords are still fitted into the same physical volatile memory (i.e., an eight-giB eight-TB drive), so the cost is the same.

[0053] For codewords, performance is moderate. For 15 complete L2P entries encapsulated in a single codeword, 64 bytes need to be read from volatile memory to read one L2P entry, and 64 bytes need to be written to write one L2P entry. However, since only four bytes change within the codeword, each L2P entry update requires a 64-byte read for read-modify-write. For L2P entries split between two codewords, 128 bytes need to be read from volatile memory to read a single L2P entry, and 128 bytes need to be written to write one split L2P entry. However, since only two bytes change within each codeword, each split L2P entry update requires two additional 64-byte reads for read-modify-write. Therefore, performance is low for L2P entries split between two codewords.

[0054] Given the minimum burst size of volatile memory, this 64-byte codeword (with read-modify-write) for 15 complete L2P entries becomes one burst transaction for reading and two burst transactions for writing. For a split L2P entry spanning two codewords, this becomes two burst transactions for reading and four burst transactions for writing. This means that volatile memory access performance is better than 128+2 codewords. When mapped to 62+2 codewords, random write workloads typically conform to a uniform random distribution of logical block addresses. Therefore, average performance is dominated by accessing one of the 15 complete L2P entries encapsulated in a single codeword.

[0055] It should be noted that if storage system 100 first reads the original L2P entry (to mark the old logical page as invalid data overwritten for garbage collection), the additional read-modify-write overhead required when updating an L2P entry can be eliminated. When reading the entire codeword (reading an L2P entry before updating), the entire codeword can be cached in controller 102, so that when an L2P update is performed to record the new logical page location, controller 102 can modify one L2P entry in the cached codeword and then write the entire modified codeword to volatile memory. Considering that controller 102 reads and writes the entire codeword regardless, the amount of data read from and written to volatile memory remains unchanged, except that caching has eliminated the need for read-modify-write operations.

[0056] As stated above, there are several advantages to using 62+2 codewords compared to 4+1 or 128+2 codewords. For example, 62+2 codewords have a small efficiency reduction, but this is practically no different from the required physical volatile memory; therefore, the cost is the same. 62+2 codewords (without a read-modify-write cache) improve drive random write performance by 12% because they only use one burst for most L2P accesses and three bursts for 128+2 codewords. Furthermore, 62+2 codewords offer significant space efficiency benefits, meaning they use 20% less raw volatile memory space than 4+1 codewords. Additionally, when compared to 4+1 codewords, 62+2 codewords (without a read-modify-write cache) have a decrease in random access performance (due to the need for read-modify-write operations). However, when adding a read-modify-write cache, two codewords use a single burst access for most L2P accesses, but because one of the sixteen L2P accesses in two bursts is required, the 62+2 codeword has slightly lower random access performance.

[0057] Finally, as mentioned above, any suitable type of memory can be used. Semiconductor memory devices include volatile memory devices, such as dynamic random access memory (“DRAM”) or static random access memory (“SRAM”) devices, non-volatile memory devices, such as resistive random access memory (“ReRAM”), electrically erasable programmable read-only memory (“EEPROM”), flash memory (which can also be considered a subset of EEPROM), ferroelectric random access memory (“FRAM”), and magnetoresistive random access memory (“MRAM”), as well as other semiconductor elements capable of storing information. Each type of memory device can have different configurations. For example, flash memory devices can be configured in a NAND configuration or a NOR configuration.

[0058] The memory device can be formed from passive and / or active components in any combination. As a non-limiting example, a passive semiconductor memory element includes a ReRAM device element, which in some embodiments includes resistivity-switching storage elements such as antifuse, phase-change materials, etc., and optionally includes guiding elements such as diodes. As a further non-limiting example, an active semiconductor memory element includes EEPROM and flash memory device elements, which in some embodiments include elements having charge storage regions, such as floating gates, conductive nanoparticles, or charge storage dielectric materials.

[0059] Multiple memory elements can be configured such that they are connected in series or that each element can be accessed individually. By way of non-limiting example, a flash memory device (NAND memory) in a NAND configuration typically contains memory elements connected in series. A NAND memory array can be configured such that the array consists of multiple strings of memory, where a string consists of multiple memory elements that share a single bit line and are accessed as a group. Alternatively, memory elements can be configured such that each element can be accessed individually, for example, in a NOR memory array. NAND memory configurations and NOR memory configurations are examples, and memory elements can be configured in other ways.

[0060] Semiconductor memory elements located within and / or above a substrate can be arranged in two or three dimensions, such as two-dimensional or three-dimensional memory structures.

[0061] In a two-dimensional memory structure, semiconductor memory elements are arranged in a single plane or a single memory device level. Typically, in a two-dimensional memory structure, the memory elements are arranged in a plane (e.g., in the xz plane) that extends substantially parallel to the main surface of the substrate supporting the memory elements. The substrate may be a wafer on which layers of the memory elements are formed, or it may be a carrier substrate attached to the memory elements after they have been formed. As a non-limiting example, the substrate may include a semiconductor, such as silicon.

[0062] Memory elements can be arranged in a single memory device level in an ordered array (such as in multiple rows and / or columns). However, memory elements can be arranged in unconventional or non-orthogonal configurations. Each memory element may have two or more electrodes or contact lines, such as bit lines and word lines.

[0063] The three-dimensional memory array is arranged such that the memory elements occupy multiple planes or multiple memory device levels, thereby forming a three-dimensional structure (i.e., in the x, y and z directions, where the y direction is substantially perpendicular to the main surface of the substrate, and the x and z directions are substantially parallel to the main surface of the substrate).

[0064] As a non-limiting example, a three-dimensional memory structure can be vertically arranged as a stack of multiple two-dimensional memory device levels. As another non-limiting example, a three-dimensional memory array can be arranged as multiple vertical columns (e.g., columns extending substantially perpendicular to the main surface of the substrate, i.e., in the y-direction), wherein each column has multiple memory elements. The columns can be arranged in a two-dimensional configuration, for example, in the xz plane, resulting in a three-dimensional arrangement of the memory elements, where the elements are located on multiple vertically stacked memory planes. Other configurations of the three-dimensional memory elements can also constitute a three-dimensional memory array.

[0065] As a non-limiting example, in a three-dimensional NAND memory array, memory elements may be coupled together to form NAND strings within a single horizontal (e.g., xz) memory device level. Alternatively, memory elements may be coupled together to form vertical NAND strings spanning multiple horizontal memory device levels. Other three-dimensional configurations are conceivable, where some NAND strings contain memory elements within a single memory level, while others contain memory elements spanning multiple memory levels. Three-dimensional memory arrays can also be designed in NOR and ReRAM configurations.

[0066] Typically, in a monolithic three-dimensional memory array, one or more memory device classes are formed over a single substrate. Optionally, the monolithic three-dimensional memory array may also have one or more memory layers at least partially within the single substrate. As a non-limiting example, the substrate may include a semiconductor, such as silicon. In a monolithic three-dimensional array, the layer constituting each memory device class of the array is typically formed on the layer of the underlying memory device class of the array. However, the layers of adjacent memory device classes in a monolithic three-dimensional memory array may be shared or may have intervening layers between memory device classes.

[0067] Two-dimensional arrays can then be formed individually and then packaged together to form a non-monolithic memory device with multiple memory layers. For example, a non-monolithic stacked memory can be constructed by forming memory stages on separate substrates and then stacking the memory stages on top of each other. The substrates can be thinned or removed from the memory device stages before stacking, but since the memory device stages are initially formed on separate substrates, the resulting memory array is not a monolithic three-dimensional memory array. Furthermore, multiple two-dimensional or three-dimensional memory arrays (monolithic or non-monolithic) can be formed on separate chips and then packaged together to form a stacked chip memory device.

[0068] Typically, associated circuitry is required to operate and communicate with the memory element. As a non-limiting example, a memory device may have circuitry for controlling and driving the memory element to perform functions such as programming and reading. This associated circuitry may be located on the same substrate as the memory element and / or on a separate substrate. For example, a controller for memory read-write operations may be located on a separate controller chip and / or on the same substrate as the memory element.

[0069] Those skilled in the art will recognize that the present invention is not limited to the two-dimensional and three-dimensional structures described herein, but covers all relevant memory structures as described herein and as understood by those skilled in the art to be of the spirit and scope of the invention.

[0070] The foregoing detailed description is intended to be understood as an illustration of the selected forms in which the invention may take place, and not as a definition of the invention. The scope of the claimed invention is intended to be defined only by the following claims (including all equivalents). Finally, it should be noted that any aspect of any embodiment described herein may be used alone or in combination with each other.

Claims

1. A storage system, the storage system comprising: Non-volatile memory; Volatile memory; and A means for storing a first codeword and a second codeword in the volatile memory, each of the first and second codewords including a data payload and an error correction code generated from the data payload; The data payload of the first codeword includes a first logical-to-physical address table entry read from the non-volatile memory and a portion of a second logical-to-physical address table entry read from the non-volatile memory; and The data payload of the second codeword includes the remainder of the second logical-to-physical address table entry.

2. A storage system, the storage system comprising: Non-volatile memory; Volatile memory; and A controller configured to store a first codeword and a second codeword in the volatile memory, each of the first codeword and the second codeword including a data payload and an error correction code generated from the data payload; The data payload of the first codeword includes a first logical-to-physical address table entry read from the non-volatile memory and a portion of a second logical-to-physical address table entry read from the non-volatile memory; and The data payload of the second codeword includes the remainder of the second logical-to-physical address table entry.

3. The storage system of claim 2, wherein the data payload of each of the first codeword and the second codeword comprises 62 bytes, and wherein the error correction code of each of the first codeword and the second codeword comprises two bytes.

4. The storage system of claim 2, wherein each of the first codeword and the second codeword is configured to store a non-integer number of logical-to-physical address table entries.

5. The storage system of claim 4, wherein each of the first codeword and the second codeword is configured to store 15 complete logical-to-physical address table entries and half of the additional logical-to-physical address table entries.

6. The storage system of claim 2, wherein the error correction code includes a first error correction code in the first codeword and a second error correction code in the second codeword, and wherein the second logical-to-physical address table entry is partially protected by the first error correction code in the first codeword and partially protected by the second error correction code in the second codeword.

7. The storage system according to claim 2 further includes a four-byte wide bus located between the controller and the volatile memory.

8. The storage system of claim 2, wherein reading the first logical-to-physical address table entry from the volatile memory requires a single minimum read transaction, and wherein reading the second logical-to-physical address table entry from the volatile memory requires two minimum read transactions.

9. The storage system of claim 2, wherein updating the first logical-to-physical address table entry in the volatile memory requires two minimum write transactions, and wherein updating the second logical-to-physical address table entry in the volatile memory requires four minimum write transactions.

10. The storage system of claim 9, wherein the four minimal write transactions comprise a read-modify-write operation, and wherein updating the second logical-to-physical address table entry in the volatile memory requires only two minimal write transactions in response to using the read-modify-write cache.

11. The storage system of claim 2, wherein the controller is further configured to change the number of logical-to-physical address table entries in the data payload of the first codeword.

12. The storage system of claim 11, wherein the controller is further configured to change the number of logical-to-physical address table entries by: The first codeword is determined to include a specified logical-to-physical address table entry; Extract the specified logic into a physical address table entry from the data payload of the first codeword to identify the previous address; and Modify the specified logical-to-physical address table entry using the previous address.

13. The storage system of claim 2, wherein the non-volatile memory includes a three-dimensional memory.

14. In a storage system comprising non-volatile memory and volatile memory, a method includes: The codeword is stored in the volatile memory, wherein the codeword includes a data portion and an error correction code generated from the data portion, wherein the data portion includes a complete logic-to-physical address table entry read from the non-volatile memory and a partial logic-to-physical address table entry read from the non-volatile memory; and At least one additional codeword is stored in the volatile memory, wherein the at least one additional codeword comprises the remainder of the partial logical-to-physical address table entry.

15. The method of claim 14, wherein the at least one additional codeword includes at least one additional error correction code, wherein the data payload of each codeword includes 62 bytes, and wherein the error correction code of each codeword includes two bytes.

16. The method of claim 14, wherein each codeword is configured to store 15 complete logical-to-physical address table entries and half of the additional logical-to-physical address table entries.

17. The method of claim 14, wherein reading the complete logic-to-physical address table entry from the volatile memory requires a single minimum read transaction, and wherein reading a logic-to-physical address table entry segmented between codewords requires two minimum read transactions.

18. The method of claim 14, wherein updating the complete logical-to-physical address table entry in the volatile memory requires two minimum write transactions, and wherein updating a logical-to-physical address table entry segmented between codewords requires four minimum write transactions without a read-modify-write cache or two minimum write transactions with a read-modify-write cache.

19. The method of claim 14, further comprising changing the number of logical-to-physical address table entries in the data payload of the codeword.

20. The method of claim 19, wherein the number of logical-to-physical address table entries is changed in the following way: The codeword is determined to include a specified logical-to-physical address table entry; Extract the specified logic into a physical address table entry from the data payload of the codeword to identify the previous address; and Modify the specified logical-to-physical address table entry using the previous address.

Images