Storage systems and methods for implementing encoders, decoders, and / or buffers using field programmable gate arrays

Abstract

Provided herein are storage systems and methods for implementing encoders, decoders, and / or buffers using field programmable gate arrays. In one embodiment, a storage system has a field programmable gate array and a memory storing a set of instruction codes for the field programmable gate array. The set of instruction codes can be used for different error decoder implementations, for providing additional encoders and / or decoders, or for implementing a host memory buffer or a controller memory buffer.

Description

BACKGROUND

[0001] A storage system can be configured with an encoder for encoding error correction code bits for data to be stored in a memory of the storage system, and a decoder for decoding error correction code bits for data read from the memory. Some storage systems implement a controller memory buffer or a host memory buffer. BRIEF DESCRIPTION OF DRAWINGS

[0002] Figure 1A Block diagram of a non-volatile storage system, according to an embodiment.

[0003] Figure 1B Block diagram of a storage module, according to an embodiment.

[0004] Figure 1C Block diagram of a hierarchical storage system, according to an embodiment.

[0005] Figure 2 Block diagram of a non-volatile storage system, according to an embodiment. Figure 1A Block diagram of components of a controller of a non-volatile storage system, according to an embodiment.

[0006] Figure 3 Block diagram of components of a non-volatile storage system, according to an embodiment. Figure 1A Block diagram of components of a non-volatile storage system, according to an embodiment.

[0007] Figure 4 Block diagram of an example of a system-on-chip (SoC) embedded field programmable gate array (eFPGA) system.

[0008] Figure 5 Block diagram of an example use case of an eFPGA.

[0009] Figures 6A to 6D Block diagram of an adaptive storage error correction code (ECC) decoder hardware design, according to an embodiment implemented by an eFPGA.

[0010] Figure 7 Flowchart of a method for adapting a storage ECC decoder for different storage life states, according to an embodiment.

[0011] Figure 8 Block diagram of a storage system, according to an embodiment.

[0012] Figure 9 Block diagram of a storage system, according to an embodiment, having fixed partitioning of error correction code (ECC) encoder and decoder cores.

[0013] FIG. 10A is a block diagram of a prior art ECC core.

[0014] Figure 10B and 10CA block diagram of an ECC core for an embodiment.

[0015] Figure 11 A flowchart of a method for an embodiment for adaptive encoding and decoding a channel.

[0016] Figure 12 A diagram of deterministic and non-deterministic windows for an embodiment.

[0017] Figure 13 A block diagram of a storage system for an embodiment.

[0018] Figure 14 A flowchart of a method for an embodiment for providing dynamic host memory buffer (HMB)-controller memory buffer (CMB) support for a storage controller.

[0019] Figure 15 A block diagram of a storage system for an embodiment. DETAILED DESCRIPTION

[0020] By way of introduction, the following embodiments relate to a storage system and method for implementing an encoder, decoder, and / or buffer using a field programmable gate array. In one embodiment, a storage system has a field programmable gate array and a memory storing a set of instruction codes for the field programmable gate array. The set of instruction codes can be used for different error decoder implementations, for providing additional encoders and / or decoders, and / or for implementing a host memory buffer or a controller memory buffer. Other embodiments are possible, and each of the embodiments can be used alone or in combination with one another. Accordingly, various embodiments will now be described with reference to the drawings.

[0021] Turning now to the drawings, a storage system suitable for implementing aspects of these embodiments is shown in Figures 1A to 1C . Figure 1A A block diagram of a non-volatile storage system 100 (sometimes referred to herein as a storage device or just a device) to illustrate embodiments in accordance with the subject matter described herein. With reference to Figure 1A , the non-volatile storage system 100 includes a controller 102 and a non-volatile memory, which can be composed of one or more non-volatile memory dies 104. As used herein, the term “die” refers to a collection of non-volatile memory cells formed on a single semiconductor substrate, and associated circuitry for managing the physical operations of those non-volatile memory cells. The controller 102 interfaces with a host system and transmits sequences of commands for read, program, and erase operations to the non-volatile memory dies 104.

[0022] The controller 102, which can be a non-volatile memory controller (e.g., a flash, resistive random access memory (ReRAM), phase change memory (PCM), or magnetoresistive random access memory (MRAM) controller), can take the form of processing circuitry, a microprocessor or processor, and a computer-readable medium storing computer-readable program code (e.g., firmware) executable by, for example, the (micro)processor, logic gates, switches, an application-specific integrated circuit (ASIC), programmable logic controller, and embedded microcontrollers. The controller 102 can be configured with hardware and / or firmware to perform the various functions described below and shown in the flowcharts. Also, some components shown as being internal to the controller can also be stored external to the controller, and other components can be used. Additionally, the phrase "in operative communication with" can mean direct communication, or indirect (wired or wireless) communication through one or more components that can or can not be shown or described herein.

[0023] As used herein, a non-volatile memory controller is a device that manages data stored on non-volatile memory and communicates with a host, such as a computer or electronic device. The non-volatile memory controller can have various functionality in addition to the specific functionality described herein. For example, the non-volatile memory controller can format the non-volatile memory to ensure that the memory operates properly, map out bad non-volatile memory cells, and allocate spare cells to replace future failed cells. Some portion of the spare cells can be used to hold firmware to operate the non-volatile memory controller and implement other features. In operation, when a host needs to read data from or write data to the non-volatile memory, the host can communicate with the non-volatile memory controller. If the host provides a logical address to which the data is to be read / written, the non-volatile memory controller can convert the logical address received from the host to a physical address in the non-volatile memory. (Alternatively, the host can provide a physical address.) The non-volatile memory controller can also perform various memory management functions, such as, but not limited to, wear leveling (spreading writes to avoid wearing out particular memory blocks that would otherwise be written repeatedly) and garbage collection (after a block is full, only valid pages of data are moved to a new block, so the full block can be erased and reused). Further, the structure of "means" recited in the claims can include some or all of the structure of a controller as described herein, programmed or manufactured as necessary to cause the controller to operate to perform the recited function.

[0024] The non-volatile memory die 104 can include any suitable non-volatile storage medium, including resistive random access memory (ReRAM), magnetic random access memory (MRAM), phase change memory (PCM), NAND flash memory cells, and / or NOR flash memory cells. The memory cells can be in the form of solid state (e.g., flash) memory cells, and can be single program, few program, or multi-programmable. The memory cells can also be single level cells (SLC), multi-level cells (MLC), triple level cells (TLC), or use other memory cell level technology known today or developed in the future. Furthermore, the memory cells can be fabricated in two-dimensional or three-dimensional fashion.

[0025] The interface between the controller 102 and the non-volatile memory die 104 can be any suitable flash interface, such as Toggle Mode 200, 400, or 800. In one embodiment, the storage system 100 can be a card-based system, such as a Secure Digital (SD) or micro Secure Digital (microSD) card. In alternative embodiments, the storage system 100 can be part of an embedded storage system.

[0026] While in the example shown in Figure 1A the non-volatile storage system 100 (sometimes referred to herein as a storage module) includes a single channel between the controller 102 and the non-volatile memory die 104, the subject matter described herein is not limited to having a single memory channel. For example, in some storage system architectures, such as those shown in Figure 1B and Figure 1C depending on controller capabilities, there can be 2, 4, 8, or more memory channels between the controller and the memory device. In any of the embodiments described herein, there can be more than a single channel between the controller and the memory die, even if a single channel is shown in the figures.

[0027] Figure 1B A storage module 200 is shown that includes multiple non-volatile storage systems 100. As such, the storage module 200 can include a storage controller 202 that interfaces with a host and with a storage system 204 that includes multiple non-volatile storage systems 100. The interface between the storage controller 202 and the non-volatile storage systems 100 can be a bus interface, such as a Serial Advanced Technology Attachment (SATA), Peripheral Component Interconnect Express (PCIe) interface, or Double Data Rate (DDR) interface. In one embodiment, the storage module 200 can be a solid state drive (SSD) or a non-volatile dual in-line memory module (NVDIMM), such as found in server PCs or portable computing devices such as laptops and tablets.

[0028] Figure 1C To illustrate a block diagram of a tiered storage system. The tiered storage system 250 includes a plurality of storage controllers 202, each of which controls a respective storage system 204. A host system 252 can access memory within the storage system via a bus interface. In one embodiment, the bus interface can be a Non-Volatile Memory Express (NVMe) interface or a Fibre Channel over Ethernet (FCoE) interface. In one embodiment, Figure 1C The system shown in FIG. 1 can be a rack-mountable mass storage system that is accessible by multiple hosts, such as would be found in a data center or other location where mass storage is needed.

[0029] Figure 2 To illustrate a block diagram of the components of the controller 102 in more detail. The controller 102 includes a front-end module 108 that interfaces with a host, a back-end module 110 that interfaces with one or more non-volatile memory dies 104, and various other modules that perform functions that will now be described in detail. Modules may, for example, be in the form of a packaged functional hardware unit designed for use with other components, a portion of program code (e.g., software or firmware) that can be executed by a (micro)processor or processing circuitry that is of a particular function that is generally performed by the processor or processing circuitry, or an autonomous hardware or software component that interfaces with a larger system. The modules of the controller 102 can include an embedded field programmable gate array (eFPGA) 111 discussed in more detail below. In some embodiments, the ECC module 124 is at least partially embedded in the eFPGA 111.

[0030] Referring again to the modules of the controller 102, a buffer manager / bus controller 114 manages buffers in random access memory (RAM) 116 and controls internal bus arbitration of the controller 102. A read-only memory (ROM) 118 stores system boot code. While shown as being located apart from the controller 102 in FIG. 1, in other embodiments, one or both of the RAM 116 and the ROM 118 can be located within the controller. In still other embodiments, portions of the RAM and the ROM can be both located within the controller 102 and located external to the controller. Figure 2

[0031] The front-end module 108 includes a host interface 120 and a physical layer interface (PHY) 122 that provide an electrical interface with a host or next level storage controller. The choice of type of host interface 120 can depend on the type of memory being used. Examples of host interfaces 120 include, but are not limited to, SATA, SATA Express, Serial Attached Small Computer System Interface (SAS), Fibre Channel, Universal Serial Bus (USB), PCIe, and NVMe. The host interface 120 generally facilitates the transfer of data, control signals, and timing signals. ​

[0032] The back end module 110 includes an error correction code (ECC) engine 124 that encodes data bytes received from the host and decodes and error corrects data bytes read from the non-volatile memory. A command sequencer 126 generates command sequences, such as program and erase command sequences, to be transmitted to the non-volatile memory die 104. A redundant array of independent drive (RAID) module 128 manages the generation of RAID parity and the recovery of failed data. The RAID parity can be used as an additional level of integrity protection for data being written into the memory device 104. In some cases, the RAID module 128 can be part of the ECC engine 124. A memory interface 130 provides command sequences to the non-volatile memory die 104 and receives status information from the non-volatile memory die 104. In one embodiment, the memory interface 130 can be a double data rate (DDR) interface, such as a Toggle Mode 200, 400, or 800 interface. A flash control layer 132 controls the overall operation of the back end module 110.

[0033] The storage system 100 also includes other discrete components 140, such as external electrical interfaces, external RAM, resistors, capacitors, or other components that can interface with the controller 102. In alternative embodiments, one or more of the physical layer interface 122, the RAID module 128, the media management layer 138, and the buffer management / bus controller 114 are optional components that are not necessarily in the controller 102.

[0034] Figure 3 A block diagram of components of the non-volatile memory die 104 is shown in more detail. The non-volatile memory die 104 includes peripheral circuitry 141 and a non-volatile memory array 142. The non-volatile memory array 142 includes non-volatile memory cells for storing data. The non-volatile memory cells can be any suitable non-volatile memory cells, including ReRAM, MRAM, PCM, NAND flash memory cells, and / or NOR flash memory cells in two-dimensional and / or three-dimensional configurations. The non-volatile memory die 104 further includes a data cache 156 that caches data. The peripheral circuitry 141 includes a state machine 152 that provides status information to the controller 102.

[0035] Returning again to Figure 2The flash control layer 132 (which will be referred to herein as a flash translation layer (FTL), or, for example, more generally, a "media management layer") handles flash errors and interfaces with the host. In particular, the FTL, which can be an algorithm in firmware, is responsible for memory internal management and converts writes from the host to writes to the memory 104. The FTL can be needed because the memory 104 can have limited endurance, can only be written in multiples of pages, and / or can not be written unless it is erased as a block. The FTL is aware of these potential limitations of the memory 104, which can not be visible to the host. Thus, the FTL attempts to convert writes from the host to writes in the memory 104.

[0036] The FTL can include a logical to physical address (L2P) map (sometimes referred to herein as a table or data structure) and an allocated cache. In this way, the FTL converts logical block addresses ("LBAs") from the host to physical addresses in the memory 104. The FTL can include other features, such as, but not limited to, power fail recovery (to enable recovery of the FTL's data structures in the event of a sudden power failure) and wear leveling (to cause wear on memory blocks to be more evenly distributed to prevent certain blocks from wearing out excessively, which would lead to a greater probability of failure).

[0037] Modern storage systems often include iterative soft decoders, such as low density parity check (LDPC) error correction code (ECC) engines, which often include several internal hardware kernels to support different bit error rate (BER) levels of the stored data. For example, a storage system can include three decoding engines (sometimes referred to herein as gears or algorithms) within an LDPC hardware module: a first engine targeted to provide a first level of correction capability; a second engine targeted to provide a second level of additional correction capability; and a third engine targeted to provide a third level of even more correction capability.

[0038] The different decoding kernels of an LDPC ECC decoder have a tradeoff between correction capability on one hand and consumed operational power and decoding duration on the other hand. The frequency of use of the different ECC kernels changes throughout the lifetime of the storage device (e.g., as measured by program / erase count). As storage conditions degrade during the lifetime of the device (e.g., as evidenced by an increase in BER), the use of the different decoder kernels also degrades.

[0039] As the throughput requirements of storage systems increase, the use of multiple decoding engines increases. For example, some consumer solid state drives (cSSDs) have various numbers of first, second, and third engines, while some enterprise solid state drives (eSSDs) have other various numbers of first, second, and third engines. Next generation storage systems can have even further variations.

[0040] The gain in cost and power consumption with the multiple decoder approach is considerable. Furthermore, the probability of success of each decoding gear can vary with the BER. In this way, different BERs can indicate the use of different decoding gears. For example, at the beginning of life, the first engine can be appropriate, but as the memory evolves, the BER increases and therefore the percentage of use of the second and third engines also increases.

[0041] To address this situation, in one embodiment, an adaptive memory decoder hardware design can be based on an embedded field programmable gate array (eFPGA) implementation. Generally, eFPGAs integrate FPGAs as part of an application specific integrated circuit (ASIC) system on a chip (SoC). eFPGA technology offers the flexible hardware programmability of FPGAs with the performance and cost benefits of ASIC SoCs. While an embedded FPGA is used in some of these examples, it should be understood that a non-embedded FPGA can be used.

[0042] Figure 4 is a block diagram of an example of a SoC eFPGA system 400. As shown in Figure 4 The system 400 includes two central processing unit (CPU) clusters 405, 410, an eFPGA 415, three level 2 (L2) cache memories 420, 425, 430, a cache coherent interconnect layer 435, a secure bridge 440, a double data rate (DDR) controller 445, a peripheral component interconnect express (PCIe) interface 450, and an Ethernet interface 455. This SoC 400 can be used with programmable acceleration.

[0043] FIG. 6 is a block diagram of example use cases of an eFPGA. This diagram demonstrates the hardware programmability flexibility of eFPGA systems that allow the FPGA component inside a SoC to be programmed to different hardware functionality. For example, in one use case 510, the eFPGA system is used as an accelerator and provides flexible input / output (I / O) functionality. In another use case 520, the eFPGA system is used as the voting logic of a dual-core processor that runs in lockstep. In yet another use case 530, the eFPGA system is used to provide parallel I / O processors that run different workloads.

[0044] The main advantage of employing FPGAs on a SoC is related to the flexibility of the FPGA while still maintaining the performance of the SoC. The direct advantage of employing FPGAs is related to the reduction of time to market (TTM) and flexible hardware vulnerability support.

[0045] Returning to the discussion of the decoding engine, the second and third decoding modes are rarely used for fresh storage devices and are practically redundant at the beginning of the life of the memory. However, when the traditional ASIC hardware is fixed, the LDPC decoding solution can be sized according to all conditions: a first decoder for the beginning of life (BOL) and a second / third decoder for the end of life (EOL). Depending on the product requirements, either both sets of decoders can be needed, or a compromise is made sacrificing power efficiency at BOL (less first decoders) or performance at EOL (less third decoders). If the hardware design can be dynamically adapted according to the typical BER (or estimate of the BER (e.g., syndrome weight) corresponding to the age of the device, the storage device performance can be significantly accelerated. However, current storage systems use a fixed hardware design of the LDPC ECC decoder, which includes the same hardware modules (e.g., different decoding kernels) to support the worst-case conditions of both BOL and EOL in terms of performance and correction capability.

[0046] In one embodiment, a dynamic adaptive storage decoder hardware design is provided with an embedded FPGA SoC implementation. This embodiment utilizes the eFPGA capability in a unique way. As mentioned above, the common use case for eFPGA aims to shorten time to market as well as improve hardware vulnerability support capabilities. However, this embodiment can be used to incorporate a different concept that utilizes the eFPGA to provide a flexible hardware design along the device life (even in the steady-state development phase of the product). In one embodiment, the FPGA portion of the SoC can be configured to implement different combinations of decoding kernels according to the age of the device (e.g., as measured by the level of program / erase cycles) to adjust the decoder characteristics to the memory characteristics to achieve optimized performance and power consumption.

[0047] In one embodiment, a flexible hardware design of the LDPC ECC module is provided by the eFPGA-based adaptive configuration of the different decoding mode sub-engines. The flexible hardware design of the storage device LDPC kernels can be provided based on the typical BER level (according to the device life stage).

[0048] As mentioned above, the expensive FP-decoding mode (i.e., high power consumption with long decoding duration) is rarely used for fresh storage devices. However, because the traditional ASIC SoC hardware design is fixed, the LDPC decoding kernels must include all decoding modules because it needs to operate well with the same hardware at the end of life (EOL) as well. In contrast, this embodiment defines several hardware mode configurations that accommodate different BER levels according to the age of the storage device, such that there are more first decoding kernels at the beginning of life, more second decoding kernels at the middle of life, and more third decoding kernels at the end of life.

[0049] For each BER level, the engines and their parallelism can be defined to optimize performance and power according to the expected BER and memory conditions. By adjusting the decoder architecture, the number of each engine type, and the parallelism of each engine to memory conditions, this embodiment can optimize decoder performance and power consumption. In other words, this optimization can result in higher performance and lower power compared to non-adaptive systems.

[0050] Figures 6A to 6D A block diagram of an adaptive error correction code (ECC) decoder hardware design for embodiments implemented by eFPGAs. These figures will be discussed in conjunction with Figure 7 The figures show transitions in eFPGA hardware design between different stages of memory life. As shown in Figure 6A In this embodiment, the eFPGA 610 is programmed to implement one or more additional decoders based on the life stage of the memory system 100.

[0051] For example, as shown in Figure 6B and shown in act 710 in Figure 7 At the beginning of life, the module is programmed as "design mode A" where the eFPGA 610 is programmed to implement three additional first decoders 611, 612, and 613. This is a super-fast decoding hardware mode that speeds up decoding parallelism at the beginning of life. In this mode, the controller 102 executes host commands (act 720) and monitors the aging of the memory system 100 to see if a change in decoding mode is needed. For example, as shown in act 730 in Figure 7 If so, it indicates that the memory has aged into the middle of life where a different error protection scheme will likely be appropriate. In this case, the controller 102 implements design mode B (act 740). As shown in Figure 6C In design mode B, the eFPGA 610 implements two second decoders 614, 615 because using second decoding as a decoding gear is appropriate for the middle of life.

[0052] The controller 102 continues to execute commands (act 750) and determines whether the memory has aged into the end of life (e.g., by determining whether the PEC or BER is greater than a second threshold) (act 760). When the memory is at the end of life, the controller 102 sets the hardware decoder to design mode C (act 770) where the eFPGA 610 implements a third decoder 616 (see Figure 6D), and the controller 102 executes host commands in this mode (act 780).

[0053] Figure 8 A block diagram of a storage system 100 for embodiments involving these features. As shown in Figure 8 The storage system 100 includes a controller 102 and a memory (here, NAND) 104. The controller 102 includes an error correction eFPGA module 800, which can be in the form of a module 620 similar to that in Figure 6A The eFPGA module 800 contains an eFPGA that can be programmed by a processor 810 with instructions read from an eFPGA configuration area / file 805 in the memory 104. For example, the eFPGA configuration area / file 805 in the memory 104 can store three different sets of instruction code to implement the design modes A-C discussed above. In operation, based on the average BER or PEC of the memory 104, the processor 810 will issue a configuration read command to the memory 104 via the flash interface module 110 to get the appropriate set of instruction code for the configuration desired, and then dynamically configure the eFPGA module 800 with the read instruction code. The other components of the controller 102 (e.g., command path 820, command scheduler 830, other processor 840, control path 850, direct memory access (DMA) module 860, and data transfer scheduler 870) will operate as usual.

[0054] In another embodiment, the parallelism of each ECC engine varies with the average BER. For example, when there is the best BER, the parallelism of each ECC engine is lower, but there are many ECC engines. After there is a worse BER, the controller 102 can reduce the number of ECC engines, but increase the parallelism in each of them.

[0055] There are several advantages associated with these embodiments. For example, adapting the decoder architecture to the memory conditions during the lifetime of the product can help ensure optimal performance and power consumption. As another example, reconfiguring the eFPGA decoder for modules used only during exceptions / maintenance can help reduce the cost of the controller 102 and / or reduce the additional cost of the eFPGA.

[0056] In another embodiment, the eFPGA can be used for adaptive encoding and decoding channels. An LDPC ECC engine can include different channels for encoding and decoding. The encoding channel operates during the write path, where the parity redundancy data is computed, while the decoding channel operates during the read path, where an iterative ECC scheme is operated using specified decoder hardware to correct bits that were flipped in error (e.g., by implementing a message passing algorithm such as belief propagation).

[0057] Because the encoding and decoding operations are strictly different from each other (e.g., encoding is a deterministic operation by nature, while decoding is random), it is common for the involved hardware implementations to produce different hardware kernels for the encoder and the decoder. However, to enhance the performance of decoding and / or encoding, an effective option is to add several copies of each of the encoder / decoder kernels in order to allow parallel processing of larger data blocks. This is shown in the block diagram 900 in Figure 9

[0058] Figure 9 A fixed partitioning of the ECC encoder and decoder kernels is shown, which is referred to as a balanced system because it has the same number of encoders and decoders. However, it can be beneficial to change the nominal balance because (1) product requirements can change, with some being more write-intensive and some being more read-intensive while using the same controller, and (2) there can be a large asymmetry in NAND program / read times depending on the number of dies, so different configurations of the same product can require different partitioning of the encoder / decoder kernels.

[0059] Since the convention to implement the encoder and decoder channels is to have a fixed hardware design with one or more encoding kernels and one or more decoding kernels, the hardware capabilities of such ECC engines are limited by the available designs of the decoding / encoding partitioning. Such inflexibility means that the system is not able to adapt its performance according to the specific host workloads and / or according to the required current performance.

[0060] For example, if a write-intensive application is regularly operating on a storage device (e.g., for an event data recorder log), the performance of the device, which was planned for a general workload, will be limited by the write path. And if the same storage device is operating in a read-intensive mode (e.g., for three-dimensional and / or high-definition maps in memory for autonomous vehicles), it will be limited by the read path. Since the memory can perform only one operation (program or sense) at a time, having a fixed partitioning of the encoding / decoding kernels (even using multiple dies, but it is difficult and sometimes impossible to maintain simultaneous read / write) is inefficient. It can be desirable to allow the ECC engine to have the ability to adaptively change the allocation of its internal hardware resources between the decoding and encoding channels according to the specific host / application workloads and / or according to the current versus the required read / write performance.

[0061] ​As mentioned above, the aforementioned approach for the hardware design of the encoding / decoding channels of the ECC engine for a storage device is a fixed partitioning of one or more encoding kernels and one or more decoding kernels, with no ability to adapt between the encoding / decoding channels according to a particular host workload and / or according to a required current performance. In contrast, in one embodiment, the ECC engine uses eFPGA technology to have a flexible partitioning of the hardware resources between the encoding and decoding channels in an optimized manner according to the current storage device performance and the particular user typical workload. This embodiment further includes tracking the statistical performance of the memory in order to adaptively prioritize the encoding / decoding resource allocation and optionally also tracking the particular host / application typical workload for the same purpose. In one embodiment, the adaptability of the encoding / decoding channels is allowed by the embedded FPGA module.

[0062] Returning to the figures, FIG. 10A is a block diagram of a prior art ECC kernel design 1. As shown in FIG. 10A, this design 1 includes a single encoder 2, a single decoder 3, and an I / O interface 4. In contrast to this fixed design, the ECC module of this embodiment provides adaptive encoding and decoding hardware channels. For example, as shown in Figure 10B and 10C The ECC module 1000 of this embodiment includes an encoder 1010, a decoder 1020, a storage device performance statistic logger 1030, a host typical workload tracking module 1040, an I / O interface, and an embedded FPGA 1060. The embedded FPGA 1060 can be programmed to implement additional encoders and / or decoders. For example, in Figure 10B the embedded FPGA 1060 is programmed to implement two additional decoders 1065 (which is a design for enhanced decoding), while in Figure 10C the embedded FPGA 1060 is programmed to implement two additional encoders 1075, 1080 (which is a design for enhanced encoding).

[0063] In one embodiment, there are two methods for adapting the eFPGA module for encoding or decoding intensive workloads: offline adaptation and online adaptation. In offline adaptation, the partitioning of the encoding / decoding is adapted according to the product requirements (e.g., a particular product identifier (stock keeping unit (SKU)), configuration, and number of connected dies. This offline adaptation of the eFPGA module for decoding or encoding intensive workloads can be performed during initialization after power on reset (POR).

[0064] In contrast, online adaptation can be used to dynamically change the configuration over time. Figure 11 A flowchart 1100 of the method for the embodiment for online adaptation. As Figure 11As shown in the middle, after receiving configuration information from the host (act 1110), the read and write performance and quality of service (QoS) of the storage device are tracked (act 1120). Other metrics can be tracked, such as but not limited to, type of application, type of configuration, type of virtual host, queue depth, power condition, and temperature condition. The read / write performance is then compared to relevant baseline performance values (act 1130). These values can be predefined or updated as input from the user during the lifetime of the device. This comparison checkpoint can be initiated at predefined time intervals according to user notification, or once an anomaly in acceptable performance criteria is identified.

[0065] To make the hardware configuration decision, the controller 102 can compare the importance level of the read and write channels to relevant thresholds (act 1140) and set the corresponding configuration of the eFPGA module in the controller 102 accordingly (act 1150). In one embodiment, the memory 104 stores a plurality of sets of predefined eFPGA configuration options, and a set of “if-else” conditions are pre-associated to decide the appropriate hardware configuration mode. The validity of the selected eFPGA hardware configuration can be tracked and optionally reported to the host when needed, or otherwise used to update the decision rules of the previous steps (act 1160).

[0066] In one embodiment, the eFPGA concept can be used to address burst operations. For example, typically, a system can incorporate ten decoders and three encoders. However, if the device detects a burst write operation that requires additional encoder parallelism, the number of decoders can be dynamically reduced while increasing the number of encoders and having performance / QoS improvement for this burst operation. The storage device can detect burst operations in various ways, such as but not limited to, online detection of many operations (e.g., read / write commands) with the same attributes, prediction of burst operations based on previous history or machine learning, host hints, and using predictable latency patterns.

[0067] With respect to the predictable latency mode, in some systems, the workload is predictable over time, and the predictable latency mode is used to achieve predictable latency for both read and write operations in the NVMe device. When the NVMe device is configured to operate in this mode, the namespace in the NVM set provides distinct separate operational windows for deterministic operations and for non-deterministic operations. The deterministic window (DTWIN) is the operational window during which the NVM set is able to provide deterministic latency for read and write operations. The non-deterministic window (NDWIN) is the operational window during which the NVM set is not able to provide deterministic latency for read and write operations due to preparation for a subsequent deterministic window. Example actions that can be performed in the non-deterministic window include background operations on the non-volatile media. The current window in which the NVM set is operating is configured by the host using the predictable latency mode window feature or by the controller 102 due to autonomous actions. Figure 12 The deterministic and non-deterministic windows are depicted as periodically switching. The motivation is to increase the deterministic window duration and decrease the non-deterministic window duration as needed. In one embodiment, more decoders are implemented during the DTWIN since the QoS and performance of read operations is critical. On the other hand, more encoder engines are implemented during the NDWIN since more write operations are issued.

[0068] Figure 13 Block diagram of a storage system 100 for embodiments that can be used with the above features. As Figure 13 shown in FIG. 1 1, the storage system 100 includes a controller 102 and a memory (here, NAND) 104. The controller 102 includes an ECC engine 1300 with an eFPGA that can be in the form of the module 1000 in FIGS. 1 and 2. The eFPGA in the ECC engine 1300 can be programmed by a processor 1310 with instructions read from an eFPGA configuration area / file 1305 in the memory 104. For example, the eFPGA configuration area / file 1305 in the memory 104 can store three different sets of instruction code to implement different ECC schemes. In operation, based on the expected workload, the processor 1310 issues a configuration read command to the memory 104 via the flash interface module 1 10 to fetch the appropriate set of instruction code for the desired configuration, and then dynamically reconfigures the eFPGA in the ECC engine 1300 with the read instruction code. The other components of the controller 102 (e.g., command path 1320, command scheduler 1330, other processor 1340, control path 1350, direct memory access (DMA) module 1360, and data transfer scheduler 1370) will operate as usual. Figure 10B and 10C The eFPGA in the ECC engine 1300 can be programmed by a processor 1310 with instructions read from an eFPGA configuration area / file 1305 in the memory 104. For example, the eFPGA configuration area / file 1305 in the memory 104 can store three different sets of instruction code to implement different ECC schemes. In operation, based on the expected workload, the processor 1310 issues a configuration read command to the memory 104 via the flash interface module 1 10 to fetch the appropriate set of instruction code for the desired configuration, and then dynamically reconfigures the eFPGA in the ECC engine 1300 with the read instruction code. The other components of the controller 102 (e.g., command path 1320, command scheduler 1330, other processor 1340, control path 1350, direct memory access (DMA) module 1360, and data transfer scheduler 1370) will operate as usual.

[0069] There are many advantages associated with these embodiments. For example, the adaptive system described above allows better utilization of available resources, requiring less overall resources for the same performance goals. Also, adaptive systems can achieve higher performance and QoS compared to non-adaptive systems. The solution is more efficient and suitable for current workloads and bursty operations. Furthermore, these embodiments allow flexibility in design (e.g., to support products that require different read / write patterns after tape-out).

[0070] In another embodiment, an embedded FPGA is used to provide dynamic host memory buffer (HMB)-controller memory buffer (CMB) support to the controller 102, which can be used in a non-volatile memory express (NVMe) environment. NVMe is a pair-based submission and completion queue mechanism. Commands are placed into submission queues by host software, and completions are placed into associated completion queues by the controller 102. Generally, submission and completion queues are allocated in host memory, and each queue can be physically located contiguously or non-contiguously in host memory.

[0071] Modern high performance SSD controllers, such as NVMe controllers, use a large amount of RAM, and there is typically a 1 GB of RAM per 1 TB of flash ratio. Controllers are typically conservative in using RAM as cache for user data to limit the damage of sudden power loss, and the RAM is instead used to store organization metadata necessary for the controller to track where what data is stored on the flash chips. The goal is that when the drive receives a read or write request, it can determine which flash memory location needs to be accessed based on a much faster lookup in the controller’s DRAM (compared to flash), and the drive does not need to update a copy of the metadata stored on the flash after every single write operation is completed. For fast, consistent performance, the data structure can be chosen to minimize the amount of computation and the number of RAM lookups needed, at the cost of requiring more RAM. As a common cost reduction, instead of adding on-board DRAM to the device, RAM can be allocated from a shared pool on the host. This feature is known as host memory buffer (HMB) in PCIe / NVMe devices, and utilizes the PCIe protocol to allow direct access to the host memory.

[0072] Typically, HMB is allocated as a fixed buffer during host initialization and left entirely to the device as a dedicated pool. The device can then directly access this buffer and treat it as an extension of its own memory. The HMB feature is extremely common in low-end client SSD applications, but is not common in other client and enterprise markets. On the other hand, the controller memory buffer (CMB) and persistent memory region (PMR) features are common in data center and enterprise markets. The CMB feature enables the host to place submission queues, completion queues, physical region pages (PRPs), scatter gather list (SGL) segments, and data buffers in controller memory. The PMR feature is an optional region of read / write persistent memory that can be used for multiple purposes. The address range of the PMR is defined by a PCI base address register (BAR) and consumes the entire address region exposed by the BAR. The PMR supports the required features of the PCI Express programming model (i.e., it never restricts what PCI Express originally permits). The contents of the PMR remain unchanged during PMR deactivation, controller and NVM subsystem reset, and power cycles.

[0073] CMB / PMR is becoming a hot topic in the next generation of enterprise markets because this feature has a direct impact on performance, especially in PCI Express fabric topologies. In addition, it reduces the amount of storage devices implemented in host DRAM.

[0074] Because each of the HMB and CMB features is unique to a particular market and is generally not relevant to other markets, the HMB and CMB features are not implemented in the same application. Furthermore, the current approach to support the HMB and CMB features in a storage controller for future products is based on implementing two hardware engines for HMB support and CMB support. However, these approaches can never be done in parallel (i.e., the storage system will operate in either HMB mode or CMB mode).

[0075] The following embodiments can be used to provide configurable hardware that will not require the redundant implementation of two engines for HMB and CMB support. In one embodiment, a storage controller has an embedded FPGA that can be configured to support either HMB or CMB (or neither mode) without redundant hardware to support both modes. In this way, the embedded FPGA can replace the current HMB and CMB support blocks in a storage device controller.

[0076] Returning to the figures, Figure 14 A flowchart 1400 of a method for configuring an embedded FPGA module in the controller 102 to operate as either an HMB support module or a CMB support module for an embodiment. As Figure 14As shown in the middle, the host first configures the eFPGA in the controller 102 to be one of the predefined HMW / CMB hardware options (act 1410). In one embodiment, the host can configure the eFPGA to be one of three options: (1) eFPGA HMB support, (2) eFPGA CMB support, and (3) non-HMA / CMB support. In option 1, the eFPGA will be configured to the predefined design of an HMB support block. In option 2, the eFPGA will be configured to the predefined design of a CMB support block. In option 3, the eFPGA will be free to utilize any other hardware acceleration option, such as adding additional processing engines like ECC, encryption and decryption engines, or additional hierarchical buffers.

[0077] Next, the firmware in the controller 102 parses the host configuration instructions (act 1420) and downloads the relevant configuration from the management area of the memory 104 (act 1430). Subsequently, the eFPGA in the controller 102 is configured to the specific hardware design according to the configuration obtained from the memory 104 (act 1440). When this dynamic configuration is complete, a completion notification can be issued to the host.

[0078] In another embodiment, the eFPGA module can be tuned to implement various features of the HMB or CMB support module, including but not limited to HMB / CMB size, HMB contiguous or non-contiguous buffers (non-contiguous is more complex as the host can provide many non-contiguous buffers representing the entire HMB), CMB usage (e.g., host submission queue, host completion queue, write user data, read user data, and PRP / SGL), and HMB / CMB performance / latency.

[0079] Turning again to the figures, Figure 15 is a block diagram of a storage system 100 that can be used to provide embodiments of eFPGA HMB / CMB configurable devices. As Figure 15As shown in the middle, the storage system 100 includes a controller 102 and a memory (here, NAND) 104. The controller 102 includes an eFPGA 1500 that can be programmed by a processor 1540 with instructions read from an eFPGA configuration area / file 1505 in the memory 104. For example, the eFPGA configuration area / file 1505 in the memory 104 can store instruction code to implement HMB and instruction code to implement CMB. In operation, based on installation instructions from a host 50 (which has a host memory 55), the processor 1510 issues a configuration read command to the memory 104 via a flash interface module 110 to fetch the appropriate instruction code, and then dynamically reconfigures the eFPGA 1500 with the read instruction code. Other components of the controller 102 (e.g., a control path 1550, a command fetcher 1551, a command executor 1552, a direct memory access (DMA) module 1560, a data transfer scheduler 1570, and an error correction block 1554) will operate as usual.

[0080] There are many advantages associated with these embodiments. For example, by using a single configurable eFPGA module in place of two different hardware modules for HMB and CMB support blocks, these embodiments can reduce the price of the controller ASIC, provide design flexibility to configure the HMB / CMB usage to different value features during the lifetime of the device, and shorten time to market (by enabling higher vulnerability immunity realized even after tape-out by the option to reconfigure the HMB / CMB support modules).

[0081] Finally, as mentioned above, any suitable type of memory can be used. Semiconductor memory devices include volatile memory devices, such as dynamic random access memory ("DRAM") or static random access memory ("SRAM") devices, non-volatile memory devices, such as resistive random access memory ("ReRAM"), electrically erasable programmable read-only memory ("EEPROM"), flash memory (which can also be considered a subset of EEPROM), and other semiconductor elements capable of storing information. Each type of memory device can have different configurations. For example, a flash memory device can be configured in a NAND or NOR configuration.

[0082] Memory devices can be formed from passive and / or active elements in any combination. By way of non-limiting example, passive semiconductor memory elements include ReRAM device elements, which in some embodiments include resistivity-switching storage elements such as resistive random access memory (ReRAM), phase change material, etc., and optionally include steering elements such as diodes, etc. Additionally, by way of non-limiting example, active semiconductor memory elements include EEPROM and flash memory device elements, which in some embodiments include elements containing charge storage regions such as floating gates, conductive nanoparticles, or charge storage dielectric material.

[0083] Multiple memory elements can be configured such that they are connected in series or such that each element can be individually accessed. By way of non-limiting example, flash memory devices in a NAND configuration (NAND memory) typically contain memory elements connected in series. A NAND memory array can be configured such that the array is composed of multiple strings of memory, where a string is composed of multiple memory elements that share a single bit line and are accessed as a group. Alternatively, memory elements can be configured such that each element can be individually accessed, such as a NOR memory array. NAND and NOR memory configurations are examples, and memory elements can be configured in other ways.

[0084] Semiconductor memory elements located within and / or over a substrate can be arranged in two dimensions or three dimensions, such as two-dimensional memory structures or three-dimensional memory structures.

[0085] In two-dimensional memory structures, semiconductor memory elements are arranged in a single plane or a single memory device level. Typically, in two-dimensional memory structures, memory elements are arranged in a plane that extends generally parallel to a major surface of a substrate that supports the memory elements (e.g., in an x-z direction plane). The substrate can be a wafer on which layers of memory elements are formed above or within, or can be a carrier substrate to which memory elements are attached after formation. By way of non-limiting example, the substrate can include a semiconductor such as silicon.

[0086] Memory elements can be arranged in a single memory device level in an ordered array, such as a plurality of rows and / or columns of memory elements. However, the memory elements can be arranged in un-ordered or non-orthogonal configurations. Memory elements can each have two or more electrodes or contact lines, such as bit lines and word lines.

[0087] Three-dimensional memory arrays are arranged such that memory elements occupy multiple planes or multiple memory device levels, thereby forming a structure in three dimensions (i.e., in the x, y, and z directions, where y is generally perpendicular to a major surface of the substrate and x and z are generally parallel to a major surface of the substrate).

[0088] As non-limiting examples, a three-dimensional memory structure can be arranged vertically as a stack of multiple two-dimensional memory device levels. As another non-limiting example, a three-dimensional memory array can be arranged as multiple vertical columns (e.g., columns extending generally perpendicular to a major surface of the substrate, i.e., in the y-direction) with each column having a plurality of memory elements in each column. The columns can be arranged in a two-dimensional configuration, e.g., in the x-z plane, resulting in a three-dimensional arrangement of memory elements with elements on multiple vertically stacked memory planes. Other configurations of memory elements in three dimensions can also constitute a three-dimensional memory array.

[0089] By way of non-limiting example, in a three-dimensional NAND memory array, memory elements can be coupled together to form NAND strings within a single horizontal (e.g., x-z) memory device level. Alternatively, memory elements can be coupled together to form vertical NAND strings that traverse multiple horizontal memory device levels. Other three-dimensional configurations can be envisioned in which some NAND strings contain memory elements in a single memory level while other strings contain memory elements that span multiple memory levels. Three-dimensional memory arrays can also be designed in NOR configurations and ReRAM configurations.

[0090] Generally, in a monolithic three-dimensional memory array, one or more memory device levels are formed over a single substrate. Optionally, a monolithic three-dimensional memory array can also have one or more memory levels at least partially within the single substrate. As a non-limiting example, the substrate can comprise a semiconductor such as silicon. In a monolithic three-dimensional array, the layers making up each memory device level of the array are generally formed on the layers of the underlying memory device levels of the array. However, the layers of adjacent memory device levels of a monolithic three-dimensional memory array can be shared, or have intervening layers between the memory device levels.

[0091] Also, two-dimensional arrays can be formed separately and then packaged together to form a non-monolithic memory device having multiple memory levels. For example, a non-monolithic stacked memory can be constructed by forming memory levels on separate substrates and then stacking the memory levels on top of one another. The substrates can be thinned or removed from the memory device levels prior to stacking, but because the memory device levels are initially formed over separate substrates, the resulting memory array is not a monolithic three-dimensional memory array. In addition, multiple two-dimensional memory arrays or three-dimensional memory arrays (monolithic or non-monolithic) can be formed on separate chips and then packaged together to form a stacked chip memory device.

[0092] Memory elements are typically operated and communicated with using associated circuitry. As non-limiting examples, a memory device can have circuitry for controlling and driving the memory elements to implement functions such as programming and reading. This associated circuitry can be on the same substrate as the memory elements and / or on a separate substrate. For example, a controller for memory read-write operations can be on a separate controller chip and / or on the same substrate as the memory elements.

[0093] Those of skill in the art will recognize that the present application is not limited to the two-dimensional and three-dimensional structures described, but covers all related memory structures as described herein and as understood by those of skill in the art within the spirit and scope of the present application.

[0094] It is intended that the foregoing detailed description of the application be understood as an illustration of selected forms that the application can take and not as a limitation to the application. Only the claims, including all equivalents, are intended to define the scope of the application. Finally, it should be noted that any aspect of any of the embodiments described herein can be used alone or in combination with other aspects.

Claims

1. A storage system comprising: a memory configured to store a plurality of sets of instruction code, wherein each set is for a different error decoder implementation; and a controller configured to: select one of the sets of instruction code; read the instruction code of the selected set from the memory; and program the controller with the instruction code of the selected set, wherein the controller implements the error decoder specified in the instruction code of the selected set; and wherein the controller comprises a field programmable gate array.

2. The storage system of claim 1, wherein the selected set is selected based on a bit error rate or an estimate of the bit error rate.

3. The storage system of claim 1, wherein the selected set is selected based on a program / erase count.

4. The storage system of claim 1, wherein the selected set is selected based on a command from a host.

5. The storage system of claim 1, wherein the memory comprises a three-dimensional memory.

6. The storage system of claim 1, wherein the controller is programmed only once to support a particular product configuration or is programmed multiple times over the lifetime of the storage system.

7. A storage system comprising: an encoder; a decoder; a controller means; a memory configured to store first instruction code for implementing an additional decoder and second instruction code for implementing an additional encoder; means for programming the controller means with the first instruction code in response to a first condition being met; and means for programming the controller means with second instruction code in response to a second condition being met; and wherein the controller means comprises a field programmable gate array.

8. The storage system of claim 7, wherein the first condition is met during a read-intensive period, and wherein the second condition is met during a write-intensive period.

9. The storage system of claim 7, further comprising: means for tracking read and write host requests or performance of the storage system; and means for comparing the tracked read and write host requests or performance of the storage system to a benchmark.

10. The storage system of claim 7, wherein the first and second conditions relate to one or more of: a type of application, a type of configuration, a type of virtual host, a queue depth, a power condition, and a temperature condition.

11. The storage system of claim 7, wherein the second condition is met when the storage system is in a burst mode.

12. The storage system of claim 11, wherein the burst mode is detected using a predictable latency period configured by a host.

13. The storage system of claim 7, wherein the field programmable gate array is programmed only once to support a particular product configuration or is programmed multiple times over the lifetime of the storage system.

14. In a storage system including a field programmable gate array and a memory configured to store first instruction code to implement a host memory buffer and second instruction code to implement a controller memory buffer, a method comprising: receiving a configuration instruction from a host; in response to the configuration instruction indicating that the storage system implement a host memory buffer, reading the first instruction code from the memory and programming the field programmable gate array with the first instruction code to implement the host memory buffer; and in response to the configuration instruction indicating that the storage system implement a controller memory buffer, reading the second instruction code from the memory and programming the field programmable gate array with the second instruction code to implement the controller memory buffer.

15. The method of claim 14, wherein the field programmable gate array is programmed only once to support a particular product configuration.

16. The method of claim 14, wherein the field programmable gate array is programmed multiple times over the lifetime of the storage system to dynamically change between host memory buffer implementations and controller memory buffer implementations.

17. The method of claim 16, wherein the implementations dynamically change in response to changes in workload.

18. The method of claim 16, wherein the implementations dynamically change in response to commands from the host.

19. The method of claim 14, wherein the field programmable gate array is embedded in an application specific integrated circuit in the storage system. ​

Images