Thermal management of storage devices to improve host write performance - Patents.com
By selectively disabling writes to overheated dies in multi-die non-volatile memory devices, the controller maintains high performance by using unaffected dies, addressing thermal throttling issues and achieving significant throughput gains.
Patent Information
- Application Number
- JP2024529571
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2022-05-19
- Filing Date
- 2023-05-02
- Publication Date
- 2024-11-27
- Estimated Expiration
- 2043-05-02
AI Technical Summary
Thermal throttling in multi-die non-volatile memory devices leads to performance oscillations and degradation during extended workloads, as all dies are throttled to prevent thermal breakdown, causing overall slowdowns and inefficiencies.
A controller in thermal communication with each die detects excessive temperature and disables writes to the affected die while continuing writes on other dies, reactivating when the temperature drops below a threshold, maintaining full performance on unaffected dies.
This approach minimizes overall performance degradation and maintains high write performance by utilizing unaffected dies at full capacity, with potential throughput increases of up to 93.7% compared to traditional throttling methods.
Smart Images

Figure 2024544047000001_ABST
Abstract
Description
[Technical field]
[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Non-provisional Application No. 17 / 748,336, filed May 19, 2022, entitled “THERMAL MANAGEMENT OF STORAGE DEVICES FOR INCREASING HOST WRITE PERFORMANCE,” the entire contents of which are incorporated by reference into this application for all purposes.
[0002] FIELD OF THEINVENTION The present disclosure relates generally to electronic devices, and more particularly to storage devices. [Background technology]
[0003] introduction Storage devices allow users to store and retrieve data. Examples of storage devices include non-volatile memory devices. Non-volatile memory generally retains data after a power cycle. One example of non-volatile memory is flash memory, which may include an array(s) of NAND cells on one or more dies. Flash memory may be found in solid state drives (SSD), secure digital (SD) cards, etc.
[0004] Thermal throttling in multi-die non-volatile memory (NVM) devices is the conventional process where high performance dissipates heat to at least one die(s), which in turn may cause the die(s) to reach or exceed a threshold temperature. To avoid thermal breakdown, the host may set thermal constraints in addition to internal limits, such as reducing speed for all dies, but with less than maximum performance. For extended workloads, such as multiple consecutive host writes, it is likely that the storage device will switch back and forth between power states, resulting in performance oscillations and overall performance degradation. Summary of the Invention
[0005] An aspect of a storage device is disclosed herein. The storage device includes a plurality of dies, each with a plurality of memory locations. Each corresponding memory location from each die forms an addressable group, such that the plurality of addressable groups traverses the plurality of dies. The storage device also includes a controller in thermal communication with each of the dies. The controller is configured to detect an excessive temperature in at least one of the dies while performing sequential host writes. As a result, the controller is configured to disable all writes to the detected at least one die while continuing to perform writes to memory locations of other dies corresponding to one or more of the addressable groups. The controller is also configured to reactivate writes to the at least one detected die when the temperature drops below a threshold.
[0006] Another aspect of a storage device is disclosed herein. The storage device includes a plurality of dies. Each of the dies includes a positive integer n set of memory locations. A corresponding one of the n sets from each die are combined together to form one metaset of the n metasets. Each metaset includes a maximum addressable unit of host input / output (I / O) operations. The storage device also includes a controller in thermal communication with each of the dies. The controller is configured to detect an excessive temperature in at least one of the dies while performing a sequence of host writes. The controller is also configured to disable all writes to the detected at least one die while continuing host writes to other dies in the corresponding one or more of the n metasets. The controller is further configured to reactivate writes to the at least one detected die when the temperature drops below a threshold.
[0007] Further aspects of a storage device are disclosed herein. The storage device includes a plurality of dies, each comprising memory locations forming one of a plurality of selectively addressable pages. The storage device includes a controller coupled to each of the plurality of dies. The controller is configured to perform host write operations to one or more pages and to detect excessive temperature in at least one of the dies. In response to the detection, the controller is configured to disable all writes to the at least one detected die while continuing to write to memory locations of other dies. The controller is configured to reactivate writes to the at least one detected die when the temperature drops below a threshold.
[0008] It is understood that other aspects of the storage device will become readily apparent to those skilled in the art from the following detailed description, in which various aspects of the device and method are shown and described by way of example. As will be understood, these aspects may be implemented in other different forms, and their several details are capable of modification in various other respects. Accordingly, the drawings and detailed description are to be regarded as illustrative in nature, and not as restrictive. [Brief description of the drawings]
[0009] Various aspects of the invention are presented in the detailed description, by way of example and not by way of limitation, with reference to the accompanying drawings, in which: [Figure 1] 1 is a block diagram illustrating an exemplary embodiment of a storage device in communication with a host device. [Diagram 2] 2 is a conceptual diagram showing an example of a logical-physical mapping table in a non-volatile memory of the storage device of FIG. 1. [Diagram 3] 2 is a conceptual diagram showing an example of a transistor level arrangement of memory cells in the storage device of FIG. 1. [Figure 4] 2 is a conceptual diagram showing an example of an arrangement of memory cells in the memory device of FIG. 1. [Diagram 5] 1 is a flow diagram illustrating an example of steps for selective thermal disablement according to one embodiment. [Figure 6] 1 is a flow diagram illustrating an example of steps for selective thermal disablement during relocation, according to one embodiment. [Figure 7] 1 is an exemplary graph showing the performance of a NAND-based solid-state drive (SSD) as a function of time and temperature. [Figure 8] 1 is a block diagram illustrating three exemplary storage devices having different capacities according to one embodiment. [Figure 9] FIG. 13 is a block diagram illustrating another example of data relocation after the temperature on the skipped die(s) returns to the thermal operating range. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] The detailed description set forth below in conjunction with the accompanying drawings is intended as a description of various exemplary embodiments of the present invention, and is not intended to represent the only embodiments in which the present invention may be practiced. The detailed description includes specific details for the purpose of providing a thorough understanding of the present invention. However, it is apparent to those skilled in the art that the present invention may be practiced without such specific details. In some instances, well-known structures and components are represented in block diagram form to avoid obscuring the concepts of the present invention. Acronyms and other descriptive terms may be used merely for convenience and clarity, and are not intended to limit the scope of the present invention.
[0011] The words "exemplary" and "example" are used herein to mean "serving as an example, instance, or illustration." Any exemplary embodiment described herein as "exemplary" should not necessarily be construed as preferred or advantageous over other exemplary embodiments. Similarly, the term "exemplary embodiment" of a device, method, or article of manufacture does not require that all exemplary embodiments of the invention include the described component, structure, feature, function, process, advantage, benefit, or mode of operation.
[0012] As used herein, the term "coupled" is used to indicate either a direct connection between two components, or, where appropriate, an indirect connection to one another through an intervening or intermediate component. In contrast, when a component is referred to as being "directly coupled" to another component, there are no intervening components present.
[0013] In the following detailed description, various aspects of a storage device that communicates with a host device are presented. These aspects are well suited to flash storage devices such as SSDs and SD cards. However, those skilled in the art will understand that these aspects can be extended to all types of storage devices that can store data. Thus, any reference to a specific device or method is intended only to illustrate various aspects of the invention, with the understanding that such aspects may have a wide range of applications without departing from the spirit and scope of the present disclosure.
[0014] Thermal throttling in conventional non-volatile memory (NVM) devices involves operating the die(s) within temperature constraints while maximizing performance. As mentioned above, during periods of intense performance, such as sustained sequential writes (SSW) or continuous host (I / O) writes, and other operations such as data relocation (including folding and compaction), high performance causes high power that eventually causes at least one of the dies in a multi-die memory system to exceed a power or temperature threshold. In a multi-die storage device, when at least one die exceeds a temperature limit, all dies are throttled (e.g., their (I / O) performance is reduced by an amount proportional to the temperature increase) to help reduce the temperature before it reaches its upper limit, such as during periods of extreme or continuous activity. For extreme workloads, the absence of throttling can cause severe performance degradation.
[0015] Data during a host write is written to metapages within a metablock, which may span multiple dies. For example, a metablock may be the largest addressable unit on a multi-die storage device or a subset of the largest addressable unit. A metadie may be formed of four dies across four flash interface modules (FIMs), with each FIM spanning one die. In other configurations, a jumbo die may be formed with 16 dies across four FIMs, with each FIM spanning four dies. Other configurations and deployments including different numbers of dies and / or FIMs are possible.
[0016] As mentioned above, multiple metapages in a metablock may therefore constitute a jumbo page. However, for the specific purposes of this disclosure, metapages in a metablock may also be broadly referred to as a metaset, where a single write across two or more sets (dies) may be performed simultaneously. Also, for the purposes of this disclosure, data may be referred to as being written (or not written) to one or more dies, which simply means that the data is written to one or more memory locations or sets on that die. Each die may be associated with a flash interface module (FIM), which may be assembled into a larger storage device having an array of multiple dies.
[0017] One such example of conventional throttling on a multi-die storage device involves heavy write loads placed on NVM devices such as NAND memory, NOR memory, and other forms of SSD or flash NVM memory. The temperature of each of the dies typically increases as performance increases until at least one die meets or exceeds an upper temperature threshold. This in turn prompts the need for throttling. In some cases, throttling can be initiated by a host device in response to thermal information from an infrastructure module on the storage device or other sources. Conventionally, a storage device can initiate throttling across all meta-dies in response to a command from a host device, which is an intentional reduction in performance with the purpose of restoring acceptable temperatures within the die while continuing to respond to new (I / O) commands, folds, etc.
[0018] In the above conventional example, it is assumed that a page or other group ("set") of one or more memory locations on each die forms an individually addressable metapage in a metablock across the six dies. Further assume that the temperature of die 3 increases. When the controller detects (e.g., via a thermal infrastructure module, etc.) that at least one of the dies has met a threshold, the host controller or a controller on the storage device can begin throttling die 3 in response to notification of the increase in die 3. Thus, due to the temperature increase on die 3, the host or storage device controller begins slowing down host writes across all dies. This in turn causes an overall slowdown of host writes, allowing the temperature on die 3 to cool. Thus, the temperature increase on die 3 impedes write performance on all dies. For example, performance of successive host writes or other actions may degrade by 50% or more across all dies until the temperature of die 3 returns to a normal range for the operation in question (e.g., host writes). In some cases, die 3 may include one or more dies that exceed a temperature threshold.
[0019] Depending on the memory technology and configuration, the amount of throttling initiated may vary based on whether at least one die has crossed light, heavy, or extreme performance, or some scale in between. At higher temperatures, more restrictive throttling is conventionally applied across all die to continue to perform writes and other operations, while using slower speeds commensurate with the over-temperature of at least one of the involved dies.
[0020] The inventors have found a superior way to handle writes and other input / output (I / O) operations or internal relocations, etc., in the face of excessive heat and temperature caused by excessive or extreme performance. The approach described herein, for example, results in higher overall write performance with much less overall performance degradation. In one aspect of the present disclosure, instead of a global slowdown of writes across a metablock or metaset during a detected temperature rise of any of the die(s) that meets or exceeds a temperature threshold, the controller (at the command of the host or independently) skips writes only to the specific die(s) whose temperature(s) exceed the threshold. The controller continues to perform write operations at full speed, without throttling, to all other dies until the temperature of the disabled die(s) returns to normal operating temperature, or at least to an acceptable value. During the disablement of a die(s), the performance of the remaining dies may be utilized at 100%. After the die(s) are restored, full performance at maximum capacity may be restored, including all metapages in the metablock(s).
[0021] FIG. 1 illustrates an exemplary block diagram 100 of a storage device 102 in communication with a host device 104 (also referred to as a “host”), according to an exemplary embodiment. The host 104 and the storage device 102 may form a system, such as a computer system (e.g., a server, a desktop, a mobile / laptop, a tablet, a smartphone, etc.). The components of FIG. 1 may or may not be physically collocated. In this regard, the host 104 may be located remotely from the storage device 102. Although FIG. 1 illustrates that the host 104 is shown separate from the storage device 102, the host 104 in other embodiments may be integrated in whole or in part into the storage device 102. Alternatively, the host 104 may be distributed across multiple remote entities, either in its entirety or, alternatively, with some functionality within the storage device 102.
[0022] Those skilled in the art will appreciate that other exemplary embodiments may include more or fewer elements than those shown in Figure 1, and that the disclosed processes may be implemented in other environments. For example, other exemplary embodiments may include a different number of hosts in communication with storage device 102, or multiple storage devices 102 in communication with a host(s).
[0023] The host device 104 can store data in and / or retrieve data from the storage device 102. The host device 104 can include any computing device including, for example, a computer server, a network attached storage (NAS) unit, a desktop computer, a notebook (e.g., laptop) computer, a tablet computer, a mobile computing device such as a smartphone, a television, a camera, a display device, a digital media player, a video game console, a video streaming device, and the like. The host device 104 can include at least one processor 101 and a host memory 103. The at least one processor 101 can include any form of hardware capable of processing data and can include a general purpose processing unit (such as a central processing unit (CPU)), dedicated hardware (such as an application specific integrated circuit (ASIC)), a digital signal processor (DSP), configurable hardware (such as a field programmable gate array (FPGA)), or any other form of processing unit configured by software instructions, firmware, and the like. The host memory 103 may be used by the host device 104 to store data or instructions to be processed by the host or data received from the storage device 102. In some examples, the host memory 103 may include non-volatile memory such as magnetic memory devices, optical memory devices, holographic memory devices, flash memory devices (e.g., NAND or NOR), phase change memory (PCM) devices, resistive random access memory (ReRAM) devices, magnetoresistive random access memory (MRAM) devices, ferroelectric random access memory (F-RAM), and any other type of non-volatile memory device. In other examples, the host memory 103 may include volatile memory such as random access memory (RAM), dynamic random access memory (DRAM), static RAM (SRAM), and synchronous dynamic RAM (SDRAM (e.g., DDR1, DDR2, DDR3, DDR3L, LPDDR3, DDR4, etc.)).Host memory 103 may also include both non-volatile and volatile memory, whether integrated together or as separate units.
[0024] The host interface 106 is configured to interface the storage device 102 with the host 104 via a bus / network 108, and may interface using, for example, bus standards such as Ethernet or WiFi, or Serial Advanced Technology Attachment (SATA), PCI Express (PCIe), Small Computer System Interface (SCSI), or Serial Attached SCSI (SAS), among other possible candidates. Alternatively, the host interface 106 may be wireless, and may interface the storage device 102 with the host 104 using, for example, cellular communications (e.g., 5G NR, 4G LTE, 3G, 2G, GSM / UMTS, CDMA One / CDMA2000, etc.), wireless distribution methods via access points (e.g., IEEE 802.11, WiFi, HiperLAN, etc.), infrared (IR), Bluetooth, Zigbee, or other wireless wide area network (WWAN), wireless local area network (WLAN), wireless personal area network (WPAN) technologies, or equivalent wide area, local area, and personal area technologies.
[0025] The storage device 102 includes a memory. For example, in the exemplary embodiment of FIG. 1, the storage device 102 can include a non-volatile memory (NVM) 110 for persistently storing data received from the host 104. The NVM 110 can include, for example, a flash integrated circuit, a NAND memory (e.g., a single-level cell (SLC) memory, a multi-level cell (MLC) memory, a triple-level cell (TLC) memory, a quad-level cell (QLC) memory, a penta-level cell (PLC) memory, or any combination thereof), or a NOR memory. The NVM 110 can include a number of memory locations 112 that can store system data for operating the storage device 102 or user data received from a host for storage in the storage device 102. For example, the NVM can have a cross-point architecture that includes a 2-D NAND array of memory locations 112 having n rows and m columns, where m and n are predefined according to the size of the NVM. In the exemplary embodiment of FIG. 1, each memory location 112 can be a die 114 that includes a number of planes, each including a number of blocks of a number of cells 116. Alternatively, each memory location 112 may be a plane including multiple blocks of cells 116. The cells 116 may be, for example, single level cells, multi-level cells, triple level cells, quad level cells, and / or penta level cells. Other examples of memory locations 112 are possible. For example, each memory location may be a block or a group of blocks. Each memory location may include one or more blocks in a 3-D NAND array. Each memory location 112 may include one or more logical blocks that are mapped to one or more physical blocks. Alternatively, the memory and each memory location may be implemented in other manners known to those skilled in the art.
[0026] The storage device 102 also includes a volatile memory 118, which may include, for example, dynamic random access memory (DRAM) or static random access memory (SRAM). Data stored in the volatile memory 118 may include data read from or written to the NVM 110. In this regard, the volatile memory 118 may include a write buffer or a read buffer for temporarily storing data. Although FIG. 1 illustrates the volatile memory 118 as separate from the controller 123 of the storage device 102, the volatile memory 118 may be integrated into the controller 123. In other embodiments, one or more firmware modules may be integrated into the controller. The controller 123 may also perform various functions associated with a flash translation layer (FTL) in deployments in which such an FTL is implemented.
[0027] The memory (e.g., NVM 110) is configured to store data 119 received from the host device 104. The data 119 may be stored in cells 116 of any of the memory locations 112. As an example, although FIG. 1 shows the data 119 stored in different memory locations 112, the data may be stored in the same memory location. In another example, the memory locations 112 may be different dies and the data may be stored in one or more of the different dies.
[0028] Each of the data 119 may be associated with a logical address. For example, the NVM 110 may store a logical-to-physical (L2P) mapping table 120 of the storage device 102 that associates each data 119 with a logical address. The L2P mapping table 120 stores a mapping between logical addresses specified for data written from the host 104 and physical addresses in the NVM 110 that indicate the location(s) at which each data is stored. This mapping may be performed by the controller 123 of the storage device. The L2P mapping table may be a table or other data structure that includes an identifier, such as a logical block address (LBA), associated with each memory location 112 in the NVM at which data is stored. Although FIG. 1 shows a single L2P mapping table 120 stored in one of the memory locations 112 of the NVM to avoid unduly obscuring the concepts of FIG. 1, the L2P mapping table 120 may actually include multiple tables stored in one or more memory locations of the NVM.
[0029] The dashed area of NVM 110 includes infrastructure module 131 in communication with controller 123 and thermal mapping table 141. Infrastructure module 131 or controller 123 may store information regarding die temperature in thermal mapping table 141 in volatile memory.
[0030] In some configurations, the infrastructure module may span all die locations, although only a portion is shown to avoid unduly obscuring the example of FIG. 1. The infrastructure module 131 may be in thermal communication with each die and may relay the temperature of each die to the controller 123. The controller may then forward this data to the host device 104, if desired in some implementations. In other configurations, the controller may forward temperature data only for one or more dies that exceed a pre-set threshold, e.g., stored in a portion of the NVM. The controller 123 may store this information in a thermal mapping table 141.
[0031] 2 is a conceptual diagram 200 of an example of an L2P mapping table 205 illustrating the mapping of data 202 received from a host device to logical and physical addresses in NVM 110 of FIG. 1. The data 202 may correspond to data 119 of FIG. 1, and the L2P mapping table 205 may correspond to L2P mapping table 120 of FIG. 1. In an exemplary embodiment, the data 202 may be stored in one or more pages 204, e.g., pages 1 through x, where x is the total number of pages of data to be written to NVM 110. Each page 204 may be associated with one or more entries 206 in the L2P mapping table 205 that identify a logical block address (LBA) 208, a physical address 210 associated with the data written to NVM, and a length 212 of the data. The LBA 208 may be a logical address specified in a write command for the data received from the host device. The physical address 210 may indicate the block and offset to which the data associated with the LBA 208 is physically written. Length 212 may indicate the size of the data written (eg, 4 KB or some other size).
[0032] Referring again to FIG. 1, the volatile memory 118 also stores a cache 122 for the storage device 102. The cache 122 contains entries indicating a mapping of a logical address specified for data requested by the host 104 to a physical address in the NVM 110 indicating the location(s) where the data is stored. This mapping may be performed by the controller 123. When the controller 123 receives a read or write command for the data 119, it checks the cache 122 for a logical-to-physical mapping for each data. If a mapping does not exist (e.g., if this is the first request for the data), the controller accesses the L2P mapping table 120 and stores the mapping in the cache 122. When the controller 123 executes a read or write command, it accesses the mapping from the cache and reads or writes the data from or to the NVM 110 at the specified physical address. The cache may be stored in the form of a table or other data structure that includes a logical address associated with each memory location 112 in the NVM from which data is being read.
[0033] The NVM 110 includes a sense amplifier 124 and a data latch 126 connected to each memory location 112. For example, the memory location 112 may be a block including cells 116 on multiple bit lines, and the NVM 110 may include a sense amplifier 124 on each bit line. Additionally, one or more data latches 126 may be connected to the bit lines and / or the sense amplifiers. The data latches may be, for example, shift registers. When data is read from a cell 116 of the memory location 112, the sense amplifiers 124 sense the data by amplifying the voltage on the bit lines to a logic level (e.g., readable as a "0" or a "1"), and the sensed data is stored in the data latches 126. The data is then transferred from the data latches 126 to the controller 123, after which the data is stored in the volatile memory 118 until it is transferred to the host device 104. When data is written to a cell 116 of memory location 112 , controller 123 stores the programmed data in a data latch 126 , and then the data is transferred from data latch 126 to cell 116 .
[0034] The memory device 102 includes a controller 123, which includes circuitry such as one or more processors for executing instructions, and may include a microcontroller, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), hardwired logic, analog circuitry, and / or combinations thereof.
[0035] The controller 123 is configured to receive data transferred from one or more of the cells 116 of the various memory locations 112 in response to a read command. For example, the controller 123 may read the data 119 by activating the sense amplifiers 124 to sense the data from the cells 116 to the data latches 126, and the controller 123 may receive the data from the data latches 126. The controller 123 is also configured to program data into one or more of the cells 116 in response to a write command. For example, the controller 123 may write the data 119 by sending the data to be programmed into the cells 116 to the data latches 126. The controller 123 is further configured to access the L2P mapping table 120 in the NVM 110 when reading and writing data to the cells 116. For example, in response to a read or write command from host device 104, controller 123 can receive a logical-to-physical address mapping from NVM 110, identify the physical address mapped to the logical address identified in the command (e.g., translate the logical address to a physical address), and access or store data in cells 116 located at the mapped physical address.
[0036] Controller 123 and its components may be implemented using embedded software that performs the various functions of the controller described throughout this disclosure. Alternatively, software for implementing each of the aforementioned functions and components may be stored in NVM 110 or memory external to storage device 102 or host device 104 and accessed by controller 123 for execution by one or more processors of controller 123. Alternatively, the functions and components of the controller may be implemented in hardware within controller 123 or using a combination of the aforementioned hardware and software.
[0037] During operation, the host device 104 stores data in the storage device 102 by sending a write command to the storage device 102 that specifies one or more logical addresses (e.g., LBAs) as well as the length of the data to be written. The interface element 106 receives the write command and the controller allocates a memory location 112 in the NVM 110 of the storage device 102 to store the data. The controller 123 stores an L2P mapping in the NVM (and cache 122) to map the logical address associated with the data to the physical address of the memory location 112 allocated to the data. The controller also stores the length of the L2P mapped data. The controller 123 then stores the data in the memory location 112 by sending the data to one or more data latches 126 connected to the allocated memory location, from which the data is programmed into the cell 116.
[0038] The host 104 may retrieve data from the storage device 102 by sending a read command that specifies one or more logical addresses associated with the data to be retrieved from the storage device 102, as well as the length of the data to be read. The interface 106 receives the read command, and the controller 123 accesses the L2P mapping in the cache 122 or NVM to translate the logical address specified in the read command into a physical address indicating the location of the data. The controller 123 then reads the requested data from the memory location 112 specified by the physical address by sensing the data using sense amplifiers 124 and storing them in data latches 126 until the read data is returned to the host 104 via the host interface 106.
[0039] 3 illustrates an example of a NAND memory array 300 of cells 302. The cells 302 may correspond to the cells 116 in the NVM 110 of FIG. 1. The cells 302 are coupled to word lines 304 and bit lines 306. For example, the memory array 300 may include n word lines and m bit lines in a block of the die 114 of the NVM 110, where n and m are predetermined according to the size of the block. Each word line and bit line may be associated with a row address and a column address, respectively, and the controller 123 may use the row address and the column address to select a particular word line and bit line (e.g., using a row decoder and a column decoder). For example, each of the word lines 0-n may be associated with their own row address (e.g., word line 0 may correspond to word line address 0, word line 1 may correspond to word line address 1, etc.), and each of the bit lines 0-m may be associated with their own column address (e.g., bit line 0 may correspond to bit line address 0, bit line 1 may correspond to bit line address 1, etc.). Select gate source (SGS) cells 308 and select gate drain (SGD) cells 310 are coupled to the memory cells 302 on each bit line 306. The SGS cells 308 and SGD cells 310 connect the memory cells 302 to a source line 312 (e.g., ground) and the bit line 306, respectively. A string 314 may include a group of cells 302 (including the SGS cells 308 and the SGD cells 310) coupled to one bit line in a block, and a page 316 may include a group of cells 302 coupled to one word line in a block.
[0040] FIG. 4 is a conceptual diagram 400 illustrating an example of an arrangement of memory cells in the memory device of FIG. 1. Each die 402, 404, 406, and 408 in this example may be associated with a respective flash interface module (FIM 0, FIM 1, FIM 2, FIM 3). Thus, while a total of four die are shown in this figure, that number may be any number greater than or equal to two. Die 0 402 of FIM 0 includes page, group, set, or storage location(s) 402-0, 402-1, 402-2, 402-3, 402-4, 402-5, etc. Similarly, die 0 404 of FIM 1 may include page, group, set, or storage location(s) 404-0 through 404-5, etc. Die 0 406 of FIM 2 may include page, group, set, or storage location(s) 406-0 through 406-5, etc. Die 0 408 of FIM 3 may include pages, groups, sets, or storage location(s) 408-0 through 408-5, etc. In this example, there are a total of N such pages, groups, sets, or storage location(s), as indicated by the designation row N-1 435 on die 0 of FIM 0.
[0041] Each one or more of the pages, groups, sets, or storage location(s) is associated with a corresponding page, group, set, or storage location(s) (hereinafter sometimes referred to as a "group," "set," or "memory location"). For example, 402-0, 404-0, 406-0, and 408-0 may form a simultaneously addressable metapage or metadata set. Similarly, 402-1, 404-1, 406-1, and 408-1 may form an individually addressable metapage, and so on through the penultimate metapage that includes memory location 421N-2, and the last metapage or metaset 423N-1. It will be appreciated that other configurations are possible, including different addressing schemes, different numbers and / or arrangements of dies, different FIMs, etc., without departing from the scope of this disclosure. The memory locations corresponding to 402-3, 404-3, 406-3, and 408-3 may be simultaneously addressable as a full data width, although in various configurations smaller portions or chunks of memory locations may be addressable. Thus, any given metaset (e.g., 423N-1) may be written using a minimal portion of the total amount of data locations, as the case may be.
[0042] For purposes of this disclosure, unless otherwise clear from the context, the terms "memory location" and "memory locations" may be used interchangeably. This definition includes other nomenclature used to describe memory cells, since a single location may be, for example, either a single bit or a series of bits.
[0043] As can be seen from the different textures corresponding to the different rows 0-N, set 402-0 may be in the same addressable metaset as sets 404-0, 406-0, and 408-0. Similar textures are described herein to identify individual metapages or metasets. In one embodiment, controller 123 may be in the process of performing multiple consecutive host writes on each metapage starting with a simultaneous write operation to sets 402-0 to 408-0 (one metapage or metaset), then the next write to the metapage defined by sets 402-1 to 408-1, and so on. Before reaching the fourth metapage, the controller may detect, for example, that die 0 in FIM 2 has exceeded a maximum temperature. In some embodiments, multiple different stages of maximum temperature are used. When the controller detects that the temperature is excessive on FIM 2, controller 123 terminates writes to die 0 in FIM 2 entirely. However, controller 123 continues to write at full speed (without throttling) to the remaining memory locations 402-4, 404-4, and 408-4 in a single write operation. Controller 123 can then perform host writes to memory locations 402-5, 404-5, and 408-5. The controller can continue to perform successive writes at full speed, except that it skips the write to die 0 (i.e., does not write to memory locations 406-4 and 406-5).
[0044] 4, the other dies 402, 404, and 408 may be utilized at their maximum performance until such time as controller 123 determines that the temperature of die 406 has dropped to an acceptable operating range, at which point the controller may perform a write, such as a relocation operation, to populate memory locations (sets) 406-4 and 406-5.
[0045] In short, during an observed temperature rise of any one or more dies above an acceptable temperature, instead of causing a slowdown of writes across an entire metapage (sometimes called a metablock), the controller skips only the writes corresponding to those particular die(s), while the performance of the remaining dies is not slowed down and is utilized at 100%. Once the skipped die(s) are back within the operating temperature range, the controller may proceed to utilize all metapages at maximum speed and capacity.
[0046] The inventors have observed that the above and similar embodiments, in which a thermally aged (over-temperature) die 406 is skipped entirely without throttling other dies until their temperature drops, have significant advantages over conventional approaches. For example, as described further below with reference to a storage device having a four-die metapage, a 75% increase in performance can be achieved (three of the four dies remain fully operational). As the number or arrangement of metapages increases, higher performance throughputs of over 90% are possible using the principles described herein (see, for example, FIG. 8). These throughputs exceed conventional throughputs where all dies in a metapage are slowed down.
[0047] Another advantage of the technique described herein is that in most configurations, it uses only firmware overhead with minimal changes without requiring additional hardware. This in turn can result in significant cost savings while potentially achieving significant throughput increases. Yet another advantage is that it can provide an earlier cooling of the die temperature since writes (I / O operations) to that particular die are turned off. Thus, the die is more likely to return to normal operating temperature than traditional approaches (which continue to write to a thermally aged die, which results in the die continuing to generate heat, albeit at a lower rate).
[0048] In some embodiments, an infrastructure module 131 of FIG. 1 is included on each die and in thermal communication with each die. When temperature information is provided to the controller 123, the controller can store the thermal mapping data in a thermal mapping table 141 in volatile memory. In other embodiments, the thermal mapping table may additionally or alternatively be maintained in NVM. The architecture of the infrastructure module may also vary depending on the memory configuration and type, all without departing from the scope of this disclosure. In some embodiments, the controller may notify the host processor 101 of an excessive temperature, in which case, in those embodiments, the host processor 101 on the host device 104 may initiate a die shutdown on one or more dies or provide a more general command to modify performance. The controller 123 on the die can then receive instructions from the host device 104 and implement the instructions in the manner described above.
[0049] FIG. 5 is a flow diagram 500 illustrating an example of steps for selective thermal disablement according to one embodiment. The steps of FIG. 5 may be performed, for example, by controller 123 (FIG. 1). The controller may perform steps specified by the FTL or firmware. In block 502, a sequence may be initiated by control in controller 123. Referring initially to block 504, it is assumed that a storage device such as a solid state drive, or flash memory, compact flash, and other forms of NVM, is in the process of processing incoming reads and writes from a host device and performing any folding in the case of MLC.
[0050] As a result, referring to block 506, it is assumed that the infrastructure module 131 for a die senses that the operating temperature of, for example, die K is excessive. For example, it meets or exceeds a defined threshold amount. The infrastructure module may sense similar information for another die. In various embodiments, after sensing the excessive temperature, in block 508, the infrastructure module may notify the controller 508 of the change in thermal configuration, for example, via the FTL.
[0051] 5, as shown in block 510, the controller may update the temperature information received from the infrastructure module in the thermal mapping table 141. Also, as shown with reference to block 512 of FIG. 5, the controller 123 (via the FTL or otherwise) may begin processing selective writes, including skipping writes to the affected die K, while maintaining full capacity writes to other dies (e.g., other memory locations), in the manner described above. Note that the infrastructure module may vary depending on the configuration and may be implemented, for example, as a simple circuit configured to sense the die temperature.
[0052] In various embodiments, the controller may periodically or continuously check the state of the temperature of the exemplary die K to determine if it is still excessive. This check is indicated by diamond 514. If the temperature remains excessive, control returns to box 512 and the controller continues to perform writes or other I / O or internal operations at full capacity while disabling (skipping writes or other operations) writes or other I / O or internal operations to die(s) K (e.g., memory locations or sets thereof that are part of a larger metaset or metapage that spans multiple dies). Multiple dies containing metasets or metapages that can be individually addressed may also be used in a manner consistent with the principles described herein.
[0053] If the controller determines that die(s) K has returned to normal operating range at diamond 514, the controller may return die(s) K to full functionality. In various embodiments, as shown in block 516, the controller may reset die K thermal information by updating the thermal mapping table. Concurrently or thereafter, the controller 123 may return to processing writes normally (e.g., via FTL). This change may involve simultaneously performing host writes to all sets in all dies in a given meta-die. This change may also involve, in various embodiments, performing a relocation of data to free locations in die K that were skipped while die K was disabled, as further described below with reference to FIG. 9. The process may end at block 520.
[0054] Those skilled in the art will appreciate upon reviewing this disclosure that the sequence of events in Figure 5 may occur in different orders in various embodiments. For example, in some embodiments, the controller may execute the steps of blocks 510 and 512 simultaneously. Other embodiments and variations are possible. Except for the skipped die(s) K, no performance throttling is implemented.
[0055] In short, as shown in FIG. 5, whenever at least one die becomes over-temperature during a host write or sequential sustained write (SSW), the infrastructure module notifies the FTL layer. Based on the messages received by the controller via the FTL, the controller can understand which meta-die(s) (e.g., multiple dies) of the multiple meta-dies (in the case of one or more meta-dies) is the problem and which die(s) in the multiple have the excessive temperature. In some embodiments, it is detected that one or more dies have the excessive temperature, in which case writes to the subject die are skipped, but full performance is maintained for other dies in the meta-die (including metasets, metapages, metablocks, etc. within that meta-die) that have the excessive temperature across the multiple dies of interest. If one or more meta-die groups are affected, the controller can perform the same set of functions on the multiple meta-dies to disable those dies that have the excessive temperature.
[0056] In various embodiments involving large arrays of die (NAND, NOR, SSD, Flash, etc.), controller 123 may include multiple controllers, each of which may coordinate with one another to perform the operations described above. Thus, the principles of this disclosure may be applied to meta-dies of any size and complexity.
[0057] In another aspect of the disclosure, after the controller closes a host-written data block (e.g., a location(s) on one of the metasets), the controller may be configured to relocate the data to a previously skipped set / block of the metaset of the meta-die to achieve optimal read performance, among other benefits. In various embodiments, to implement this sequence of steps, a thermal relocation state machine may be used to repopulate data into blocks or metasets to which fewer die(s) were written due to excessive temperature(s). In this manner, host reads may be maximized to include data across an entire metapage.
[0058] In these embodiments, the bitmap may be retrieved in a log table along with the thermal mapping table in NVM and / or cached in volatile memory. The bitmap may be part of the thermal mapping table or may be separate. The bitmap in the log table may be included in or adjacent to the L2P mapping table in NVM in some examples. The bitmap may also be used in conjunction with SLC or MLC (e.g., TLC, QLC, PLC, etc.) memory architectures. The bitmap may be retrieved such that, for example, when a closed SLC block becomes a candidate for a relocation operation due to thermal information (less than excessive), its priority is lower than read scrub (RS) and higher than static relocation. The priority level may vary based on the design.
[0059] Figure 6 is a flow diagram 600 illustrating an example of steps for selective thermal disablement during relocation, according to one embodiment. In Figure 6, it can be assumed that the temperature of die K has returned to within normal operating range. Beginning at block 602, the storage device, such as an SSD, can return to processing reads and writes as normally prescribed, with accesses to entire metapages.
[0060] Referring to block 606, the controller may determine whether a relocation is recommended or necessary. Examples of relocation types may include, for example, SLC relocation, RS relocation, thermal relocation, SLC-SLC static relocation, SLC-SLC dynamic relocation, SLC-QLC fold, QLC-QLC static relocation, QLC-QLC dynamic relocation, etc., depending on the memory architecture. Upon determining the required type of relocation and its associated priority, the controller performs operations to process the respective relocation, such as a priority determined via one or more of a bitmap, a log table, and a thermal mapping table.
[0061] After performing each relocation operation, the controller may be configured to determine, such as at diamond 610, whether any of the relocation operations were thermal operations, thereby filling, for example, memory locations on a skipped die K. If not, control may end at block 614. Conversely, if a thermal relocation occurred, then at block 612, the thermal mapping table and / or bitmap, respectively, may be updated to account for the thermal read locations.
[0062] In short, this aspect of the disclosure also improves read performance. In the example of FIG. 5 where die K was disabled due to excessive temperature, conventionally, if a host read command is received, data will be read from three FIMs instead of four FIMs, which impacts read performance. In the embodiment of FIG. 6, thermal relocation can populate the memory locations on the skipped die(s) at full capacity. Any host read on the same set of meta-dies will result in the host receiving data from four FIMs (four dies in this example) instead of three. Thus, after closing the host write data block, the disabled die(s) can be relocated as soon as possible to make them available for reading through four FIMs during a subsequent host read to maximize performance.
[0063] FIG. 7 is an exemplary graph 700 showing the performance of a conventional NAND-based solid-state drive (SSD) as a function of time and temperature. Although details of a NAND-based SSD are described for illustrative purposes, other memory architectures may exhibit similar behavior. The purpose of FIG. 7 is to show the performance degradation when using conventional throttling. Referring to FIG. 7, two plots are shown, as explained by the legend in the upper right. To that end, the right vertical axis corresponds to temperature in degrees Celsius. This vertical axis is particularly applicable to the SMART Temperature (C) data plot, which is a solid line. The right vertical axis corresponds to storage device performance (i.e., write speed) in megabytes per second (MBps) and is directed to the dashed plot line labeled Drive Performance (MBps) in the legend. The horizontal axis indicates time in seconds and applies to both plots.
[0064] Various events occur on the graph at 780, 782, and 784 and are described below. At initial low and normal temperatures ranging from about 30° C. to just under 70° C., higher performance can be obtained up to about 72 MBps. At event 780, when the temperature reaches 70° C. in one of the dies, the host can initiate an initial throttling of all dies of the meta-die, and the controller 123 can respond by performing an initial throttling of all dies of the meta-die. This initial throttling corresponds to a first downgrade in performance (e.g., to TMT1), as the write speed of all dies is reduced to near about 57 MBps. As the host writes continue at the first reduced rate, the temperature may continue to increase.
[0065] 7, at event 782, the controller determines that the temperature of at least one of the dies in the target meta-die has exceeded 85° C. Thus, throttling is increased by the next gradation (e.g., ETT), resulting in a much larger performance hit as the write speed across all dies decreases to below 40 MBps. Finally, the temperature finally begins to decrease, as shown by the peak temperature at 786 of just under 90° C. Once the temperature has decreased to approximately 80-85° C., the controller can decrease the throttling to TMT1, and the write speed performance in this example increases to just under approximately 60 MBps.
[0066] In short, during normal temperature, NAND speed is even around 75 MBps. During high temperature around 90°C, performance can degrade by 50% in worst case, which is the effect across all dies, meaning the overall performance degradation using conventional methods is around 50%.
[0067] FIG. 8 is a block diagram 800 illustrating three exemplary storage devices 802-1, 804-2, and 804-4 of different capacities, according to one embodiment. FIG. 8 is exemplary in nature. Different numbers of addressable dies and different capacities are equally possible across different memory architectures. The storage device 802-1 on the left side contains the NVM of one meta-die 802, which in this example corresponds to four dies including F0 D0, F1 D0, F2 D0, and F3 D0. The meta-die 802 on the left side may correspond to 512 GB of storage. The same texture (or its absence) in each row represents concurrent input / output (I / O) operations that may occur on different ones of the dies. In this example, a total number of metapages 802a-802e are available. In various implementations, writing to an individual die D0 or to portions of die D0 is possible.
[0068] Storage device 804-2, for purposes of illustration, may refer to a 1 terabyte (TB) device that includes two meta-dies 804. The two meta-dies 802 are arranged in an array of separately addressable meta-pages 804a-804j. The two meta-dies 804 of storage device 804-2 include a total of eight dies with four dies on each meta-die. Storage device 804-4 may refer to a 2 terabyte (TB) device that includes four meta-dies 806. The addressable meta-pages may include meta-pages 806a-806t. Storage device 804-4 includes four dies for each meta-die of the four meta-dies 806 for a total of sixteen dies.
[0069] It should be noted that in the exemplary embodiment of FIG. 8, much higher write performance can be achieved as opposed to the conventional throttling described with reference to FIG. 7. Referring to storage device 802-1 as an example, if die F1 DO is disabled on all five rows 802 a-3, then 3 / 4 or 75% total throughput performance can still be achieved since the other three dies are operating at full host write capacity. Referring now to the example storage device 804-2, if die F1 D1 has an excessive temperature and is disabled across five metapages 804 f-j, then 7 / 8 or 87.5% total throughput performance can still be achieved by operating the other seven dies at full capacity. Referring to the remaining exemplary storage device 804-4, similarly, if die F2 D3 is disabled for all five metapages 806 p-t, then the remaining 15 / 16 dies operating at full capacity can achieve 93.7% throughput using the principles of the present disclosure. This is true even at extreme temperatures.
[0070] In short, a range of 75% (one meta-die capacity) to 93% (four meta-die capacity) of total throughput can be obtained on the illustrated exemplary device. In other configurations including a larger number of meta-dies, even higher performance may be possible. These performances can be implemented with minimal firmware overhead and generally without additional hardware. Furthermore, as mentioned, since the die(s) are disabled and therefore do not independently dissipate much, if any, heat, the disabled die(s) can accurately expect to cool down faster. Thus, the turn-off or disable time of a particular die(s) can be much shorter in time before it / them can be reactivated. Furthermore, in some embodiments, to maximize host read performance, data can be relocated to the skipped memory locations of the disabled die(s). Subsequent reads can benefit from the maximum amount of data received at one time.
[0071] FIG. 9 is a block diagram 900 illustrating another example of data relocation after the temperature on the skipped die returns to an acceptable thermal operating range. FIG. 9 illustrates an exemplary metablock 970. The metablock 970 includes 128 wordlines (WL) from WL0 to WL127. The metablock 970 then includes four dies - F0 D0, F1 D1, F2 D2, and F3, D3. In this exemplary embodiment, assume that after a full host write to WL 0-2, the controller detects an excessive temperature on die F2 D0 from the infrastructure module and disables die F2 D0 while continuing a series of host writes (e.g., SSW) from WL3 to WL 127 to each of the remaining three dies at full capacity without throttling. The full capacity may be the same capacity of writes that occurred before F2 D0 was disabled, or more simply, 100% capacity of writes on the storage device.
[0072] After performing sequential writes at full capacity while disabling die F2 D0 from memory locations WL3-WL127 due to excessive temperature on die F2 D0, the controller may decide to relocate the data to die F2 D0 once the temperature drops within the operating range. The right side of FIG. 9 shows metablock 970 after the relocation has occurred. Subsequent host reads are more efficient because all the dies in the metablock are filled with data. Overall performance is improved.
[0073] Various aspects of the present disclosure are provided to enable those skilled in the art to practice the present invention. Various modifications to the exemplary embodiments presented throughout this disclosure will be readily apparent to those skilled in the art, and the concepts disclosed herein may be extended to other magnetic storage devices. Thus, the claims are not limited to the various aspects of the present disclosure, but should be accorded the full scope consistent with the language of the claims. All structural and functional equivalents to the various components of the exemplary embodiments described throughout this disclosure that are known or that later become known to those skilled in the art are expressly incorporated herein by reference and are intended to be encompassed by the claims. Furthermore, nothing disclosed herein is intended to be dedicated to the public, regardless of whether such disclosure is expressly recited in the claims. No element of a claim should be construed under the provisions of 35 U.S.C. 112(f) in the United States, or under any similar statute or rule of law in another jurisdiction, unless the element is expressly recited using the phrase "means for" or, in the case of a method claim, unless the element is recited using the phrase "step for".
Claims
1. 1. A storage device comprising: a plurality of dies, each comprising a plurality of memory locations, where corresponding memory locations from each die form addressable groups such that the addressable groups span the plurality of dies; a controller in thermal communication with each of the dies, Detecting an excessive temperature on at least one of the dies while performing sequential host writes; disable all writes to the at least one detected die while continuing to perform writes to the memory locations of the other die corresponding to one or more of the addressable groups; a controller configured to reactivate writing to the at least one detected die when the temperature drops below a threshold.
2. 2. The apparatus of claim 1, wherein the controller is configured to continue performing the host writes to the other die at full capacity while the at least one detected die is disabled.
3. 2. The apparatus of claim 1, wherein the controller is configured to continue performing the host writes to the other die without throttling the other die while the at least one detected die is disabled.
4. The apparatus of claim 1 , further comprising an infrastructure module configured to communicate thermal information to the controller, the infrastructure module identifying the at least one die having the excessive temperature.
5. The apparatus of claim 4 , wherein the controller is further configured to update thermal information of the at least one detected die in a thermal mapping table.
6. 2. The apparatus of claim 1, wherein the controller is configured to perform a function in a flash translation layer (FTL) to disable the writes to the at least one detected die.
7. The apparatus of claim 5 , wherein the controller is further configured to update the thermal mapping table when the temperature falls below the threshold.
8. The apparatus of claim 1 , wherein the controller is configured to perform the relocation and update the thermal mapping table in response to a relocation required due to a thermal disable operation.
9. 10. The apparatus of claim 9, wherein the controller is configured to perform a last good page (FLGP) discovery action in recovery after an ungraceful shutdown (UGSD) event based on the updated thermal mapping table.
10. 1. A storage device comprising: a plurality of dies each including n sets of memory locations, where a corresponding one of the n sets from each die are combined together to form one metaset of the n metasets, each metaset including a maximum addressable unit of a host input / output (I / O) operation; a controller in thermal communication with each of the dies, Detecting an excessive temperature on at least one of the dies while performing a sequence of host writes; Disabling all writes to the at least one detected die while continuing the host writes to the other die in the corresponding one or more of the n metasets; a controller configured to reactivate writing to the detected at least one die when the temperature drops below a threshold.
11. 11. The apparatus of claim 10, wherein the controller is configured to continue performing the host writes to the other die at full capacity while the at least one detected die is disabled.
12. 11. The apparatus of claim 10, wherein the controller is configured to continue performing the host writes to the other die without throttling the other die while the at least one detected die is disabled.
13. 11. The apparatus of claim 10, wherein during a disabled state of the at least one detected die, the controller is configured to disable writes to the memory locations in the set corresponding to the at least one detected die while performing host writes without throttling to memory locations in the other sets on other die in one or more of the n metasets.
14. The apparatus of claim 10 , further comprising an infrastructure module configured to communicate thermal information to the controller, the infrastructure module identifying the at least ten dies having the excessive temperature.
15. The apparatus of claim 14 , wherein the controller is further configured to update thermal information of the at least one detected die in a thermal mapping table.
16. The apparatus of claim 15 , wherein the controller is further configured to update a thermal mapping table when the temperature falls below the threshold.
17. 1. A storage device comprising: a plurality of dies each having memory locations forming one of a plurality of selectively addressable pages; a controller coupled to each of the plurality of dies, Performing a host write operation to one or more pages; Detecting an excessive temperature in at least one of the dies; in response to said detecting, disabling all writes to said detected at least one die while continuing to perform writes to said memory locations of said other die; a controller configured to reactivate writing to the detected at least one die when the temperature drops below a threshold.
18. 20. The apparatus of claim 17, wherein the controller is further configured to continue to perform writes to the memory locations of the other die at the same write speed as before the detection of the excessive temperature.
19. 20. The apparatus of claim 17, wherein the controller is further configured to continue performing writes to the memory locations of the other die without throttling.
20. 20. The apparatus of claim 17, wherein the controller is further configured to reactivate the writes to the detected die at the same rate as the writes to the other die in the one or more pages.
Citation Information
Patent Citations
Memory system
JP2012018648A
Data storage device and access control method
JP2015032317A
System and Method for Memory System Management Based on Temperature Information of Memory System
JP2016510930A
Temperature management in open-channel memory devices
US20190043559A1
