Disabling pages in a NAND flash memory system

Abstract

Method for disabling pages (700) in a data storage system (120) containing an array (140) of non-volatile random access memory (NVRAM array), wherein the method comprises: Storing data (702) in the NVRAM array (140) in page stripes (600, 1102) distributed over several memory blocks (500, 1100), wherein at least two of the several memory blocks (500, 1100) over which a particular page stripe of the page stripes (600, 1102) is distributed contain different numbers of active pages (700), wherein a page (700) is the smallest unit that can be accessed in the NVRAM array (140), and a memory block (500, 1100) containing several pages (700) is the smallest unit that can be erased in the NVRAM array (140); Detecting (910, 914) an error in a specific page (700) of a specific block (500, 1100) of the NVRAM array (140); in response to detection (910, 914) of the error, disable (916) only the specific page (700) of the specific block (500, 1100); subsequent disabling (920) a physical memory area in the NVRAM array (140) containing the specified page (700) and several other pages (700) in response to disabling (918) a threshold number of pages (700) in the physical memory area; Disabling (924) a larger physical memory area in the NVRAM array (140) containing the specified page (700) in response to disabling (922) a threshold number of physical memory areas in the larger physical memory area.

Description

Technical area

[0001] The present invention relates to processing and storing data, and more particularly to improving the ability of a data processing system to efficiently perform page deactivation. Technical background

[0002] In certain data storage systems, data is stored in multiple storage units. For example, some of these systems use multiple individual hard disks or memory chips to store data, and the data stored in one or more storage units is mapped to data stored in other storage units in such a way that data errors in one or more storage units can be detected and possibly corrected. One such approach is to store a given amount of data across multiple storage locations, dividing the amount of data into equal-sized sections, where each section is sometimes referred to as "data pages," and then storing the data pages in multiple storage locations so that a data page is stored in each of multiple storage units.In conjunction with this approach, another storage unit may be used to store a page of backup information, where a given page of backup information is associated with a specific group of data pages stored in the multiple storage locations. In some cases, the group of memory pages at the multiple locations used to store associated data is referred to as a "data stripe" or "page stripe."

[0003] In addition to the data protection information for each data stripe, individual data pages can also be protected by error-correcting code (ECC), which can be used to detect errors and correct a specific number of errors on the page. ECC protection is provided at a specific codeword, the size of which is often referred to as the codeword (or block) length. Multiple codewords can be present on a data page. Upon a certain number of errors or upon the occurrence of errors, the data storage system can prohibit the use of (or disable) sections of the data storage that represent the source of the error.In data storage systems using NAND flash memory, a data page is the smallest unit of memory that can be accessed by read and write operations, and a block (or erase block) containing many pages is the smallest unit of memory that can be erased. Therefore, it is common practice for data storage systems to disable an entire erase block in response to an ECC error if only one codeword is detected on a single page in the erase block.

[0004] US 2011 / 0 078 496 A1 discloses methods and apparatus for stripe-based storage operation. One example includes writing data in a first stripe across a volume of a plurality of storage devices. A portion of the first stripe is updated by writing updated data to a portion of a second stripe across the volume of the plurality of storage devices. The portion of the first stripe is invalidated. The invalidated portion of the first stripe and a remainder of the first stripe are retained until the first stripe is restored.

[0005] US Pat. No. 8,176,284 B2 discloses methods and apparatus for protecting data stored in a FLASH memory system, comprising a controller capable of protecting data using memory page stripes of varying sizes. The controller is configured to store data in FLASH memory devices in the form of page stripes, each page stripe comprising a plurality of information pages, and each information page stored on a different FLASH memory chip. The controller stores the data such that the pages comprising each page stripe comprise multiple data pages and at least one backup page.In one implementation, the page stripes stored by the controller include a first page stripe with N data pages and one backup page and a second page stripe with M data pages and one backup page, where N is an integer greater than three and M is an integer less than N. Brief description of the inventionTechnical problem

[0006] The present disclosure recognizes that this conventional strategy for disabling bad blocks is completely excessive and may unnecessarily shorten the lifetime of a NAND flash memory device because the NAND flash memory device itself is disabled when a threshold of its erase blocks are disabled. Problem solving

[0007] The problems underlying the invention are solved by the features of the independent claims. Embodiments of the invention are the subject of the dependent claims.

[0008] In a data storage system containing an array of non-volatile random access memory (NVRAM) (e.g., a NAND flash memory array), a page is the smallest unit of the NVRAM array that can be accessed by read and write operations, and in a memory block containing multiple pages, it is the smallest unit of NVRAM that can be erased. Data is stored in the NVRAM array in page stripes distributed across multiple memory blocks. In response to detecting an uncorrectable error in a particular codeword of a particular page of a particular block of the NVRAM array, only the particular page of the particular block is disabled, such that only at least two of the multiple memory blocks across which a particular one of the page stripes is distributed contain different numbers of active (non-disabled) pages. Short description of drawings Fig. 1 is an overview block diagram of a computing environment according to one embodiment. Fig. 2 illustrates an exemplary arrangement of physical memory in a NAND flash memory array according to the present disclosure. Fig. 3 illustrates an exemplary arrangement of physical memory in a NAND flash memory array according to the present disclosure. Fig. 4 illustrates an exemplary arrangement of physical memory in a NAND flash memory array according to the present disclosure. Fig. 5 illustrates an exemplary arrangement of physical memory in a NAND flash memory array according to the present disclosure. Fig. 6 shows an exemplary implementation of page stripes according to the present disclosure. Fig. 7 illustrates an exemplary data page according to the present disclosure. Fig. 8 shows an exemplary privacy page according to the present disclosure. Fig. 9 is a logical overview flow diagram of an exemplary embodiment of a data read process in which physical memory in a NAND flash memory array is disabled by page rather than by block. Fig. 10 is a logical overview flow diagram of an exemplary embodiment of a data write process in which data is written to a NAND flash memory array in a variable length page stripe across variable size data blocks. Fig. Figure 11 illustrates an example page stripe in a NAND flash memory array where physical memory is disabled page by page. Description of embodiments

[0009] One skilled in the art will appreciate that aspects of the present invention may be embodied as a system, method, or computer program product. Accordingly, aspects of the present invention may take the form of a purely hardware embodiment, a purely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, all of which may be generally referred to herein as a "circuit," "module," or "system." Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer-readable media having computer-readable program code executing therein.

[0010] Any combination of one or more computer-readable media may be used. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or unit, or any suitable combination of the foregoing.More specific examples (a non-exhaustive list) of the computer-readable storage medium would include: an electrical connection having one or more wires, a portable computer diskette, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer-readable storage medium may be any tangible medium capable of containing or storing a program for use by or in connection with an instruction execution system, apparatus, or device.

[0011] A computer-readable signal medium may contain a propagating data signal with computer-readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagating signal may take any of a variety of forms, including, but not limited to, electromagnetic, optical, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium, other than a computer-readable storage medium, capable of transmitting, disseminating, or transporting a program for use by or in connection with an instruction execution system, apparatus, or device.

[0012] Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including, but not limited to, wireless, wireline, fiber optic cable, radio frequency (RF) media, etc., or any suitable combination of the foregoing.

[0013] Computer program code for performing operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object-oriented programming language such as Java, Smalltalk, C++, or the like, and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages. The program code may execute only on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or only on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g.,over the Internet using an Internet service provider).

[0014] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be appreciated that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to form a machine, such that instructions executing via the processor of the computer or other programmable data processing apparatus generate means for implementing the functions / effects specified in the flowchart and / or block diagram block or blocks.

[0015] These computer program instructions may also be stored in a computer-readable medium that can instruct a computer, other programmable data processing apparatus, or other devices to function in a particular manner such that the instructions stored in the computer-readable medium produce an article of manufacture that includes instructions that implement the function / effect specified in the flowchart and / or block diagram block(s).

[0016] Computer program instructions may also be loaded into a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other devices to produce a computer-implemented process, such that the instructions executing on the computer or other programmable device provide processes for implementing the functions / effects specified in the flowchart and / or block diagram block or blocks.

[0017] In the characters and especially in Fig. 1 is a high-level diagram of an exemplary computing environment 100 including a data storage system in which NAND flash memory is disabled on a page-by-page basis rather than a block-by-block basis, as described below. As shown, the computing environment 100 includes at least one processor system 102 having one or more processors 104 that process instructions and data. The processor system 102 may additionally include local memory 106 (e.g., dynamic random access memory (DRAM) or disk storage) that may store program code, operands, and / or execution results of the processing performed by the processor(s) 104. In various embodiments, the processor system 102 may be, for example, a mobile computing device (such asIt can be a mobile device (e.g., a smartphone or tablet computer), a laptop or desktop personal computer system, a server computer system (such as a POWER series system available from International Business Machine Corporation), or a mainframe computer system. It can also be an embedded processor system, using various processors such as ARM, PowerPC, Intel X86, or any other processor in combination with memory caches, memory controllers, local storage, I / O bus hubs, etc.

[0018] The processor system 102 further includes an input / output (I / O) adapter 108, which is connected directly (i.e. without any intervening unit) or indirectly (i.e. via at least one intervening unit) to a data storage system 120 via an I / O channel 110. In various embodiments, the I / O channel may use any one or a combination of known or future-developed data transmission protocols, including, for example, Fibre Channel (FC), FC over Ethernet (FCoE), Internet Small Computer System Interface (iSCSI), Transport Control Protocol / Internet Protocol (TCP / IP), Peripheral Component Interconnect Express (PCIe), etc. The I / O operations (IOPs) transmitted over the I / O channel 110 include read IOPs, through which the processor system 102 requests data from the data storage system 120, and write IOPs, through which the processor system 102 requests data to be stored in the data storage system 120.

[0019] In the illustrated embodiment, data storage system 120 includes an interface 122 through which data storage system 120 receives and responds to IOPs 102 via I / O channel 110. Interface 122 is coupled to a flash controller 124 (e.g., an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA)) that includes associated flash controller memory 126 (e.g., dynamic random access memory (DRAM)). Flash controller 124 is also coupled to a CPU 128 that includes associated CPU memory 130 (e.g., DRAM) and further to a NAND flash memory array 140. In embodiments where the flash controller 124 is implemented with an FPGA, the CPU 128 may program and configure the flash controller 124 during boot-up of the data storage system 120.After power-up, during general operation, the flash controller 124 receives read and write IOPs via the I / O channel 110 and the interface 122 to read data stored in the NAND flash memory array 140 and / or to store data in the NAND flash memory array 140. The flash controller 124 services these IOPs, for example, by accessing the NAND flash memory array 140 to read or write the requested data from the NAND flash memory array 140, or by accessing a buffer (not shown) associated with the NAND flash memory array 140.

[0020] Flash controller 124 implements a flash translation layer (FTL), which provides logical-to-physical address translation to enable access to specific memory locations on NAND flash memory array 140. Generally, an IOP received by flash controller 124 from a host device, such as processor system 102, contains the logical block address (LBA) where the data to be accessed (read or written) is located and, if a write IOP, contains the write data to be stored in data storage system 120. The IOP may also specify the extent (size) of the data to be accessed. Depending on the protocol and features supported by data storage system 120, additional information may also be transmitted.As mentioned above, NAND flash memory, such as that used in NAND flash memory array 140, is limited by its design such that the smallest unit of data accessible by a read or write IOP is the size of a single flash memory page, e.g., 16 kilobytes (KB). The LBA provided by the host device thus corresponds to a page in a logical address space. The flash translation layer translates this LBA into a physical address assigned to a corresponding physical location in NAND flash memory array 140. Flash controller 124 may perform address translation and / or store mappings between logical and physical addresses in a logical / physical translation data structure, such as a translation table 150, which may be conveniently stored in flash controller memory 126.

[0021] In some embodiments, the components of data storage system 120 are mounted on the same printed circuit board, for example, using techniques such as surface mounting, through-hole plating, sockets and socket supports, and / or other mounting techniques. In such embodiments, data storage system 120 may optionally include one or more capacitors 132 on the printed circuit board to provide power to data storage system 120 during a power failure. The data storage system may therefore serve as a non-volatile storage system even though it utilizes various volatile components.

[0022] The NAND flash memory array 140 can take many forms in various embodiments. Fig. Figures 2 to 5 show an exemplary arrangement of a physical memory with a NAND flash memory array 140 according to the present disclosure.

[0023] As in Fig. 2, the NAND flash memory array 140 may be formed of thirty-two (32) individually addressable storage units of NAND flash memory. In the illustrated example, each of the flash memory storage units M0a through M15b takes the form of a printed circuit board-mounted flash memory module, such as a single-level cell (SLC) or multi-level cell (MLC) NAND flash memory module. The thirty-two NAND flash memory modules are arranged in sixteen groups of two (M0a, M0b) through (M15a, M15b). For purposes of the physical addressing scheme, each group of two modules forms a "lane," sometimes referred to as a "channel," so that the NAND flash memory array 140 includes sixteen channels or lanes (Lane0 through Lane15).

[0024] In a preferred embodiment, each of the individual lanes has a corresponding associated bus connecting it to the flash controller 124. Thus, by directing its data transfers to one of the dedicated data transfer buses, the flash controller 124 can direct its data transfers to one of the lanes of memory modules. Because each data transfer bus for a given lane is independent of the data transfer buses for the other lanes, the flash controller 124 can issue commands and send or receive data over the various data transfer buses simultaneously, thereby allowing the flash controller 124 to access the memory modules corresponding to each lane simultaneously or nearly simultaneously.

[0025] In Fig. 3 illustrates an exemplary embodiment of a flash memory module 300 that may be used to implement the flash memory modules M0a to M15b of Fig. 2 to implement. As in Fig. Figure 3 shows the physical memory locations provided by the Flash Memory Module 300, further subdivided into physical locations that can be addressed and / or identified by chip enable signals (CEs). In the example of Fig. 3, the physical memory of each flash memory chip 300 is divided into four regions for chip enable signals, each with a CE line (CE0, CE1, CE2, and CE3), which is asserted by the flash memory controller 124 to enable access to or from the physical memory locations in the corresponding CE region. Each CE region is in turn divided into multiple semiconductor die (Die0 and Die1), each of which has two planes (Plane0 and Plane1). Each plane represents a collection of blocks (described below) that are physically interconnected due to the physical layout of the flash memory chips and that share common circuitry (e.g., I / O buffers) to perform various operations, such as read and write operations.

[0026] As further stated in the Fig. 4 and Fig. 5 includes an exemplary level 400 that can be used to configure each of the levels in the flash memory module 300 of Fig. 3, 2048 blocks of physical memory. A block 500 is generally a collection of physical pages that are usually physically connected to one another. This connection is such that a block is defined to represent the smallest unit of physical memory locations that can be erased in the NAND flash memory array 140. In the embodiment of Fig. 5, each block 500 contains 256 physical pages, where a physical page is defined as the smallest individually addressable unit of data for read and write access. In the exemplary system, each physical page of data contains a common capacity (e.g., 16 KB) for data storage plus additional storage for metadata, which is described in more detail below. Thus, data is written to or read from the NAND flash memory array 140 page by page, but erased block by block. As further described in Fig. 5, each block 500 preferably includes page status information (PSI) 502 indicating the status of each physical page in that block 500 as deactivated (i.e., retired from use) or non-deactivated (i.e., active or still in use). In various implementations, PSI 502 may be collected in a single data structure (e.g., a vector or table) in block 500, distributed within block 500 (e.g., as one or more bits of metadata appended to each physical page), or maintained elsewhere in data storage system 120 (e.g., in a data structure in memory 126 of the flash controller).

[0027] As stated above, the data storage system 120 generally does not allow external devices to directly address and / or access the physical memory locations in the NAND flash memory array 140. Instead, the data storage system 120 is configured to present a single, contiguous logical address space to the external devices, allowing host devices to read and write data from and to LBAs in the logical address space while allowing the flash controller 124 and CPU 128 to control where the data associated with the various LBAs is actually located in the physical memory locations included in the NAND flash memory array 140. In this way, the performance and longevity of the NAND flash memory array 140 can be intelligently managed and optimized.

[0028] When write operations occur in the data storage system 120, the flash controller 124 and CPU 128 are required to invalidate the data stored in one of the physical pages of one of the erase blocks in the NAND flash memory array 140. The new data is then merged with data being written and eventually stored in other locations in the NAND flash memory array 140. It may then be detected that pages or portions of pages are being invalidated, and therefore portions of the NAND flash memory array 140 become unused. The flash controller 124 and CPU 128 must eventually reclaim this space through a process called garbage collection. Certain erase blocks are selected based on a number of factors, such as what proportion of the data in the erase blocks is invalid.Valid data is read and written to new erase blocks along with new writes from the host.

[0029] Because the flash translation layer implemented by data storage system 120 separates the logical address space provided to host devices from the physical memory in NAND flash memory array 140, the size of NAND flash memory array 140 does not need to be equal to the size of the logical address space offered to host devices. In most embodiments, it is useful to offer a logical address space that is smaller than the total available physical memory (i.e., over-provisioning NAND flash memory array 140). Over-provisioning thus ensures that physical memory resources are available when the logical address space is fully utilized, even in the presence of a certain amount of invalid data, as described above.In addition to invalid data that has not yet been reclaimed, the space provided by over-provisioning can be used to ensure that sufficient logical space is available even when memory errors are present, and the memory overhead necessitates the use of data protection schemes such as error-correcting code (ECC), cyclic redundancy check (CRC), and parity.

[0030] In a preferred embodiment, data is typically written in groups of associated physical pages of NAND flash memory array 140, referred to herein as "page stripes." While the length of the various page stripes stored in NAND flash memory array 140 can vary, and preferably does, in one embodiment, each page stripe contains two to fifteen data pages of write data (typically provided by a host device) and an additional page (a "data protection page") used to store backup information for the write data. Fig. 6 illustrates an exemplary page stripe 600 that includes N data pages (i.e., Dpage00 through DpageN-1) and one privacy page (i.e., PpageN).

[0031] Fig. 7 illustrates an exemplary format of each data page in the page stripe 700. In this example, a data page 700 includes a 16 KB data field 702, along with additional fields for metadata describing the data page. In the illustrated example, these metadata fields include an LBA field 704 containing the LBA of the data page 700, a CRC field 706 containing the CRC value calculated for the combination of the data field 702 and the LBA field 704, and an ECC field 708 containing an ECC value calculated, in the illustrated example, from a combination of the contents of the data field 702, the LBA field 704, and the CRC field 706.

[0032] Fig. 8 shows an example format of the data protection page of page stripe 800. In the example shown, data protection page 800 includes a data XOR field 802 containing the bitwise exclusive-or (XOR) of data fields 702 of data pages 700 in data stripe 600. Data protection page 800 further includes an LBA XOR field 804 containing the bitwise XOR of LBA fields 704 of data pages 700 in data stripe 600. Finally, data protection page 800 includes a CRC field 806 and an ECC field 808 that store a CRC value and an ECC value, respectively, for data protection page 800. Such a protection scheme is commonly referred to as RAID5 because the parity field is not always located at a particular flash level. However, it should be clear that alternative data protection schemes such as Reed-Solomon can be used.

[0033] The data page and data protection page formats described above protect data stored in a page stripe using multiple protection mechanisms. First, the use of ECC bits allows for the correction of a specific number of bit errors in the flash page. Depending on the ECC method used, potentially hundreds or even thousands of bits in a NAND flash page can be corrected. After an ECC check and correction is performed, the corrected CRC field is used to validate the corrected data. When used together, these two mechanisms allow for the correction of relatively benign errors and the detection of more serious errors using only local, intra-page information.If an uncorrectable error occurs on a data page, for example, due to a failure of the physical page used to store the data page, the contents of the data field and the LBA field of the faulty data page can be reconstructed from the other data pages and the data protection page for the page stripe.

[0034] While the physical storage locations where the data pages and the data protection page are located will vary within a NAND flash memory array 140, in one embodiment, the data pages and the data protection page comprising a given data stripe are preferably stored in physical storage locations selected to optimize the overall function of the data storage system 120. For example, in some embodiments, the data pages and the data protection page comprising a data stripe are stored such that different physical tracks are used to store each of the data pages and the data protection page. Such embodiments support efficient access to a data stripe because the flash controller 124 can access all pages of data comprising the data stripe simultaneously or nearly simultaneously.It should be noted that assigning pages to lanes does not need to be sequential (i.e., data pages can be stored in any order in each lane), and unless a page stripe is a full-length page stripe (e.g., containing fifteen data pages and one privacy page), the lanes used to store the page stripe do not need to be adjacent.

[0035] Having described the general physical structure of the data storage system 120 and aspects of how data is addressed and stored in the form of data stripes in the NAND flash memory array 140, certain functional aspects of the data storage system 120 will now be described, including aspects relating to reading and writing data from and to the NAND flash memory array 140.

[0036] In Fig. Figure 9 illustrates a logical high-level flowchart of an exemplary method for performing a read operation in data storage system 120, in which physical memory may be disabled on a page-by-page basis. The method may be performed, for example, by flash controller 124 and / or CPU 128 in hardware, software, or a combination thereof. For ease of explanation, the following assumes that the process is performed by flash controller 124. As with the other flowcharts provided herein, steps are presented in logical order rather than in strict chronological order, and in some embodiments, at least some of the steps may be performed in a different order or concurrently.

[0037] The illustrated process begins at block 900 and then proceeds to block 902, which illustrates a flash controller 124 awaiting receipt of a read IOP from an external host device, such as a processor system 102. The read IOP generally specifies the LBA of the target data pages requested by the host device. In response to receiving the read IOP, the flash controller 124 translates the LBA (e.g., by referring to a logical / physical translation table (TT) 150 in the flash controller memory 126) to determine the physical address of the physical page where the requested data page is stored (block 904).After the physical address is determined, the flash controller 124 accesses the target data page using the physical address by issuing a read request to read the target data page from the physical page associated with the physical address (block 906). The read IOP may request various amounts of data, but for simplicity, it is assumed that the schedule 900 is executed once for each logical page designated by the read IOP.

[0038] At block 910, the flash controller 124 calculates the ECC value for the data page and compares the calculated ECC value with the ECC contents of the ECC array to determine if the data page may contain a data error. Such data errors can occur as a result of several problems, including trapped charges in the oxide layer of the physical device or charges that have leaked from the floating gate. These errors can be permanent, rendering the cells unable to store or read data correctly, or the errors can occur as a result of problems affecting data storage or electrical noise that occurs during write or read operations of neighboring cells. Such errors are not present if the cell is erased and then rewritten.True soft errors are essentially random and are typically uniquely associated with specific physical pages, blocks, or other physical regions of the NAND flash memory array 140. A hard error, on the other hand, is the corruption of one or more data bits caused by a physical error. Hard errors can be caused by a variety of factors, including the physical failure of one or more components within a given memory chip (such as the failure of a charge pump), the physical failure of an entire memory chip or the external power structure for that chip (e.g., the interruption of a power line or address line to a chip), the physical failure of an entire chip or a portion thereof as a result of environmental factors (e.g., excessive temperature, magnetic field, humidity, etc.).Because hard errors generally arise from the physical design of the memory system, hard errors are uniquely associated with a specific group of memory chips, a specific memory chip, or specific physical regions within a chip (such as a CE, plane, block, or page).

[0039] In response to a determination at block 910 that the data page does not contain a data error, the process proceeds to block 930, which is described below. However, in response to a determination at block 910 that the data page contains a data error, the flash controller 124 may correct the error in the data page using ECC (e.g., if more than 50 bits are in error in the codeword and ECC is capable of correcting more than 50 in error bits in the codeword). At block 914, the flash controller 124 determines whether the detected error may be a hard error. The flash controller 124 may conclude that a data error is a hard error based, for example, on a frequency or number of data errors in a given physical memory region.In other words, the flash controller 124 may determine that all physical pages in a given erase block are corrupted. In response to a determination at block 914 that the data error is a correctable error, the process proceeds to block 930, described below. On the other hand, if the flash controller 124 determines that the data error is a non-correctable error, the process proceeds to block 916.

[0040] At block 916, the flash controller 124 simply marks the physical page storing the target data page as disabled (i.e., no longer available for allocation to store a data page of a new page stripe), e.g., in the PSI 502 of the block containing that physical page. Thus, unlike prior art systems that disable physical memory in NAND flash memory on a block-by-block basis, the data storage system 120 disables physical memory in the NAND flash memory array 140 on a page-by-page basis in response to detecting a single data error. It is understood that disabling a physical page of physical memory (e.g., 16 KB) rather than a block (e.g., 4 MB) conserves physical memory resources, improves performance, and extends the lifetime of the NAND flash memory array 140.However, as a result of disabling a page, the effective sizes of blocks of physical memory of the NAND flash memory system 140 will vary as described below.

[0041] In this example implementation, it waits until a physical page contains one or more codewords containing uncorrectable errors before performing deactivation. However, in certain implementations, deactivation of a page may be chosen at a point before it becomes uncorrectable. For example, if an implementation uses BCH ECC over approximately 1024 bytes and 50 erroneous bits can be corrected, the flash controller 124 may decide to deactivate a page when the number of uncorrectable bits reaches a number less than 50, for example, 48. Furthermore, it will be understood by one of ordinary skill in the art that, depending on the flash technology used, the flash controller 124 may choose to perform additional steps in block 914 before deactivating the page.For example, the flash manufacturer may request that the flash controller 124 change certain parameters for that page or block and perform a reread of the page. If the data is now correctable, the flash controller 124 would follow path No from block 914. Thus, block 914 may include many additional steps, not described here, to determine whether the page contains a hard error or a truly uncorrectable error.

[0042] At block 918, the flash controller 124 further determines, by reference to PSI 502, whether the number of disabled pages of physical memory in a physical memory region containing the page disabled at block 916 may now be at (e.g., greater than and / or equal to) a first threshold T1. In at least some embodiments, the first threshold T1 is configurable. Performance testing has shown that the performance of the data storage system 120 is optimized when the first threshold T1 is less than about 10% of the physical pages in the physical memory region, and in particular, is configured to be about 5% of the physical pages in the physical memory region. It is particularly preferable that the first threshold T1 be configured to be about 4% of the physical pages in the physical memory region.These figures are merely examples and can vary significantly depending on the use and requirements of the data storage system. For instance, the physical storage area to which the first threshold T1 refers could be a block, a layer, a chip, a CE area, or the entire flash module.

[0043] In response to the flash controller 124 determining at block 918 that the first threshold is not met, the process proceeds to block 930. However, in response to the flash controller 124 determining at block 918 that the first threshold is met, the flash controller 124 marks the entire region of physical memory containing the disabled physical page as disabled and thus unavailable for allocation to store new page stripes (block 920). If the physical memory region determined to be disabled at blocks 918-920 is smaller than an entire flash memory module, the flash controller 124 may optionally determine whether or not to disable larger physical memory regions containing the disabled physical page, as shown at optional blocks 922-924.As indicated, the additional determination regarding disabling larger physical memory regions may be made based on a comparison of the number of disabled physical memory regions (e.g., pages, blocks, planes, chips, or CE regions) in the larger physical memory region and a second threshold T2 (which may be different from the first threshold T1). The process proceeds either from block 920 or from one of blocks 922 and 924 to block 930. It may also be determined that there are now too many disabled memory resources in the NAND flash memory system 140 to be able to achieve required performance levels, and the flash controller 124 may send a response to the user indicating that it is time to replace the physical resource that the NAND flash memory system 140 contains.

[0044] In block 930, the flash controller 124 attempts to validate the data field 702 and the LBA field 704 of the target data page by calculating a cyclic redundancy code and comparing the calculated CRC with the one contained in the CRC field 706. In response to a successful validation of the data page, the process proceeds to block 934, described below. However, in response to a CRC validation failure, the flash controller 124 reads the entire data stripe and reconstructs the correct contents of the target data page from the other data page(s) and the data protection page of the page stripe (block 932). In addition, the flash controller 124 moves the page stripe (containing the reconstructed target data page) to another physical location in the NAND flash memory system 140, for example, the Fig. 10 is used, which is described below. From block 930 or block 932, the process proceeds to block 934, which illustrates that the flash control unit 124 transfers the target data page to the requesting host unit. The process then ends at Fig. 9 in Block 940. The procedure of Fig. 9 may be executed again at a later unspecified time depending on other ongoing operations.

[0045] As mentioned above, the sequence of operations performed in Fig. 9, by way of example, and embodiments are contemplated where the order of operations differs from that illustrated above. For example, embodiments are contemplated where flash controller 124 transfers the data page to the host device before performing CRC validation and subsequently transfers an indication of whether the data page is valid. Additionally, in some embodiments, the read IOP may access multiple data pages instead of a single data page.

[0046] In Fig. Figure 10 illustrates a logical overview flow diagram of an exemplary method for performing a write operation in data storage system 120. The method may be executed, for example, by flash controller 124 and / or CPU 128 in hardware, firmware, software, or a combination thereof. For ease of explanation, the following assumes that the process is performed by flash controller 124.

[0047] The procedure begins in block 1000 and proceeds to block 1002, which illustrates a flash control unit 124 that expects to receive a write IOP from a host unit, such as a processor system 102. The write IOP contains, for example, data to be written to the NAND flash memory array 140 and a specification of the target LBA(s) where the host unit wants to store the data. In response to receiving a write IOP, the process goes from block 1002 to blocks 1004 to 1106, which illustrate that the flash control unit 124 determines (e.g., by referring to a logic / physical translation table 150) whether the target LBA(s) specified in the write IOP are currently allocated to one or more physical memory pages, and if so, the status of each data page currently allocated to a target LBA is changed to indicate that it is no longer valid.As is known in the art, invalidated pages (and the blocks they contain) are eventually erased and made available again for allocation to store data by a garbage collection process of the flash controller 124.

[0048] In block 1008, the Flash Control Unit 124 allocates physical pages to form a page stripe for storing write data. As mentioned above, the Flash Control Unit 124 preferentially selects each of the physical pages allocated for write data storage from a different block in a different track. Furthermore, any disabled pages are ignored by the Flash Control Unit 124 when allocating physical pages for storing the page stripe, as specified by PSI 502. As specified in block 101, the Flash Control Unit 124 then begins storing the first data page of the write data on the first physical page allocated to the page stripe. In block 1012, the Flash Control Unit 124 also calculates the CRC and ECC values ​​for the data page and stores these values ​​in CRC field 706 and ECC field 708 of the data page.As indicated by block 1014, the flash controller 124 determines whether all write data has been written to the NAND flash memory system 140. If the flash controller 124 determines that not all write data has been written to the NAND flash memory system 140, the process returns to block 1010 described above. In response to determining that all write data has been stored to the NAND flash memory system 140, the flash controller 124 calculates parity information for the data stripe (e.g., by performing a bitwise XOR operation of all data pages in the page stripe) and stores the parity information in the data protection page of the page stripe (block 1016). Again, any physical pages that are disabled and would otherwise be assigned to the data stripe are excluded from the parity calculation.Flash controller 124 also updates the logical / physical address translation table to map the one or more physical pages for storing the write data to the LBA(s) specified by the host device (block 1018). The process then ends at block 1020.

[0049] It should be noted that blocks 1008, 1010, and 1012 contain steps that verify that the programming of that physical page is being executed correctly. If the flash controller 124 determines that the write operation was not executed correctly, the flash controller 124 uses various recovery techniques, such as retrying the write operation. When the write operation finally times out, the flash controller 124 proceeds to the next block. However, if the write operation is unsuccessful, the flash controller 124 disables that page in the same way as if a read operation were unsuccessful. After that page is disabled, the flash controller 124 modifies its tables accordingly and proceeds to the next non-disabled page to execute blocks 1008, 1010, or 1012.

[0050] In Fig. 11 shows an exemplary page stripe 1102 generated according to the method of Fig. 10 was written to the NAND flash memory system 140. As mentioned above, page stripes written to a NAND flash memory system 140 can vary in length from two to fifteen data pages plus one privacy page. However, in the example shown, the flash control unit 124 wrote a data stripe with six data pages and one privacy page. In this example, the flash control unit 124 selected the physical pages for storing the data stripe from a set of blocks, which includes blocks 1100a to 1100h in tracks 0 to 7. In each of the blocks 1100a to 1100h, one or more pages were disabled, as shown in Fig. 11 is graphically indicated by an "X" and recorded by the flash control unit 124 in PSI 502. Thus, for example, Page1, Page255 of block 1100a are disabled; Page0 and Page86 of block 1100g are disabled; and Page23 of block 1100h are disabled.

[0051] In flash memory, it is advantageous to write data in blocks sequentially per page in a specific order specified by the flash manufacturer. Thus, when writing a block of the NAND flash memory system 140, it may be desirable and useful to first write to Page0, then Page1, and so on, until the block is filled or nearly filled with stored data. In some implementations, it may be desirable to write in a specific order other than Page0, Page1, etc., but the end result is still sequential processing of pages in a specific order for the entire block. In the embodiment of Fig.11, this form of writing data to NAND flash memory 140 is implemented by having flash controller 124 maintain a count such that the first page stripe written to a given group of blocks is written over Page0 of the blocks in the group of blocks, the next page stripe is written over Page1 of the blocks in the group of blocks, and so on. It should be noted, however, that all disabled pages, such as Page0 of block 1100g, are excluded from being assigned to a page stripe, such as page stripe 1102. Likewise, when the next page stripe is written to the group of blocks, Page1 of block 1100a is not used.

[0052] The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or section of code comprising one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions specified in the block may not occur in the order indicated in the figures. For example, two blocks shown sequentially may actually execute substantially concurrently, or the blocks may occasionally execute in the reverse order depending on the functionality involved.It is also noted that each block in the block diagram and / or flowchart illustrations, and combinations of blocks in the block diagram and / or flowchart illustrations, may be implemented by systems based on special purpose hardware that perform the specified functions or effects, or combinations of special purpose hardware and computer instructions.

[0053] As described, in some embodiments of a data storage system including a NAND flash memory array, a page is a smallest unit of the NAND flash memory array that can be accessed by read and write operations, and a memory block containing multiple pages is the smallest unit of the NAND flash memory array that can be erased. Data is stored in the NAND flash memory array in data stripes distributed across multiple memory blocks. In response to detecting an error in a particular page of a particular block of the NAND flash memory array, only the particular page of the particular block is disabled, such that at least two of the plurality of memory blocks across which a particular data stripe of the data stripes is distributed contain different numbers of active (non-disabled) pages.

[0054] Although the present invention has been presented in a certain manner and described with reference to one or more preferred embodiments, it should be clear to a person skilled in the art that various modifications to the form and details can be made without departing from the inventive concept and scope of the invention. For example, although aspects have been described with reference to a data storage system that includes a flash control unit which directs certain functions, it should be clear that the present invention can alternatively be implemented as a program product that includes a storage unit which stores program code which can be processed by a processor to execute these functions or which causes these functions to be executed.A “storage unit” as used here is specifically defined to include only objects of manufacture in the true sense of the word, and perishable media of propagation are per se excluded.

[0055] While embodiments involving the use of NAND flash memory have also been described, it should be understood that the inventions are not limited to the use of NAND flash memory, but can be applied to any other non-volatile random access memory (NVRAM) technology that may allow the definition of erase blocks larger than physical pages. For example, the disclosed techniques can be applied to phase-change memory (PCM), magnetoresistive RAM (MRAM), and resistive RAM (RRAM).

Claims

[1] A method for deactivating pages (700) in a data storage system (120) including a non-volatile random access memory (NVRAM) array (140), the method comprising: Storing data (702) in the NVRAM array (140) in page stripes (600, 1102) distributed across a plurality of memory blocks (500, 1100), wherein at least two of the plurality of memory blocks (500, 1100) over which a particular page stripe of the page stripes (600, 1102) is distributed contain different numbers of active pages (700), wherein a page (700) is a smallest unit that can be accessed in the NVRAM array (140), and a memory block (500, 1100) containing a plurality of pages (700) is a smallest unit that can be erased in the NVRAM array (140); Detecting (910, 914) an error in a particular page (700) of a particular block (500, 1100) of the NVRAM array (140); in response to detecting (910, 914) the error, deactivating (916) only the particular page (700) of the particular block (500, 1100); subsequently disabling (920) a physical memory region in the NVRAM array (140) containing the particular page (700) and a plurality of other pages (700) in response to disabling (918) a threshold number of pages (700) in the physical memory region; Disabling (924) a larger physical memory region in the NVRAM array (140) containing the particular page (700) in response to disabling (922) a threshold number of physical memory regions in the larger physical memory region. [2] The method of claim 1, wherein at least two of the plurality of page stripes (600, 1102) are distributed across different numbers of blocks (500, 1100). [3] The method of claim 1, wherein the detecting (910, 914) includes detecting a mismatch between a code calculated for the particular page (700) and a code stored in the NVRAM array (140) in association with the particular page (700). [4] The method of claim 1, wherein the physical memory area comprises the particular block (500, 1100). [5] The method of claim 1, further comprising: Calculating backup information (800) for one of the page stripes (600, 1102) as a whole by performing an operation on corresponding pages (700) of the plurality of blocks (500, 1100), wherein the calculating includes excluding deactivated pages (700) from the operation. [6] An apparatus for an NVRAM array (140) including a plurality of blocks (500, 1100) of physical memory, each containing a plurality of pages (700) of physical memory, wherein a page (700) is a smallest unit that can be accessed in the NVRAM array (140) and a block (500, 1100) is a smallest unit that can be erased in the NVRAM array (140), the apparatus comprising: a memory control unit (124) that stores data (702) in the NVRAM array (140) in page stripes (600, 1102) that are distributed across a plurality of memory blocks (500, 1100), wherein the memory control unit (124), in response to detecting (910, 914) an error in a particular page (700) of a particular block (500, 1100) of the NVRAM array (140), deactivates (916) only the particular page (700) of the particular block (500, 1100) so that at least two of the plurality of memory blocks (500, 1100) over which a particular page stripe of the page stripes (600, 1102) is distributed contain different numbers of active pages (700), wherein the memory controller (124), after deactivating (916) the particular page (700), deactivates (920) a physical memory region in the NVRAM array (140) containing the particular page (700) and a plurality of other pages (700) in response to deactivating (918) a threshold number of pages (700) in the physical memory region; wherein the memory controller (124) deactivates (924) a larger physical memory region in the NVRAM array (140) containing the particular page (700) in response to deactivating (922) a threshold number of physical memory regions in the larger physical memory region. [7] The apparatus of claim 6, wherein the memory controller (124) distributes at least two of the plurality of page stripes (600, 1102) across different numbers of blocks (500, 1100). [8] The apparatus of claim 6, wherein the memory control unit (124) detects (910, 914) the error by detecting a mismatch between a code calculated for the particular page (700) and a code stored in the NVRAM array (140) in association with the particular page (700). [9] The apparatus of claim 6, wherein the physical memory area comprises the particular block (500, 1100). [10] The apparatus of claim 6, wherein the memory control unit (124) calculates backup information (800) for one of the page stripes (600, 1102) as a whole by performing an operation on corresponding pages (700) of the plurality of blocks (500, 1100), excluding deactivated pages (700) from the operation. [11] A program product for a data storage system (120) including an NVRAM array (140) having a plurality of blocks (500, 1100) of physical memory, each containing a plurality of pages (700) of physical memory, wherein a page (700) is a smallest unit that can be accessed in the NVRAM array (140), and a memory block (500, 1100) is a smallest unit that can be erased in the NVRAM array (140), the program product comprising: a storage unit; and Program code stored in the memory unit that, when executed by a memory controller (124) coupled to the NVRAM array (140), causes the memory controller (124) to execute: Storing data (702) in the NVRAM array (140) in page stripes (600, 1102) distributed across a plurality of memory blocks (500, 1100), wherein at least two of the plurality of memory blocks (500, 1100) across which a particular page stripe of the page stripes (600, 1102) is distributed contain different numbers of active pages (700), wherein a page (700) is a smallest unit that can be accessed in the NVRAM array (140), and a memory block (500, 1100) containing a plurality of pages (700) is a smallest unit that can be erased in the NVRAM array (140); Detecting (910, 914) an error in a particular page (700) of a particular block (500, 1100) of the NVRAM array (140); in response to detecting (910, 914) the error, deactivating (916) only the particular page (700) of the particular block (500, 1100); subsequently disabling (920) a physical memory region in the NVRAM array (140) containing the particular page (700) and a plurality of other pages (700) in response to disabling (918) a threshold number of pages (700) in the physical memory region; Disabling (924) a larger physical memory region in the NVRAM array (140) containing the particular page (700) in response to disabling (922) a threshold number of physical memory regions in the larger physical memory region. [12] The program product of claim 11, wherein the memory control unit (124) distributes at least two of the plurality of page stripes (600, 1102) across different numbers of blocks (500, 1100). [13] The program product of claim 11, wherein the detecting (910, 914) includes detecting a mismatch between a code calculated for the particular page (700) and a code stored in the NVRAM array (140) in association with the particular page (700). [14] The program product of claim 11, wherein the physical memory area comprises the particular block (500, 1100). [15] The program product of claim 11, wherein the program code further causes the memory control unit (124) to execute: Calculating backup information (800) for one of the page stripes (600, 1102) as a whole by performing an operation on corresponding pages (700) of the plurality of blocks (500, 1100), wherein the calculating includes excluding deactivated pages (700) from the operation. [16] The program product of claim 11, wherein the program code further causes the memory control unit (124) to execute: Mapping logical addresses (704) specified in input / output operations (IOPs) to physical addresses of pages (700) in the NVRAM array (140) by referring to a translation table (150). [17] A program product according to claim 11, wherein: the NVRAM array (140) includes a plurality of chips arranged in tracks; and the program code further causes the memory control unit (124) to select the plurality of memory blocks (500, 1100) across which each page stripe (600, 1102) is distributed, each from different tracks.

Images