Cooperative memory subsystem data recovery

Abstract

Embodiments of the present application relate to cooperative memory subsystem data recovery. Exemplary methods, apparatus, and systems include detecting a failure of a first memory subsystem of a plurality of memory subsystems. First recovery instructions are sent to a second memory subsystem of the plurality of memory subsystems. The first recovery instructions direct the second memory subsystem to recover a first subset of data stored by the first memory subsystem. Second recovery instructions are sent to a third memory subsystem of the plurality of memory subsystems. The second recovery instructions direct the third memory subsystem to reconstruct a second subset of data stored by the first memory subsystem. The first and second subsets of data are different from each other.

Description

TECHNICAL FIELD

[0001] The present disclosure generally relates to recovery of memory subsystem data, and more particularly, to multiple memory subsystems that recover data of another memory subsystem in coordination and in parallel. BACKGROUND

[0002] A memory subsystem can include one or more memory devices that store data. The memory devices can be, for example, non-volatile memory devices and volatile memory devices. Generally, a host system can utilize a memory subsystem to store data at and retrieve data from the memory devices. SUMMARY

[0003] According to embodiments of the present disclosure, a method is provided. The method includes detecting a failure of a first memory subsystem of a plurality of memory subsystems, sending a first recovery instruction to a second memory subsystem of the plurality of memory subsystems, the first recovery instruction directing the second memory subsystem to recover a first subset of data stored by the first memory subsystem, and sending a second recovery instruction to a third memory subsystem of the plurality of memory subsystems, the second recovery instruction directing the third memory subsystem to recover a second subset of data stored by the first memory subsystem, the first and second subsets of data being different from each other.

[0004] According to embodiments of the present disclosure, a non-transitory computer- readable storage medium is provided. The medium includes instructions that, when executed by a processing device, cause the processing device to implement the following: detecting a failure of a first memory subsystem of a plurality of memory subsystems, sending a first recovery instruction to a second memory subsystem of the plurality of memory subsystems, the first recovery instruction directing the second memory subsystem to recover a first subset of data stored by the first memory subsystem, and sending a second recovery instruction to a third memory subsystem of the plurality of memory subsystems, the second recovery instruction directing the third memory subsystem to recover a second subset of data stored by the first memory subsystem, the first and second subsets of data being different from each other.

[0005] According to embodiments of the disclosure, a system is provided. The system includes a plurality of memory devices; and a processing device operably coupled with the plurality of memory devices to implement the following: detecting a failure of a first memory sub-system of a plurality of memory sub-systems; sending a first recovery instruction to a second memory sub-system of the plurality of memory sub-systems, the first recovery instruction directing the second memory sub-system to recover a first subset of data stored by the first memory sub-system, wherein the first recovery instruction further directs the second memory sub-system to write the recovered first subset of data to a spare memory sub-system; and sending a second recovery instruction to a third memory sub-system of the plurality of memory sub-systems, the second recovery instruction directing the third memory sub-system to recover a second subset of data stored by the first memory sub-system, the first and second subsets of data being different from one another, wherein the second recovery instruction further directs the third memory sub-system to write the recovered second subset of data to the spare memory sub-system. BRIEF DESCRIPTION OF DRAWINGS

[0006] The present disclosure will become more fully understood from the detailed description given herein and from the accompanying drawings, wherein: The figure is not to be considered in a limiting sense, as the present disclosure is to be measured only in accordance with the appended claims.

[0007] Figure 1 An example computing system including a memory sub-system according to some embodiments of the disclosure is illustrated.

[0008] Figure 2 An example illustrating additional details of a memory sub-system controller including a recovery component according to some embodiments of the disclosure is illustrated.

[0009] Figure 3 Another example computing system including a memory sub-system according to some embodiments of the disclosure is illustrated.

[0010] Figure 4 is a flow diagram of an example method for orchestrating a coordinated recovery of data of a failed storage sub-system according to some embodiments of the disclosure.

[0011] Figure 5 is a flow diagram of an example method for coordinating a recovery of data of another storage sub-system according to some embodiments of the disclosure.

[0012] Figure 6 is a flow diagram of another example method for orchestrating a coordinated recovery of data of a failed storage sub-system according to some embodiments of the disclosure.

[0013] Figure 7 is a block diagram of an example computer system in which embodiments of the present disclosure can operate. DETAILED DESCRIPTION

[0014] Aspects of the present disclosure relate to coordinating multiple memory subsystems to collaboratively and in parallel recover data of another storage subsystem. A memory subsystem can be a storage device, a memory module, or a hybrid of a storage device and a memory module. Examples of storage devices and memory modules are described below in connection with Figure 1 A host system can utilize a memory subsystem including one or more components, such as a storage device that stores data. The host system can provide data to be stored at the memory subsystem and can request data to be retrieved from the memory subsystem.

[0015] A memory device can be a non-volatile memory device. A non-volatile memory device is a package of one or more chips. One example of a non-volatile memory device is a NAND (also known as not-and) memory device. Examples of non-volatile memory devices are described below in connection with Figure 1 Other examples of non-volatile memory devices are described. The chips in a package can be assigned to one or more channels for communication with a memory subsystem controller. Each die can be composed of one or more planes. A plane can be grouped into logical units (LUNs). For some types of non-volatile memory devices (e.g., NAND memory devices), each plane is composed of a group of physical blocks, which are groups of memory cells for storing data. A cell is an electronic circuit that stores information.

[0016] Depending on the type of cell, a cell can store one or more bits of binary information and have various logical states related to the number of bits stored. A logical state can be represented by a binary value, such as “0” and “1” or a combination of such values. There are various types of cells, such as single-level cells (SLCs), multi-level cells (MLCs), triple-level cells (TLCs), and quad-level cells (QLCs). For example, an SLC can store one bit of information and have two logical states.

[0017] The capacity of memory subsystems, such as solid state drives, continues to increase. As a result, the impact of a single memory subsystem failure also continues to increase. Data protection techniques are employed to mitigate memory subsystem failures within a storage network. The computational overhead of modern data protection techniques that include computations of exclusive OR (XOR) and / or erasure coding (e.g., Reed-Solomon polynomials) can be significant for both encoding data and decoding data. These protection techniques can be implemented, for example, in a storage service stack located on a host system as a software-based storage stack solution. In addition to the computational overhead, the dispersion of the host system from the memory subsystems also adds latency and network traffic overhead. Even when implemented locally on a data processing unit or similar processing unit / controller, such as on a per storage shelf basis, the centralization of the encoding and decoding computations adds to the burden of already overburdened storage service stacks and increases network / switch bandwidth utilization. As a result, quality of service (QoS) can be significantly impacted.

[0018] Aspects of the disclosure address the above and other deficiencies by having multiple memory subsystems perform computations to recover data (user data and / or parity data) in a distributed and parallel manner. A host, processing unit, or other controller allocates portions of the recovery work to each of the multiple memory subsystems. Each memory subsystem performs its portion of the erasure computation to recover data stored by the failed memory subsystem and writes the recovered data to a spare memory subsystem. Since the memory subsystems operate in parallel, the recovery process takes less time than using conventional centralized recovery. This results in increased network / switch efficiency and better QoS.

[0019] Figure 1 An example computing system 100 including a memory subsystem 110 is described in accordance with some embodiments of the disclosure. The memory subsystem 110 can include media, such as one or more volatile memory devices (e.g., memory devices 140), one or more non-volatile memory devices (e.g., memory devices 130), or a combination thereof.

[0020] The memory subsystem 110 can be a storage device, a memory module, or a hybrid of a storage device and a memory module. Examples of storage devices include a solid state drive (SSD), a flash drive, a universal serial bus (USB) flash drive, an embedded Multi-Media Controller (eMMC) drive, a Universal Flash Storage (UFS) drive, a Secure Digital (SD) card, and a hard disk drive (HDD). Examples of memory modules include a dual in-line memory module (DIMM), a small outline DIMM (SO-DIMM), and various types of non-volatile dual in-line memory modules (NVDIMM).

[0021] The computing system 100 can be, for example, a desktop computer, a laptop computer, a network server, a mobile device, a vehicle (e.g., an airplane, a drone, a train, a car, or other transportation vehicle), an Internet of Things (IoT) capable device, an embedded computer (e.g., an embedded computer included in a vehicle, industrial equipment, or a networked commercial appliance), or a computing device that includes a memory and a processing device.

[0022] The computing system 100 can include a host system 120 coupled to one or more memory sub-systems 110. In some embodiments, the host system 120 is coupled to different types of memory sub-systems 110. Figure 1 One example of a host system 120 coupled to one memory sub-system 110 is illustrated. As used herein, “coupled to” or “coupled with” generally refers to a connection between components that can be an indirect communicative connection or a direct communicative connection (e.g., without intervening components), whether wired or wireless, including connections such as electrical, optical, magnetic, etc.

[0023] The host system 120 can include a processor chipset and a software stack executed by the processor chipset. The processor chipset can include one or more cores, one or more caches, a memory controller (e.g., NVDIMM controller), and a storage protocol controller (e.g., PCIe controller, SATA controller). The host system 120 uses the memory sub-system 110 to, for example, write data to and read data from the memory sub-system 110.

[0024] The host system 120 can be coupled to the memory sub-system 110 via a physical host interface. Examples of a physical host interface include, but are not limited to, a Serial Advanced Technology Attachment (SATA) interface, a Peripheral Component Interconnect Express (PCIe) interface, Compute Express Link (CXL), a Universal Serial Bus (USB) interface, Fibre Channel, Serial Attached SCSI (SAS), Small Computer System Interface (SCSI), Double Data Rate (DDR) memory bus, a Dual In-line Memory Module (DIMM) interface (e.g., a DIMM socket interface that supports Double Data Rate (DDR)), Open NAND Flash Interface (ONFI), Double Data Rate (DDR), Low Power Double Data Rate (LPDDR), or any other interface. The physical host interface can be used to transfer data between the host system 120 and the memory sub-system 110. When the memory sub-system 110 is coupled with the host system 120 through a PCIe interface, the host system 120 can further access components (e.g., memory devices 130) utilizing a NVM Express (NVMe) interface. The physical host interface can provide an interface for passing control, address, data, and other signals between the memory sub-system 110 and the host system 120.Figure 1 The memory sub-system 110 is illustrated as an example. In general, the host system 120 can access multiple memory sub-systems via the same communication connection, multiple separate communication connections, and / or a combination of communication connections.

[0025] The memory devices 130, 140 can include any combination of different types of non-volatile memory devices and / or volatile memory devices. Volatile memory devices (e.g., the memory device 140) can be, but are not limited to, random access memories (RAMs), such as dynamic random access memories (DRAMs), static random access memories (SRAMs), and synchronous dynamic random access memories (SDRAMs).

[0026] Some examples of non-volatile memory devices (e.g., the memory device 130) include negative-and (NAND) type flash memory and in-place write memory, such as a three-dimensional cross-point (“3D cross-point”) memory device, which is a cross-point array of non-volatile memory cells. A cross-point array of non-volatile memory can perform bit storage based on bulk resistance changes in conjunction with a stackable cross-grid format data access array. Further, in contrast to many flash-based memories, a cross-point non-volatile memory can perform in-place write operations, where a non-volatile memory cell can be programmed without previously erasing the non-volatile memory cell. NAND type flash memory includes, for example, two-dimensional NAND (2D NAND) and three-dimensional NAND (3D NAND).

[0027] Although non-volatile memory devices such as NAND type memory (e.g., 2D NAND, 3D NAND) and 3D cross-point arrays of non-volatile memory cells are described, the memory device 130 can be based on any other type of non-volatile memory, such as read-only memory (ROM), phase change memory (PCM), self-selecting memory, other chalcogenide-based memory, ferroelectric transistor random access memory (FeTRAM), ferroelectric random access memory (FeRAM), magnetic random access memory (MRAM), spin-transfer torque (STT)-MRAM, conductive-bridge RAM (CBRAM), resistive random access memory (RRAM), oxide-based RRAM (OxRAM), or negative-or (NOR) flash memory and electrically erasable programmable read-only memory (EEPROM).

[0028] The memory sub-system controller 115 (or, for simplicity, the controller 115) can communicate with the memory devices 130 to perform operations such as reading data, writing data, or erasing data at the memory devices 130 and other such operations (e.g., in response to commands scheduled by the controller 115 on a command bus). The memory sub-system controller 115 can include hardware such as one or more integrated circuits and / or discrete components, a buffer memory, or a combination thereof. The hardware can include digital circuitry with specialized (i.e., hard-coded) logic for performing the operations described herein. The memory sub-system controller 115 can be a microcontroller, special purpose logic circuitry (e.g., a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), etc.), or another suitable processor.

[0029] The memory sub-system controller 115 can include a processing device 117 (processor) configured to execute instructions stored in local memory 119. In the illustrated example, the local memory 119 of the memory sub-system controller 115 includes embedded memory configured to store instructions for performing various processes, operations, logic flows, and routines that control operation of the memory sub-system 110, including handling communications between the memory sub-system 110 and the host system 120.

[0030] In some embodiments, the local memory 119 can include memory registers that store memory pointers, fetched data, etc. The local memory 119 can also include read-only memory (ROM) for storing microcode. Although the example memory sub-system 110 has been illustrated as including the memory sub-system controller 115, in another embodiment of the disclosure, the memory sub-system 110 does not include the memory sub-system controller 115, but instead relies upon external control (e.g., provided by an external host or by a processor or controller separate from the memory sub-system 110). Figure 1

[0031] ​In general, memory sub-system controller 115 can receive commands or operations from host system 120 and can convert the commands or operations into instructions or appropriate commands to achieve the desired access to memory devices 130 and / or memory devices 140. Memory sub-system controller 115 can be responsible for other operations such as wear leveling operations, garbage collection operations, error detection and error-correcting code (ECC) operations, encryption operations, caching operations, and address translations between logical addresses (e.g., logical block addresses (LBAs), namespaces) and physical addresses (e.g., physical block addresses) associated with memory devices 130. Memory sub-system controller 115 can further include host interface circuitry to communicate with host system 120 via a physical host interface. The host interface circuitry can convert commands received from the host system into command instructions to access memory devices 130 and / or memory devices 140, as well as convert responses associated with memory devices 130 and / or memory devices 140 into information for host system 120.

[0032] Memory sub-system 110 can also include additional circuitry or components not illustrated. In some embodiments, memory sub-system 110 can include a cache or buffer (e.g., DRAM) and address circuitry (e.g., row decoder and column decoder) that can receive an address from memory sub-system controller 115 and decode the address to access memory devices 130.

[0033] In some embodiments, memory devices 130 include a local media controller 135 that operates in conjunction with memory sub-system controller 115 to perform operations on one or more memory units of memory devices 130. An external controller (e.g., memory sub-system controller 115) can externally manage memory devices 130 (e.g., perform media management operations on memory devices 130). In some embodiments, memory devices 130 are managed memory devices that are raw memory devices combined with a local controller (e.g., local controller 135) for media management within the same memory device package. An example of a managed memory device is a managed NAND (MNAND) device.

[0034] The host system 120 includes a recovery processor 123 that can orchestrate the recovery process by sending scripts or other instructions to each memory sub-system 110 to cause it to perform a portion of the recovery. For example, a recovery script can be implemented in a high-level language that describes the operations (per memory sub-system 110) to complete the recovery process. This can be implemented as a lightweight messaging protocol between the host system 120 and each participating memory sub-system 110. In another embodiment, the recovery processor 123 is external to the host system 120. For example, the recovery processor 123 can be a data processing unit (DPU), an infrastructure processing unit (IPU), or another controller that provides acceleration of data processing for a group of memory sub-systems 110. Additional details regarding the operations of the recovery processor 123 are described below.

[0035] The memory sub-system 110 includes a recovery component 113 that can perform a portion of the recovery of a failed memory sub-system 110 in parallel with one or more other memory sub-systems 110. In some embodiments, the controller 115 includes at least a portion of the recovery component 113. For example, the controller 115 can include a processor 117 (processing device) configured to execute instructions stored in local memory 119 for performing the operations described herein.

[0036] The recovery component 113 can receive and implement recovery instructions from the recovery processor 123. For example, the recovery component 113 can read data for a stripe of blocks from other memory sub-systems 110, perform an erasure computation to recover data for a stripe of blocks stored by the failed memory sub-system, and write the recovered data to a spare memory sub-system. In addition, the recovery component 113 can send recovery status messages to the recovery processor 123 reporting the progress of the recovery. The recovery component 113 can also receive and implement regeneration requests to determine the priority of recovery for portions of data. Additional details regarding the operations of the recovery component 113 are described below.

[0037] Figure 2 An example illustrating additional details of a memory sub-system controller 115 including a recovery component 113 according to some embodiments of the disclosure is described. The recovery component 113 includes a recovery processor 205. The recovery processor 205 performs an erasure computation to recover data stored by a failed memory sub-system. For example, the recovery processor 205 performs an XOR computation, a matrix computation, a Galois field multiplication, a Reed-Solomon code computation, or another error correction code or erasure code computation on data stored by the operating memory sub-system to recover data lost due to a failure of the memory sub-system.

[0038] The recovery component 113 further includes a multi-lane direct memory access (DMA) component 210. The multi-lane DMA component 210 performs read and write operations to the memory devices 130, 140 coupled to the memory sub-system controller 115 via multiple lanes.

[0039] The recovery component 113 further includes a memory device interface 220. The memory device interface 220 is a high-speed computer bus that provides access to the memory devices 130, 140 of the memory sub-system 110, access to other memory sub-systems, and address translation services and caching. In one embodiment, the memory device interface 220 operates according to an interface standard such as Peripheral Component Interconnect Express (PCIe).

[0040] Additional details regarding the operation of these components are described below.

[0041] Figure 3 Another example computing system 300 including memory sub-systems in accordance with some embodiments of the present disclosure is illustrated. For example, each of the memory sub-systems of the computing system 300 is similar to the memory sub-system 110 described above. However, to simplify the description, the memory sub-systems of the computing system 300 are each illustrated with only the recovery component 113 and the memory devices 130. The other components described above can also be included in each of the memory sub-systems of the computing system 300. Moreover, each of the memory sub-systems can have a different number and / or type of memory devices 130 (again, the common reference number is used to simplify the illustration of the example).

[0042] The memory sub-systems of the computing system 300 store chunk stripes, with a portion of each chunk stripe residing in a portion of a corresponding memory device 130. For example, the memory sub-system 305 stores user data portions of chunk stripe A, Al, user data portions of chunk stripe B, Bl, and user data portions of chunk stripe C, C1. The failed memory sub-system 310 stores another user data portion of chunk stripe A, A2, another user data portion of chunk stripe B, B2, and parity / error correction data of chunk stripe C, CP. The memory sub-system 315 stores another user data portion of chunk stripe A, A3, parity / error correction data of chunk stripe B, BP, and another user data portion of chunk stripe C, C2. The memory sub-system 320 stores another user data portion of chunk stripe A, A3, parity / error correction data of chunk stripe B, BP, and another user data portion of chunk stripe C, C2. The spare memory sub-system 325 stores a recovery copy of the user data portions of chunk stripe A, A2*, the user data portions of chunk stripe B, B2*, and the parity / error correction data of chunk stripe C, CP*. The following describes the operation of the memory sub-systems of the computing system 300 in more detail. Figures 4 to 6An example of a coordinated recovery of this data is described. The memory device 130 can store additional block stripes. Further, the computing system 300 can include additional memory sub-systems, increasing the user data and / or parity / error correction data included in the block stripes. The illustrated example is for ease of explanation and does not limit the embodiments to a particular number of memory sub-systems, portions of user data, portions of parity / error correction data, organization of data across block stripes, size of portions of data striped across memory sub-systems, etc.

[0043] The memory sub-systems of the computing system 300 are coupled to a switch 330. In one embodiment, the switch 330 provides a peer-to-peer network to enable the memory sub-systems to perform reads and writes between each other, for example, during a recovery process, minimizing the impact on input / output (I / O) bandwidth used for normal read and write operations. In another embodiment, this peer-to-peer traffic for the recovery process uses a portion of the backbone bandwidth of another interface, for example, the memory device interface 220, leaving the remainder of the backbone to normal operations.

[0044] Figure 4 is a flow diagram of an example method 400 for orchestrating a coordinated recovery of data of a failed memory sub-system in accordance with some embodiments of the present disclosure. The method 400 can be performed by processing logic that can include hardware (e.g., processing device, circuitry, dedicated logic, programmable logic, microcode, hardware of a device, integrated circuit, etc.), software (e.g., instructions run or executed on a processing device), or a combination thereof. In some embodiments, the method 600 is performed by the recovery processor 123 of Figure 1 is performed by the recovery processor 123. Although shown in a particular sequence or order, unless otherwise specified, the order of the processes can be modified. Thus, illustrated embodiments should be understood only as examples, and that the illustrated processes can be performed in a different order, and that some processes can be performed in parallel. Additionally, one or more processes can be omitted in various embodiments. Thus, not all processes are required in every embodiment. Other process flows are possible.

[0045] At operation 405, the processing device detects a failure of a memory sub-system. For example, the recovery processor 123 detects a lack of a heartbeat signal, a lack of a response from the failed memory sub-system 310, or another indicator that the memory sub-system 310 has failed.

[0046] At operation 410, the processing device sends a recovery instruction to two or more other memory sub-systems. For example, in response to detecting the failure, the recovery processor 123 accesses a map or other data structure to determine the range of logical addresses of data stored by the failed memory sub-system 310. In one embodiment, the recovery instruction is in the form of a script or another high-level language that describes the operations to be performed by each memory sub-system to complete the recovery process.

[0047] The recovery processor 123 divides or otherwise allocates the range of addresses (i.e., the data to be recovered) across the operational memory sub-systems, e.g., memory sub-systems 305, 315, and 320. In one embodiment, the recovery processor 123 also allocates the data to be recovered to one or more spare memory sub-systems, e.g., spare memory sub-system 325 that will be used as a replacement for the failed memory sub-system 310.

[0048] In one embodiment, the recovery processor 123 allocates the data to be recovered equally (or nearly equally if the tasks cannot be divided equally) across the memory sub-systems, e.g., using a round-robin algorithm or similar algorithm. In another embodiment, the recovery processor 123 allocates the data to be recovered unequally across the memory sub-systems, e.g., based on workload history, bandwidth, processing power, and / or other resource allocation data. In one example, the spare memory sub-system 325 is allocated a greater amount of recovery work because normal / non-recovery I / O operations for the spare memory sub-system 325 are less than for the other operational memory sub-systems before the spare memory sub-system 325 stores a large amount of recovered data. Thus, the recovery process has less impact on QoS for ongoing read and write operations, e.g., for data stored on the operational memory sub-systems.

[0049] At operation 415, the processing device aggregates status reports from the recovery processes of the memory sub-systems. For example, as the memory sub-systems perform the recovery operations, the recovery component 113 of each memory sub-system sends a status report message to the recovery processor 123. The status report can include an indication of the portion of memory that has been recovered (e.g., since the last status report), recovery errors, responses to regeneration requests, etc.

[0050] At operation 420, the processing device determines whether the recovery of the data stored by the failed memory sub-system is complete. For example, the recovery processor 123 uses the aggregated status reports to determine whether all of the data has been recovered and written to the spare memory sub-system 325. If the recovery of the data stored by the failed memory sub-system is complete, the method 400 proceeds to operation 445. If the recovery of the data stored by the failed memory sub-system is not complete, the method 400 proceeds to operation 425.

[0051] At operation 425, the processing device determines whether the read or write operation (e.g., from a client or host system) is for data stored by the failed memory subsystem. For example, the recovery processor 123 can receive or internally generate a read or write operation for an application or process unrelated to the recovery process. If the read or write operation is not for data stored by the failed memory subsystem, the method 400 returns to operation 415 to continue aggregation of status reports. If the read or write operation is for data stored by the failed memory subsystem, the method 400 proceeds to operation 430.

[0052] At operation 430, the processing device determines whether the operation is for a portion of memory that has been recovered. For example, the recovery processor 123 uses the aggregated status reports to determine a range of data / addresses that have been successfully recovered and written to the spare memory subsystem 325. If the operation is for a portion of memory that has been recovered, the method 400 proceeds to operation 440. If the operation is for a portion of memory that has not been recovered, the method 400 proceeds to operation 435. Alternatively, if the recovery processor 123 determines that recovery of data was attempted and failed (via an error message in the status report), the recovery processor 123 can respond to the I / O operation with an error message.

[0053] At operation 435, the processing device sends a regeneration request to the memory subsystem to prioritize recovery of the stripe of blocks containing data subject to the received read or write operation. For example, the recovery processor 123 sends a regeneration request to the memory subsystem 305, 315, 320, 325 and the memory subsystem 305, 315, 320, 325 prioritizes recovery of the stripe of blocks, as described below with reference to Figure 5 described.

[0054] At operation 440, the processing device redirects the I / O operation to a memory subsystem that replaces the failed memory subsystem. For example, the recovery processor 123 forwards the I / O operation to the spare memory subsystem 325 to perform the read or write request (or corresponding portion thereof) on the recovered data. In one embodiment, a regeneration request for a read operation causes the recovery memory subsystem, rather than the replacement / spare memory subsystem, to return the data. For example, a read request for user data B2 that has not been recovered can trigger the recovery processor 123 to send a regeneration request to the memory subsystem 315 for the stripe of blocks B, the task of which is to recover the stripe of blocks B. The regeneration request can include the read request, such that once the memory subsystem 315 has recovered the user data B2, the memory subsystem 315 can both return the data B2 in response to the read request and write the data B2 to the spare memory subsystem 325.

[0055] At operation 445, the processing device sends a report of the completion of the recovery and / or resumes normal operations. For example, the recovery processor 123 can send a recovery completion message to one or more host systems 120 and / or one or more memory sub-systems 110 to indicate one or more of: the recovery is complete, the backup memory sub-system is in use, and / or a summary of errors that occurred during the recovery. The recovery processor 123 can resume normal operations and stop monitoring the recovery job or otherwise dedicate resources to recovering data stored in the failed memory sub-system 310.

[0056] Figure 5 is a flow diagram of an example method 500 for cooperatively recovering data of another storage sub-system according to some embodiments of the present disclosure. The method 500 can be performed by processing logic that can include hardware (e.g., processing device, circuitry, dedicated logic, programmable logic, microcode, hardware of a device, integrated circuit, etc.), software (e.g., instructions run or executed on a processing device), or a combination thereof. In some embodiments, the method 500 is performed by the recovery component 113 of the Figure 1 Although shown in a particular sequence or order, unless otherwise specified, the order of the processes can be modified. Thus, the illustrated embodiments should be understood only as examples, and the illustrated processes can be performed in a different order, and some processes can be performed in parallel. Additionally, one or more processes can be omitted in various embodiments. Thus, not all processes are required in every embodiment. Other process flows are possible.

[0057] At operation 505, the processing device receives a failure recovery instruction. For example, the recovery component 113 receives a recovery instruction from the recovery processor 123, as described above with reference to operation 410.

[0058] At operation 510, the processing device reads data of a current chunk stripe from an operational memory sub-system. For example, if the task of the memory sub-system 305 is to recover chunk stripe A, the recovery component 113 of the memory sub-system 305 uses the switch 330 or the memory interface 220 to read user data A3 from the memory sub-system 315 and parity data AP from the memory sub-system 320. Additionally, the recovery component 113 of the memory sub-system 305 can read user data Al from its own memory device 130.

[0059] At operation 515, the processing device recovers data for the current chunk stripe stored in the failed memory sub-system. Continuing the example described above, the recovery component 113 of the memory sub-system 305 uses user data Al, user data A3, and parity data AP to recover user data A2 via error correction, an erasure code, or other data protection calculation. The recovery component 113 of the memory sub-system 305 writes the recovered user data A2*to the spare memory sub-system 325.

[0060] In one embodiment, the recovery component 113 selects current chunk stripes in an order provided by the recovery processor 123. Alternatively, the recovery component 113 selects current chunk stripes in order of address value, randomly, based on frequency of I / O requests for the chunk stripe, etc. In one embodiment, the recovery component 113 divides time and / or processing resources between recovering and processing normal I / O operations. For example, the recovery component 113 can use time division multiple access (TDMA) or another resource sharing scheme to allocate resources to the memory sub-system for recovering chunk stripe data, responding to read requests from other memory sub-systems for its recovery work, and normal I / O operations. While the example of recovery is described herein as being performed one stripe at a time, the memory sub-system can divide the recovery task into smaller or larger portions of memory.

[0061] At operation 520, the processing device reports a recovery status. For example, the recovery component 113 sends an indication to the recovery processor 123 that recovery of the current chunk stripe was successful or that a recovery error occurred. In one embodiment, the recovery component 113 processes I / O requests as part of a regeneration request, as described above with reference to operation 440.

[0062] At operation 525, the processing device determines whether there is another chunk stripe to be recovered. For example, the recovery component 113 determines whether there is any remaining data in the address range of the recovery instruction received from the recovery processor 123 to be recovered. If there are no other chunk stripes to be recovered, the method 500 proceeds to operation 545. If there is another chunk stripe to be recovered, the method 500 proceeds to operation 530.

[0063] At operation 530, the processing device determines whether a regeneration request has been received. For example, the recovery component 113 can receive a regeneration request from the recovery processor 123, as described above with reference to operation 435. If a regeneration request has been received, the method 500 proceeds to operation 540. If a regeneration request has not been received, the method 500 proceeds to operation 535.

[0064] At operation 535, the processing device sets the current stripe to the next stripe. For example, the recovery component 113 selects the next chunk stripe for recovery according to the order as described above. The method 500 returns to operation 510 to continue recovering the new current chunk stripe.

[0065] At operation 540, the processing device sets the current stripe to the stripe subject to the regeneration request. For example, the recovery component 113 prioritizes recovery of the chunk stripe in the regeneration request to minimize the impact of recovery on QoS of normal I / O operations. Thus, the regeneration request triggers the recovery component 113 to process recovery of the regeneration chunk stripe in a different order than described above. The method 500 returns to operation 510 to continue recovering the new current chunk stripe.

[0066] At operation 545, the processing device resumes normal operations. For example, in response to determining that no chunk stripe is pending recovery, the recovery component 113 reallocates the resources bandwidth used to read, compute, and write data during the recovery process to processing normal I / O operations.

[0067] Figure 6 is another example method 600 for orchestrating coordinated recovery of data of a failed storage subsystem according to some embodiments of the present disclosure. The method 600 can be performed by processing logic that can comprise hardware (e.g., processing device, circuitry, dedicated logic, programmable logic, microcode, hardware of a device, integrated circuit, etc.), software (e.g., instructions run or executed on a processing device), or a combination thereof. In some embodiments, the method 600 is performed by the recovery processor 123 of Figure 1 exhibited in a particular sequence or order, unless otherwise specified, the order of the processes can be modified. Thus, the illustrated embodiments should be understood only as examples, and the illustrated processes can be performed in a different order, and some processes can be performed in parallel. Additionally, one or more processes can be omitted in various embodiments. Thus, not all processes are required in every embodiment. Other process flows are possible.

[0068] At operation 605, the processing device detects a failure of the memory subsystem. For example, the recovery processor 123 detects a lack of heartbeat signal, a lack of response, or another indicator that the memory subsystem 310 has failed.

[0069] At operation 610, the processing device sends, to the first memory sub-system, a failure recovery instruction directing the first memory sub-system to recover a first subset of data stored by the failed memory sub-system. For example, in response to detecting the failure, the recovery processor 123 allocates a range of addresses of data to be recovered to the memory sub-system 305 and sends a script or other instructions to the memory sub-system 305 to recover the corresponding data lost in the failure of the memory sub-system 310, as described above with reference to operation 410.

[0070] At operation 615, the processing device sends, to the second memory sub-system, a failure recovery instruction directing the second memory sub-system to recover a second subset of data stored by the failed memory sub-system. For example, in response to detecting the failure, the recovery processor 123 allocates another range of addresses of data to be recovered to the memory sub-system 315 and sends a script or other instructions to the memory sub-system 315 to recover the corresponding data lost in the failure of the memory sub-system 310, as described above with reference to operation 410. The first and second subsets of data are different from each other. For example, the first and second subsets of data are referenced by different ranges of addresses.

[0071] Figure 7 An example machine of a computer system 700, within which a set of instructions, for causing the machine to perform any one or more of the methodologies discussed herein, can be executed, is illustrated in FIG. 7. In some embodiments, the computer system 700 can correspond to a host system (e.g., the host system 120 of FIG. 1) that includes, is coupled to, or utilizes a memory sub-system (e.g., the memory sub-system 110 of FIG. 1) or can be used to perform operations of a controller (e.g., execute an operating system to perform operations corresponding to the recovery component 113 of FIG. 1). Figure 1 Figure 1 Figure 1 In alternative embodiments, the machine can be connected (e.g., networked) to other machines in a LAN, an intranet, an extranet, and / or the Internet. The machine can operate in the capacity of a server or a client machine in client-server network environments, as a peer machine in peer-to-peer (or distributed) network environments, or as a server or a client machine in cloud computing infrastructure or environments.

[0072] The machine can be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular telephone, a web appliance, a server, a network router, a switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while a single machine is illustrated, the term "machine" shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein. ​​

[0073] The example computer system 700 includes a processing device 702, a main memory 704 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) (e.g., synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM)), static memory 706 (e.g., flash memory, static random access memory (SRAM)), and a data storage system 718, which communicate with each other via a bus 730.

[0074] Processing device 702 represents one or more general-purpose processing devices, such as microprocessors, central processing units, or the like. More specifically, the processing device may be a Complex Instruction Set Computing (CISC) microprocessor, a Reduced Instruction Set Computing (RISC) microprocessor, a Very Long Instruction Word (VLIW) microprocessor, or a processor implementing other instruction sets, or multiple processors implementing combinations of instruction sets. Processing device 702 may also be one or more special-purpose processing devices, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), network processors, hardware logic (accelerators), or the like (including combinations of one or more of the listed devices). Processing device 702 is configured to execute instructions 726 for performing the operations and steps discussed herein. Computer system 700 may further include a network interface device 708 for communication via network 720.

[0075] The data storage system 718 may include a machine-readable storage medium 724 (also referred to as computer-readable medium) thereon storing one or more sets of instructions 726 or software embodying any or more of the methodologies or functions described herein. The instructions 726 may also reside wholly or at least partially within main memory 704 and / or processing device 702 during execution by computer system 700, which also constitute machine-readable storage media. The machine-readable storage medium 724, data storage system 718, and / or main memory 704 may correspond to... Figure 1 The memory subsystem 110.

[0076] In one embodiment, instruction 726 includes implementing a component corresponding to the recovery component (e.g., Figure 1 The functional instructions of the recovery component 113). Although the machine-readable storage medium 724 is shown as a single medium in the exemplary embodiment, the term "machine-readable storage medium" should include a single medium or multiple media storing one or more sets of instructions. The term "machine-readable storage medium" should also include any medium capable of storing or encoding a set of instructions for machine execution and causing the machine to perform any or more of the methodologies of this disclosure. Therefore, the term "machine-readable storage medium" should be understood to include (but is not limited to) solid-state memory, optical media, and magnetic media.

[0077] Some portions of the preceding detailed descriptions have been presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. The algorithms presented herein are not inherently related to any particular computer or other apparatus. Various general purpose systems can be used with programs in accordance with the teachings herein, or it can prove convenient to construct more specialized apparatus to perform the methods. The structure for a variety of these systems will appear as follows.

[0078] It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. The present disclosure can refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage systems.

[0079] The present disclosure also relates to an apparatus for performing the operations herein. This apparatus can be specially constructed for the intended purposes, or it can include a general purpose computer selectively activated or reconfigured by a computer program stored in the computer. For example, a computer system or other data processing system (e.g., controller 115 and / or host system 120) can perform the computer-implemented methods 400, 500, and 600 in response to its processor executing a computer program (e.g., a sequence of instructions) contained in a memory or other non-transitory machine-readable storage medium. Such a computer program can be stored in a computer readable storage medium, such as, but not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, and magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, or any type of media suitable for storing electronic instructions, each coupled to a computer system bus.

[0080] The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various general purpose systems can be used with programs in accordance with the teachings herein, or it can prove convenient to construct more specialized apparatus to perform the methods. The structure for a variety of these systems will appear as follows. In addition, the present disclosure is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages can be used to implement the teachings of the disclosure described herein.

[0081] The present disclosure can be provided as a computer program product or software, which can include a machine-readable medium having stored thereon instructions that can be used to program a computer system (or other electronic devices) to perform a process according to the present disclosure. A machine-readable medium includes any mechanism for storing information in a form accessible by a machine (e.g., a computer). In some embodiments, a machine- readable (e.g., computer-readable) medium includes a machine (e.g., a computer) readable storage medium, such as a read only memory ("ROM"), random access memory ("RAM"), magnetic disk storage media, optical storage media, flash memory components, etc.

[0082] In the foregoing specification, embodiments of the disclosure have been described with reference to specific examples thereof. It will be evident that various modifications can be made thereto without departing from the broader spirit and scope of embodiments of the disclosure as set forth in the following claims. The specification and drawings are, accordingly, to be regarded in an illustrative sense rather than a restrictive sense.

Claims

1. A method comprising: receiving a first recovery instruction to recover a first subset of data stored by a first memory sub-system of a plurality of memory sub-systems, wherein the first recovery instruction is received from a second memory sub-system and the second memory sub-system sends a second recovery instruction to a third memory sub-system of the plurality of memory sub-systems, the second recovery instruction directing the third memory sub-system to recover a second subset of data stored by the first memory sub-system, the first and second subsets of data being different from one another; reading data from one or more other memory sub-systems of the plurality of memory sub-systems; recovering the first subset of data by performing data protection calculations on the data read from the one or more other memory sub-systems; and reporting a recovery status to the second memory sub-system in response to recovering the first subset of data.

2. The method of claim 1, further comprising: writing the first subset of data to be recovered to a spare memory sub-system.

3. The method of claim 2, wherein writing the first subset of data to be recovered to the spare memory sub-system is in response to an instruction received from the second memory sub-system, and wherein the second recovery instruction further directs the third memory sub-system to write the second subset of data to be recovered to the spare memory sub-system.

4. The method of claim 2, wherein the second memory sub-system sends a third recovery instruction to the spare memory sub-system, the third recovery instruction directing the spare memory sub-system to reconstruct a third subset of data stored by the first memory sub-system.

5. The method of claim 1, further comprising: receiving a request from the second memory sub-system to prioritize recovery of data in the first subset of data, wherein the request to prioritize recovery of the data is in response to a read or write operation received prior to recovery of the first subset of data.

6. The method of claim 1, wherein the first subset of data is a portion of a stripe of data across a plurality of memory sub-systems of the plurality of memory sub-systems, and the reading data from the one or more other memory sub-systems includes reading other portions of the stripe.

7. The method of claim 1, further comprising: re-allocating resource bandwidth to internal read or write operations unrelated to recovery data stored by the first memory sub-system in response to recovery of the first subset of data having been completed.

8. A non-transitory computer-readable storage medium comprising instructions that, when executed by a processing device, cause the processing device to: receive a first recovery instruction to recover a first subset of data stored by a first memory sub-system of a plurality of memory sub-systems, wherein the first recovery instruction is received from a second memory sub-system and the second memory sub-system sends a second recovery instruction to a third memory sub-system of the plurality of memory sub-systems, the second recovery instruction directing the third memory sub-system to recover a second subset of data stored by the first memory sub-system, the first and second subsets of data being different from each other; read data from one or more other memory sub-systems of the plurality of memory sub-systems; recover the first subset of data by performing data protection calculations on the data read from the one or more other memory sub-systems; and report a recovery status to the second memory sub-system in response to recovering the first subset of data.

9. The non-transitory computer-readable storage medium of claim 8, wherein the processing device is further to: write the first subset of data recovered to a spare memory sub-system.

10. The non-transitory computer-readable storage medium of claim 9, wherein writing the first subset of data recovered to the spare memory sub-system is in response to an instruction received from the second memory sub-system, and wherein the second recovery instruction further directs the third memory sub-system to write the second subset of data recovered to the spare memory sub-system.

11. The non-transitory computer-readable storage medium of claim 9, wherein the second memory sub-system sends a third recovery instruction to the spare memory sub-system, the third recovery instruction directing the spare memory sub-system to reconstruct a third subset of data stored by the first memory sub-system.

12. The non-transitory computer-readable storage medium of claim 8, wherein the processing device is further to: receive a request from the second memory sub-system to prioritize recovery of data in the first subset of data, wherein the request to prioritize recovery of the data is in response to a read or write operation received prior to recovery of the first subset of data.

13. The non-transitory computer-readable storage medium of claim 8, wherein the first subset of data is a portion of a stripe of data across a plurality of memory sub-systems of the plurality of memory sub-systems, and the reading data from the one or more other memory sub-systems includes reading other portions of the stripe.

14. The non-transitory computer-readable storage medium of claim 8, wherein the processing device is further to: reallocate resource bandwidth to internal read or write operations unrelated to recovery data stored by the first memory sub-system in response to recovery of the first subset of data having been completed.

15. A system comprising: a plurality of memory sub-systems; and a processing device operably coupled with the plurality of memory sub-systems to: receive a first recovery instruction to recover a first subset of data stored by a first memory sub-system of a plurality of memory sub-systems, wherein the first recovery instruction is received from a second memory sub-system and the second memory sub-system sends a second recovery instruction to a third memory sub-system of the plurality of memory sub-systems, the second recovery instruction directing the third memory sub-system to recover a second subset of data stored by the first memory sub-system, the first and second subsets of data being different from each other; read data from one or more other memory sub-systems of the plurality of memory sub-systems; recover the first subset of data by performing data protection calculations on the data read from the one or more other memory sub-systems; and report a recovery status to the second memory sub-system in response to recovering the first subset of data. receive a first recovery instruction to recover a first subset of data stored by a first memory sub-system of the plurality of memory sub-systems, wherein the first recovery instruction is received from a second memory sub-system and the second memory sub-system sends a second recovery instruction to a third memory sub-system of the plurality of memory sub-systems, the second recovery instruction directing the third memory sub-system to recover a second subset of data stored by the first memory sub-system, the first and second subsets of data being different from each other; read data from one or more other memory sub-systems of the plurality of memory sub-systems; recover the first subset of data by performing data protection calculations on the data read from the one or more other memory sub-systems; report a recovery status to the second memory sub-system in response to recovering the first subset of data; and write the recovered first subset of data to a spare memory sub-system.

16. The system of claim 15, wherein writing the recovered first subset of data to the spare memory sub-system is in response to an instruction received from the second memory sub-system, and wherein the second recovery instruction further directs the third memory sub-system to write the recovered second subset of data to the spare memory sub-system.

17. The system of claim 15, wherein the second memory sub-system sends a third recovery instruction to the spare memory sub-system, the third recovery instruction directing the spare memory sub-system to reconstruct a third subset of data stored by the first memory sub-system.

18. The system of claim 15, wherein the processing device further: receives a request from the second memory sub-system to prioritize recovery of data in the first subset of data, wherein the request to prioritize recovery of the data is in response to a read or write operation received prior to recovery of the first subset of data.

19. The system of claim 15, wherein the first subset of data is a portion of a stripe of data across a plurality of memory sub-systems of the plurality of memory sub-systems, and the reading data from the one or more other memory sub-systems includes reading other portions of the stripe.

20. The system of claim 15, wherein the processing device further: re-allocates resource bandwidth to internal read or write operations unrelated to recovery data stored by the first memory sub-system in response to recovery of the first subset of data having been completed. ​

Images