System and method for parity-based fault protection for storage devices

Abstract

The present disclosure relates generally to systems and methods for parity-based failure protection for storage devices. Various implementations described herein relate to systems and methods for providing data protection and recovery for drive failures, including receiving a request from a host by a controller of a first storage device. In response to receiving the request, the controller transmits new data from a second storage device. The controller determines an XOR result by performing an XOR operation on the new data and existing data, the existing data stored in a non-volatile storage device.

Description

[0001] Cross-reference of related patent applications

[0002] This application claims priority to U.S. Provisional Patent Application Serial No. 63 / 108,196, filed October 30, 2020, entitled "System and Methods for Parity-Based Failure Protection for Storage Devices," the entire contents of which are hereby incorporated herein by reference as if fully described herein for all purposes. Technical Field

[0003] This disclosure generally relates to systems, methods, and non-transitory processor-readable media for data protection and recovery in the event of drive failure in a data storage device. Background Technology

[0004] Redundant Array of Inexpensive Drives (RAID) can be implemented on drives based on non-volatile memory devices to provide protection against drive failures. Various forms of RAID can be broadly categorized based on whether data is replicated or protected against parity checks. Replication is more expensive in terms of storage costs because it doubles the number of devices required.

[0005] On the other hand, parity protection typically requires lower storage costs than replication. In RAID 5 instances, an additional unit is needed to provide protection against a single device failure at a given time by storing parity data from at least two data devices. When using RAID 5 parity protection, the percentage of additional storage costs in the total cost typically decreases as the number of protected devices in the RAID group increases.

[0006] For RAID 6, which provides protection against the simultaneous failure of up to two devices, two additional devices are required to store parity data for at least two data devices. Similarly, when using RAID 6 parity protection, the percentage of additional storage cost in the total cost decreases as the number of protected devices in the RAID group increases. To mitigate the risk of the drive storing parity data failing, the drive storing parity data is rotated.

[0007] Other variations of parity protection include combining replication with parity protection (e.g., like RAID 51 and RAID 61) to change the stripe size used between devices to match a given application, and so on. Summary of the Invention

[0008] In some arrangements, a first storage device includes a non-volatile storage device and a controller. The controller is configured to: receive a request from a host operatively coupled to the first storage device; in response to receiving the request, transfer new data from a second storage device; and determine an XOR result by performing an XOR operation on the new data and existing data, the existing data being stored in the non-volatile storage device.

[0009] In some arrangements, a first storage device includes a non-volatile storage device and a controller. The controller is configured to: receive a request from a second storage device; in response to receiving the request, transfer new data from the second storage device; and determine an XOR result by performing an XOR operation on the new data and existing data, the existing data being stored in the non-volatile storage device. Attached Figure Description

[0010] Figure 1 A block diagram illustrating an example of a system including a storage device and a host according to some implementation schemes.

[0011] Figure 2A This is a block diagram illustrating instance methods for performing data updates according to some implementation schemes.

[0012] Figure 2B This is a flowchart illustrating an example method for performing data updates according to some implementation schemes.

[0013] Figure 3A This is a block diagram illustrating instance methods for performing parity updates according to some implementation schemes.

[0014] Figure 3B This is a flowchart illustrating an instance method for performing parity updates according to some implementation schemes.

[0015] Figure 4A This is a block diagram illustrating example methods for performing data recovery according to some implementation schemes.

[0016] Figure 4B This is a flowchart illustrating an example method for performing data recovery according to some implementation schemes.

[0017] Figure 5A This is a block diagram illustrating an example method for putting a backup storage device into use according to some implementation schemes.

[0018] Figure 5B This is a flowchart illustrating an example method for putting a backup storage device into use according to some implementation schemes.

[0019] Figure 6This is a flowchart illustrating an example method for providing data protection and recovery in the event of a driver failure, based on some implementation schemes.

[0020] Figure 7A This is a block diagram illustrating instance methods for performing parity updates according to some implementation schemes.

[0021] Figure 7B This is a flowchart illustrating an instance method for performing parity updates according to some implementation schemes.

[0022] Figure 8A This is a block diagram illustrating example methods for performing data recovery according to some implementation schemes.

[0023] Figure 8B This is a flowchart illustrating an example method for performing data recovery according to some implementation schemes.

[0024] Figure 9A This is a block diagram illustrating an example method for putting a backup storage device into use according to some implementation schemes.

[0025] Figure 9B This is a flowchart illustrating an example method for putting a backup storage device into use according to some implementation schemes.

[0026] Figure 10 This is a flowchart illustrating an example method for providing data protection and recovery in the event of a driver failure, based on some implementation schemes.

[0027] Figure 11 This is a flowchart illustrating an example method for providing data protection and recovery in the event of a driver failure, based on some implementation schemes.

[0028] Figure 12 This is a schematic diagram illustrating a host-side view used for updating data according to some implementation schemes.

[0029] Figure 13 This is a schematic diagram illustrating the placement of parity check data according to some implementation schemes. Detailed Implementation

[0030] Various challenges arise in parity-based protection. Currently, the vast majority of implementations rely on a dedicated disk array controller (DAC). The DAC calculates parity data by performing an XOR operation on each stripe of data on each data disk in a given RAID group and stores the resulting parity data on one or more parity disks. The DAC is typically attached to the main central processing unit (CPU) via a PCIe bus or network, while using storage-specific interconnects and protocols (interfaces) such as (but not limited to) AT Attach (ATA), Small Computer System Interface (SCSI), Fibre Channel, and Serial Attach SCSI (SAS) to connect to and communicate with the disks. For storage-specific interconnects, a dedicated hardware controller is required to translate between the PCIe bus and the storage interface (such as SCSI or Fibre Channel).

[0031] The applicant recognizes that the evolution of storage devices based on non-volatile memory (e.g., solid-state drives (SSDs)) has fundamentally changed system architecture, as storage devices are directly attached to the PCIe bus via the Non-Volatile Memory Fast (NVMe) interface, thus eliminating path inefficiencies and optimizing cost, power consumption, and performance. While SSD fault protection still requires DAC functionality, this functionality is migrating from dedicated hardware controllers to software running on general-purpose CPUs.

[0032] Because hard disk drive (HDD) access times are in the millisecond range, DAC inefficiencies are not exposed. The advent of SSDs has reduced data access times, thus placing very stringent requirements on DACs to derive and aggregate the performance of a large number of SSDs. The applicant recognizes that as SSD access times have decreased to the tens of microseconds, conventional DAC implementations become inefficient because DAC performance translates into SSD performance.

[0033] With the advent of NVMe, the interfaces used for HDDs needed to be upgraded to be compatible with SSDs. These interfaces define the way commands are passed, statuses are returned, and data is exchanged between the host and the storage device. The interfaces optimize and simplify direct connections to the CPU, eliminating the need for intermediate interface translators.

[0034] Furthermore, as SSD adoption has increased, the cost of HDDs (per GB) has decreased significantly, partly due to the increased capacity offered in HDDs to differentiate them from SSDs on the market. However, increased capacity comes at the cost of performance, especially in rebuilding data when a drive fails in a RAID group. Therefore, DAC vendors have moved from using parity-based protection to replication-based HDD protection. HDD data storage and access times are already slower than SSD data storage and access times, so packing more capacity makes HDDs perform worse on average. In this regard, DAC vendors do not want to further slow down HDDs by using parity-based protection. Therefore, replication-based protection is practically ubiquitous in standard DACs for HDDs. When SSDs impose even greater demands on DACs, the DAC can also reuse replication-based protection only for SSDs.

[0035] Therefore, parity-based SSD protection did not continue with system-wide architectural changes. Furthermore, cost and access barriers were imposed on the main CPU in the form of special stock units (SKUs), along with limited availability to selected customers. DAC vendors lost the freedom to provide parity-based protection for SSDs, just as they did with replication-based SSDs.

[0036] Therefore, implementing parity protection for SSDs based on RAID 5 and RAID 6 becomes more difficult.

[0037] Typically, RAID (e.g., RAID 5 and RAID 6) redundancy for storage devices (such as SSDs) is created by relying on the host to perform XOR calculations and update parity data on the SSD. SSDs perform their common function of reading data from or writing data to storage media (e.g., memory arrays) without knowing whether the data is parity data. Therefore, in RAID 5 and RAID 6, computational overhead and the additional data generation and movement often become performance bottlenecks for the storage media.

[0038] The arrangements disclosed herein relate to parity-based protection schemes, which are cost-effective solutions for SSD failure protection without compromising the need to meet business demands more quickly. This disclosure improves upon parity-based protection while creating solutions that are compatible with current system architectures and evolving changes. In some arrangements, this disclosure relates to collaboratively performing data protection and recovery operations among two or more components of a storage system. Although non-volatile memory devices are presented as examples herein, the disclosed schemes can be implemented on any storage system or device that interfaces with a host and temporarily or permanently stores data for later retrieval by the host.

[0039] To help illustrate the embodiments of the present invention, Figure 1 This diagram illustrates a system comprising storage devices 100a, 100b, ..., 100n (collectively referred to as storage device 100) coupled to a host 101, according to several examples. The host 101 may be a user device operated by a user or an autonomous central controller of the storage device 100, wherein the host 101 and storage device 100 correspond to a storage subsystem or storage device. The host 101 may be connected to a communication network 109 (via a network interface 108) so that other host computers (not shown) can access the storage subsystem or storage device via the communication network 109. Examples of this storage subsystem or device include an all-flash array (AFA) or a network-attached storage (NAS) device. As shown, the host 101 includes a memory 102, a processor 104, and a bus 106. The processor 104 is operatively coupled to the memory 102 and the bus 106. The processor 104 is sometimes referred to as the central processing unit (CPU) of the host 101 and is configured to perform processing on the host 101.

[0040] Memory 102 is the local memory of host 101. In some instances, memory 102 is a buffer, sometimes referred to as a host buffer. In some instances, memory 102 is a volatile storage device. In other instances, memory 102 is a non-volatile permanent storage device. Examples of memory 102 include (but are not limited to) random access memory (RAM), dynamic random access memory (DRAM), static RAM (SRAM), magnetic RAM (MRAM), phase-change memory (PCM), and so on.

[0041] Bus 106 includes one or more of software, firmware, and hardware that provides an interface for communication between components of host 101. Examples of components include (but are not limited to) processor 104, network interface card (NIC), storage device, memory 102, graphics card, etc. Additionally, host 101 (e.g., processor 104) can use bus 106 to communicate with storage device 100. In some instances, storage device 100 is directly attached to or communicatively coupled to bus 106 via a suitable interface 140. Bus 106 is one or more of serial, PCIe bus or network, PCIe root complex, internal PCIe switch, etc.

[0042] Processor 104 can execute an operating system (OS) that provides a file system and applications that use the file system. Processor 104 can communicate with storage devices 100 (e.g., controllers 110 of each of the storage devices 100) via a communication link or network. In this regard, processor 104 can use interface 140 connected to the communication link or network to send data to and receive data from one or more of the storage devices 100. Interface 140 allows software running on processor 104 (e.g., a file system) to communicate with storage devices 100 (e.g., their controllers 110) via bus 106. Storage devices 100 (e.g., their controllers 110) are operatively coupled directly to bus 106 via interface 140. Although interface 140 is conceptually shown as a dashed line between host 101 and storage devices 100, interface 140 may include one or more controllers, one or more physical connectors, and one or more data transfer protocols (including namespaces, ports, transport mechanisms, and their connectivity). Although the connection between host 101 and storage devices 100a, b...n is shown as a direct link, in some implementations, the link may include a network structure that may include network components such as bridges and switches.

[0043] To send and receive data, processor 104 (and the software or file system running thereon) communicates with storage device 100 using a storage data transfer protocol running on interface 140. Examples of protocols include (but are not limited to) SAS, Serial ATA (SATA), and NVMe protocols. In some instances, interface 140 includes hardware (e.g., a controller) implemented on or operatively coupled to bus 106, storage device 100 (e.g., controller 110), or another device operatively coupled to bus 106 and / or storage device 100 via one or more suitable networks. Interface 140 and the storage protocols running thereon also include software and / or firmware executing on this hardware.

[0044] In some instances, processor 104 may communicate with communication network 109 via bus 106 and network interface 108. Other host systems (not shown) attached to or communicatively coupled to communication network 109 may communicate with host 101 using suitable network storage protocols, examples of which include (but are not limited to) NVMe over Fibre Channel (NVMeoF), iSCSI, Fibre Channel (FC), Network File System (NFS), Server Message Block (SMB), etc. Network interface 108 allows software running on processor 104 (e.g., storage protocols or file systems) to communicate with external hosts attached to communication network 109 via bus 106. In this manner, network storage commands can be issued by the external host and processed by processor 104, which can issue storage commands to storage device 100 as needed. Thus, data can be exchanged between the external host and storage device 100 via communication network 109. In this example, any exchanged data is buffered in memory 102 of host 101.

[0045] In some instances, storage device 100 is located in a data center (not shown for brevity). The data center may contain one or more platform or rack units, each supporting one or more storage devices (e.g., but not limited to, storage device 100). In some implementations, host 101 and storage device 100 together form a storage node, where host 101 acts as a node controller. An example of a storage node is a Kioxia Kumoscale storage node. One or more storage nodes within a platform are connected to a top-of-rack (TOR) switch, each storage node connected to the TOR via one or more network connections (e.g., Ethernet, Fibre Channel, or InfiniBand), and can communicate with each other via the TOR switch or another suitable intra-platform communication mechanism. In some implementations, storage device 100 may be a network-attached storage device (e.g., an Ethernet SSD) connected to a TOR switch, where host 101 is also connected to the TOR switch and can communicate with storage device 100 via the TOR switch. In some implementations, at least one router facilitates communication between storage devices 100 in storage nodes across different platforms, racks, or cabinets via a suitable network architecture. Examples of storage device 100 include non-volatile devices such as (but not limited to) solid-state drives (SSDs), Ethernet-attached SSDs, non-volatile dual in-line memory modules (NVDIMMs), universal flash memory (UFS), secure digital storage (SD) devices, and so on.

[0046] Each of the storage devices 100 includes at least a controller 110 and a memory array 120. For simplicity, other components of the storage device 100 are not shown. The memory array 120 includes NAND flash memory devices 130a to 130n. Each of the NAND flash memory devices 130a to 130n includes one or more individual NAND flash memory dies, which are NVMs capable of storing data without power. Therefore, NAND flash memory devices 130a to 130n refer to multiple NAND flash memory devices or dies within the flash memory device 100. Each of the NAND flash memory devices 130a to 130n includes one or more dies, each having one or more planes. Each plane has multiple blocks, and each block has multiple pages.

[0047] Although NAND flash memory devices 130a to 130n are shown as examples of memory array 120, other examples of non-volatile memory technologies used to implement memory array 120 include (but are not limited to) non-volatile (battery-backup) DRAM, magnetic random access memory (MRAM), phase-change memory (PCM), ferroelectric RAM (FeRAM), and so on. The arrangements described herein can also be implemented on memory systems using such memory technologies and other suitable memory technologies.

[0048] Instances of controller 110 include (but are not limited to) SSD controllers (such as client SSD controllers, data center SSD controllers, enterprise SSD controllers, etc.), UFS controllers, or SD controllers, etc.

[0049] Controller 110 can combine raw data storage from multiple NAND flash memory devices 130a to 130n, such that the NAND flash memory devices 130a to 130n are logically used as a single storage unit. Controller 110 may include a processor, a microcontroller, buffer memory 111 (e.g., buffers 112, 114, 116), an error correction system, a data encryption system, a flash translation layer (FTL), and a flash interface module. Such functionality may be implemented in hardware, software, and firmware, or any combination thereof. In some arrangements, the software / firmware of controller 110 may be stored in memory array 120 or any other suitable computer-readable storage medium.

[0050] Controller 110 includes processing and storage capabilities suitable for performing the functions described herein and other functions. As described, controller 110 manages various features of NAND flash memory devices 130a to 130n, including (but not limited to) I / O handling, reading, writing / programming, erasing, monitoring, logging, error handling, discarded item collection, wear leveling, logical-to-physical address mapping, data protection (encryption / decryption, cyclic redundancy check (CRC)), error correction coding (ECC), data scrambling, etc. Therefore, controller 110 provides visibility of NAND flash memory devices 130a to 130n.

[0051] Buffer memory 111 is a memory device locally and operatively coupled to controller 110. For example, buffer memory 111 may be on-chip SRAM memory located on a chip of controller 110. In some embodiments, buffer memory 111 may be implemented using a memory device of storage device 110 external to controller 110. For example, buffer memory 111 may be on-chip DRAM located outside the chip of controller 110. In some embodiments, buffer memory 111 may be implemented using memory devices both internal and external to controller 110 (e.g., on-chip and off-chip of controller 110). For example, buffer memory 111 may be implemented using both internal SRAM and external DRAM, which is transparent / exposed and accessible via interface 140 by other devices (e.g., host 101 and other storage devices 100). In this example, controller 110 includes an internal processor using memory addresses within a single address space, and a memory controller that controls both internal SRAM and external DRAM selects whether to place data on internal SRAM and external DRAM based on efficiency. In other words, the internal SRAM and external DRAM are addressed like a single memory. As shown, the buffer memory 111 includes a buffer 112, a write buffer 114, and a read buffer 116. In other words, the buffer 112, write buffer 114, and read buffer 116 can be implemented using the buffer memory 111.

[0052] Controller 110 includes a buffer 112, sometimes referred to as a drive buffer or controller memory buffer (CMB). In addition to being accessible by controller 110, buffer 112 can also be accessed via interface 140 by other devices (e.g., host 101 and other storage devices 100a, 100b, ... 100n). In this way, buffer 112 (e.g., addresses of memory locations within buffer 112) is exposed across bus 106, and any device operatively coupled to bus 106 can issue commands (e.g., read commands, write commands, etc.) corresponding to the addresses of memory locations within buffer 112 to read data from and write data to those memory locations within buffer 112. In some instances, buffer 112 is a volatile storage device. In some instances, buffer 112 is a non-volatile permanent storage device that provides improved protection against accidental power loss of one or more of the storage devices 100. Examples of buffer 112 include (but are not limited to) RAM, DRAM, SRAM, MRAM, PCM, etc. Buffer 112 may refer to multiple buffers, each configured to store different types of data, as described herein.

[0053] In some implementation schemes, such as Figure 1 As shown, buffer 112 is local memory of controller 110. For example, buffer 112 may be on-chip SRAM memory located on the chip of controller 110. In some embodiments, buffer 112 may be implemented using memory devices of storage device 110 external to controller 110. For example, buffer 112 may be on-chip DRAM located other than the chip of controller 110. In some embodiments, buffer 112 may be implemented using memory devices both internal and external to controller 110 (e.g., on-chip and off-chip of controller 110). For example, buffer 112 may be implemented using both internal SRAM and external DRAM, which is transparent / exposed and accessible by other devices (e.g., host 101 and other storage devices 100) via interface 140. In this example, controller 110 includes an internal processor using memory addresses within a single address space, and a memory controller that controls both internal SRAM and external DRAM selects whether to place data on internal SRAM and external DRAM based on efficiency. In other words, internal SRAM and external DRAM are addressed like a single memory.

[0054] In one example of a write operation, in response to receiving data from host 101 (via host interface 140), controller 110 acknowledges the write command to host 101 after writing the data to write buffer 114. In some embodiments, write buffer 114 may be implemented in a separate memory separate from buffer 112, or write buffer 114 may be a defined region or portion of memory including buffer 112, wherein only the CMB portion of the memory, not write buffer 114, is accessible by other devices. Controller 110 may write the data stored in write buffer 114 to memory array 120 (e.g., NAND flash memory devices 130a to 130n). Once the physical address of the data written to memory array 120 is complete, FTL updates the mapping between the logical address (e.g., logical block address (LBA)) used by host 101 to associate with the data and the physical address used by controller 110 to identify the physical location of the data. In another instance concerning a read operation, controller 110 includes a separate buffer 116 (e.g., a read buffer) distinct from buffers 112 and 114 to store data read from memory array 120. In some embodiments, read buffer 116 may be implemented in a separate memory distinct from buffer 112, or read buffer 116 may be a defined area or portion of the memory that includes buffer 112, wherein only the CMB portion of the memory, other than read buffer 116, is accessible by other devices.

[0055] Although non-volatile memory devices (such as NAND flash memory devices 130a to 130n) are presented as examples herein, the disclosed schemes can be implemented on any storage system or device connected to host 101 via an interface, wherein the system temporarily or permanently stores data for later retrieval by host 101.

[0056] In some instances, storage devices 100 form a RAID group for parity protection. That is, one or more of the storage devices 100 store parity data (e.g., parity bits) for the data stored on the devices and / or the data stored on the other devices 100.

[0057] Typically, updating parity data (or parity) on the parity drive in a RAID 5 group requires two read I / O operations, two write I / O operations, four transfers via bus 106, and four memory buffer transfers. All such operations require CPU cycles on processor 104, commit queue (SQ) / complete queue (CQ) entries, context switching, and so on. Furthermore, transfers performed between processor 104 and memory 102 consume buffer space and bandwidth between processor 104 and memory 102. In addition, data communication between processor 104 and bus 106 consumes the bandwidth of bus 106, which is considered a valuable resource because bus 106 serves as the interface between different components of host 101. Therefore, conventional parity update schemes consume significant resources on host 101 (e.g., bandwidth, CPU cycles, and buffer space).

[0058] Some arrangements disclosed herein relate to parity-based drive failover (e.g., RAID) based on peer-to-peer (P2P) transfers between storage devices 100. In drive failover schemes using P2P transfers, a local memory buffer (e.g., buffer 112) of storage device 100 is used to perform data transfers from one storage device (e.g., storage device 100a) to another storage device (e.g., storage device 100b). Therefore, data is no longer copied to the memory 102 of host 101, thus reducing the latency and bandwidth required to transfer data to and from memory 102. Using the drive failover schemes described herein, the number of I / O operations can be reduced from 4 to 2 using host-directed P2P transfers or from 4 to 1 using device-directed P2P transfers whenever data is updated on a RAID 5 device. The efficiency gains not only improve performance but also reduce cost, power consumption, and network utilization.

[0059] To achieve such efficiency improvements, and in some instances, a buffer 112 of existing capacity within storage device 100 is exposed to host 101 for use across bus 106 (e.g., via a base address register).

[0060] In some configurations, host 101 cooperates with storage device 100 to perform XOR calculations not only on parity data reads and writes but also on non-parity data reads and writes. Specifically, controller 110 can be configured to perform XOR calculations instead of receiving XOR results from host 101. Therefore, host 101 does not need to consume additional computational or memory resources for such operations, does not need to consume CPU cycles to send additional commands for performing XOR calculations, does not need to allocate hardware resources for associated direct memory access (DMA) transfers, does not need to consume commit and completion queues for additional commands, and does not need to consume additional bus / network bandwidth.

[0061] Given the improvements achieved by enabling storage device 100 to perform XOR calculations internally within storage device 100, not only is the total system cost (for a system including host 101 and storage device 100) lower, but system performance is also improved. Therefore, compared to conventional RAID redundancy mechanisms, this disclosure relates to allowing storage device 100 to offload functionality and functional repartitioning, resulting in fewer operations and less data movement.

[0062] In addition to offloading the XOR operation from host 101, the arrangement disclosed herein also utilizes P2P communication between storage devices 100 to perform data computation and transmission, further improving performance, cost, power consumption, and network utilization. Specifically, host 101 no longer needs to send instantaneous data (e.g., instantaneous XOR data results) determined from data stored on the data device to the parity check device. That is, host 101 no longer needs to transfer the instantaneous XOR result from the data device to memory 102 and then transfer the instantaneous XOR result from memory 102 to the parity check device.

[0063] Instead, memory 102 is bypassed, and the transient XOR data result is transferred from the data device to the parity check device. For example, the buffer 112 (e.g., CMB) of each of the storage devices 100 is identified by a reference. Examples of references include (but are not limited to) an address, CMB address, address descriptor, identifier, pointer, or other suitable indicator identifying the buffer 112 of the storage device. Data can be transferred from host 101 or other storage devices to the storage device based on the storage device's address. The address of the buffer 112 of storage device 100 is stored in a shared address register (e.g., a shared PCIe base address register) known to host 101. For example, in NVMe, the CMB is defined by the NVMe controller register CMBLOC, which defines the PCI address location where the CMB begins and the controller register CMBSZ (which is the size of the CMB). In some instances, the data device stores the transient XOR result in buffer 112 (e.g., in the CMB). Since the address of the data device is in the shared PCIe base address register of host 101, host 101 sends a write command containing the address of buffer 112 of the data device to the parity check device. The parity check device can directly obtain the contents of buffer 112 of the data device (e.g., XOR instantaneous result) using a transfer mechanism (e.g., DMA transfer mechanism), thus bypassing memory 102.

[0064] Traditionally, to update data (regular, non-parity data) on a data drive in a RAID 5 group, the following steps are performed. Host 101 submits an NVMe read request to the controller 110 of the data drive via interface 140. In response, controller 110 performs a NAND read to the read buffer. In other words, controller 110 reads the data requested in the read request from memory array 120 (one or more of NAND flash devices 130a to 130n) and stores it in read buffer 116. Controller 110 transfers the data from buffer 116 across interface 140 to memory 102 (e.g., the old data buffer). Thus, host 101's old data buffer stores the old data read from memory array 120. Next, host 101 submits an NVMe write request to controller 110, presenting new data in host 101's new data buffer for controller 110 to write. In response, controller 110 performs a data transfer to transfer the new data from host 101's new data buffer across the NVMe interface to write buffer 114. Next, controller 110 updates existing data by writing new data to memory array 120 (e.g., one or more of NAND flash memory devices 130a to 130n). The new and old data share the same logical address (e.g., LBA) but have different physical addresses (e.g., stored in different NAND pages of NAND flash memory devices 130a to 130n). Then, host 101 performs an XOR operation between (i) the new data already residing in host 101's new data buffer and (ii) the existing data read from storage device 100 and residing in host 101's old data buffer. Host 101 stores the result of the XOR operation (referred to as transient XOR data) in host 101's transient XOR host buffer. In some cases, the transient XOR buffer may potentially be identical to either the old or new data buffer, as the transient XOR data can replace existing content in said buffers to save memory resources.

[0065] On the other hand, some arrangements for updating data stored in a data drive (e.g., storage device 100a as an example) include performing XOR calculations within the controller 110 and via an interface 140 (e.g., an NVMe interface) to update the data stored in the memory array 120 of the data drive. In this respect, Figure 2A This is a block diagram illustrating an example method 200a for performing data updates according to some implementation schemes. (Reference) Figures 1 to 2AMethod 200a provides improved I / O efficiency, host CPU efficiency, and memory resource efficiency compared to the conventional data update methods mentioned above. Method 200a can be executed by host 101 and storage device 100a. Memory 102 includes host buffer (new data) 201. NAND page (old data) 203 and NAND page (new data) 205 are different pages in NAND flash memory devices 130a to 130n.

[0066] In method 200a, host 101 submits a new type of NVMe write command or request to controller 110 of storage device 100a via bus 106 and interface 140. In some embodiments, the new type of NVMe write command or request may be similar to a conventional NVMe write command or request, but with a different command opcode or flag to indicate that the command is not a normal NVMe write command and that the command should be processed according to the methods described herein. Host 101 presents host buffer (new data) 201 to controller 110 for writing. In response, at 211, controller 110 performs a data transfer to obtain new data (regular, non-parity data) from host buffer (new data) 201 across interface 140 via bus 106 and stores the new data in write buffer (new data) 202. The write request contains the logical address (e.g., LBA) of the new data.

[0067] At 212, the controller 110 of storage device 100a performs a NAND read into the read buffer (old data) 204. In other words, the controller 110 reads old and existing data from memory array 120 (e.g., one or more NAND pages (old data) 203) corresponding to the logical address in the host write request received at 211 and stores the old data in the read buffer (old data) 204. The one or more NAND pages (old data) 203 are pages of one or more of the NAND flash memory devices 130a to 130n of storage device 100a. The new data and old data are data (e.g., regular, non-parity data). In other words, the old data is updated with the new data.

[0068] At 213, controller 110 then updates the old data with new data by writing new data from write buffer (new data) 202 into NAND page (new data) 205. NAND page (new data) 205 is a physical NAND page location different from NAND page (old data) 203, given that it is a physical property of NAND memory and cannot physically overwrite existing data in the NAND page. Instead, a new NAND physical page is written and the logical-to-physical (L2P) address mapping table is updated to indicate the new NAND page corresponding to the logical address used by host 101. Controller 110 (e.g., FTL) updates the L2P address mapping table to make the physical address of NAND page (new data) 205 correspond to the logical address. Controller 110 marks the physical address of NAND page (old data) 203 for obsolete item collection.

[0069] At 214, controller 110 performs an XOR operation between new data stored in write buffer (new data) 202 and old data stored in read buffer (old data) 204 to determine an instantaneous XOR result, and stores the instantaneous XOR result in CMB (instantaneous XOR) 206. In some arrangements, write buffer (new data) 202 is a specific implementation of write buffer 114 of storage device 100a. Read buffer (old data) 204 is a specific implementation of read buffer 116 of storage device 100a. CMB (instantaneous XOR) 206 is a specific implementation of buffer 112 of storage device 100a. In other arrangements, to save memory resources, CMB (instantaneous XOR) 206 may be the same as read buffer (old data) 204 and a specific implementation of buffer 112 of storage device 100a, such that the instantaneous XOR result can overwrite the contents of read buffer (old data) 204. In this way, only one data transfer is performed from the NAND page to the read buffer (old data) 204, and then the XOR result is calculated appropriately at the same location, without transferring any data.

[0070] The instantaneous XOR result from CMB (Instantaneous XOR) 206 is not transmitted to host 101 across interface 140. Instead, the instantaneous XOR result from CMB (Instantaneous XOR) 206 can be directly transmitted to the parity drive (e.g., storage device 100b) to update the parity data corresponding to the updated, new data. This will be referred to... Figure 3A and 3B Further detailed discussion is needed.

[0071] Figure 2B This is a flowchart illustrating an example method 200b for performing data updates according to some implementation schemes. (Reference) Figure 1 , 2AAnd 2B, method 200b corresponds to method 200a. Method 200b can be executed by the controller 110 of storage device 100a.

[0072] At 221, controller 110 receives a new type of write request from host 101, which is operatively coupled to storage device 100a. At 222, in response to receiving the new type of write request, controller 110 transfers new data (new regular, non-parity data) from host 101 (e.g., from host buffer (new data) 201) across bus 106 and via interface 140 to write buffer (e.g., write buffer (new data) 202) of storage device 100a. Thus, controller 110 receives new data from host 101 corresponding to the logical address identified in the write request. At 223, controller 110 performs a read operation to read existing (old) data from non-volatile storage (e.g., from NAND page (old data) 203) into an existing data drive buffer (e.g., read buffer (old data) 204) located in a memory area (i.e., CMB) accessible by other devices. The existing data has the same logical address as the new data, as identified in the write request.

[0073] In step 224, controller 110 writes new data stored in the new data drive buffer of storage device 100a to a non-volatile memory device (e.g., NAND page (new data) 205). As mentioned, the new data and existing data correspond to the same logical address but reside in different physical NAND pages. The existing data resides at a first physical address of the non-volatile memory device (e.g., at NAND page (old data) 203). Writing new data to the non-volatile memory device includes writing the new data to a second physical address of the non-volatile memory device (e.g., at NAND page (new data) 205) and updating the logical-to-physical (L2P) mapping so that the logical address corresponds to the second physical address. Steps 223 and 224 can be performed in any suitable order or simultaneously.

[0074] At 225, controller 110 determines the XOR result by performing an XOR operation on the new data and the existing data. The XOR result is called the transient XOR result. At 226, after determining the transient XOR result, controller 110 temporarily stores the transient XOR result in a transient XOR result driver buffer (e.g., CMB (transient XOR) 206).

[0075] Since updating the parity on the parity drive (e.g., methods 300a and 300b) after updating the data on the data drive (e.g., methods 200a and 200b) is necessary to maintain the integrity of RAID 5 group protection, the efficiency is actually the sum of the two processes. In other words, each write in the conventional mechanism results in four I / O operations (reading old data, reading old parity, writing new data, and writing new parity). In the mechanism described in this paper, the number of I / O operations is halved to two (writing new data and writing new parity).

[0076] Typically, to update parity data (or parity) on the parity drive in a RAID 5 group, the following steps are performed. Host 101 submits an NVMe read request to the parity drive controller 110 via interface 140. In response, controller 110 performs a NAND read to the drive buffer. In other words, controller 110 reads the data requested in the read request (old, existing parity data) from memory array 120 (one or more of NAND flash devices 130a to 130n) and stores the data in read buffer 116. Controller 110 transfers the data from read buffer 116 across interface 140 to memory 102 (e.g., an old data buffer). Thus, host 101's old data buffer stores the old data read from memory array 120. Next, host 101 performs an XOR operation between (i) data previously calculated by host 101 (referred to as transient XOR data) and residing in memory 102 (e.g., a transient XOR buffer) and (ii) old data read from memory array 120 and stored in an old data buffer of host 101. The result (referred to as new data) is then stored in memory 102 (e.g., a new data buffer). In some cases, the new data buffer may potentially be identical to the old data buffer or the transient XOR buffer, as the new data can replace existing content in said buffers to save memory resources. Host 101 then submits an NVMe write request to controller 110 and presents the new data from the new data buffer to controller 110. In response, controller 110 then performs a data transfer to obtain the new data from host 101's new data buffer across interface 140 and stores the new data in write buffer 116. Next, controller 110 updates the old, existing data by writing new data into memory array 120 (e.g., one or more of NAND flash memory devices 130a to 130n). Due to the nature of NAND flash memory operation, the new and old data share the same logical address (e.g., the same LBA) but have different physical addresses (e.g., stored in different NAND pages of NAND flash memory devices 130a to 130n). Controller 110 also updates the logical-to-physical mapping table to record the new physical address.

[0077] On the other hand, some arrangements for updating parity data stored in the memory array 120 of the parity driver (e.g., storage device 100b as an example) not only involve performing XOR calculations within the controller 110 of the parity driver rather than within the processor 104, but also involve transferring instantaneous XOR data directly from the buffer 112 of the data driver (e.g., storage device 100a as an example), without using the memory 102 of the host 101. In this respect, Figure 3A This is a block diagram illustrating an example method 300a for performing parity updates according to some implementation schemes. (Reference) Figures 1 to 3A Method 300a provides improved I / O efficiency, host CPU efficiency, memory resource efficiency, and data transfer efficiency compared to the conventional parity update method mentioned above. Method 300a can be executed by host 101, storage device 100a (data driver), and storage device 100b (parity driver storing parity data stored on the data driver).

[0078] NAND page (old data) 303 and NAND page (XOR result) 306 are different pages in the NAND flash memory devices 130a to 130n of storage device 100b. NAND page (old data) 203a and NAND page (new data) 206b are different pages in the NAND flash memory devices 130a to 130n of storage device 100b.

[0079] In method 300a, at 311, host 101 submits a new type of NVMe write command or request to controller 110 of storage device 100b via interface 140. In some embodiments, the new type of NVMe write command or request may be similar to a conventional NVMe write command or request, but with a different command opcode or flag to indicate that the command is not a normal NVMe write command and that the command should be processed according to the methods described herein. The write request contains a reference to the address of CMB (transient XOR) 206 of storage device 100a. Examples of the reference include (but are not limited to) an address, CMB address, address descriptor, identifier, pointer, or another suitable indicator identifying buffer 112 of storage device 100a. As described in methods 200a and 200b, CMB (transient XOR) 206 stores the transient XOR result determined by controller 110 of storage device 100a by performing an XOR operation (e.g., at 214) on new data and existing data. Therefore, the write request received at 311 by the controller 110 of the storage device 100b does not contain the transient XOR result, but instead contains the address of the buffer of the data driver that temporarily stores the transient XOR result. The write request further contains the logical address (e.g., LBA) of the storage device 100b where new data (e.g., the transient XOR result, which is parity data) will be written.

[0080] At 312, controller 110 performs a NAND read into the read buffer (new data) 304. In other words, controller 110 reads old and existing data from memory array 120 (e.g., one or more NAND pages (old data) 303) corresponding to the logical address in the write request received from the host at 311 and stores the old data in the read buffer (new data) 304. The old and existing data is located at the old physical address corresponding to the LBA of the new data provided in the write request received at 311. The old physical address is obtained by controller 110 using a logical-to-physical lookup table. The one or more NAND pages (old data) 303 are pages in one or more of the NAND flash memory devices 130a to 130n of storage device 100b.

[0081] At 313, the controller 110 of storage device 100b performs a P2P read operation to transfer new data from the CMB (instant XOR) 206 of storage device 100a to the write buffer (new data) 302 of storage device 100b. In other words, storage device 100b can directly access the contents (e.g., the XOR instant result) of the CMB (instant XOR) 206 of storage device 100a using a suitable transfer mechanism, thus bypassing the memory 102 of host 101. The transfer mechanism can identify the source (CMB (instant XOR) 206) of the new data to be transferred using the address of the CMB (instant XOR) 206 received from host 101 at 311. The transfer mechanism can then transfer the data from the source to the write buffer (write buffer (new data) 302) of storage device 100b. In some implementations, the read operation is performed by the controller 110 of storage device 100b as part of the normal processing of any NVMe write command received from host 101, except that the address of the data to be written is referenced to the CMB (instantaneous XOR) 206 in storage device 100a instead of the host buffer 102. Examples of transfer mechanisms include (but are not limited to) DMA transfer mechanisms, transfer over wireless or wired networks, transfer over a bus or serial network, intra-platform communication mechanisms, or another suitable communication channel connecting the source and destination buffers.

[0082] In some instances, after new data is successfully transferred from CMB (transient XOR) 206 to write buffer (new data) 302, the controller 110 of storage device 100b may acknowledge the write request received at 311 to host 101. In some instances, after new data is successfully transferred from CMB (transient XOR) 206 to write buffer (new data) 302 (e.g., via a transfer mechanism or by the controller 110 of storage device 100b acknowledging the transfer to the controller 110 of storage device 100a), the controller 110 of storage device 100a removes the contents of CMB (transient XOR) 206 or indicates that the contents of CMB (transient XOR) 206 are invalid.

[0083] Regarding method 300a, the new data and the old data are parity data (e.g., one or more parity bits). In other words, the old data (old parity data) is updated to the new data (new parity data). For clarity, other components of storage device 100a are not shown.

[0084] At 314, controller 110 performs an XOR operation between new data stored in write buffer (new data) 302 and old data stored in read buffer (new data) 304 to determine the XOR result, and stores the XOR result in write buffer (XOR result) 305. In some arrangements, write buffer (new data) 302 is a specific implementation of write buffer 114 of storage device 100b. Read buffer (new data) 304 is a specific implementation of read buffer 116 of storage device 100b. Write buffer (XOR result) 305 is a specific implementation of buffer 112 of storage device 100b. In other arrangements, to save memory resources, write buffer (XOR result) 305 may be the same as write buffer (new data) 302 and a specific implementation of buffer 114 of storage device 100b, such that the XOR result may overwrite the contents of write buffer (new data) 302.

[0085] At 315, controller 110 then updates the old data with the new data by writing the XOR result to NAND page (XOR result) 306. Controller 110 (e.g., FTL) updates the logical-to-physical addressing mapping table so that the physical address of NAND page (XOR result) 306 corresponds to the logical address. Controller 110 marks the physical address of NAND page (old data) 303 as containing invalid data, ready for use in obsolete item collection.

[0086] Figure 3B This is a flowchart illustrating an example method 300b for performing parity updates according to some implementation schemes. (Reference) Figures 1 to 3B Method 300b corresponds to method 300a. Method 300b can be executed by the controller 110 of the storage device 100b.

[0087] At 321, controller 110 receives a new type of write request from host 101, operatively coupled to storage device 100b, which contains the address of a buffer (e.g., CMB (Instantaneous XOR) 206) of another storage device (e.g., storage device 100a). At 322, in response to receiving the write request, controller 110 uses a transfer mechanism to transfer new data (new parity data) from the buffer of the other storage device to a new data drive buffer (e.g., write buffer (new data) 302). Thus, controller 110 receives new data from the buffer of the other storage device, not host 101, corresponding to the address of the buffer identified in the write request. At 323, controller 110 performs a read operation to read existing (old) data (existing old parity data) from a non-volatile storage device (e.g., from NAND page (old data) 303) into an existing data drive buffer (e.g., read buffer (new data) 304). Blocks 322 and 323 can be performed in any suitable order or simultaneously.

[0088] At 324, controller 110 determines the XOR result by performing an XOR operation on the new data and the existing data. At 325, after determining the XOR result, controller 110 temporarily stores the XOR result in an XOR result driver buffer (e.g., write buffer (XOR result) 305). At 326, controller 110 writes the XOR result stored in the XOR result driver buffer to a non-volatile memory device (e.g., NAND page (XOR result) 306). As mentioned, the new data and the existing data correspond to the same logical address. The existing data is located at a first physical address of the non-volatile memory device (e.g., at NAND page (old data) 304). Writing the XOR result to the non-volatile memory device includes writing the XOR result to a second physical address of the non-volatile memory device (e.g., at NAND page (XOR result) 306) and updating the L2P mapping so that the logical address corresponds to the second physical address.

[0089] Methods 300a and 300b improve upon conventional parity data update methods by performing an XOR operation in the driver hardware (e.g., in the hardware of the mentioned storage device 100b) to avoid requiring any XOR operation at the host level. Furthermore, new parity data is transferred directly from the data driver (e.g., storage device 100a) to the parity driver (storage device 100b) without going through the host 101. Therefore, methods 300a and 300b improve I / O efficiency, host CPU efficiency, memory resource efficiency, and data transfer efficiency compared to conventional parity update methods.

[0090] Regarding I / O performance efficiency, host 101 needs to submit only one request (the write request at 311 / 321) instead of two requests to update the parity data, and this request only contains the buffer address, not the transient XOR data or the buffer address located in the host's memory 102. In some instances, the work involved in each request includes: 1) host 101 writes the command to the commit queue; 2) host 101 writes the updated commit queue tail pointer to the doorbell register; 3) storage device 100b (e.g., controller 110) retrieves the command from the commit queue; 4) storage device 100b (e.g., controller 110) processes the command; 5) storage device 100b (e.g., controller 110) writes details about the completion status to the completion queue; 6) storage device 100b (e.g., controller 110) notifies host 101 that the command has been completed; 7) host 101 completes processing; and 8) host 101 writes the updated completion queue head pointer to the doorbell register.

[0091] Therefore, host 101 does not need to read existing parity data and perform a transient XOR on the data and an XOR on the existing parity data. Host 101 also does not need to transfer the transient XOR data to its memory 102 and then from memory 102 to the parity data device. The mechanism disclosed herein consumes approximately 10% of the total elapsed time to retrieve 4KB of data from storage device 100b, excluding all time elapsed within storage device 100b for fetch commands, processing commands, fetching data from storage media (e.g., memory array 120), and completing the XOR operation. Therefore, this arrangement can reduce the number of host requests by at least half (from two to one), representing a significant efficiency improvement.

[0092] Regarding host CPU efficiency, host computation is more expensive than computation on storage device 100 because the cost of a host CPU is significantly higher than that of a CPU on storage device 100. Therefore, saving host CPU computation cycles results in higher efficiency. Research estimates that the number of CPU clocks required per NVMe request is approximately 34,000. Therefore, a CPU saving of 34,000 clock cycles occurs each time a parity update needs to be performed. For comparison, a 12Gb SAS interface request on an SSD consumes approximately 79,000 clock cycles per request. Using NVMe interface technology, this requirement is reduced to approximately 34,000, saving approximately 45,000 clock cycles. Considering the elimination of host-level XOR computation along with the reduction in requests, the efficiency improvement can be compared to the efficiency provided by the NVMe interface via the SAS interface.

[0093] Regarding memory resource efficiency, savings are achieved not only in CPU-level host 101 but also in memory consumption. DRAM memory remains a valuable resource for host 101, not only because the limited number of dual in-line memory module (DIMM) slots that can be added to host 101 is limited, but also due to the inherent capacity expansion limitations of DRAM technology itself. Furthermore, modern applications (such as machine learning, in-memory databases, and big data analytics) increase the demand for additional memory at host 101. Therefore, given that DRAM cannot meet this increased memory demand, a new category of devices called Storage Class Memory (SCM) has emerged to bridge the gap. Although this technology is still in its early stages, most existing systems are still seeking solutions that can help reduce memory resource consumption without compromising other attributes such as cost or performance. This arrangement reduces memory consumption and thus lowers costs by eliminating the need to potentially allocate up to two buffers in host 101 (saving up to 200% per request).

[0094] Regarding data transfer efficiency, the number of cross-NVMe interface data transfers for copying data from the drive buffer (e.g., buffer 112) to the host buffer (e.g., memory 102) can be reduced by more than half, reducing the consumption of hardware resources sought by DMA transfers and the utilization of the PCIe bus / network. Furthermore, in example implementations, storage devices 100a, 100b, ... 100n can reside in remote storage devices connected to a PCI switch and then connected to the host via the PCI switch structure, such that transfers between storage devices only need to travel "upstream" from the source device to the nearest switch before returning "downstream" to the target device, thus consuming no bandwidth and avoiding network structure latency upstream of the first switch. This reduction in resources and network latency not only lowers power consumption but also improves performance.

[0095] Typically, to recover data from a failed device in a RAID 5 group, the following steps are performed. Host 101 submits an NVMe read request to the controller 110 of the first storage device in a series of storage devices in the RAID 5 group via interface 140. In the RAID 5 group, the nth storage device is the failed device, and the first through (n-1)th storage devices are active devices. Each storage device in the RAID 5 group is one of storage devices 100. In response, the controller 110 of the first storage device performs a NAND read to the drive buffer of the first storage device. In other words, the controller 110 reads the data requested in the read request from the memory array 120 of the first storage device (one or more of the NAND flash memory devices 130a through 130n) and stores the data in the read buffer 116 of the first storage device. The controller 110 of the first storage device transfers the data from the read buffer 116 of the first storage device across interface 140 to the host 101's memory 102 (e.g., a previous data buffer).

[0096] Next, host 101 submits an NVMe read request to the controller 110 of the second storage device in the RAID 5 group via interface 140. In response, the controller 110 of the second storage device performs a NAND read to the drive buffer of the second storage device. In other words, the controller 110 of the second storage device reads the data requested in the read request from the memory array 120 of the second storage device (one or more of the NAND flash memory devices 130a to 130n) and stores the data in the read buffer 116 of the second storage device. The controller 110 of the second storage device transfers the data from the read buffer 116 of the second storage device across interface 140 to the memory 102 of host 101 (e.g., the current data buffer). Then, host 101 performs an XOR operation between (i) the data in the previous data buffer of host 101 and (ii) the data in the current data buffer of host 101. The result (instant XOR result) is then stored in the instant XOR buffer of host 101. In some cases, the transient XOR buffer of host 101 may potentially be identical to the previous data buffer or the current data buffer, because transient XOR data can replace the existing contents in the buffer to save memory resources.

[0097] Next, host 101 submits an NVMe read request to the controller 110 of the next storage device in the RAID 5 group via interface 140 to read the current data of the next storage device in accordance with the description of the second storage device. Then, host 101 performs an XOR operation between (i) the data in the previous data buffer (which is the instantaneous XOR result determined in the previous iteration involving the previous storage device) and (ii) the current data of the next storage device. This process is repeated until host 101 determines the recovered data by performing an XOR operation between the current data of the (n-1)th storage device and the instantaneous XOR result determined in the previous iteration involving the (n-2)th storage device.

[0098] On the other hand, some arrangements for recovering data from a failed device in a RAID 5 group not only involve performing XOR calculations within the controller 110 of the storage device rather than within the processor 104, but also involve transferring instantaneous XOR data directly from a buffer 112 of another storage device, without using the memory 102 of the host 101. In this respect, Figure 4A This is a block diagram illustrating an example method 400a for performing data recovery according to some implementation schemes. (Reference) Figure 1 and 4A Method 400a provides improved host CPU efficiency and memory resource efficiency compared to the conventional data recovery methods mentioned above. Method 400a can be executed by host 101, storage device 100a (previous storage device), and storage device 100b (current storage device). NAND page (saved data) 403 refers to one or more pages of the NAND flash memory devices 130a to 130n of storage device 100b.

[0099] Figure 4A This demonstrates one iteration of the data recovery method for a failed nth device (as an example, storage device 100n) in a RAID 5 group containing storage devices 100. The current storage device refers to... Figure 4A The storage device currently performing the XOR operation in this iteration is shown in the diagram (storage device 100b, for example). Previous storage device refers to the storage device from which the current storage device obtained previous data (storage device 100a, for example). Therefore, the current storage device can be any of the second to (n-1)th storage devices in the RAID 5 group.

[0100] CMB (Previous Data) 401 is an example and specific implementation of the buffer 112 of storage device 100a. For clarity, Figure 4A The components of storage device 100a other than CMB (previous data) 401 are not shown.

[0101] In an example where storage device 100a is the first storage device in a RAID 5 group, host 101 submits an NVMe read request for a logical address to controller 110 of storage device 100a via bus 106 and interface 140. In response, controller 110 of storage device 100a performs a NAND read into the drive buffer (e.g., CMB (Previous Data) 401) of storage device 100a. In other words, controller 110 reads the start data corresponding to the logical address requested in the read request from memory array 120 of storage device 100a (one or more of NAND flash memory devices 130a to 130n) and stores the start data in CMB (Previous Data) 401. Instead of transferring the start data from CMB (Previous Data) 401 to host 101's memory 102 across interface 140, controller 110 of storage device 100a temporarily stores the start data in CMB (Previous Data) 401 for direct transfer to subsequent storage devices in the RAID 5 group. Therefore, in an instance where storage device 100a is the first storage device in a RAID 5 group, the previous data is the starting data.

[0102] In any instance where storage device 100a is between the first storage device and the current storage device 100b in a RAID 5 group, the content of CMB (Previous Data) 401 (e.g., transient XOR data) is determined in the same manner as the content of the drive buffer (transient XOR) 405 of the current storage device 100b is determined. In other words, storage device 100a is the current storage device in a previous iteration of the data recovery method. Therefore, in any instance where storage device 100a is between the first storage device and the current storage device 100b in a RAID 5 group, previous data refers to transient XOR data.

[0103] In the current iteration and as Figure 4AAs shown, at 411, host 101 submits a new type of NVMe command or request to the controller 110 of the current storage device 100b via bus 106 and interface 140. In some embodiments, the new type of request may be similar to a conventional NVMe write command, but with a different command opcode or flag to indicate that the command is not a normal NVMe write command and that the command should be processed according to the methods described herein. The new type of NVMe command or request contains a reference to the address of the CMB (previous data) 401 of the previous storage device 100a. Examples of the reference include (but are not limited to) an address, CMB address, address descriptor, identifier, pointer, or another suitable indicator identifying the buffer 112 of the storage device. Therefore, the new type of request received by the controller 110 of storage device 100b at 411 does not contain the previous data, but rather the address of the buffer of the previous storage device where the previous data is temporarily stored. The new type of request further contains the logical address (e.g., LBA) where the data is stored. However, in the new type of NVMe command, this LBA refers to the address of the stored data to be read, rather than the address of the data to be written as in regular NVMe write commands.

[0104] At 412, controller 110 performs a NAND read into read buffer (saved data) 404. In other words, controller 110 reads saved data from memory array 120 (e.g., one or more NAND pages (saved data) 403) corresponding to the logical address in a new type of request received from host 101 and stores the saved data in read buffer (saved data) 404. One or more NAND pages (saved data) 403 are pages in one or more of the NAND flash memory devices 130a to 130n of storage device 100b.

[0105] At 413, the controller 110 of storage device 100b performs a P2P read operation to transfer previous data from the CMB (Previous Data) 401 of storage device 100a to the write buffer (New Data) 402 of storage device 100b. In other words, storage device 100b can directly access the contents (e.g., previous data) in the CMB (Previous Data) 401 of storage device 100a using a suitable transfer mechanism, thus bypassing the memory 102 of host 101. The transfer mechanism can identify the source (CMB (Previous Data) 401) of the previous data to be transferred using the address of the CMB (Previous Data) 401 received from host 101 at 411. The transfer mechanism can transfer data from the source to the target buffer (write buffer (New Data) 402). Examples of transfer mechanisms include (but are not limited to) DMA transfer mechanisms, transfer over wireless or wired networks, transfer over a bus or serial network, intra-platform communication mechanisms, or another suitable communication channel connecting the source and target buffers.

[0106] In some instances, after previous data is successfully transferred from CMB (Previous Data) 401 to write buffer (New Data) 402, the controller 110 of storage device 100b may acknowledge the new type of request received at 411 to host 101. In some instances, after previous data is successfully transferred from CMB (Previous Data) 401 to write buffer (New Data) 402 (e.g., via a transfer mechanism or by the controller 110 of storage device 100b acknowledging the new data to the controller 110 of storage device 100a), the controller 110 of storage device 100a may release the memory used by CMB (Previous Data) 401 or indicate that the contents of CMB (Previous Data) 401 are invalid.

[0107] At 414, controller 110 performs an XOR operation between previous data stored in write buffer (new data) 402 and saved data stored in read buffer (saved data) 404 to determine an instantaneous XOR result, and stores the instantaneous XOR result in CMB (instantaneous XOR) 405. In some arrangements, write buffer (new data) 402 is a specific implementation of write buffer 114 of storage device 100a. Read buffer (saved data) 404 is a specific implementation of read buffer 116 of storage device 100a. CMB (instantaneous XOR) 405 is a specific implementation of buffer 112 of storage device 100b. In other configurations, to save memory resources, CMB (instantaneous XOR) 405 may be the same as read buffer (saved data) 404 and a specific implementation of buffer 112 of storage device 100a, such that the instantaneous XOR result can overwrite the contents of read buffer (saved data) 404.

[0108] At this point, the iteration of the current storage device 100b is complete, and the instantaneous XOR result becomes the previous data of the next storage device after the current storage device 100b. In the next iteration, CMB (instantaneous XOR) 405 becomes CMB (previous data) 401. The instantaneous XOR result from CMB (instantaneous XOR) 405 is not transmitted across interface 140 to the memory 102 of host 101, but is stored in CMB (instantaneous XOR) 405 for direct transmission to the next storage device in an operation similar to 413. In the case where the current storage device 100b is the (n-1)th storage device in the RAID 5 group, the instantaneous XOR result is actually the recovery data of the failed nth storage device 100n.

[0109] Figure 4B This is a flowchart illustrating an example method 400b for performing data recovery according to some implementation schemes. (Reference) Figure 1 , 4AAnd 4B, method 400b corresponds to method 400a. Method 400b can be executed by the controller 110 of the current storage device 100b.

[0110] At 421, controller 110 receives a new type of request from host 101, which is operatively coupled to storage device 100b. This new type of request contains the address of a buffer (e.g., CMB (Previous Data) 401) on another storage device (e.g., storage device 100a). At 422, in response to receiving the new type of request, controller 110 uses a transfer mechanism to transfer the previous data from the buffer on the other storage device to a new data drive buffer (e.g., write buffer (new data) 402). Thus, controller 110 receives the previous data from the buffer on the other storage device, not the buffer on host 101, corresponding to the address of the buffer identified in the new type of request. The new type of request further contains a logical address (e.g., LBA) of the saved data. However, in the new type of request, this LBA refers to the address of the saved data to be read, rather than the address of the data to be written as in a regular NVMe write command. At 423, controller 110 performs a read operation to read existing (saved) data (located at the physical address corresponding to the LBA) from a non-volatile storage device (e.g., from a NAND page (saved data) 403) into an existing data drive buffer (e.g., a read buffer (saved data) 404). Boxes 422 and 423 can be performed in any suitable order or simultaneously.

[0111] At 424, controller 110 determines the XOR result by performing an XOR operation on the previous data and the saved data. The XOR result is called the transient XOR result. At 425, after determining the transient XOR result, controller 110 temporarily stores the transient XOR result in a transient XOR result driver buffer (e.g., CMB (transient XOR) 405).

[0112] Typically, to make a spare storage device in a RAID 5 group available, the following steps are performed. Host 101 submits an NVMe read request to the controller 110 of the first storage device in a series of storage devices in the RAID 5 group via interface 140. In the RAID 5 group, the nth storage device is a spare, and the first through (n-1)th storage devices are currently active. Each storage device in the RAID 5 group is one of storage devices 100. In response, the controller 110 of the first storage device performs a NAND read to the drive buffer of the first storage device. In other words, the controller 110 reads the data requested in the read request from the memory array 120 of the first storage device (one or more of the NAND flash memory devices 130a through 130n) and stores the data in the read buffer 116 of the first storage device. The controller 110 of the first storage device transfers the data from the read buffer 116 of the first storage device across interface 140 to the host 101's memory 102 (e.g., a previous data buffer).

[0113] Next, host 101 submits an NVMe read request to the controller 110 of the second storage device in the RAID 5 group via interface 140. In response, the controller 110 of the second storage device performs a NAND read to the drive buffer of the second storage device. In other words, the controller 110 of the second storage device reads the data requested in the read request from the memory array 120 of the second storage device (one or more of the NAND flash memory devices 130a to 130n) and stores the data in the read buffer 116 of the second storage device. The controller 110 of the second storage device transfers the data from the read buffer 116 of the second storage device across interface 140 to the memory 102 of host 101 (e.g., the current data buffer). Then, host 101 performs an XOR operation between (i) the data in the previous data buffer of host 101 and (ii) the data in the current data buffer of host 101. The result (instant XOR result) is then stored in the instant XOR buffer of host 101. In some cases, the transient XOR buffer of host 101 may potentially be identical to the previous data buffer or the current data buffer, because transient XOR data can replace the existing contents in the buffer to save memory resources.

[0114] Next, host 101 submits an NVMe read request to the controller 110 of the next storage device in the RAID 5 group via interface 140 to read the current data of the next storage device in accordance with the description of the second storage device. Then, host 101 performs an XOR operation between (i) the data in the previous data buffer (which is the instantaneous XOR result determined in the previous iteration involving the previous storage device) and (ii) the current data of the next storage device. This process is repeated until host 101 determines the recovered data by performing an XOR operation between the current data of the (n-1)th storage device and the instantaneous XOR result determined in the previous iteration involving the (n-2)th storage device. Host 101 stores the recovered data in its recovery data buffer.

[0115] Recovery data is written to a spare nth storage device at the logical address. For example, host 101 submits an NVMe write request to the nth device and presents its recovery data buffer to be written. In response, the nth storage device performs a data transfer to obtain recovery data from host 101 by transferring the recovery data from host 101's recovery data buffer across the NVMe interface to the drive buffer of the nth storage device. Then, the controller 110 of the nth storage device updates the old data stored in the NAND pages of the nth storage device with the recovery data by writing the recovery data from the drive buffer of the nth storage device into one or more new NAND pages. The controller 110 (e.g., FTL) updates the addressing mapping table to correspond the physical address of the new NAND page to its logical address. The controller 110 marks the physical address of the NAND page on which the old data was stored for obsolete item collection.

[0116] On the other hand, some of the measures used to enable the spare storage devices in the RAID 5 group not only involve performing XOR calculations within the controller 110 of the storage devices rather than within the processor 104, but also involve transferring instantaneous XOR data directly from the buffer 112 of another storage device, without using the memory 102 of the host 101. In this respect, Figure 5A This is a block diagram illustrating an example method 500a for putting a backup storage device into service, according to some implementation schemes. (See reference) Figure 1 and 5A Method 500a provides improved host CPU efficiency and memory resource efficiency compared to the conventional methods mentioned above for enabling backup storage devices. Method 500a can be executed by host 101, storage device 100a (previous storage device), and storage device 100b (current storage device). NAND page (data saved) 503 refers to one or more pages of the NAND flash memory devices 130a to 130n of storage device 100b.

[0117] Figure 5A This demonstrates one iteration that brings the spare nth device (in this example, storage device 100n) into service within a RAID 5 group containing storage devices 100. The current storage device refers to... Figure 5A The storage device currently performing the XOR operation in this iteration is shown in the diagram (storage device 100b, for example). Previous storage device refers to the storage device from which the current storage device obtained previous data (storage device 100a, for example). Therefore, the current storage device can be any of the second to (n-1)th storage devices in the RAID 5 group.

[0118] CMB (Previous Data) 501 is an example and specific implementation of the buffer 112 of storage device 100a. For clarity, Figure 5A The components of storage device 100a other than CMB (previous data) 501 are not shown.

[0119] In an example where storage device 100a is the first storage device in a RAID 5 group, host 101 submits an NVMe read request for a logical address to controller 110 of storage device 100a via bus 106 and interface 140. In response, controller 110 of storage device 100a performs a NAND read into the drive buffer (e.g., CMB (Previous Data) 501) of storage device 100a. In other words, controller 110 reads the start data corresponding to the logical address requested in the read request from the memory array 120 of the first storage device (one or more of the NAND flash memory devices 130a to 130n) and stores the start data in buffer 112 of storage device 100a. Instead of transferring the start data from CMB (Previous Data) 501 to host 101's memory 102 across interface 140, controller 110 of storage device 100a temporarily stores the start data in CMB (Previous Data) 501 for direct transfer to subsequent storage devices in the RAID 5 group. Therefore, in an instance where storage device 100a is the first storage device in a RAID 5 group, the previous data is the starting data.

[0120] In any instance where storage device 100a is between the first storage device and the current storage device 100b in a RAID 5 group, the content of CMB (Previous Data) 501 (e.g., transient XOR data) is determined in the same manner as the content of CMB (Transient XOR) 505 for the current storage device 100b. In other words, storage device 100a is the current storage device in a previous iteration of the data recovery method. Therefore, in any instance where storage device 100a is between the first storage device and the current storage device 100b in a RAID 5 group, "previous data" refers to transient XOR data.

[0121] In the current iteration and as Figure 5A As shown, at 511, host 101 submits a new type of NVMe command or request to the controller 110 of the current storage device 100b via bus 106 and interface 140. In some embodiments, the new type of request may be similar to a conventional NVMe write command, but with a different command opcode or flag to indicate that the command is not a normal NVMe write command and that the command should be processed according to the methods described herein. The new type of NVMe command or request contains a reference to the address of the CMB (previous data) 501 of the previous storage device 100a. Examples of the reference include (but are not limited to) an address, CMB address, address descriptor, identifier, pointer, or another suitable indicator identifying the buffer 112 of the storage device. Therefore, the new type of request received by the controller 110 of storage device 100b at 511 does not contain the previous data, but instead contains the address of the buffer of the previous storage device where the previous data is temporarily stored. The new type of request further contains the logical address (e.g., LBA) of the stored data. However, in the new type of NVMe command, this LBA refers to the address of the saved data to be read, rather than the address of the data to be written as in regular NVMe write commands.

[0122] At 512, controller 110 performs a NAND read into the read buffer (saved data) 504. In other words, controller 110 reads saved data from memory array 120 (e.g., one or more NAND pages (saved data) 503) corresponding to the logical address in a new type of request received from host 101 and stores the saved data in the read buffer (saved data) 504. The one or more NAND pages (saved data) 503 are pages in one or more of the NAND flash memory devices 130a to 130n of storage device 100b.

[0123] At 513, the controller 110 of storage device 100b performs a P2P read operation to transfer previous data from the CMB (previous data) 501 of storage device 100a to the write buffer (new data) 502 of storage device 100b. In other words, storage device 100b can directly access the contents (e.g., previous data) in the CMB (previous data) 501 of storage device 100a using a suitable transfer mechanism, thus bypassing the memory 102 of host 101. The transfer mechanism can identify the source (CMB (previous data) 501) of the previous data to be transferred using the address of the CMB (previous data) 501 received from host 101 at 511. The transfer mechanism can transfer data from the source to the target buffer (write buffer (new data) 502). Examples of transfer mechanisms include (but are not limited to) DMA transfer mechanisms, transfer over wireless or wired networks, transfer over a bus or serial network, intra-platform communication mechanisms, or another suitable communication channel connecting the source and target buffers.

[0124] In some instances, after previous data is successfully transferred from CMB (Previous Data) 501 to write buffer (New Data) 502, the controller 110 of storage device 100b may acknowledge the write request received at 511 to host 101. In some instances, after previous data is successfully transferred from CMB (Previous Data) 501 to write buffer (New Data) 502 (e.g., via a transfer mechanism or by the controller 110 of storage device 100b acknowledging the request to the controller 110 of storage device 100a), the controller 110 of storage device 100a may release the memory used by CMB (Previous Data) 501 or indicate that the contents of CMB (Previous Data) 501 are invalid.

[0125] At 514, controller 110 performs an XOR operation between previous data stored in write buffer (new data) 502 and saved data stored in read buffer (saved data) 504 to determine an instantaneous XOR result, and stores the instantaneous XOR result in CMB (instantaneous XOR) 505. In some arrangements, write buffer (new data) 502 is a specific embodiment of write buffer 114 of storage device 100b. Read buffer (saved data) 504 is a specific embodiment of read buffer 116 of storage device 100b. CMB (instantaneous XOR) 505 is a specific embodiment of buffer 112 of storage device 100b. In other arrangements, to save memory resources, CMB (instantaneous XOR) 505 may be the same as read buffer (saved data) 504 and a specific embodiment of buffer 112 of storage device 100b, such that the instantaneous XOR result may overwrite the contents of read buffer (saved data) 504.

[0126] At this point, the iteration of the current storage device 100b is complete, and the instantaneous XOR result becomes the previous data of the next storage device after the current storage device 100b. In the next iteration, CMB (instantaneous XOR) 505 becomes CMB (previous data) 501. The instantaneous XOR result from CMB (instantaneous XOR) 505 is not transmitted across interface 140 to the memory 102 of host 101, but is stored in CMB (instantaneous XOR) 505 for direct transmission to the next storage device in an operation similar to 513. In the case where the current storage device 100b is the (n-1)th storage device in the RAID 5 group, the instantaneous XOR result is actually the recovery data of the spare nth storage device 100n and is stored in the memory array 120 of storage device 100n.

[0127] Figure 5B This is a flowchart illustrating an example method 500b for putting a backup storage device into use, according to some implementation schemes. (Reference) Figure 1 , 5A Method 500b corresponds to method 500a. Method 500b can be executed by the controller 110 of the storage device 100b.

[0128] At 521, controller 110 receives a new type of request from host 101, operatively coupled to storage device 100. This new type of request contains the address of a buffer (e.g., CMB (Previous Data) 501) on another storage device (e.g., storage device 100a). At 522, in response to receiving the new type of request, controller 110 uses a transfer mechanism to transfer the previous data from the buffer on the other storage device to a new data drive buffer (e.g., write buffer (new data) 502). Thus, controller 110 receives the previous data from the buffer on the other storage device, not the buffer on host 101, corresponding to the address of the buffer identified in the new type of request. The new type of request further contains a logical address (e.g., LBA) of the saved data. However, in the new type of request, the LBA refers to the address of the saved data to be read, rather than the address of the data to be written as in a regular NVMe write command. At 523, controller 110 performs a read operation to read existing (saved) data from a non-volatile storage device (e.g., from a NAND page (saved data) 503) into an existing data drive buffer (e.g., a read buffer (saved data) 504). Boxes 522 and 523 can be executed in any suitable order or simultaneously.

[0129] At 524, controller 110 determines the XOR result by performing an XOR operation on the previous data and the saved data. The XOR result is called the transient XOR result. At 525, after determining the transient XOR result, controller 110 temporarily stores the transient XOR result in a transient XOR result driver buffer (e.g., CMB (transient XOR) 505).

[0130] Figure 6 This is a process flowchart illustrating an example method 600 for providing data protection and recovery in case of driver failure, according to some implementation schemes. (Reference) Figures 1 to 6 Method 600 is executed by the controller 110 of the first storage device (e.g., storage device 100b). Methods 200a, 200b, 300a, 300b, 400a, 400b, 500a, and 500b are specific instances of method 600.

[0131] At 610, the controller 110 of the first storage device receives a new type of request from the host 101. The host 101 is operatively coupled to the first storage device via interface 140. In some instances, the new type of request contains the address of a buffer in a second storage device (e.g., storage device 100a). At 620, in response to receiving the new type of request, the controller 110 of the first storage device transfers new data from the second storage device. At 630, the controller 110 of the first storage device determines the XOR result by performing an XOR operation on the new data and the existing data. The existing data is stored in a non-volatile storage device of the first storage device (e.g., in memory array 120).

[0132] In some arrangements, in response to receiving a new type of request, the controller 110 of the first storage device uses a transfer mechanism to transfer new data from the buffer of the second storage device to a new data drive buffer of the first storage device based on the address of the buffer of the second storage device. The controller 110 performs a read operation to read existing data from a non-volatile storage device (e.g., in memory array 120) into an existing data drive buffer.

[0133] As described with reference to updating parity data (e.g., methods 300a and 300b), the controller 110 of the first storage device is further configured such that the XOR result, after being determined, is stored in an XOR result driver buffer (e.g., write buffer (XOR result) 305) and the XOR result is written to a non-volatile storage device (e.g., to a NAND page (XOR result) 306). The new data and the existing (old) data correspond to the same logical address (same LBA). The existing data is located at a first physical address of the non-volatile storage device (e.g., NAND page (old data) 303). The controller 110 of the first storage device writes the XOR result to a second physical address of the non-volatile storage device (e.g., writes the XOR result to a second physical address of the non-volatile storage device (e.g., NAND page (XOR result) 306) and updates the L2P mapping to correspond to the logical address of the second physical address. The existing data and the new data are parity bits.

[0134] As described in reference to data recovery (e.g., methods 400a and 400b), the XOR result corresponds to the instantaneous XOR result. The instantaneous XOR result from the instantaneous XOR result driver buffer (e.g., CMB (Instantaneous XOR) 405) of the first storage device is transmitted as previous data to the third storage device without being sent across interface 104 to host 101. The third storage device is the next storage device in a series of storage devices after the first storage device.

[0135] As described with reference to activating the backup storage device (e.g., methods 500a and 500b), the XOR result corresponds to the instantaneous XOR result. The instantaneous XOR result from the instantaneous XOR result driver buffer (e.g., CMB (Instantaneous XOR) 505) of the first storage device is transmitted as recovery data to the third storage device without being sent across interface 104 to the host 101. The third storage device is the backup storage device that is activated. The recovery data is stored by the controller of the third storage device in the non-volatile memory of the third storage device.

[0136] In the host-directed P2P transfer mechanism disclosed in methods 300a, 300b, 400a, 400b, 500a, and 500b, host 101 sends a new type of command or request to storage device 100b to trigger data transfer from buffer 112 of storage device 100a to buffer of storage device 100b. The resulting status of the write command or request is reported by storage device 100b to host.

[0137] In the device-directed P2P transmission mechanism, storage device 100a sends a new type of command or request (including a buffer address) to storage device 100b to trigger data transmission from buffer 112 of storage device 100a to buffer of storage device 100b. Figures 7A to 10 The device is guided by a P2P transmission mechanism. The status of a new type of command or request is reported by storage device 100b to storage device 100a. Storage device 100a considers the status of storage device 100b before reporting the status of the host command or data write update request it first received, thus triggering a parity write update by storage device 100b.

[0138] In a host-oriented P2P transmission mechanism, since a new type of request is sent to the parity driver during parity updates, the parity driver is responsible for returning the state to host 101. In a device-oriented P2P transmission mechanism, host 101 does not send the request to the parity driver, thus eliminating another I / O (and thus from 2 to 1). Instead, when host 101 first issues a request to update data, host 101 implicitly delegates the responsibility to the data driver. The data driver, after calculating the instantaneous XOR, sends a new type of request (on behalf of host 101) to the parity driver using the CMB address. Because the parity driver receives the request from the data driver, it returns the resulting state to the data driver, not to host 101. Host 101 is unaware of this transaction, but of course requests it to occur implicitly when host 101 makes the first write request by providing the parity driver's CMB address. The data driver itself does not know which is the parity driver and does indeed need the parity driver's CMB address information provided by host 101.

[0139] Figure 7A This is a block diagram illustrating an example method 700a for performing parity updates according to some implementation schemes. (Reference) Figures 1 to 3BMethod 700a differs from method 300a in that, at 311', the controller 110 of storage device 100a (e.g., a data drive) submits a new type of NVMe write command or request to the controller 110 of storage device 100b (e.g., a parity drive) via a wireless or wired network, bus or serial communication, intra-platform communication mechanism, or another suitable communication channel between storage device 100a and storage device 100b. The new type of request includes a reference to the buffer address of the CMB (Instantaneous XOR) 206 of storage device 100a and the LBA of the location of the data used for the XOR operation of the data to be written. Upon receiving the request, the controller 110 of storage device 100b reads the data located at the CMB (Instantaneous XOR) buffer at 313 and transfers it to the write buffer (new data) 302. After the controller 110 of storage device 100b notifies storage device 100a that the request is complete, storage device 100a releases the memory used for the CMB (Instantaneous XOR) buffer 206. Reference examples include (but are not limited to) an address, CMB address, address descriptor, identifier, pointer, or another suitable indicator for identifying the buffer 112 of the storage device. The transfer at 313 is performed in response to a write request received at 311'.

[0140] Figure 7B This is a flowchart illustrating an example method 700b for performing parity updates according to some implementation schemes. (Reference) Figures 1 to 3B Methods 700b, 7A and 7B, correspond to method 700a. Method 700b can be executed by the controller 110 of storage device 100b. Method 700b differs from method 300b in that, at 321', the controller 110 of storage device 100b (e.g., a parity drive) receives a write request from storage device 100a (e.g., a data drive). Block 322 is executed in response to the request received at 321'.

[0141] Figure 8A This is a block diagram illustrating an example method 800a for performing data recovery according to some implementation schemes. (Reference) Figure 1 , 4AMethods 800a and 400a differ in that, at 411', the controller 110 of storage device 100a (e.g., a previous storage device) submits a new type of NVMe write command or request to the controller 110 of storage device 100b (e.g., the current storage device) via a wireless or wired network, bus or serial communication, intra-platform communication mechanism, or another suitable communication channel between storage device 100a and storage device 100b. The request includes a reference to the buffer address of the CMB (Previous Data) 401 of storage device 100a and the LBA of the location of the data to be used with the data located at the calculation buffer address used for XOR operations. Upon receiving the request, the controller 110 of storage device 100b reads the data located at the CMB (Instant XOR) buffer at 413 and transfers it to the write buffer (New Data) 402. After the controller 110 of storage device 100b notifies storage device 100a of a request completion, storage device 100a releases memory used in buffer CMB (instantaneous XOR) 206. Reference examples include (but are not limited to) addresses, CMB addresses, address descriptors, identifiers, pointers, or other suitable indicators identifying buffer 112 of the storage device. The transfer at 413 is performed in response to receiving a write request at 411'.

[0142] Figure 8B This is a flowchart illustrating an example method 800b for executing data according to some implementation schemes. (Reference) Figure 1 , 4A Methods 800b, 4B, 8A, and 8B, correspond to method 800a. Method 800b can be executed by the controller 110 of storage device 100b. The difference between method 800b and method 400b is that at 421', the controller 110 of storage device 100b (e.g., the current storage device) receives a new type of write request from storage device 100a (e.g., the previous storage device). Block 422 is executed in response to the request received at 421'.

[0143] Figure 9A This is a block diagram illustrating an example method 900a for putting a backup storage device into service, according to some implementation schemes. (See reference) Figure 1 , 5AMethods 900a and 500a differ in that, at 511', the controller 110 of storage device 100a (e.g., a previous storage device) submits a new type of NVMe write command or request to the controller 110 of storage device 100b (e.g., a current storage device) via a wireless or wired network, bus or serial communication, intra-platform communication mechanism, or another suitable communication channel between storage device 100a and storage device 100b. The request includes a reference to the buffer address of the CMB (Previous Data) 501 of storage device 100a and the LBA of the location of the data to be used with the data located at the buffer address used for XOR operations. Upon receiving the request, the controller 110 of storage device 100b reads the data located at the buffer CMB (Instant XOR) at 413 and transfers it to the write buffer (New Data) 402. After the controller 110 of storage device 100b notifies storage device 100a that the request is complete, storage device 100a releases the memory used for the buffer CMB (Instant XOR) 206. Reference examples include (but are not limited to) an address, CMB address, address descriptor, identifier, pointer, or another suitable indicator for identifying the buffer 112 of the storage device. The transfer at 513 is performed in response to a write request received at 511'.

[0144] Figure 9B This is a flowchart illustrating an example method 900b for putting a backup storage device into service, according to some implementation schemes. (Reference) Figure 1 , 5A Methods 900b, 5B, 9A, and 9B, correspond to method 900a. Method 900b can be executed by the controller 110 of storage device 100b. Method 900b differs from method 500b in that, at 521', the controller 110 of storage device 100b (e.g., the current storage device) receives a new type of write request from storage device 100a (e.g., the previous storage device). Block 522 is executed in response to the request received at 521'.

[0145] Figure 10 This is a flowchart illustrating an example method 1000 for providing data protection and recovery in case of driver failure, according to some implementation schemes. (Reference) Figure 1 , 6Methods 1000, from 7A to 10, are executed by the controller 110 of the first storage device (e.g., storage device 100b). Methods 700a, 700b, 800a, 800b, 900a, and 900b are specific instances of method 1000. Method 1000 differs from method 600 in that, at 610', the controller 110 of the first storage device (e.g., storage device 100b) receives a new type of write request from the second storage device (e.g., storage device 100a) instead of from the host 101. In some instances, the new type of write request includes the address of a buffer in the second storage device (e.g., storage device 100a) and the LBA of the location of the data to be used with the data located at the buffer address used for XOR operations. Box 620 is executed in response to box 610'.

[0146] Figure 11 This is a flowchart illustrating an example method 1100 for providing data protection and recovery in case of drive failure, according to some implementation schemes. (Reference) Figures 1 to 11 Method 1100 is executed by the controller 110 of the first storage device (e.g., storage device 100b). Methods 200a, 200b, 300a, 300b, 400a, 400b, 500a, 500b, 600, 700a, 700b, 800a, 800b, 900a, 900b, and 1000 are specific instances of method 1100. At 1110, the controller 110 of the first storage device (e.g., storage device 100b) receives a new type of write request. In some arrangements, the new type of write request may be received from host 101, as disclosed at block 610 in method 600. In other arrangements, the new type of write request may be received from a second storage device (e.g., storage device 100a), as disclosed at block 610' in method 1000. Box 620 (in methods 600 and 1000) is executed in response to box 1110 in method 1100. Box 630 (in methods 600 and 1000) is executed in response to box 620 in method 1100.

[0147] Figure 12 This is a schematic diagram illustrating a host-side view 1200 used for updating data according to some implementation schemes. (Reference) Figures 1 to 12The RAID stripe written by host 101 includes logical blocks 1201, 1202, 1203, 1204, and 1205. Logical blocks 1201 to 1204 contain regular, non-parity data. Logical block 1205 contains parity data for the data in logical blocks 1201 to 1204. As shown, in response to determining that new data 1211 will be written to logical block 1202 (initially containing old data) in one of the storage devices 100, instead of performing two XOR operations or storing transient data (e.g., transient XOR results) as before, host 101 only needs to update logical block 1205 with the new data 1211. The controller 110 of storage device 100 performs the XOR operation and performs the described P2P transfer. Both new data 1211 and old data 1212 are used to update the parity data in logical block 1205.

[0148] Figure 13 This is a diagram illustrating the placement of parity data according to some implementation schemes. (Reference) Figures 1 to 13 RAID group 1300 (e.g., RAID 5 group) comprises four drives: Drive 1, Drive 2, Drive 3, and Drive 4. Each drive instance is one of the storage devices 100. Each of drives 1 to 4 stores data and parity data in its corresponding memory array 120. Four RAID stripes A, B, C, and D are depicted, where stripe A includes data A1, A2, A3, and parity A, and stripes B, C, and D, and so on. Parity A is generated by XORing data A1, A2, and A3 and stored on drive 4. Parity B is generated by XORing data B1, B2, and B3 and stored on drive 3. Parity C is generated by XORing data C1, C2, and C3 and stored on drive 2. Parity D is generated by XORing data D1, D2, and D3 and stored on drive 1.

[0149] Typically, if A3 is modified (updated) to A3', host 101 reads A1 from drive 1 and A2 from drive 2, performs an XOR operation on A1, A2, and A3' to produce parity A', writes A3' to drive 3, and writes parity A' to drive 4. Alternatively, and typically, to avoid having to reread all other drives (especially when there are more than four drives in a RAID group), host 101 can also produce parity A' by reading A3 from drive 3, reading parity A from drive 4, performing an XOR operation on A3, A3', and parity A, and then writing A3' to drive 3 and parity A' to drive 4. In both conventional cases, modifying A3 will require host 101 to perform at least two reads from the drive and two writes to the drive.

[0150] The arrangement disclosed herein eliminates the need for host 101 to read parity A by enabling the drive (e.g., its controller 110) to not only perform XOR operations internally but also to perform P2P data transfer to generate and store parity A'. In some instances, the drive may support new types of write commands (which may be vendor-unique commands (VUCs) or new commands in the new NVMe specification according to an extended command set) for calculating and storing the results of XOR operations and coordinating P2P data transfer between drives.

[0151] Host 101 can send a VUC containing an LBA to a first drive. The LBA is used for (1) data or parity data stored on the first drive and (2) the address of a second drive containing data XORed with the data or parity data corresponding to the LBA. The first drive can read the data or parity data corresponding to the LBA sent by host 101. The read is an internal read, and the read data is not sent back to host 101. The first drive XORs the data obtained from the second drive based on the address with the internally read data or parity data, stores the result of the XOR operation in the LBA corresponding to the read data or parity data, and confirms the successful completion of the command to host 101. This command can be executed in a time not exceeding that required for read and write commands of the same size.

[0152] Host 101 may send a command to driver 4, the command containing an LBA for parity check A and an address of another driver storing the result of an XOR operation between A3 and A3'. In response, driver 4 calculates parity check A' and stores it in the same LBA that previously contained parity check A.

[0153] In some instances, the command can trigger an XOR operation performed within controller 110. The command can also trigger a P2P transfer. The command can be implemented through any interface used to communicate with the storage device. In some instances, NVMe interface commands can be used. For example, host 101 can send an XFER command (with an indication LBA) to controller 110 to cause controller 110 to perform a computation function (CF) on data from memory array 120 corresponding to the indicated LBA, specifying the type of CF to be performed, the timing of the CF execution, and which data is CF-specific. For example, before writing data to memory array 120, the CF operation can invoke an XOR operation between read data from memory array 120 and write data transferred from another drive.

[0154] In some instances, no CF operation is performed on the metadata. Host 101 can specify the protection information to be included as part of the CF operation. In other instances, the XFER command causes controller 110 to operate on the data and metadata according to the CF specified for the logical block indicated in the command. Host 101 can also specify the protection information to be included as part of the CF operation.

[0155] In some configurations, host 101 may invoke a CF (Creative Message) regarding data sent to storage device 100 (e.g., for write operations) or requested from storage device 100 (e.g., for read operations). In some instances, the CF is applied before data is saved to memory array 120. In some instances, the CF is applied after data is saved to memory array 120. In some instances, storage device 100 sends data to host 101 after performing the CF. Instances of CF include the XOR operation described herein, such as for RAID 5.

[0156] The preceding description is provided to enable those skilled in the art to practice the various aspects described herein. Various modifications to these aspects will readily be understood by those skilled in the art, and the general principles defined herein can be applied to other aspects. Therefore, the claims are not intended to be limited to the aspects shown herein, but are given the full scope consistent with the language of the claims, wherein an element referred to in the singular does not mean “one and only one” (unless expressly stated otherwise), but rather “one or more.” Unless expressly stated otherwise, the term “some” refers to one or more. All structural and functional equivalents of elements throughout the various aspects described in the preceding description that are known to those skilled in the art thereafter are expressly incorporated herein by reference and are intended to be covered by the claims. Furthermore, whatever is disclosed herein, whether or not it is expressly stated in the claims, is not intended to be made public. Unless an element is expressly stated using the phrase “component for…”, no element of a claim should be construed as a component plus a function.

[0157] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an illustrative method. Based on design preferences, it will be understood that the specific order or hierarchy of steps in the process may be rearranged, while still remaining within the scope of the previously described method. The appended method claims present the elements of each step in the order shown in the sample and are not intended to limit one to the specific order or hierarchy presented.

[0158] The prior description of the disclosed embodiments is provided to enable those skilled in the art to make or use the disclosed object. Those skilled in the art will readily understand various modifications to these embodiments and can apply the general principles defined herein to other embodiments without departing from the spirit or scope of the prior description. Therefore, the prior description is not intended to be limited to the embodiments shown herein, but should be given the broadest scope consistent with the principles and novel features disclosed herein.

[0159] The various examples illustrated and described are provided merely as examples to illustrate the various features of the claims. However, the features shown and described with respect to any given example are not necessarily limited to the associated example, but may be used or combined with other examples shown and described. Furthermore, the claims are not intended to be limited to any single example.

[0160] The foregoing method descriptions and process flowcharts are provided as illustrative examples only and are not intended to require or imply that the steps in the various examples must be performed in the presented order. Those skilled in the art will understand that the order of the steps in the foregoing examples can be performed in any order. Words such as "afterwards," "next," "next," etc., are not intended to limit the order of steps; these words are only used to guide the reader in understanding the description of the method. Furthermore, any claim element referred to in the singular form (e.g., using the articles "a / an" or "the") should not be construed as limiting the element to the singular.

[0161] The various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the examples disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above in general terms of their functionality. Whether this functionality is implemented in hardware or software depends on the specific application and the design constraints imposed on the system as a whole. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as departing from the scope of this disclosure.

[0162] The hardware used to implement the various illustrative logics, logic blocks, modules, and circuits described in conjunction with the examples disclosed herein may be implemented or executed using a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic device, discrete gate or transistor logic, discrete hardware component, or any combination thereof designed to perform the functions described herein. The general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Alternatively, some steps or methods may be performed by a circuit system specific to a given function.

[0163] In some exemplary instances, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality may be stored as one or more instructions or code on a non-transitory computer-readable storage medium or a non-transitory processor-readable storage medium. The steps of the methods or algorithms disclosed herein may be embodied in a processor-executable software module that may reside on a non-transitory computer-readable or processor-readable storage medium. A non-transitory computer-readable or processor-readable storage medium may be any storage medium accessible by a computer or processor. By way of example, and not limitation, such non-transitory computer-readable or processor-readable storage media may include RAM, ROM, EEPROM, flash memory, CD-ROM or other optical drive storage devices, magnetic drive storage devices or other magnetic storage devices, or any other medium that may be used to store desired program code in the form of instructions or data structures and is accessible by a computer. As used herein, drives and optical discs include compact optical discs (CDs), laser optical discs, optical discs, digital versatile optical discs (DVDs), floppy disk drives, and Blu-ray discs, wherein drives typically magnetically copy data, while optical discs use laser optical copying of data. The combinations described above also fall within the scope of non-transitory computer-readable and processor-readable media. Furthermore, the operation of a method or algorithm may reside as one or any combination or set of code and / or instructions on non-transitory processor-readable and / or computer-readable storage media, which may be incorporated into a computer program product.

[0164] The foregoing description of the disclosed examples is provided to enable those skilled in the art to make or use this disclosure. Various modifications to these examples will readily be apparent to those skilled in the art, and the general principles defined herein can be applied to some examples without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not intended to be limited to the examples shown herein, but should be given the broadest scope consistent with the principles and novel features disclosed herein and in the appended claims.

Claims

1. A first storage device in a system comprising a plurality of storage devices communicating with a host, the first storage device comprising: Non-volatile storage devices; and The controller in the first storage device is configured to: Receive a request to update the parity bit, wherein the request is separate from the request to write data associated with the parity bit and includes a logical address corresponding to the parity bit. In response to receiving the request to update the parity bit, new data is transferred from the second storage device of the plurality of storage devices to the first storage device; and The XOR result is determined by performing an XOR operation on the new data and existing data from the second storage device, wherein the existing data is stored at the logical address in the non-volatile storage device of the first storage device. The XOR operation mentioned therein includes a single XOR operation that updates the old parity bit.

2. The first storage device of claim 1, wherein the request to update the parity bit is received from the host communicating with the first storage device.

3. The first storage device according to claim 1, wherein the request to update the parity bit is received from the second storage device.

4. The first storage device of claim 1, wherein the request to update the parity bit includes a reference to the buffer of the second storage device.

5. The first storage device according to claim 4, further comprising an existing data drive buffer and a new data drive buffer, wherein... In response to receiving the request to update the parity bit, the controller uses a transfer mechanism based on the reference of the buffer in the second storage device to transfer the new data from the buffer in the second storage device to the new data driver buffer; and The controller performs a read operation to read the existing data from the non-volatile storage device into the existing data driver buffer.

6. The first storage device of claim 4, further comprising an XOR result driver buffer, wherein the controller is further configured to: The XOR result is stored in the XOR result driver buffer after it is determined; and The XOR result is written to the non-volatile storage device.

7. The first storage device according to claim 6, wherein Both the new data and the existing data correspond to the logical address; The existing data is located at the first physical address of the non-volatile storage device; and Writing the XOR result to the non-volatile storage device includes: Write the XOR result to the second physical address of the non-volatile storage device; and Update the logical-to-physical mapping to correspond to the logical address to the second physical address.

8. The first storage device according to claim 5, further comprising a transient XOR result driver buffer, wherein The XOR result corresponds to the instantaneous XOR result; The instantaneous XOR result from the instantaneous XOR result driver buffer is transmitted as previous data to a third storage device without being sent across the interface to the host; and The third storage device is the next storage device in a series of storage devices after the first storage device.

9. The first storage device according to claim 5, further comprising a transient XOR result driver buffer, wherein The XOR result corresponds to the instantaneous XOR result; The instantaneous XOR result from the instantaneous XOR result driver buffer is transmitted as recovered data to a third storage device, without being sent across the interface to the host; The third storage device is a standby storage device that is in use; and The recovered data is stored by the controller of the third storage device in the non-volatile memory of the third storage device.

10. The first storage device according to claim 1, further comprising an existing data driver buffer, a new data driver buffer, and an XOR result driver buffer, wherein... In response to receiving the request to update the parity bit, the controller transfers the new data from the second storage device to the new data drive buffer; and The controller performs a read operation to read the existing data from the non-volatile storage device into the existing data driver buffer; and After determining the XOR result, the controller stores the XOR result in the XOR result driver buffer.

11. A method for managing data in a system comprising multiple storage devices communicating with a host, the method comprising: A request to update a parity bit is received by the controller of the first storage device among the plurality of storage devices, wherein the request is separate from a request to write data associated with the parity bit and includes a logical address corresponding to the parity bit. In response to receiving the request to update the parity bit, the controller of the first storage device transmits new data from the second storage device among the plurality of storage devices to the first storage device; and The controller of the first storage device determines the XOR result by performing an XOR operation on the new data and existing data from the second storage device, wherein the existing data is stored at the logical address in the non-volatile storage device of the first storage device. The XOR operation mentioned therein includes a single XOR operation that updates the old parity bit.

12. The method of claim 11, wherein the request to update the parity bit is received from the host communicating with the first storage device.

13. The method of claim 11, wherein the request to update the parity bit is received from the second storage device.

14. The method of claim 11, wherein the request to update the parity bit includes a reference to the buffer of the second storage device.

15. The method of claim 14, further comprising: In response to receiving the request to update the parity bit, the controller uses a transfer mechanism based on the reference of the buffer of the second storage device to transfer the new data from the buffer of the second storage device to the new data driver buffer of the first storage device. and The controller performs a read operation to read the existing data from the non-volatile storage device into the existing data driver buffer.

16. A non-transitory computer-readable medium comprising computer-readable instructions that, when executed, cause a processor of a first storage device in a system having a plurality of storage devices in communication with a host: Receive a request to update the parity bit, wherein the request is separate from the request to write data associated with the parity bit and includes a logical address corresponding to the parity bit. In response to receiving the request to update the parity bit, new data is transferred from the second storage device of the plurality of storage devices to the first storage device; and The XOR result is determined by performing an XOR operation on the new data from the second storage device and the existing data, wherein the existing data is stored at the logical address in the non-volatile storage device of the first storage device, and The XOR operation mentioned therein includes a single XOR operation that updates the old parity bit.

17. The non-transitory computer-readable medium of claim 16, wherein the request to update the parity bit is received from the host communicating with the first storage device.

18. The non-transitory computer-readable medium of claim 16, wherein the request to update the parity bit is received from the second storage device.

19. The non-transitory computer-readable medium of claim 16, wherein the request to update the parity bit includes a reference to the buffer of the second storage device.

20. The non-transitory computer-readable medium of claim 19, further causing the processor to: In response to receiving the request to update the parity bit, the new data is transferred from the buffer of the second storage device to the new data drive buffer of the first storage device using a transfer mechanism based on the reference of the buffer of the second storage device; and Perform a read operation to read the existing data from the non-volatile storage device into the existing data driver buffer.

Images