CXL module fault tolerance using RAID

Abstract

The invention discloses a CXL module fault tolerance using RAID (Redundant Array of Independent Disks). Related systems, methods, and computer-readable storage devices can include a fabric network for isolating and correcting faults in the fabric network and device faults. The fabric network is connected to a group of devices and a host. The device group includes at least one target data device, other data devices, and parity devices. A standalone device redundant array (RAID) engine coupled to the group of devices performs access operations. A fault tolerant engine is provided in a leaf switch of the fabric network. A routing processor determines a path of a request received from the host to the target data device. The routing processor is coupled to the fault tolerant engine, and the routing processor is provided in the leaf switch.

Description

[0001] CROSS REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of all of U.S. Provisional Patent Applications No. 63 / 517,647, 63 / 517,653, 63 / 517,632, 63 / 517,658, and 63 / 517,660, filed on August 4, 2023, the disclosures of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure relates generally to fabric networks, and more generally to isolating and correcting faults in fabric networks. Background Art

[0004] A fabric interconnect system connects multiple components by providing a fabric through which these components can communicate. A fabric interconnect system provides peer-to-peer communication, thereby allowing decentralized communication between components. There may be many dependencies between interconnected components in a fabric interconnect system. Due to these dependencies, a failure in one component may trigger a failure condition in another component.

[0005] Fabric interconnect systems can have numerous sources of failure, ranging from software failures to hardware failures. The interoperability of a system is often characterized by its level of fault tolerance, safety, and reliability. Fault tolerance is often associated with defining redundancy of system components and / or the connections between those components.

[0006] In view of the foregoing deficiencies, there is a need for a fabric interconnect system that is more adaptable to its environment, so that failures can be quickly detected and efficiently contained without shutting down the entire system. There is a further need for a fault-tolerant system in which hardware and / or software systems can be applied to memory devices in a cluster to maintain normal cluster operation even if a component fails without warning. There is also a need for a fault-tolerant system that supports global fabric attached memory (GFAM) devices that allow different types of memory to be directly attached to and accessible by multiple processor nodes. Summary of the invention

[0007] One aspect of the present disclosure relates to a system, which includes at least one group of memory devices, wherein for a destination address in a memory request, one of the devices is a target device and another of the devices is a parity device. The system also includes at least one group of memory devices, at least one host, and a memory switch, the memory switch including at least one fault tolerance engine, the at least one fault tolerance engine coupled to the at least one group of memory devices, configured to perform: a read access and a read recovery when one of the memory devices is damaged during the read access; or a write access including atomically updating data of the target device and parity data of the parity device and a write recovery when one of the memory devices is damaged during the write access. The system also includes at least one routing processor, which is coupled to the memory switch, configured to determine a path of the memory request received from the at least one host to the target device and direct the memory request to the target device.

[0008] In some embodiments, the system further includes a fabric manager configured to execute storage instructions to scan metadata contents of the devices of the at least one group using a fast consistency recovery process to recover from a switch failure, access metadata bits and data bits from the parity devices, and access the data bits from each data device. The metadata bits in the parity devices define a parity state, and the parity state of the parity devices is an idle state, a locked state, or an invalid state.

[0009] In some embodiments, the fabric manager is configured to determine whether the metadata bit of the parity device is set to the locked state. When the parity state of the parity device is not set to the locked state, the fabric manager omits checking consistency between the data device and the parity device and takes no further action.

[0010] In some embodiments, the fabric manager is configured to determine whether the metadata bit of the parity device is set to the locked state. When the parity state of the parity device is set to the locked state, a consistency check is performed to determine whether the data of the parity device is equal to the XOR of all data of the data device, and when the data of the parity device is equal to the XOR of all data of the data device, the fabric manager clears the lock and rewrites the parity data.

[0011] In some embodiments, the fabric manager is configured to determine whether the metadata bit of the parity device is set to the locked state. When the parity state of the parity device is set to the locked state, a consistency check is performed to determine whether the data of the parity device is not equal to the XOR of all data of the data device, and when the data of the parity device is not equal to the XOR of all data of the data device, the fabric manager clears the lock and writes new parity data resulting from the XOR operation of all data.

[0012] In some embodiments, the metadata bits include poison metadata bits.

[0013] Another aspect of the present disclosure is a non-transitory computer-readable medium having computer-executable instructions stored thereon, which when executed by a computing device causes the computing device to perform a method used by a memory controller of a memory system. This includes at least one group of memory devices, wherein for a destination address in a memory request, one of the devices is a target device and the other of the devices is a parity device, and at least one fabric is coupled to the at least one group of memory devices and at least one host and includes a memory switch including at least one fault tolerance engine, the at least one fault tolerance engine being coupled to each of the memory devices in the at least one group. The method includes: performing a read access and a read recovery when one of the memory devices is damaged during the read access; or performing a write access including atomically updating data of the target device and parity data of the parity device and a write recovery when one of the memory devices is damaged during the write access, wherein at least one routing processor coupled to the memory switch is configured to determine a path of the memory request received from the at least one host to the target device and direct the memory request to the target device.

[0014] In some embodiments, the read recovery is performed when one of the memory devices is damaged. The at least one fault tolerance engine atomically reconstructs the data by: locking a location in the memory device associated with the parity data to provide fault tolerance to prevent subsequent read or write requests from accessing the parity data; and unlocking the locked location in the memory device when the reconstruction operation is complete so that the subsequent read or write request can be performed.

[0015] In some embodiments, the rebuild operation performs an atomic read access operation. This includes: if the target device responds to the read access request from the requesting host with a valid data response, then forwarding the request data to the requesting host; or if the target device responds to the read access request with a poison response or a timeout response, then performing an atomic read access operation via the fault-tolerant engine. The execution includes: locking the cache line of the parity device; reading existing data from the data lines of other data devices of the at least one group; calculating a regeneration target line value by performing an exclusive OR (XOR) operation on the check data of the parity device and the data of the data lines from the other data devices; writing the regeneration target line value to the target device; sending the regeneration target line value to the requesting host; and writing the check data to the cache line and unlocking the cache line.

[0016] In some embodiments, when one of the memory devices is damaged during the write access, the fault-tolerant engine is configured to perform an atomic write access operation by: locking an entire parity row group containing the target device to prevent subsequent write access requests to the target device from accessing the target device while the fault-tolerant engine is writing to the target device; and unlocking the entire parity row group when the write access is completed so that the subsequent write access request can be executed.

[0017] In some embodiments, the fault-tolerant engine is configured to generate new data by performing the atomic write access operation and a partial cache line write operation, wherein the partial cache line write operation prevents a portion of existing data from being overwritten by the new data when the new data is written to the target device.

[0018] In some embodiments, the memory system further comprises: providing a hot spare device that is pre-configured so that metadata marks an entire row of the hot spare device as invalid; and providing instructions for controlling the operation of the at least one fabric to a software device coupled to the memory. The method further comprises: in response to a single device failure detected by the fault-tolerant engine, receiving a notification of the single device failure from the fault-tolerant engine via the software device; and replacing the single failed device with the hot spare device via the software device so that subsequent read and write accesses regenerate data previously stored on the single failed device to the hot spare device.

[0019] In some embodiments, the fault tolerance engine is configured to perform a rebuild operation to reconstruct data in response to detecting a single memory device failure per device group or a failure of one or more switch links connected to the memory device.

[0020] In some embodiments, the fabric manager is configured to execute storage instructions to scan metadata contents of the devices of the at least one group using a fast consistency recovery process to recover from a switch failure, access metadata bits and parity bits from the parity devices, and access the data bits from each data device. The metadata bits in the parity devices define a parity state and the parity state of the parity devices is an idle state, a locked state, or an invalid state.

[0021] In some embodiments, the fabric manager is further configured to determine whether the metadata bit of the parity device is set to the locked state. When the parity state of the parity device is not set to the locked state, the fabric manager omits checking consistency between the data device and the parity device and takes no further action.

[0022] In some embodiments, the fabric manager is further configured to determine whether the metadata bit of the parity device is set to the locked state. When the parity state of the parity device is set to the locked state, a consistency check is performed to determine whether the data of the parity device is equal to the XOR of all data of the data device, and when the data of the parity device is equal to the XOR of all data of the data device, the fabric manager clears the lock and rewrites the parity data.

[0023] In some embodiments, the fabric manager is further configured to determine whether the metadata bit of the parity device is set to the locked state. When the parity state of the parity device is set to the locked state, a consistency check is performed to determine whether the data of the parity device is not equal to the XOR of all data of the data device, and when the data of the parity device is not equal to the XOR of all data of the data device, the fabric manager clears the lock and writes new parity data resulting from the XOR operation of all data.

[0024] In some embodiments, the entire row of the hot spare is marked as invalid via a metadata tag of the hot spare device, a fabric manager is coupled to the at least one fabric, the fabric manager is configured to manage operation of the at least one fabric, and the fabric manager is configured to decide to replace a failed device with the hot spare device.

[0025] In some embodiments, the non-transitory computer-readable medium further comprises coupling a fabric manager to the at least one fabric to manage its operation and at least one device configured as a self-diagnostic data device, the self-diagnostic data device sending a ready-to-be-replaced warning to the fabric manager when one or more errors detected exceed a threshold indicating an impending failure. In response to the warning, the fabric manager is configured to: preemptively add a hot spare device as a new data device; update the fault-tolerant engine, wherein each fault-tolerant engine maintains a record of the hot spare device location of the device group; instruct all tenants in the device group to perform a hot spare device data reconstruction to rebuild the data to the new data device before the impending failure occurs by each tenant reading the location of its corresponding shared memory device; update the fault-tolerant engine; and remove the ready-to-be-replaced device.

[0026] Another aspect of the present disclosure relates to a method of utilizing one or more processors and one or more memories. The method includes: providing at least one group of memory devices; for a destination address in a memory request, one of the devices is a target device and the other of the devices is a parity device; and providing at least one fabric coupled to the at least one group of memory devices and at least one host and including a memory switch including at least one fault tolerance engine, the at least one fault tolerance engine coupled to each of the memory devices in the at least one group of memory devices, configured to perform: a read access and a read recovery when one of the memory devices is damaged during the read access; or a write access including atomically updating data of the target device and parity data of the parity device and a write recovery when one of the memory devices is damaged during the write access; and providing at least one routing processor coupled to the memory switch and configured to determine a path of the memory request received from the at least one host to the target device and direct the memory request to the target device.

[0027] In some embodiments, the read recovery is performed when one of the memory devices is damaged, and the at least one fault-tolerant engine reconstructs the data in an atomic manner by: locking a location in the memory device associated with the parity data to provide fault tolerance to prevent subsequent read or write requests from accessing the parity data; and unlocking the locked location in the memory device when the reconstruction operation is completed so that the subsequent read or write request can be executed.

[0028] In some embodiments, when one of the memory devices is damaged during the write access, the fault-tolerant engine is configured to perform an atomic write access operation by: locking an entire parity row group containing the target device to prevent subsequent write access requests to the target device from accessing the target device while the fault-tolerant engine is writing to the target device; and unlocking the entire parity row group when the write access is completed so that the subsequent write access request can be executed.

[0029] In some embodiments, the fault-tolerant engine is configured to generate new data by performing the atomic write access and a partial cache line write operation, wherein the partial cache line write operation prevents a portion of existing data from being overwritten by the new data when the new data is written to the target device.

[0030] In some embodiments, the memory system further comprises: providing a hot spare device that is pre-configured so that metadata marks an entire row of the hot spare device as invalid; and providing instructions for controlling the operation of the at least one fabric to a software device coupled to the memory. The method further comprises: in response to a single device failure detected by the fault-tolerant engine, receiving a notification of the single device failure from the fault-tolerant engine via the software device; and replacing the single failed device with the hot spare device via the software device so that subsequent read and write accesses regenerate data previously stored on the single failed device to the hot spare device.

[0031] In some embodiments, the method further includes configuring the fault tolerance engine to perform a rebuild operation to reconstruct data in response to detecting a single memory device failure per device group or a failure of one or more switch links connected to the memory device.

[0032] In some embodiments, the method further includes providing a fabric manager configured to execute storage instructions to scan metadata contents of the devices of the at least one group to recover from a switch failure using a fast consistency recovery process, access metadata and data bits from the parity device, and access the data bits from each data device. The metadata bits in the parity device define a parity state, and the parity state of the parity device is an idle state, a locked state, or an invalid state.

[0033] In some embodiments, the method further includes configuring the fabric manager to determine whether the metadata bit of the parity device is set to the locked state. When the parity state of the parity device is not set to the locked state, the fabric manager omits checking consistency between the data device and the parity device and takes no further action.

[0034] In some embodiments, the method further includes configuring the fabric manager to determine whether the metadata bit of the parity device is set to the locked state. When the parity state of the parity device is set to the locked state, performing a consistency check to determine whether the data of the parity device is equal to the XOR of all data of the data device, and when the data of the parity device is equal to the XOR of all data of the data device, the fabric manager clears the lock and rewrites the parity data.

[0035] In some embodiments, the method further includes configuring the fabric manager to determine whether the metadata bit of the parity device is set to the locked state. When the parity state of the parity device is set to the locked state, performing a consistency check to determine whether the data of the parity device is not equal to the XOR of all data of the data device, and when the data of the parity device is not equal to the XOR of all data of the data device, the fabric manager clears the lock and writes new parity data resulting from the XOR operation of all data.

[0036] In some embodiments, the method further includes: marking an entire row of the hot spare as invalid via a metadata tag of the hot spare device; coupling a fabric manager to the at least one fabric, wherein the fabric manager is configured to manage the operation of the at least one fabric; and configuring the fabric manager to decide to replace a failed device with the hot spare device.

[0037] In some embodiments, the method further includes coupling a fabric manager to the at least one fabric to manage its operation and providing at least one device configured as a self-diagnostic data device, the self-diagnostic data device sending a ready-to-be-replaced warning to the fabric manager when one or more errors detected exceed a predetermined threshold indicating an impending failure. In response to the warning, the fabric manager is configured to: preemptively add a hot spare device as a new data device; update the fault-tolerant engine, wherein each fault-tolerant engine maintains a record of the hot spare device location of the device group; instruct all tenants in the device group to perform a hot spare device data reconstruction to rebuild the data to the new data device before the impending failure occurs by each tenant reading the location of its corresponding shared memory device; update the fault-tolerant engine; and remove the ready-to-be-replaced device. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The illustrative embodiments may form various components and component arrangements. The illustrative embodiments are shown in the accompanying drawings, in which the same reference numerals may indicate corresponding or similar components in the various figures. The drawings are only used to illustrate the embodiments and should not be construed as limiting the present disclosure. In view of the following feasible description of the drawings, a person of ordinary skill in the relevant art should understand the novel aspects of the present disclosure.

[0039] Figure 1A A functional block diagram is illustrated in the form of a cluster topology having a redundant array of independent devices (RAID) architecture with a single switch topology in accordance with an embodiment of the present disclosure.

[0040] Figure 1B illustrate Figure 1A An exemplary detailed block diagram of a fault-tolerant engine depicted in FIG.

[0041] Figure 2 An alternative functional block diagram in the form of a cluster topology having a RAID architecture with a redundant switch topology is illustrated in accordance with an embodiment.

[0042] Figure 3A Example parity groups according to an embodiment are described.

[0043] Figure 3B An example of interleaving parity groups according to an embodiment is described.

[0044] Figure 4 is a swim lane diagram illustrating a read operation according to an embodiment.

[0045] Figure 5 is a swim lane diagram illustrating a write operation according to an embodiment.

[0046] Figure 6 A flow chart illustrating an example method of operating a write operation according to an embodiment.

[0047] Fig. 7A is a swim-lane diagram illustrating a conflict resolution operation according to an exemplary embodiment of the present disclosure.

[0048] Figure 7B is a table illustrating a fast consistency recovery algorithm according to an exemplary embodiment of the present disclosure.

[0049] Figure 8 A flow chart illustrating an example method of operating a read access operation according to an exemplary embodiment of the present disclosure.

[0050] Figures 9A to 9B A flow chart illustrating an example method of operating a write access operation according to an exemplary embodiment of the present disclosure.

[0051] Fig.10Illustrated is a table showing an example of an error handling algorithm for initiating a rebuild in response to various read and write operations according to the present disclosure.

[0052] Fig.11 is a swim-lane diagram showing a reconstruction of a read operation according to an exemplary embodiment of the present disclosure.

[0053] Fig.12 is a swim-lane diagram showing a reconstruction of a write operation according to an exemplary embodiment of the present disclosure.

[0054] Fig.13 A simplified version of a fabric fault-tolerant RAID design according to the present disclosure is described.

[0055] Fig.14 A detailed version of a fabric fault-tolerant RAID design according to the present disclosure is described.

[0056] Fig.15A is a swim-lane diagram showing a write flow with a write protection operation according to an exemplary embodiment of the present disclosure.

[0057] Fig. 15B An example result of a conflict check according to the present disclosure is described.

[0058] Fig. 15C A detailed block diagram illustrating an exemplary locking controller according to the present disclosure.

[0059] Fig.16 Table illustrating read and write responses according to the present disclosure.

[0060] Fig.17 is a table illustrating a Compute Express Link (CXL) proprietary extension according to an exemplary embodiment of the present disclosure.

[0061] Figures 18A to 18C A preemptive maintenance process according to an exemplary embodiment of the present disclosure is described.

[0062] Fig.19 exhibit Figures 18A to 18C An example flow chart of the preemptive maintenance process depicted in .

[0063] Fig. 20 An alternative embodiment of a fabric fault-tolerant RAID design according to the present disclosure is described.

[0064] Fig.21 A flow chart illustrating an exemplary method of practicing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0065] The present disclosure addresses the need to isolate and correct faults in fabric networks.Various embodiments may include a dynamically configurable fabric network.

[0066] RAID devices (which are understood in the art as memory devices that combine multiple physical drives into a single logical unit) are able to implement data recovery using a checksum. This checksum may be an exclusive OR (XOR) of the rows of data in the checksum group. In some embodiments, the term RAID refers to implementation within the context of a CXL device (i.e., a memory device that implements the CXL protocol). In some embodiments, the term refers to implementing CXL device fault tolerance using RAID logic. In other embodiments, the term RAID refers to configuring fault tolerance in a cluster using a RAID design. Systems, apparatus, and methods related to RAID architectures are described herein. In a basic design, a RAID architecture creates memory redundancy such that a single failure of a device or its associated components (such as its power delivery system) can be tolerated.

[0067] Although embodiments are not limited in this regard, one example may be different serial interconnect protocols for interconnecting or coupling devices. Available protocols that may be used in some embodiments include the Peripheral Component Interconnect Express (PCIe) protocol, such as in accordance with any existing version of the PCIe specification or any future updates, versions, or variants thereof. Other protocols may include the CXL protocol, such as in accordance with any existing version of the CXL specification (e.g., CXL Specification Version 1.0 / 1.1, CXL Specification Version 2.0, and CXL Specification Version 3.0), any future updates, versions, or variants thereof. Of course, other protocols are possible.

[0068] In some embodiments, the memory system may be a CXL-compatible memory system (e.g., the memory system may include a PCIe / CXL interface). CXL is a high-speed central processing unit (CPU)-to-device and CPU-to-memory interconnect designed to accelerate next-generation data center performance. CXL technology maintains memory coherency between the (host's) CPU memory space and memory on attached devices, which allows resource sharing to improve performance, reduce software stack complexity, and reduce overall system cost.

[0069] CXL is designed as an industry open interconnect standard interface for high-speed communications as accelerators are increasingly used to complement CPUs to support emerging applications such as artificial intelligence and machine learning. CXL technology is built on PCIe infrastructure, leveraging the PCIe physical and electrical interfaces to provide high-level protocols such as input / output (I / O) protocols, memory protocols (e.g., initially allowing the host and accelerator to share memory), and coherent interfaces in the field.

[0070] The present disclosure includes apparatus and methods related to CXL 3.0 technology. The CXL 3.0 specification expands upon previous versions CXL 1.0 / 1.1 and CXL 2.0. CXL 3.0 focuses on major improvements to the interconnect standard. CXL 3.0 greatly expands the logical capabilities of the standard, allowing for complex connection topologies and configurations. CXL 3.0 extends the topology from server scale to rack scale. CXL switches can now support any array of topologies. A rack or many racks of servers can now be networked with a leaf and spine topology. The spine-leaf architecture routes traffic through the top-level spine node to the lower-level (leaf) nodes that may contain hosts / devices. These changes greatly expand the potential size of a CXL network from a few servers to many racks of servers.

[0071] CXL 3.0 enables peer-to-peer communication, allowing devices to directly access each other's memory using enhanced consistency semantics and without having to go through a host to inform each other of their status. CXL 3.0 introduces memory sharing, which allows each host to see the latest data at that location without software management coordination. This allows system designers to build machine clusters to solve large problems through shared memory fabrics.

[0072] Multi-level switching is enabled by CXL 3.0, which allows for multiple layers of switches, where switches plug into other switches. This greatly increases the variety and complexity of supported network topologies. CXL 3.0 introduces fabric capabilities, which enable non-tree topologies such as rings, meshes, and other fabric settings. CXL fabrics can support up to 4096 nodes, which can communicate with each other using a new scalable addressing mechanism called port-based routing (PBR).

[0073] With CXL 3.0, GFAM is supported which allows different types of memory to be directly attached to and accessible by multiple processor nodes. The GFAM architecture extends the memory expansion board of CXL by factoring out memory from a given host and implements a shared large memory pool. GFAM devices may also be referred to as GFAM memory devices (GFDs) or memory devices. These terms are used interchangeably herein. GFAM devices are similar to traditional CXL 3-type devices, except that they can be accessed by multiple nodes (up to 4095) in a flexible manner using PBR.

[0074] Some embodiments are directed to a RAID architecture that can operate with all types of CXL 3.x GFAM devices without modification. Optionally, CXL GFAM devices can be configured to implement enhanced read request responses that will help improve the availability of clusters that implement the basic RAID architecture. In other words, the RAID architecture can make GFAM devices highly available. In various embodiments, these CXL design changes are optional.

[0075] Some embodiments relate to a RAID architecture including hardware and software features that may be implemented in a CXL switch or software device called a CXL fabric manager (FM) for implementing storage RAID in a cluster. As background, in CXL, a fabric manager is a resource management capability (i.e., a trusted element of a fabric) that is configured to manage access to a fabric and storage resources within a fabric. In embodiments, for example, a fabric manager may be implemented in hardware, firmware, or software, or a combination thereof.

[0076] Some embodiments relate to a fabric fault tolerant RAID design, where the basic RAID architecture can be extended to gracefully cope with CXL switch failures as well as device failures in a RAID fabric fault tolerant CXL device. Such a fabric fault tolerant CXL device may include a memory protocol unit (MPU) and an advanced lock management block. More specifically, only parity groups are locked. A parity group is N data cache lines (or check lines) and one parity cache line.

[0077] Although the present disclosure uses the term cache line, it should be recognized that the terms check line and cache line can be used interchangeably. The RAID architecture creates memory redundancy such that failure of a single CXL device or its associated components (such as its power delivery system) can be tolerated. In some embodiments, the RAID architecture tolerates only one device failure per parity group at a time, but this limitation can be overcome through online replacement mechanisms and cluster instant repair.

[0078] Some embodiments of the present disclosure are directed to mechanisms for supporting redundant arrays of independent CXL GFAM devices, which may provide a number of significant advantages. The mechanisms may include a fault tolerance engine within a CXL leaf switch. The fault tolerance engine may receive read / write requests from a host processor and perform a series of operations on a group of CXL GFAM devices to complete the host's request. These advantages include, but are not limited to, the ability to support N+1 GFAMs with N+1 redundancy, support single switch parity groups, and support single fabric parity groups (no MPU).

[0079] Advantages also include support for redundant fabric parity groups (with or without MPU), support for active-active redundant switch fabrics, support for identification and recovery of parity group checksums in all scenarios, support for memory atomic operations via the fault tolerance engine, and support for shadowed write operations. As a result, these advantages can maximize the throughput of memory regions under RAID support and minimize the latency of memory regions under RAID support.

[0080] In the following detailed description of the present disclosure, reference is made to the accompanying drawings, which form a part of the present disclosure and in which are shown by way of illustration how one or more embodiments of the present disclosure may be practiced. These embodiments are described in sufficient detail to enable one of ordinary skill in the art to practice the embodiments of the present disclosure, and it is to be understood that other embodiments may be utilized and process, electrical and structural changes may be made without departing from the scope of the present disclosure.

[0081] It should also be understood that the terminology used herein is only used to describe specific embodiments and is not intended to be limiting. As used herein, the singular forms "a", "an", and "the" may include both singular and plural referents unless the context clearly indicates otherwise. In addition, "several", "at least one", and "one or more" (e.g., several memory banks) may refer to one or more memory banks, while "plurality" is intended to refer to more than one such thing.

[0082] In addition, the words "can" and "may" are used throughout this application in a permissive sense (i.e., having the potential to, being able to), rather than a mandatory sense (i.e., having to). The terms "including" and its derivatives mean "including (but not limited to)". As the context requires, the terms "coupled" and "coupled" mean physically directly or indirectly connected or used to access and move (transfer) commands and / or data. As the context requires, the terms "data" and "data value" are used interchangeably herein and may have the same meaning.

[0083] Embodiments described herein relate to a fabric network that supports a spine-and-leaf topology consisting of any combination of leaf switches, including a single switch or multiple spine-and-leaf switches. Some embodiments described herein provide a fabric network that can consist of a single switch topology, a multiple spine-and-leaf topology including redundant connections, or a multiple spine-and-leaf topology including redundant connections with failover paths. It should be understood that these are merely examples and that the embodiments described herein can be implemented in different types of networks to manage any type of content.

[0084] Figure 1A is an example of a basic RAID system 100 with a cluster topology. Generally speaking, a computer cluster (which is a group of linked computers that can work together as a single system) can be linked by a high-speed network that allows sharing of computer resources and memory. Data transfer to and from the computers can be performed through a high-speed PCIe switch bus.

[0085] Figure 1A 1 is a block diagram illustrating a basic RAID system 100 according to an embodiment. More specifically, Figure 1A RAID system 100 may be managed by any type of computing platform, from a relatively small device, such as a laptop or another personal computer, to larger devices, such as servers and data center-based implementations.

[0086] exist Figure 1A , RAID system 100 includes one or more host devices 102a, 102b and one or more memory devices 106a, 106b, 106c, such as a fault-tolerant memory device. One or more memory devices 106a, 106b, and 106c may also be collectively referred to herein as a memory device group or a portion of a memory device group. In an alternative embodiment, there may be multiple memory device groups in system 100. System 100 also includes a fabric 108. Host devices 102a and 102b and memory devices 106a, 106b, and 106c may be coupled to fabric 108. Fabric 108 is capable of carrying bidirectional traffic from host devices 102a, 102b to one or more of memory devices 106a, 106b, 106c and vice versa.

[0087] The host devices 102a, 102b may be personal laptop computers, desktop computers, digital cameras, mobile phones, memory card readers, or Internet of Things (IoT) enabled devices, as well as various other types of systems. The host devices 102a, 102b may include a number of processing resources (e.g., one or more processors, microprocessors, or some other type of control circuitry) capable of accessing a memory system. The host devices 102a, 102b may include a system motherboard and / or backplane and may include a number of processing resources (e.g., one or more processors, microprocessors, or some other type of control circuitry). In an embodiment, the host components may include the CXL server, all local memory, local storage devices, and I / O cards. The host components may also include the power delivery system within the server. The host devices may be physical, virtual (e.g., implemented in software), or a combination of both.

[0088] Host devices 102a, 102b and fabric 108 are connected via connections 110a and 110b. Multiple connection and transaction layers may be provided in the interface to provide communications via I / O connections 110a and 110b.

[0089] Coupling host devices 102a, 102b to fabric 108 may include, for example, a physical interface that employs a suitable protocol (e.g., a data bus, an address bus, and a command bus or a combined data / address / command bus). This protocol may be custom or proprietary or may employ a standardized protocol such as PCIe, CXL, Gen-Z, Cache Coherent Interconnect for Accelerators (CCIX), or the like. With respect to the CXL transaction layer, various components may be included to enable transaction layer processing of PCIe / CXL.io communications and CXL.cache and CXL.memory transactions.

[0090] In one embodiment, a PCIe 6.0 interface used by the CXL 3.0 specification may be coupled to connections 110a and 110b. In some embodiments, fabric 108 may receive access requests involving at least one of memory devices 106a, 106b, and 106c via the PCIe 6.0 interface according to the CXL protocol. Fabric 108 may receive data from hosts 102a and 102b via connections 110a and 110b.

[0091] refer to Figure 1A , fabric 108 includes a single switch topology including a single spine switch 112 and a single leaf switch 114. Spine switch 112 is connected to leaf switch 114 via connections 118a and 118b. In an embodiment, two levels of switching are provided within the equipment rack and optically connected to the rack via rack switches 132. At the top level, spine switch 112 routes data traffic to lower level leaf switches 114. Leaf switch 114 may communicate with one or more memory devices 106a, 106b, and 106c via multiple connections 116a, 116b, and 116c. Although Figure 1A Two host devices 102a, 102b and three memory devices 106a, 106b, and 106c are depicted, but it is understood that more or fewer host devices 102 and memory devices 106 may be used.

[0092] exist Figure 1A In the exemplary embodiment shown in FIG. 1 , the leaf switch 114 is configured as a leaf switch with a RAID, which includes several components including a routing processor or a memory request routing block (e.g., a RAID routing block) 120a, 120b. The RAID routing blocks 120a, 120b analyze the data or data packets to find the best upstream communication path to ensure good data transmission. The leaf switch 114 also includes a switch (e.g., a switch core) 122, a RAID controller 124a, 124b, 124c, a fault-tolerant engine 126a, 126b, 126c, an upstream port (USP) 128a, 128b, and a downstream port (DSP) 130a, 130b, 130c. The RAID routing blocks 120a and 120b are associated with each USP 128a, 128b, respectively. The fault-tolerant engines 126a, 126b, and 126c correspond to the DSPs 130a, 130b, and 130c, respectively.

[0093] For example and as described herein, RAID routing block 120a and RAID routing block 120b may be collectively referred to as RAID routing block 120. RAID controller 124a, RAID controller 124b, and RAID controller 124c may be collectively referred to as RAID controller 124; fault tolerance engine 126a, fault tolerance engine 126b, and fault tolerance engine 126c may be collectively referred to as fault tolerance engine 126; and USP 128a and USP 128b may be collectively referred to as USP 128. This approach applies to other components and figures discussed throughout.

[0094] As background and those skilled in the art will appreciate, a fault-tolerant engine is a component within a RAID system that distributes data among different GFAM devices and optimizes RAID logic to achieve optimal performance. In an embodiment, the fault-tolerant engine may be implemented in software, firmware, hardware, and / or a combination thereof.

[0095] As used herein, a USP is a device that provides a communication path for data flow between a controller (e.g., a fault-tolerant engine) and an upper-level device (e.g., an external computer) within a RAID system. A DSP is a device that provides a communication path for data flow between a fault-tolerant engine and a lower-level device (e.g., a memory device) within a RAID system.

[0096] Figure 1B is a detailed description of a single fault-tolerant engine 126, which may represent Figure 1A One or more of the fault-tolerant engines 126a, 126c depicted in FIG.

[0097] like Figure 1A , spine switches 112 are connected to leaf switches 114 to route communications between one or more hosts 102a, 102b, memory devices 106a, 106b, 106c, and other network elements in fabric 108. In fabric 108, spine switches 112 can communicate with any number of hosts 102a, 102b, and leaf switches 114 can communicate with any number of memory devices 106a, 106b, 106c.

[0098] Various connections 118a, 118b may be used to connect the spine switch 112 with the leaf switch 114. One or more of the connections 118a, 118b may be operable as an uplink for transmitting data from the leaf switch 114 to the spine switch 112, a downlink for transmitting data from the spine switch 112 to the leaf switch 114, or both an uplink and a downlink. As used herein, the term "link" may refer to one or more uplinks, downlinks, link groups, or any other communication paths that may exist between a spine switch and a leaf switch or any network element.

[0099] The spine switches 112 and leaf switches 114 may include any number of uplink and downlink ports operable to support connections to various transmission rates. Although the description herein refers to spine switches and leaf switches, various nodes including switches, routers, or other network devices including network switching or routing elements configured to perform forwarding functions are contemplated.

[0100] exist Figure 1A In the example embodiment of the RAID switch shown in FIG. 1 , a RAID routing block 120 a, 120 b is associated with each USP 128 a, 128 b. A fault-tolerant engine block 126 a, 126 b, 126 c is associated with each DSP 130 a, 130 b, 130 c. The RAID routing blocks 120 a, 120 b are located in the switch between the USPs 128 a, 128 b and the switch core 122. The switch core 122 changes the routing of data packets so that all requests are sent to the correct RAID controller 124 a, 124 b, 124 c.

[0101] As described above, the RAID routing blocks 120a, 120b analyze the data packets to find the best path to ensure good data transfer. In an embodiment, the RAID routing blocks 120a, 120b use the destination port in the request packet, the address in the packet, and the interleaved configuration status register (CSR) that defines the parity group to determine the destination fault-tolerant engine for the request. This destination is called the primary (or home) fault-tolerant engine.

[0102] RAID routing block 120 and fault tolerance engine 126 are configured by FM 134. FM 134 is configured as a management service to support network management of fabric 108. The network of fabric 108 may include network devices (such as switches, routers, servers, and the like) and their connections. FM 134 allows complex network fabrics to be reconfigured, updated, or analyzed for performance.

[0103] FM 134 is involved in fault, configuration, accounting, performance, and security management of network fabric 108. In some examples, FM 134 may allow maintenance and analysis of individual switches. In other examples, FM 134 may allow creation, installation, and / or maintenance of fabric-wide switch configurations. Components of FM 134 may be implemented in software, hardware, or a combination thereof. In some embodiments, FM 134 may be a CXL FM. Figure 1A In the single switch embodiment shown in , FM may be optional.

[0104] In an embodiment, the system 100 employs a plurality of distributed RAID controllers 124a, 124b, 124c to perform a plurality of read and write operations. In some embodiments, each RAID controller 124a, 124b, 124c may include a corresponding fault-tolerant engine 126a, 126b, 126c. As the context requires, the RAID controller and the fault-tolerant engine may be referred to interchangeably herein. The fault-tolerant engine 126a, 126b, 126c may be configured to provide data protection that tolerates predetermined device failures of data encoded across numerous devices. For example, the fault-tolerant engine 126a, 126b, and 126c may be configured to tolerate only one device failure per parity group 136 at a time. In some embodiments, a memory device group may include one or more parity groups, hereinafter referred to as a parity group 136.

[0105] When servicing read and write operations, the RAID controllers 124a, 124b, 124c may perform RAID rebuild operations. In various embodiments, there are two different rebuilds: a read rebuild and a write rebuild (as described below).

[0106] RAID controller 124 can implement logic embedded in fault tolerance engine 126. Fault tolerance engine 126 can be pre-programmed with RAID code error detection and correction in a wide range of RAID code schemes. Many RAID schemes employ an error protection scheme commonly referred to as "parity", which is a method widely used in information technology to provide tolerance in a given set of data. For example, RAID is basically a striped array with fault tolerance (in the form of distributed "parity").

[0107] In a RAID data structure, data is striped across hard drives, with each stripe having a dedicated parity block. A percentage of each member of the array is reserved for parity calculations; whenever usable data is written to a RAID array, the RAID array will generate unique metadata called "parity bits" (data that can only be read by the RAID controller 124). The parity bits are distributed among all members of the array and can be used to reconstruct usable data when any one of the devices fails.

[0108] The parity block is calculated by running an XOR comparison on each data block in the stripe. Parity is responsible for data fault tolerance. In operation, according to an embodiment of the present disclosure, if a device fails, a new device can be put in place and the fault tolerance engine can automatically rebuild the data using the parity data.

[0109] In various embodiments, the memory devices 106a, 106b, and 106c may be CXL devices (i.e., memory devices that implement the CXL protocol), GFAMs, memory devices, or combinations thereof. The memory devices 106a, 106b, and 106c may be connected to the respective DSPs 130a, 130b, 130c of the leaf switch 114 via connections 116a, 116b, 116c, respectively. In some embodiments, the RAID controller 124 may be configured as a CXL interface controller that enables communication with one or more memory devices 106a, 106b, 106c (e.g., CXL memory devices) via the CXL connections 116a, 116b, 116c to which the memory devices 106a, 106b, 106c are connected. The memory devices 106a, 106b, 106c are capable of communicating via the CXL protocol.

[0110] In an embodiment, the RAID architecture extends CXL technology to support better availability. The technology disclosed herein proposes a RAID architecture that supports two main methods: basic RAID design ( Figures 1A to 2 ) and fault-tolerant RAID design ( Figures 13 to 14 ). Thus, the techniques disclosed herein define two new types of CXL devices: memory devices 106, 206 and fabric fault-tolerant memory devices 1306, 1406, as described herein.

[0111] Additional types of data devices that can be supported by the basic RAID structure and the fabric fault-tolerant RAID structure can include GFAM devices. GFAM devices are highly scalable memory resources that can be accessed by all hosts and all peer devices within the CXL fabric. GFAM devices can be dedicated to a single host or can be shared by multiple hosts. The GFAM architecture extends the memory expansion board of CXL by factoring memory out of a given host and implementing a shared large memory pool. In this regard, the GFAM device itself is functionally a shared memory pool so that hosts and devices can communicate as needed. GFAM devices are similar to traditional CXL type 3 devices, except that GFAM devices can be accessed by multiple nodes (up to 4095) using port-based routing in a flexible manner. Root ports on the fabric 108 can be part of the same or different domains.

[0112] By implementing CXL 3.0, the present disclosure supports GFAM that allows different types of memory to be directly attached to and accessible by multiple processor nodes. For example, in the present disclosure, a node may be a CPU, a host, a CXL accelerator with or without memory, a PCIe device, or a GFAM device. A GFAM device may contain both volatile and non-volatile memory, such as dynamic random access memory (DRAM) and flash memory. Data devices that support sharing, such as GFAM devices, may support hardware-managed cache coherence across root ports in multiple domains.

[0113] refer to Figure 1A In various embodiments, the memory devices 106a, 106b, 106c may be combined into clusters. For example, some memory devices 106a, 106b, 106c (eg, CXL memory devices in a cluster) may be organized into parity groups, such as parity group 136.

[0114] like Figure 3A , for a given parity group 136, there may be one parity device 344, some number of data devices 346a, 346b, and one optional "hot spare" 348. As used herein, a hot spare is a CXL memory device that is configured to be dynamically swapped in by the FM to replace a failed device. Parity responsibility may be distributed among all non-spare devices in the group, but for convenience, in some embodiments, the parity device 344 may be designed as the device that assumes the parity role of the first cache line in the parity group.

[0115] That is, a CXL device in a cluster may or may not be part of a parity group 136. If a device is part of a parity group 136, then all bytes associated with the device are assigned to the parity group 136. Because parity responsibilities are interleaved, some bytes of a device may be used to store server data and some bytes of the device may be used to store parity.

[0116] exist Figure 3A In the example, the data is XORed with the poison bit indication to form parity. Once the parity is created, the 64 bytes of cache line data and the poison bit indication can be recovered by XORing the parity and all other data devices 346a, 346b.

[0117] like Figure 3B, RAID data interleaving can be performed on a device granularity. All bytes on a device can be configured so that they are assigned to one parity group with a fixed "N" device or the device is not a RAID at all. To further simplify the design, in some embodiments, all data devices 346a, 346b and parity device 344 of parity group 136 can be connected to the same last stage switch.

[0118] Figure 3B An example of simple interleaving of RAID is depicted in the case where N=4. Figure 3B In the RAID data structure, data is striped or distributed across all or most memory controllers of the device, with each stripe having a dedicated parity block. In this example, interleaving can be performed for 256B or larger granulators so that the effectiveness of the memory-side cache is not destroyed.

[0119] In use, the RAID routing block 120 in the switch located between the USP 128 and the switch core 122 changes the routing of data packets so that all requests are sent to the correct fault-tolerant engine. The RAID routing block 120 uses the destination port in the CXL request packet, the address in the CXL packet, and the interleaved CSRs of the devices defining the parity group 136 to determine the specific destination fault-tolerant engine 126a for the request. This specific destination fault-tolerant engine is called the primary fault-tolerant engine.

[0120] In some embodiments, all devices in a parity group 136 may be directly connected to the same CXL leaf switch. In a preferred embodiment, the leaf switch 114 may be a RAID-capable switch. However, the spine switch 112 and the rack-to-rack switch 132 may not necessarily be a RAID-capable switch.

[0121] The configuration of the parity group 136 may include details such as the number of data devices 106a, 106b, 106c in the group and whether the group has a hot spare 348. The memory devices 106a, 106b, 106c may be configured by the FM 134 via a set of commands defined in a FM application programming interface (FM API). The memory devices 106a, 106b, 106c may be statically or dynamically configured via the FM 134. The FM 134 may configure the RAID routing blocks 120a, 120b and the fault tolerance engines 126a, 126b, 126c before the parity group 136 may be used by any host 102a, 102b. To change the RAID configuration, the FM 134 stops all traffic to the memory devices 106a, 106b, 106c.

[0122] FM 134 provides support for the management of fabric 108. FM 134 serves as a trusted element of fabric 108. FM 134 is responsible for the initialization and setup of fabric 108 and the assignment of devices to different groups 136. FM 134 can establish a protection scheme and initialize aspects of the switch so that hosts 102a, 102b can gain access through the switch. FM 134 establishes a protection scheme so that each host 102a, 102b can be granted access to only a subset of all memory devices 106a, 106b, 106c. The protection scheme helps mitigate additional software attacks. FM 134 can be implemented as software.

[0123] In an embodiment, FM 134 can add or remove memory devices 106a, 106b, 106c from the system. In an embodiment, the memory devices 106a, 106b and 106c that can be added or removed can be hot spare devices 148, 348 and / or new data devices. The example system and method disclosed herein utilize values ​​representing the state of the memory devices 106a, 106b, 106c measured by FM 134. This method promotes active monitoring and / or analysis of memory devices 106a, 106b, 106c to automatically trigger replacement or maintenance of data devices. In an embodiment, FM 134 can implement preventive / preemptive maintenance to replace "ready to be replaced" components. In an embodiment, if a "hot swappable" component can be replaced without cutting the cluster, then FM 134 can swap out the component. The preventive / preemptive maintenance process and the hot swappable process are described below.

[0124] As discussed above, Figure 1A The exemplary embodiments of relate to a single switch topology. Figure 1A The disadvantage of the single switch topology in is the loss of connectivity to many storage devices after a switch failure. The solution is to deploy a redundant switch fabric and use multi-port CXL devices such as Figure 2 By deploying a redundant switch architecture and using multi-port CXL devices, if one switch fails, access to all CXL devices is supported. The architecture of the basic RAID system can support Figure 1A Single switch topology and Figure 2 Redundant switch topology for both.

[0125] Figure 2 One advantage of the redundant topology in is that if a switch fails, the cluster can be restarted and reconnected to all units without human intervention. Figure 2In the redundant switch fabric including at least two switch fabrics shown in FIG, when a switch fails, the servers in the cluster can be restarted by switching to a single switch-plane configuration mode. In the single switch-plane configuration, the failed switch can be disabled and the host can still access the data (and execute the fast consistency algorithm, as described below).

[0126] Thus, the process of reconfiguring the system to single switch mode, restarting all hosts and switches while saving memory power, performing a fast consistency recovery, and starting work on the system can all be done without human intervention. Thus, if a switch failure occurs when human intervention is not available (e.g., during the middle of the night), the cluster can restart itself.

[0127] and Figure 1A Compared with the single switch topology, Figure 2 is an alternative example of a cluster topology having a RAID system 200 configured with a redundant configuration. The cluster topology of the RAID system 200 is similar to Figure 1A For the sake of brevity, only the cluster topology of the RAID system 100 will be described below. Figure 1A The RAID system 100 differs from aspects of the RAID system 200. Figure 1A To the extent described in Figure 2 Similar components in the FIG. 100 are identified by similar reference numerals (e.g., fault-tolerant engines 112 / 212, 126 / 226). These similar components will not be Figure 2 This approach applies to many other components and figures discussed in this article.

[0128] refer to Figure 2 , shows a block diagram of a RAID system 200 including a first fabric switch 208a, a second fabric switch 208b, and a leaf switch connection 240 provided between the first fabric switch 208a and the second fabric switch 208b. Redundant connections 210a, 210b, 210c, 210d are provided from each host 202a, 202b to each spine switch 212a, 212b. Figure 2 , fabric switches 208a, 208b provide a primary access path and a secondary access path that is redundant to the primary access path. The RAID architecture of system 200 can create redundancy in memory so that failure of a single one of the memory devices 206a, 206b, or 206c or its associated components (such as its power delivery system) can be tolerated. In an embodiment, the RAID architecture can be configured to tolerate only one device failure per parity group 236 at a time, but this limitation can be overcome through online replacement mechanisms and cluster real-time repair.

[0129] RAID system 200 includes a communication topology by which storage devices 206a, 206b, 206c are coupled via a plurality of fabric switches 208a, 208b. Figure 2 The example in 208 depicts two fabric switches 208a, 208b, but it should be understood that embodiments of the present disclosure may implement multi-stage switching enabled by CXL 3.0, which allows for multi-layer switches where switches are plugged into other switches. This greatly increases the variety and complexity of supported network topologies.

[0130] For example, the switch fabric of the system 200 may implement a mesh connection to connect the memory devices 206a, 206b, 206c as endpoints with the switching mechanism (fabric switches 208a, 208b) serving as the intermediate nodes of the mesh connection. This provides redundancy so that if, for example, the connection between the memory devices 206a fails in one fabric switch 208a, the memory devices 206a may remain connected via another fabric switch 208b. In addition, if a component of the switch fabric fails, a communication path may be established that excludes the failed component and includes functionally redundant components.

[0131] As described above, the memory devices 206a, 206b, 206c may be GFAM devices. The memory devices 206a, 206b, 206c are attached to the fabric switches 208a, 208b and they are accessible, and the data within the memory devices 206a, 206b, 206c may be shared across the hosts 202a, 202b. The memory devices 206a, 206b, 206c are connected to the respective DSPs 230a, 230b, 230c, 230d, 230e, 230f of the leaf switches 214a, 214b via connections 216a, 216b, 216c, 216d, 216e, 216f. The DSP 230 from the first fabric switch 208a and the DSP 230 from the second fabric switch 208b are connected to each memory device 206. For example, DSP 230a from the first fabric switch 208a and DSP 230d from the second fabric switch 208b are connected to the memory device 206a.

[0132] In some embodiments, host device 202a and host device 202b may be collectively referred to as host device 202; memory device 206a, memory device 206b, and memory device 206c may be collectively referred to as memory device 206; DSP 230a, DSP 230b, DSP 230c, DSP 230d, DSP 230e, and DSP 230f may be collectively referred to as DSP 230.

[0133] use Figures 1A to 2The systems of the RAID architecture depicted in can potentially have large memory pools. For example, these systems can have thousands of GFAM devices and numerous hosts, which can construct very large devices of many rack sizes. In such systems, there are many components. Therefore, there is the possibility that a device fails before a long-running application is completed. Rather than restarting the application to eventually complete the application, the system should be configured to tolerate some of these failures and continue to operate the application without having to implement a master restart mode.

[0134] Embodiments of the present disclosure relate to applying RAID concepts to GFAM devices to provide a very large solution to device failures. In order to handle failures, RAID concepts can also be applied to GFAM devices to tolerate failures of GFAM devices so that the administrator of the system can identify that there is a problem, potentially take the device offline, replace, repair or upgrade it. At the same time, the application does not detect the failure and can continue its operation.

[0135] exist Figures 1A to 2 During operation, the RAID controller 124, 224 or the fault tolerance engine 126, 226 may be configured to perform read operations and write operations, such as Figures 4 to 5 A swimlane diagram shows the relationships between the boxes in a process and the components responsible for those boxes.

[0136] Full-featured reading process

[0137] Figure 4 Is to show and Figures 1A to 2 A swim lane diagram of a process 400 for performing a RAID read operation consistent with an embodiment of the present invention. Although the following examples are applicable to Figures 1A to 2 Both systems 100 and 200 are used, but for simplicity, the example uses Figure 2 to 3 reference numbers. Figure 4 The read operation method is described. In a non-faulty situation, the RAID read operation is very simple and the process 400 can be performed by the host 202a, 202b, the fault tolerance engine 226a-226f, the corresponding RAID controller 224a-224f and the memory device 206a, 206b.

[0138] In some embodiments, RAID controller 224a, RAID controller 224b, RAID controller 224c, RAID controller 224d, RAID controller 224e and RAID controller 224f may be collectively referred to as RAID controller 224; fault-tolerant engine 226a, fault-tolerant engine 226b, fault-tolerant engine 226c, fault-tolerant engine 226d, fault-tolerant engine 226e and fault-tolerant engine 226f may be collectively referred to as fault-tolerant engine 226.

[0139] There are two message categories: read request (Req) and data response (DRS) messages. The response message may be generated based on the S2M DRS message of the CXL protocol. In the exemplary process 400 of a non-failure case of a fully functional read flow, a memory read request is serviced by the primary RAID controller 224 and the primary memory device 206. The host 202 sends a read data request to one of the fault-tolerant engines 226a to 226f of the corresponding one of the RAID controllers 224a to 224f using MemRd (operation 402), which sends the read data request to the primary memory device 206 using MemRd (operation 404). The primary memory device 206 responds to one of the RAID controllers 224 with the requested data using MemData response (operation 406), and the RAID controller 224 sends the requested data to the host 202 using MemData response (operation 408).

[0140] Full-featured writing process

[0141] Figure 5 Is to show and Figures 1A to 2 A swim lane diagram of an exemplary process 500 for performing a RAID write operation in accordance with the systems 100 and 200 of FIG. Figures 1A to 2 Both systems 100 and 200 are used, but for simplicity, the example uses Figure 2 to 3 reference numbers. Figure 5 In the full-function write process, RAID write operations are more Figure 4 The RAID read operations described in the above are more complex. In some cases, this increased complexity may affect the overall latency and throughput. Process 500 may be performed by host 202, fault tolerance engine 226 of RAID controller 224, main memory device 206, and parity device 344 ( Figure 3A )implement.

[0142] Here, there are four message categories: Write Request (RwD), No Data Response (NDR), Req and DRS messages. In an exemplary process 500 of a full-function write flow, the host 202 initiates a write request to the primary fault-tolerant engine 226 of one of the RAID controllers 224 using MemWr (operation 502).

[0143] exist Figure 5In operation 504, according to some embodiments, when the primary fault-tolerant engine 226 requests to read data from the parity device 344 and uses MemRdLock to lock the parity cache line (a 64-byte line in the parity device 344), the parity device 344 updates the state of the metadata bit of the parity device 344 to locked (506). As described above, only the parity group is locked. The parity group is N data cache lines and 1 parity cache line. In operation 522, the old data and parity data are read and the lock is set. In response (operation 512), the parity device 344 transmits the request data (old data and parity data) to the primary fault-tolerant engine 226 along with the lock state of the parity group.

[0144] By locking the parity cache line, the fault-tolerant engine 226 performs an atomic operation, wherein the fault-tolerant engine 226 locks the cache line, performs the operation on the data, and then unlocks the data. Generally speaking, atomic means "one at a time". In computer programming, if an operation is guaranteed to be isolated from other operations that may occur simultaneously, it is considered atomic. For example, it means that only one read or write operation can be performed at a time. If a program reads an attribute in an atomic manner, this means that the attribute cannot be changed during this read operation. When data is used by atomic operations, it can be implemented in a sequential processing environment or a locking mechanism must be used to avoid data errors.

[0145] exist Figure 5 In the embodiment of the present invention, the fault-tolerant engine 226 can perform a read-modify-write on the cache lines of the parity device 344, and due to the lock, the system prevents all other execution threads on the same host or different hosts from accessing and modifying the parity group during the time when one of the processors performs a read-modify-write. According to the present disclosure, this is an example of an atomic aspect, where the system modifies the memory in an atomic manner to obtain consistent results. The system performs reads, modifications, and writes without performing other reads or writes in between. During an atomic memory operation, a processor can read a location, modify it, and write it back in a single operation. A read-modify-write is an atomic memory operation that both reads a memory location and writes a new value to it as part of the same operation.

[0146] refer to Figure 5 In operation 508, the primary fault-tolerant engine 226 uses MemRdRaw to read the target primary memory device 206. The target device is the location where the data will be written.

[0147] In operation 510 , the target main memory device 206 responds with the requested data using a MemData response to the fault tolerance engine 226 .

[0148] In operation 512, the old data and parity data are sent from the parity device 344 to the primary fault-tolerant engine 226 using MemData.

[0149] In operation 514, when the fault-tolerant engine 226 requests to write data to the target main memory device row 206 using MemWr, the write request writes the data to the target main memory device row 206 (operation 524), and the target main memory device row 206 transmits a response (Cmp) in operation 516 to notify the fault-tolerant engine 226 that the write is completed in operation 520. Generally speaking, Cmp indicates that a write, a read, or an invalidation is completed.

[0150] The primary fault-tolerant engine 226 may then generate new parity checksum data based on the existing parity data, the existing data, and the new data. In one example, generating the new parity data may involve a read-modify-write operation and an XOR operation. The read-modify-write operation may use the existing data and the existing parity checksum data accessed from the parity data 344 before they are overwritten and the XOR operation may use the existing parity checksum data, the existing data, and the new data to generate the new parity checksum data.

[0151] In operation 518, the fault tolerance engine 226 requests to write the new parity checksum data and unlocks the cache line to the parity device 344 using MemWrUnlock. The write request writes the data to the parity device 344 and releases the cache line lock (operation 528), which unlocks the entire parity group so that another operation can be performed and the metadata is updated to the idle state (operation 526).

[0152] In operation 520, the parity device cache line 344 transmits a response (Cmp) to inform the fault tolerance engine 226 that the write and unlock are complete.

[0153] Next, in operation 530, the fault tolerance engine 226 sends a response (Cmp) to notify the host 202 that the write is complete.

[0154] exist Figure 6 In the flowchart, process 600 is summarized Figure 5 5. In general, during a write operation, an atomic operation is performed in which both the master data and the parity data are atomically updated to inhibit other requests from accessing the cache line during the transaction. Once both the master data and the parity data are atomically updated, this results in the memory being in a consistent state again and the lock on the cache line can be released to allow other read or write operations to be processed.

[0155] like Figure 6As illustrated in the flowchart of , a method for implementing a parity group write sequence includes the following blocks: in block 610, the fault-tolerant engine reads and locks a parity device cache line; in block 620, the fault-tolerant engine reads a target device row; in block 630, the fault-tolerant engine writes to the target device row; and in block 640, the fault-tolerant engine calculates new parity data by performing an XOR operation on the original parity data, the original data, and the new data, then writes the new parity data to the parity device cache line, and unlocks the parity device cache line.

[0156] If the parity device cache line is already locked, any new subsequent write or read request received must wait until unlocked or a timeout occurs. In this case, one or more fault-tolerant engines attempt to access the parity device cache line simultaneously. Subsequent requesting fault-tolerant engines must wait until the previous operation is completed and the lock is cleared or retry the access attempt.

[0157] Return to reference Figure 3A In order to implement the locked feature, according to the present disclosure, a 2-bit meta-state of the device (e.g., GFAM device meta-state) is added per parity group 136, 236, 1436. The 2-bit meta-state is: (1) cache line is locked, which is required for cache lines; and (2) cache line invalid, which is required for all cache lines. These 2 bits are included in the parity device 344. The locked bit is an extra bit of state. Therefore, it is called a meta-state because it is not included in the 64 bytes of data, but is in addition to those data. In some embodiments, the system can be enhanced to support a 256-byte approach.

[0158] In some embodiments, the RAID lock meta-state can only be used on the parity device cache line, set when the parity device cache line is read, and cleared when the parity device cache line is written. In fact, all devices of the parity group can be configured to include a 2-bit meta-state. However, the parity device can be configured to be the only device using it. If the lock times out or the GFAM device loses power when the lock state is set, the lock state is cleared and the RAID invalid metadata is set.

[0159] The timed out parity device cache line maintains a state indicating that the parity device cache line is invalid. In some embodiments, if the memory device 106, 206, 1306, 1406 is a persistent medium (e.g., a NAND device) that maintains its stored value or data bit when in an unpowered state, then the invalid state must also be persistent. The lock bit need not be persistent. If the timeout occurs while the cache line is locked ( Figure 5 , 7, 12), then the lock bit is cleared and the invalid bit is set.

[0160] Thus, the existing value is retained until the next operation and in response, an invalid message is returned to the RAID controller 124, 224, 1424. The invalid message informs the RAID controller 124, 224, 1424 that the previous operation was not successfully completed and the assumed value is "invalid". This indicates that the parity group 136, 236, 1436 needs to be regenerated or rebuilt. Therefore, reading an invalid parity device cache line returns an invalid status to the RAID controller 124, 224, 1424 indicating that regeneration is required.

[0161] If the lock times out or the GFAM device loses power while the lock state is set, then the RAID invalid metastate is set. The RAID invalid metastate is cleared when the parity device cache line is written.

[0162] If the request is blocked because the parity device cache line is locked, then the subsequent request fault-tolerant engine 126, 226, 1426 must wait until the previous operation is completed and the lock is cleared or the access attempt is retried. The blocked request can be queued at the GFAM device or retried via a lock response sent back to the RAID controller 124, 224, 1424. The RAID controller 124, 224, 1424 can store the request in the pending operation queue until the lock is completed. This means that the host 102, 202, 1402 sends the request once to the RAID controller 124, 224, 1424 and the RAID controller 124, 224, 1424 waits until it can perform the operation.

[0163] An alternative to waiting until the lock is complete is to retry the operation. A response indicating the lock status is sent back to the RAID controller 124, 224, 1424 and the request is retried.

[0164] Since write operations can be more complex than read operations, one assumption according to the present teachings is that the RAID controller 124, 224, 1424 is located at the leaf switch. However, it is within the scope of the present teachings that the RAID controller 124, 224, 1424 can be included at other locations and / or other components of the switch. For example, the fault tolerance engine can be located as a single component between the USP 128, 228a / 228b / 228c / 228d, 1428a / 1428b / 1428c / 1428d and the switch core 122, 222a / 222b, 1422a / 1422b or the fault tolerance engine can be included in one or more of the USP 128, 228a / 228b / 228c / 228d, 1428a / 1428b / 1428c / 1428d.

[0165] Conflict Resolution

[0166] Fig. 7A An example process 700 of conflict resolution for an application (e.g., RAID) implemented according to an embodiment of the present teaching is shown. In general, with respect to conventional RAID applications, one of the most difficult technical issues is maintaining a consistent state of data and parity in a parity group, because conflicts may occur when multiple requests want to access and update the same cache line simultaneously. In a distributed configuration containing multiple fault-tolerant engines, loss of access to data may be caused by other fault-tolerant engines waiting for a long time for the fault-tolerant engine to release its lock. Data corruption may be caused by inconsistent data and parity values ​​caused by interrupting the locking RAID sequence when a fault-tolerant engine occurs. If inconsistent parity relies on any of the reconstructed data cache lines, corruption occurs.

[0167] To maintain consistency, the fault-tolerant engines work together to ensure that only one parity group is updated at a time. One challenge is to perform this conflict resolution for the extremely high request rate of the memory system. In one example, in an extreme case, each fault-tolerant engine receives a new write request every clock. For performance reasons, in this case, the write processing rate preferably needs to be set to close to one cache line write per two clocks.

[0168] Fig. 7A Shows what happens when multiple write requests want to access and update the same cache line. Figure 5 An example of conflict resolution in a fully functional write application. Fig. 7A A write operation is shown with conflict checking prevention and synchronization points enforced by the fault tolerance engine. Thus, in process 700, multiple request conflicts are resolved using conflict resolution performed at the primary fault tolerance engine (226a) and the parity fault tolerance engine (226b). Essentially, for each write operation, a read-modify-write operation is performed on the parity data.

[0169] In operation 702, all write requests are routed using MemWr to the primary fault-tolerant engine 226a, which acts as a master for coordinating writes (and reads). The primary fault-tolerant engine 226a performs conflict resolution with other requests of the same home agent.

[0170] If there is a conflict at the master fault tolerant engine 226a, then in operation 732, the parity group conflict prevention structure is activated and subsequent conflicting requests are blocked and held in the conflict list queue and processed after the previously received request is completed.

[0171] In operation 704, to process the first received request, the master fault tolerant engine 226a requests to read data from the master fault tolerant engine 226b and locks the parity device cache line using MemRdLock.

[0172] In operation 722, the old data and parity data are read from the parity fault tolerant engine parity 226b and a lock is set. In operation 708, the primary fault tolerant engine 226a reads the primary memory device row 206 using MemRdRaw. In operation 710, the primary memory device 206 responds with the requested data using a MemData response to the primary fault tolerant engine 226a.

[0173] In response, the parity fault tolerant engine 226b transmits the requested data (old data and parity data) along with the lock status to the primary fault tolerant engine 226a in operation 712. In operation 734, RAID parity is calculated for the parity device cache line and poison bit.

[0174] In operation 714, when the primary fault-tolerant engine 226a requests to write data to the target main memory device row 206 using MemWr, the data requested to be written to the target main memory device row 206 is written in operation 724 and the target main memory device row 206 transmits a response (Cmp) in operation 716 to notify the primary fault-tolerant engine 226a that the write is completed.

[0175] In operation 736, the primary fault-tolerant engine 226a waits for the primary memory device 206 to complete the write.

[0176] In operation 718, the master fault tolerant engine 226a requests to write new parity checksum data and unlocks the cache line to the parity fault tolerant engine 226b using MemWrUnlock. In operation 726, the write requests to write data to the parity fault tolerant engine 226b and releases the cache line lock, which unlocks the entire parity group.

[0177] In operation 720, the parity fault tolerant engine 226b transmits a response (Cmp) to notify the master fault tolerant engine 226a that the write and unlock are complete. In operation 738, the parity group conflict prevention structure is cleared.

[0178] Returning to operation 702, if there is no conflict at the primary fault-tolerant engine 226a when the write request is sent to the primary fault-tolerant engine 226a, the primary fault-tolerant engine 226a allocates a valid modification entry and forwards the lock request to the parity fault-tolerant engine 226b using MemRdLock in operation 704. The parity fault-tolerant engine 226b performs another collision detection to ensure that only one request modifies the parity data associated with the parity group at a time.

[0179] If there is a conflict at the parity fault tolerant engine 226b, then in operation 740, the parity group conflict prevention structure is activated and subsequent lock requests are blocked and held in the conflict list queue and processed after the previously received lock request is completed.

[0180] In operation 742, to process the previously received lock request, the parity fault tolerant engine 226b requests to read data from the parity device 344 and locks the parity cache line using MemRdLock.

[0181] In operation 744, the parity device 344 responds with the request data using a MemData response to the parity fault tolerance engine 226b.

[0182] In operation 746, the parity fault tolerance engine 226b requests to write new parity checksum data and unlocks the parity cache line to the parity device 344 using MemWrUnlock. In operation 726, the write requests the data written to the parity device 344 and releases the parity cache line lock, which unlocks the entire parity group.

[0183] In operation 748, the parity device 344 transmits a response (Cmp) to notify the parity fault-tolerant engine 226b that the write and unlock are complete. In operation 750, the conflict prevention structure is cleared. In operation 730, the fault-tolerant engine 226a sends a response (Cmp) to notify the host 202 that the write is complete.

[0184] The conflict queue size should be designed to be large enough so that there is always enough space for storing conflicting requests. In an embodiment, the conflict queue size can be managed and controlled by the architectural limit on the number of memory devices in the parity group, and each host can have only one outstanding outstanding write request per address.

[0185] Fast consistency recovery technology

[0186] In some embodiments of RAID architectures, systems and methods may employ a fast consistency recovery algorithm. Figure 7B Table 1 shows the fast consistency recovery algorithm; and Fig. 7A The process of maintaining consistency is described. When a switch fails in the middle of a RAID group update, the parity devices may become inconsistent with the memory devices, leaving the memory in a state susceptible to data corruption.

[0187] Fig. 7A Two control structures are included: a main fault-tolerant engine 226a and a parity fault-tolerant engine 226b. The main fault-tolerant engine 226a is associated with a connection 216a, which is directly connected to the memory device 206a that stores the data of the accessed location. The parity fault-tolerant engine 226b is associated with a connection 216b, which is connected to the memory device 206b that stores the parity of the parity group 236.

[0188] In some embodiments, the RAID architecture requires that the parity lock metadata bit be set in the media when acquiring the parity lock after a switch failure occurs. The fabric management software can then scan all RAID storage and "repair" any inconsistent RAID groups. This feature has significant availability advantages for GFAM devices built with persistent media or clusters built with always-powered storage devices and is accessible even after a switch failure occurs. It is advantageous to employ fast consistency recovery techniques because the time required to rebuild a large GFAM installed from storage can be significant.

[0189] like Figure 7B As shown in Table 1 of FIG. 1 , the fast consistency recovery scheme can scan all RAID storages to execute the algorithm outlined in Table 1. In addition, Figure 3A , metadata in a RAID group can be used to define a lock state. In Table 1, if the lock bit state in the media is set, then the fast consistency recovery algorithm can determine whether the data is consistent with the parity by checking whether the parity device data is equal to the XOR of the data of all other memory devices. If the parity device data is equal to the XOR of the data of all other memory devices, then action is taken to clear the lock and rewrite the parity data. If the parity device data is not equal to the XOR of the data of all other memory devices, then action is taken to clear the lock and write the parity data resulting from XORing the data.

[0190] Read access scenario

[0191] Data access may include a read access scenario and a write access scenario. The operation methods of the read access scenario and the write access scenario may be different. Figure 8 An example flow diagram 800 is shown for a read access scenario. Figures 9A to 9B Example flow diagrams 900A and 900B are shown for a write access scenario.

[0192] exist Figure 8 In flowchart 800 of , in block 805, the read access is passed through the RAID controller. If the target device responds with valid data in operation 810, the response is forwarded to the requester in block 815. This is a very simple solution that provides very high throughput.

[0193] In block 820, if the target device responds with a poison or timeout (if a timeout occurs, the FM is notified), the poison bit indicates bad data and valid data cannot be recovered. This means that the target device row needs to be rebuilt using data from other devices in the parity group to obtain valid data, which can then be sent to the host. To recover valid data, an XOR operation is performed on all devices of the parity group excluding the target device. For example, in a 4+1 parity group, when the target device returns a poison response, the RAID controller performs an XOR operation on the other three devices plus the parity device. Therefore, the XOR operation uses data from all four devices to reconstruct valid data and then sends the valid data back to the host.

[0194] To begin recovery of valid data, in box 825, the RAID controller locks the parity cache line. In box 830, the RAID controller reads all other data rows in the parity group. In box 835, the RAID controller calculates the value of the target row by XORing the check data and the data rows (64 bytes) of the other devices. In box 840, the RAID controller writes the regenerated value of the target row. In box 845, the RAID controller returns the regenerated value to the requester as a response. In box 850, the RAID controller writes (and unlocks) the parity cache line (with the original check value).

[0195] In a read access scenario, if an MPU (as described below) is present and a poison response or timeout occurs, the MPU issues a request on an alternate fabric to attempt to access the row. The request may be successfully transmitted to the target device and the data sent back to the requester. However, the RAID controller on the alternate fabric may receive a poison response (i.e., from a previous timeout lock). If the RAID controller on the alternate fabric also receives a poison response, this situation is handled by recovering valid data, as described above in blocks 820 to 850. The MPU enables the system to handle a bad switch and send requests in an alternate path.

[0196] Write access scenario

[0197] exist Figures 9A to 9BIn the exemplary flow charts 900A and 900B of the write access scenario in FIG. 900A, in block 905 of flow chart 900A, the write access is intercepted by the RAID controller and the RAID controller issues a read and lock to the parity cache line. In block 910, if the check device responds with valid check data, then in block 915 the RAID controller reads the target data row. In block 920, the RAID controller generates new check data by performing an XOR operation on the new data, the original data, and the original check data. In block 925, the fault-tolerant engine writes the target row with the new data. In block 930, the RAID controller writes and unlocks the parity cache line (with the new check data). In block 935, the RAID controller responds to the host with a completion response.

[0198] exist Fig. 9B In block 940 of flowchart 900B shown in FIG. 1 , if the check device responds with poison, then the RAID controller locks the parity cache line. The RAID controller receives an invalid response; therefore, the check value must be regenerated. In block 945, the RAID controller reads the other data lines. In block 950, the RAID controller regenerates a new check by XORing the new data with the original data of the other lines. In block 955, the RAID controller writes the new data to the target line. In block 960, the RAID controller writes the new check and unlocks the parity cache line. In block 965, the RAID controller responds to the host with a completion response.

[0199] In block 970, if the inspection device is unreachable, information about the status of the inspection device may be obtained from the last stage switch in communication with the inspection device. The inspection device may be unreachable because the link to the inspection device is disconnected. In block 975, the RAID controller writes the new data to the target row. In block 980, the RAID controller responds to the host with a completion response.

[0200] In various embodiments, when a device is unreachable, an alternate path for authentication is provided. The system may perform automatic failover. During automatic failover, requests that would otherwise be processed through a first path are routed to an alternate path of an alternate fabric. In embodiments, an active-active redundant switch fabric may provide access to a memory device simultaneously along multiple paths, such as Figure 2 and Figures 13 to 14 If the response indicates that the device is unreachable and the device is disconnected at the first port on the first path, the leaf switch connection can be used to switch to an alternate path and direct the transmission through the second port to verify that the device is disconnected.

[0201] As an example of using an alternative path to verify, Figure 2 and Figures 13 to 14, the leaf switch connections 240, 1340, 1440 may provide an alternative path to link the first fabric switch 208a, 1408a and the second fabric switch 208b, 1408b to the memory devices 206, 1306, 1406. Two ports are provided for each device. The DSP 230, 1430 from the first fabric switch 208a, 1308a, 1408a and the DSP 230 from the second fabric switch 208b, 1308b, 1408b are connected to the devices 206, 1306, 1406, respectively.

[0202] For example, a DSP 230a, 1330a, 1430a from a first fabric switch 208a, 1308a, 1408a and a DSP 230d, 1330d, 1430d from a second fabric switch 208b, 1308b, 1408b are connected to a device 206a, 1306a, 1406a. If a response to a request sent along a first path through a fabric switch 208a, 1308a, 1408a indicates that the memory device 206, 1306, 1406 cannot be reached, then a leaf switch connection 240, 1340, 1440 may be used to switch to a second path through a fabric switch 208b, 1308b, 1408b to transmit a request to confirm whether the memory device 206, 1306, 1406 is actually disconnected or remains operational. If the memory device 206, 1306, 1406 is operational, it processes the request.

[0203] Various embodiments may perform atomic operations with masked writes. In some embodiments, the fault-tolerant engine may perform atomic operations and masked writes (using CXL.mem or CXL.UIO protocols) to generate new check data. The masked write operation masks data to inhibit and / or prevent data in a write operation from being written to some memory cell blocks constituting a memory cell array. By performing the masked write operation, some memory cell blocks to which data is not written retain previous data and new data is written to the remaining memory cell blocks.

[0204] Embodiments may perform a wide range of atomic operations since a lock has been obtained to perform RAID operations. Example atomic operations that may be implemented according to the present teachings include extract and increment, extract and decrement, extract and add, extract and decrement, extract and decrement, extract and AND, extract and OR, extract and XOR, extract and min, extract and max, extract and swap, and compare and swap. For example, an atomic extract and increment operation atomically increments the contents of a memory location by a specified value. Extract and increment atomically increments a value in memory. Extract and increment implements a lock with an increment of +1. In this embodiment, a mask write instruction may specify data of a memory to which data is written and data not written.

[0205] Atomic operations that allow locking of data to perform read-modify-write operations can be combined with shadow write operations to generate new check data, as previously described with respect to Fig. 9A In addition to block 920, the execution of Fig. 9A The same sequence of operations described in blocks 910 to 935 of FIG. First, the RAID controller sets a lock and reads the target data row. However, instead of performing an XOR operation to generate new check data (block 920), the RAID controller performs an atomic fetch and increment operation with a masked write.

[0206] When performing a shadow write, the CXL.mem protocol can only write a subset of bytes out of the 64 bytes. A shadow write operation is a partial cache line write operation that prevents a portion of existing data from being overwritten by new data when new data is written to the target device to preserve some of the existing data. For example, during a write operation, the RAID controller may write a subset of 64 bytes (e.g., 8 bytes) instead of writing all 64 bytes. Thus, an 8-byte value exists in memory. A fetch and increment operation increments the original data at a memory location by a specified value (e.g., increment value +1).

[0207] In this example, the RAID controller uses the appropriate 8 bytes of the original data and increments it by an increment value of 1. Thus, the RAID controller calculates new data based on the original data. The RAID controller then writes the new data based on the mask value to the target data and other portions that were not written to preserve the previous data. In essence, the embodiments perform a wide range of atomic operations well because a lock has been obtained to perform RAID operations.

[0208] For example, conventional atomic fetch and increment implements a mutually exclusive lock with an increment value of 1, wherein the entire function is performed atomically so that no other process interrupts the function. According to the present disclosure, RAID operations and atomic operations can be performed simultaneously with the same lock. In some embodiments, a batch of atomic operations can be efficiently performed at the RAID controller based on per clock.

[0209] Rebuild Operation

[0210] In an embodiment, the system 100, 200, 1400 may perform a data reconstruction operation in response to detecting a failure of a device included in a parity group. In some implementations shown in FIGS. 1, 2, and 14, the RAID controller 124, 224, 1424 may use one or more hot spare devices 148, 248, 348, 1448 to provide additional data protection for the parity group 136, 236, 1436. In an example where a device that is part of a parity group fails, the hot spare device 148, 248, 348, 1448 may be used to replace the failed device. The hot spare device 148, 248, 348, 1448 is not used during normal access operations, but is reserved to replace any device 106 that fails.

[0211] The hot spare device 148, 248, 348, 1448 does not contain any data until a failure occurs and the hot spare device is needed. Hot spare replacement allows the RAID controller 124, 224, 1424 to perform a reconstruction. Reconstruction is a background process performed in a RAID system to regenerate data from a failed device. In this example, the data of the failed device can be rebuilt in the hot spare device 148, 248, 348, 1448 using data from other devices that are part of the parity group. In this way, the parity group 136, 236, 1436 can be returned to its redundant state, and the hot spare device 148, 248, 348, 1448 becomes part of the parity group.

[0212] More specifically, with respect to the operation of the system 100, 200, 1400 and more particularly, the operation of the RAID controller 124, 224, 1424, in response to sensing a failure of a group 136, 236, 1436, the RAID controller 124, 224, 1424 writes an invalid value to each row of the hot spare device 148, 248, 348, 1448 before swapping in the hot spare device 148, 248, 348, 1448 to replace the failed device. The invalid message indicates that the parity group 136, 236, 1436 needs to be regenerated or rebuilt, as described above with reference to Figure 8 and 9B describe.

[0213] After a failure occurs, a rebuild operation is performed where invalid values ​​are written to the rows of the hot spare device 148, 248, 348, 1448 before the hot spare device is swapped in to replace the failed device. The data of the failed device can then be rebuilt in the hot spare device 148, 248, 348, 1448 using data from other devices that are part of the parity group.

[0214] In various embodiments, the systems 100, 200, 1400 may implement two different reconstructions: read reconstruction ( Fig.11) and Write Rebuild( Fig.12 After the hot spare 148, 248, 348, 1448 is swapped in, a subsequent read or write to the invalid row of the hot spare 148, 248, 348, 1448 returns the invalid state to the RAID controller 124, 224, 1424 to initiate a rebuild or regeneration.

[0215] Fig.10 Table 2 shows an example of an error handling algorithm for a fault-tolerant engine that responds to various read and write operations and whose failures may initiate a rebuild. Fig.10 As shown in Table 2 of FIG. 1 , a rebuild operation is automatically initiated after a link down fault (read operation), a media error fault (write operation), or an invalid fault (write operation). A link down fault is at least one indication that a link is down and is communicated via a response to a read request. An invalid fault indicates that the parity device is not set.

[0216] In the case of a media error failure, the memory device may respond in a manner to indicate that the poison return was due to a media error. In some embodiments, the link timeout should be set for millisecond level (e.g., approximately 10 ms) detection so that when a CXL device crashes in the middle of a read or write operation, the fault tolerance engine has enough time to perform a rebuild operation before the host processor times out. For example, in a 16-way RAID rebuild operation, the fault tolerance engine may be configured to perform the operation in less than 10 ms, which is the time allotted for the rebuild in the timeout hierarchy.

[0217] Essentially, during the rebuild of the hot spare device 148, 248, 348, 1448 based on a read access request or a write access request, the hot spare device 148, 248, 348, 1448 is initialized when the GFAM device or parity check device is replaced, where all rows are set with invalid data (poisoned). This forces the invalid rows to be regenerated on all subsequent accesses. Therefore, the host access will regenerate the hot spare device per access row. In other words, subsequent read and write operations can be used to force the fault-tolerant engine to rebuild the values ​​in the hot spare device to be consistent with other devices.

[0218] For example, in operation, the first host access will regenerate the first row of the hot spare device, and then a different host access will regenerate the second row of the hot spare device. When the rebuild process is complete, all data previously stored on the failed device is written to the hot spare device. Thereafter, during the next read operation or write operation, the system will operate as usual.

[0219] Preferably, in some embodiments, FM 134, 234, 1434 (or another entity) "walks" all rows in hot spare devices 148, 248, 348, 1448 to ensure that all rows are regenerated concurrently with the rebuild of a read or write operation. FM 134, 234, 1434 may be configured to walk the rows to provide a consistent network update to perform a transition from an initial configuration including the original device to a final configuration including the hot spare device while maintaining correctness.

[0220] The performance of a RAID system degrades while a failed disk is being rebuilt. Longer rebuild times increase the probability that another device may fail, which may result in permanent data loss or RAID failure. Thus, ideally, the RAID is rebuilt quickly to prevent RAID failures while also maintaining adequate performance. Therefore, in some embodiments, a system using FM 134, 234, 1434 (or another entity) may force a rebuild in a relatively short period of time (e.g., within a few minutes to a few hours).

[0221] Fig.11 is a swim lane diagram showing an exemplary process 1100 for performing a reconstruction of a read operation. The reconstruction of the read operation of process 1100 corresponds to Figure 8 Boxes 820 to 850 in FIG. 8 depict a read access scenario when the device responds with a poison or timeout. If a timeout occurs, the FM is notified. The poison bit indicates bad data and valid data cannot be recovered.

[0222] That is, the target device row needs to be rebuilt using data from other devices in the parity group to obtain valid data. When a host access to one of the invalid rows of hot spare devices is received during a read request, this instructs the fault tolerance engine to rebuild the values ​​of the hot spare device to be consistent with the other devices in the parity group. In operation 1102, the reconstruction of the read request to the hot spare device is first queued to be written to the fault tolerance engine 226 using "read rebuild".

[0223] In operation 1104, the fault-tolerant engine 226 sends a read data request using MemRd to the parity device 344 and using MemRdRaw to all other memory devices 206, which use MemData in operation 1106 to send the read data request from the parity device 344 and other memory devices to the fault-tolerant engine 226. In operation 1108, the fault-tolerant engine 226 then sends the valid data back to the hot spare device.

[0224] When multiple write requests want to access and update the same cache line, a parity group conflict prevention structure is provided and may be activated in operation 1110. In operation 1112, the parity group conflict prevention structure is cleared.

[0225] Fig.12is a swim lane diagram showing a reconstruction of a process 1200 for performing a write operation. The reconstruction of the write operation process 1200 corresponds to Fig. 9B , depicts a write access scenario during a write access when the fault tolerance engine 226 receives an invalid response from a hot spare device such that the value of the hot spare data must be regenerated. When a host access to one of the invalid rows of the hot spare device is received during a write request, this instructs the fault tolerance engine 226 to rebuild the value of the hot spare device to be consistent with the other devices in the parity group. Fig.12 In operation 1202, a rebuild of a write request to the hot spare is queued to be written to the fault tolerance engine 226 using "write rebuild".

[0226] In operation 1204, the fault-tolerant engine 226 sends a read parity data request to the parity device 344 and sets a lock using MemRdLock, and the parity device 344 updates the status of the metadata bit of the parity device to locked. In operation 1206, the fault-tolerant engine 226 sends the read data using MemRdRaw to all other memory devices 206. In operation 1208, the other memory devices 206 send a read data request to the fault-tolerant engine 226 using MemData.

[0227] In operation 1210, the parity device 344 sends a read parity data request using MemData to the fault tolerance engine 226. In operation 1212, the fault tolerance engine 226 generates new data and writes the new data to the hot spare device 148 using MemWr when the new data matches.

[0228] In operation 1214, the hot spare device 148 uses a response (Cmp) transmission to notify the fault tolerance engine 226 that the writing of the new data is complete. In operation 1216, the fault tolerance engine 226 writes the new parity data to the cache line of the parity device 344 using MemWrUnlock and in response, the cache line lock is released and the metadata is updated to the "free" state.

[0229] In operation 1218, the parity device 344 transmits a response (Cmp) to inform the fault tolerance engine 226 that the write and unlock are complete.

[0230] Next, in operation 1220, the fault-tolerant engine 226 sends a response (Cmp) to notify the queue manager that the requested rebuild is complete. Fig.12 In operation 1222, when multiple write requests want to access and update the same cache line, a parity group conflict prevention structure is provided and activated in operation 1222. In operation 1224, the parity group conflict prevention structure is cleared.

[0231] Fault-tolerant RAID

[0232] Figures 13 to 14 An embodiment of a fabric fault-tolerant RAID design according to the present disclosure is presented. In conventional CXL systems, preventive maintenance alone fails to increase the availability of the system to a desired level because too many failures occur without warning. For example, a CXL fabric failure occurs when a CXL switch crashes. CXL fabric components include a group of CXL switches in a cluster, CXL cables, and a power delivery system for the CXL switches. A CXL device fails when the device does not respond to CXL requests. CXL device components include CXL memory devices and their power delivery systems. The present system and method provide a solution to the problem by introducing a fault-tolerant system including hardware and / or software systems that can keep the cluster operating normally even if components crash without warning.

[0233] Fig.13 A method for discussing data packet flow during read and write operations implemented in a fabric fault-tolerant RAID design according to the present disclosure is described. Fig.14 A simplified version of . Fig.13 In the example of , system 1300 includes a host device 1302, an MPU 1342, a first spine switch 1312a, a second spine switch 1312b, a first leaf switch 1314, a second leaf switch 1320, a leaf switch connection 1340a, a leaf switch connection 1340b, and a memory device 1306. Host device 1302 may represent one or more host devices coupled to MPU 1342 via link 1305.

[0234] In this embodiment, the system 1300 has a leaf-spine topology where the MPU 1342 has a link to each spine switch 1312a, 1312b, each spine switch 1312a, 1312b having a link to a leaf switch 1314. For example, the MPU 1342 is connected to a first spine switch 1312a by connection 1310a and is also connected to a second spine switch 1312b by connection 1310b. The first spine switch 1312a is connected to a first leaf switch 1314a by connection 1318a. The second spine switch 1312b is connected to a second leaf switch 1314b by connection 1318b. Leaf switch links 1340a, 1340b may be provided to connect the first spine switch 1312a and the second spine switch 1312b.

[0235] Both the first spine switch 1312a and the second spine switch 1312b can be connected to the memory device 1306. The memory device 1306 can represent one or more memory devices or GFAM devices coupled to the first spine switch 1312a and the second spine switch 1312b. The connection between the leaf switches 1314a, 1314b and the memory device 1306 can be through a downlink port. Thus, the leaf switch 1314 is configured to communicate with the spine switch 1312 via an uplink port and is configured to communicate with the memory device 1306 via a downlink port.

[0236] It should be understood that two spine switches 1312a, 1312b, two leaf switches 1314a, 1314b, and memory devices 1306 are illustrated in this embodiment. However, a network, such as system 1300, may include any number of switches and / or GFAM devices. For example, in one embodiment, system 1300 may be a data center and may include hundreds, thousands, or more switches, memory devices, and / or host devices.

[0237] In the network system 1300 having a leaf-spine topology, the MPU 1342 is used as a fabric switch. The host device 1302 can transmit a request to the memory device 1306 by instructing the request to traverse the MPU 1342 and travel along a specific path through the switch hierarchy to ultimately reach a specific leaf switch connected to the destination memory device. In this example, the MPU 1342 provides two different paths to the memory device 1306 through two different redundant switching topologies.

[0238] As an example, a first path may include spine switch 1312a connected to leaf switch 1314 via link 1318a, and a redundant second path may include spine switch 1312b connected to leaf switch 1314b via link 1318b. Both switching topologies may be active simultaneously, such that, for example, a host may direct the first half of a traffic flow to travel through the first path and the second half of the traffic flow to travel through the second path, with both traffic flows reaching memory device 1306. A response to the request is then transmitted from memory device 1306 back to host device 1302.

[0239] If either the first or second path fails, the MPU 1342 may perform an automatic switchover and retry issuing the request down through the functionally redundant path. For example, if the first path fails, the MPU 1342 may retry and issue the request down through the second path. In operation, the MPU 1342 makes a fabric plane selection based on the destination address and the current state of the fabric plane. A fabric plane is a path for transmitting data. Before the MPU 1342 issues a request, it stores a copy of the request and the initial fabric plane selection. Upon issuance, the MPU 1342 starts a timeout counter. If the timeout counter triggers, the MPU 1342 issues the same request using an alternate fabric plane.

[0240] Some goals achieved by the network system of the present disclosure include (but are not limited to) the following capabilities: support N+1 GFAM, support single switch parity group, support single fabric parity group (without MPU), support redundant fabric parity group (with or without MPU), support active-active redundant switch fabric, support identification and recovery of parity group checksums in all scenarios, support memory atomic operations via fault tolerance engine, and support shadowed write operations. As a result, these advantages can maximize the throughput of memory regions under RAID support and minimize the latency of memory regions under RAID support.

[0241] In an embodiment, there is a single switch, e.g. Figure 1A In the example illustrated in , the MPU may be optional since there is only a single path.

[0242] In an embodiment without an MPU, the system may include two ports extending from the host. When a failure occurs, the host will receive a response back to indicate that there is a failure. In response, the host may shut down at least that particular application. Then, the FM responsible for managing the fabric may recognize that one of the two redundant fabrics has a failure. The FM may then notify all hosts to use the other fabric that is still functioning normally. In this embodiment, the process of maintaining the application using the FM may be a manual process rather than an automatic switching process using the MPU, as described above. In this manual process, when the application is shut down, the FM may reconfigure the system and the host may restart the application.

[0243] Therefore, the system of the present disclosure can provide support for both automatic failover using MPU and manual failover using FM.

[0244] In various embodiments, the system provides support for active-active redundant switch fabrics. The system can provide access to memory devices simultaneously along multiple paths, such as Fig.13. Multiple paths can access essentially the same row in memory simultaneously, but using different paths. From a cost perspective, it is desirable to have multiple paths active so that the customer does not pay for an entire redundant switch fabric that is not always used.

[0245] In an embodiment, the system supports identifying and recovering parity group checksums in all scenarios. The system can be configured to predict all possible scenarios of errors and address all of them.

[0246] Embodiments of the present disclosure provide support via a fault-tolerant engine in memory atomic operations. In the present disclosure, by using a host (e.g., an X86 host), the system may have the ability to lock a portion of data in its cache, perform operations on the data, and then unlock the data. The system may perform a read-modify-write on that location in the cache, and because of the lock, the system prevents all other execution threads on that same host or a different host from accessing it and modifying it during the time one of the processors performs a read-modify-write.

[0247] According to the present disclosure, this is an example of an atomic aspect, where the system modifies memory in an atomic manner to obtain consistent results. The system performs reads, modifications, and writes without performing other reads or writes in between.

[0248] Therefore, in an embodiment of the present disclosure, the system can perform read-modify-write on a location in a memory device memory by utilizing a fault-tolerant engine mechanism in a leaf switch. Therefore, the system enables a large number of hosts to perform atomic operations through a leaf switch by accessing memory very efficiently, so that there is very high throughput.

[0249] In contrast, if the atomic operations are performed in the host (first host), the number of operations performed will be relatively slow. In this scenario, if the host (second host) wants to access data, the second host will have to send a request to the first host requesting the data. The first host will then send the requested data to the second host. This results in a large amount of latency while the second host waits for the data to travel through all the switches. Performing atomic operations at the leaf switch results in a very high number of atomic operations and much higher throughput.

[0250] In various embodiments, the system maximizes the throughput of the memory area under RAID support. One goal of the system is to maintain very high read and write rates as much as possible at the memory device. Although the RAID operation is performed at the memory device, through this implementation, the system can achieve very high throughput of read and write operations, which leads to high bandwidth. At the memory device, the RAID scheme can be used to provide an enhanced form of parity check protection. The second group of parity data is written across all devices in the memory group to avoid data loss when an error occurs. When the memory device in the group does not pass its parity check, the data is rebuilt using parity information coupled with data on other devices in the group.

[0251] In an embodiment, the system minimizes latency of the memory banks under RAID support because the host processor cannot tolerate high latency well. Therefore, the system minimizes latency.

[0252] Fig.14 A block diagram illustrating an exemplary fabric fault-tolerant RAID design 1400 in accordance with the present disclosure. Fig.13 Presents a general approach for providing fault-tolerant systems, while Fig.14 A more detailed embodiment of this fault-tolerant system is presented, which may be a hardware or software system that enables a cluster to maintain normal operation even if components fail without warning.

[0253] Embodiments of the fabric fault-tolerant RAID design can extend the basic RAID architecture ( Figures 1A to 2 ) to gracefully cope with CXL switch failures and device failures. Therefore, many components are different from the basic RAID architecture ( Figures 1A to 2 ) are labeled with similar reference numbers and will not be described again. In this embodiment, the new components are MPUs 1442a, 1442b and more complex lock controllers 1438a, 1438b, 1438c. Leaf switch connections 1440 are provided between the first fabric switch 1408a and the second fabric switch 1408b.

[0254] The MPUs 1442a, 1442b may be configured to keep a record of all outstanding read and write requests, and if any request exceeds the MPU 1442a, 1442b request timeout, the MPU 1442a, 1442b reissues the request along an alternate fabric plane using the leaf switch connection 1440. In some embodiments, a lock controller 1438a, 1438b, 1438c may be included in one or more of the memory devices 1406 to retain or save "old" write data for all active parity group updates. Thus, if a switch failure occurs in the middle of a parity group update, the lock controller 1438a, 1438b, 1438c will contain the data needed to keep the parity groups in a consistent state.

[0255] Since CXL memory requests (memory reads and memory writes) do not have any side effects, it is safe for the system 1300, 1400 to read the same memory location multiple times and to write the same location multiple times with the same write data. This scenario can occur when a switch failure occurs just before a read or write response is delivered to the host.

[0256] use Fig.13 , 14 The fault-tolerant design of the systems 1300 and 1400 in FIG. Fig.15A As described above, in some scenarios, a switch may crash after it updates the home location and before it updates the parity location. To address this situation where consistency may be lost, the write process is updated so that the lock controller 1438 saves the state of the home location data. If the fault-tolerant engine 1426 running on the alternate fabric plane needs to complete the write operation, the fault-tolerant engine 1426 may retrieve the original home data from the lock controller 1438 in the fault-tolerant memory device 1406. Fig.14 In 15, the fault-tolerant engine is an enhanced fault-tolerant engine because the basic write algorithm is enhanced.

[0257] The lock controller 1438 in the memory device 1406 ensures that only one write request is modifying the parity group at a time. The lock controller 1438 also allows modifications that are partially performed by one fabric plane to be picked up and completed by the redundant fabric plane. To eliminate the need for end-to-end retries and forward progress screens, the lock controller 1438 maintains a conflict list and determines the order in which read and write requests are serviced.

[0258] In addition, the lock controller 1438 may also temporarily block memory write requests that attempt to access the same parity group as the in-flight update. One fault-tolerant engine waits for the lock fault-tolerant engine to respond with a "normal" response. In some embodiments, the lock controller 1438 may use a lock pipeline (not shown) that can accept new requests at a rate of, for example, 1 per core clock. The lock pipeline may complete after only a few cycles and may report the results of the conflict check, such as Fig. 15B Displayed in.

[0259] Active requests in the lock controller 1438 may time out. When a tracker entry times out, the timeout bit in the entry may be set, causing the lock pipeline to begin detecting timeout hits. To avoid deadlocks, in some embodiments, there must be at least one entry in the active lock tracker dedicated to servicing fabric A requests. Fig.14 In , fabric A may include spine switch 1412a and leaf switch 1414a. Likewise, at least one entry in the active lock tracker must be dedicated to servicing fabric B requests. Fig.14 , fabric B may include spine switch 1412b and leaf switch 1414b. This retention scheme ensures that the lock controller 1438 continues to make forward progress even in the event that one of the fabric planes is invalid.

[0260] exist Fig.13 , 14 In the example, the MPU 1342, 1442a, 1442b records all CXL.mem read requests and write requests that the processor makes active in the CXL fabric in an active memory request tracker (not shown). If the MPU 1342, 1442a, 1442b runs out of resources in the active memory request tracker, it may flow control incoming read and / or write requests. This may be possible in embodiments where the RAID architecture is configured to support only software consistency. The MPU 1342, 1442a, 1442b may perform fabric plane selection based on the destination address and the current state of the fabric plane. Before the MPU 1342, 1442a, 1442b issues a request, it stores a copy and the initial fabric plane selection. Upon issuance, the MPU 1342, 1442a, 1442b starts a timeout counter. If the timeout counter triggers, the MPU 1342, 1442a, 1442b issues the same request using an alternate fabric plane.

[0261] Fig.15A1430 is a swim lane diagram showing a write flow with a write protection operation according to an exemplary embodiment of the present disclosure. In some embodiments, a switch may crash after the system updates the home location and before it updates the parity location. To address this situation where consistency may be lost, the write flow is updated so that the lock controller 1438 saves the state of the home location data.

[0262] In operation 1502, all write requests are routed using MemWr to the fault-tolerant engine 1426a, which serves as the master for coordinating writes (and reads). Fig.14 In the embodiment of 15, the fault tolerance engine is an enhanced fault tolerance engine because the basic write algorithm is enhanced. This fault tolerance engine 1426a performs conflict resolution with other requests of the same home agent.

[0263] If there is a conflict at the fault tolerant engine 1426a, then in operation 1532, the parity group conflict prevention structure is activated and subsequent conflicting requests are blocked and held in the conflict list queue and processed after the previously received request is completed.

[0264] If there is no conflict at the fault tolerant engine 1426a, then in operation 1504, to process the first received request, the fault tolerant engine 1426a sends a request to the fault tolerant engine parity device 1426b to read the data and lock the parity cache line using MemRdLock.

[0265] Parity engine 1426b performs additional conflict detection to ensure that only one request is modifying the parity of the parity group at a time. If there is a conflict at the fault-tolerant engine parity device 1426b, then in operation 1542, the parity group conflict prevention structure is activated and subsequent lock requests are blocked and kept in the conflict list queue and processed after the lock request received first is completed.

[0266] In operation 1544, to process the previously received lock request, the fault-tolerant engine 1426b requests to read data from the parity device 344 and locks the parity cache line using MemRdLock.

[0267] If there is a conflict at the parity device 344, then in operation 1558, the parity group conflict prevention structure is activated. The lock controller 1438 ensures that only one write request is modifying the parity group at a time. The lock controller 1438 maintains a conflict list and determines the order in which requests are serviced.

[0268] In operation 1522, the old data and parity data are read and a lock is set.

[0269] In operation 1508 , the primary fault-tolerant engine 1426 a reads a cache line of the primary memory device 1406 using MemRdRaw.

[0270] In operation 1510, the memory device 1406 responds with the request data using a MemData response to the fault-tolerant engine 1426a. In operation 1546, the parity device 344 responds with the request data using a MemData response to the fault-tolerant engine 1426b. In operation 1512, in response, the fault-tolerant engine 1426b transmits the request data (old data and parity data) to the primary fault-tolerant engine 1426a along with the lock status. In operation 1534, the RAID parity for the cache line and the poison bit is calculated.

[0271] In some embodiments, a lock controller 1438 may be included in one or more of the memory devices 1406 to maintain or save "old" write data for all active parity group updates in operation 1524. The lock controller 1438 also keeps track of all parity groups in the process of being modified. Thus, if a switch failure occurs in the middle of a parity group update, the lock controllers 1438a, 1438b, 1438c will contain the data needed to keep the parity groups in a consistent state.

[0272] In an embodiment, the switch may crash after the system updates the fault-tolerant engine 1426a and before it updates the parity device 344. To address this situation where consistency may be lost, the write flow is updated so that the lock controller 1438 saves the state of the home location data. Therefore, before the new data is written, in operation 1514, the fault-tolerant engine 1426a requests to write the old write data to save the old write data at the fault-tolerant engine 1426b using MemWrPreserve.

[0273] Operation 1548 writes the old write data to save the old write data at the parity device 344 using MemWrPreserve. In response, operation 1524 saves the old write data. The parity device 344 transmits a save completed response (Cmp) in operation 1550. The fault tolerance engine 1426b transmits a save completed response (Cmp) in operation 1516.

[0274] In operation 1536, fault-tolerant engine 1426a waits for fault-tolerant engine 1426b to respond with a "normal" response.

[0275] In operation 1518, when the fault-tolerant engine 1426a requests to write data to the cache line of the target main memory device 1406 using MemWr, the data requested to be written to the cache line of the target main memory device 1406 is written in operation 1526 and the target main memory device 1406 transmits a response (Cmp) in operation 1520 to notify the fault-tolerant engine 1426a that the write is completed.

[0276] In operation 1538, the fault-tolerant engine 1426a waits for the master CXL device 1406 to complete the write. In operation 1562, the fault-tolerant engine 1426a requests to write the new parity checksum data and unlock the cache line to the fault-tolerant engine 1426b using MemWrUnlock.

[0277] In operation 1552, the fault tolerance engine 1426b requests to write new parity checksum data and unlocks the cache line to the parity device 344 using MemWrUnlock. In operation 1528, the write requests the data written to the parity device 344 and releases the cache line lock, which unlocks the entire parity group.

[0278] In operation 1560, the conflict prevention structure is cleared. In operation 1554, the parity device 344 transmits a response (Cmp) to inform the fault tolerance engine 1426b that the write and unlock are complete.

[0279] In operation 1556, the conflict prevention structure is cleared. In operation 1564, the fault-tolerant engine 1426b transmits a response (Cmp) to inform the fault-tolerant engine 1426a that the write and unlock are complete.

[0280] In operation 1540, the fault tolerance engine 1426a conflict prevention structure is cleared. Next, in operation 1530, the fault tolerance engine 1426a sends a response (Cmp) to notify the host 1402 that the write is complete.

[0281] In some embodiments, the lock controller 1438 may include a lock pipeline. Fig. 15B and Fig. 15C , the lock pipeline may generate different types of reports indicating the results of the conflict check performed by the lock controller 1438. For example, the result types may include LockNoCnflt, LockCnflt, LockHitTimeoutNoPreservedData, and LockHitTimeoutNoPreservedData.

[0282] System failure scenarios

[0283] The system and method of the present invention provide solutions to diagnose and handle some failure scenarios that may occur in the system, including CXL switch disconnection and CXL connection disconnection between leaf switches and GFAM devices, which can be used Figures 13 to 14 Now refer to the example in Figures 13 to 14 , describing a failure scenario where a CXL switch connection is disconnected. When a CXL switch connection is disconnected and a request is sent from a host device 1302, 1402 to traverse an uplink 1310, 1410 or a downlink 1318, 1418 of a spine switch 1312, 1412, the host device 1302, 1402 can recognize that it is attempting to send a data packet over an inoperable connection. According to the CXL standard protocol, the host device 1302, 1402 can recognize that the data packet cannot be sent, so it will generate a poison response. The poison response indicates that somewhere along the path, the correct data cannot be obtained.

[0284] If the system 1300, 1400 includes the MPU 1342, 1442, then in a failure scenario where the CXL switch connection is disconnected and data transmission is rendered inoperable via the first path, the system 1300, 1400 performs automatic failover. During automatic failover, requests that were originally processed via the first path are routed to an alternate path of the alternate fabric.

[0285] If the system 1300, 1400 does not include an MPU 1342, 3442, then the host device 1302, 1402 is directly connected to the CXL switch 1312, 1412 via the connection 1310, 1410. In this case, in a failure scenario where the CXL switch is disconnected, a poison message is sent to the host device 1302, 1402 in response to the CXL switch disconnect failure. As described above, a retry request may be attempted. If the attempt to retry the request is unsuccessful, then in some embodiments, the application may have to be closed if there is no other available way to retry the request.

[0286] Reference Figures 13 to 14, describing a failure scenario where the CXL connection between the leaf switch and the GFD is disconnected. In a redundant system embodiment failure scenario, if the CXL connection 1316a / 1316b, 1416a to 1416f between the leaf switch 1314a / 1314b, 1414a / 1414b and the memory device 1306, 1406 is disconnected due to the redundant configuration and the inter-leaf switch connection 1340, 1440 remains connected, then the fault tolerance engine 1426 retries and sends the request on the leaf switch connection 1340, 1440 on the alternate CXL connection. If a valid completion response message is received indicating that the request has been completed using the alternate CXL connection, then the fault tolerance engine 1426 responds to the host device 1302, 1402 with a response and notifies the FM 1434 of the memory device link failure (i.e., the memory device 1306, 1406 has a disconnected connection 1316, 1416).

[0287] If an invalid completion response message is received indicating that the request was not completed, the fault tolerance engine 1426 performs a line regeneration using the cache line to send a response back to the host device 1302, 1402 and informs the FM 1434 of the failure of the memory device 1306, 1406. The fault tolerance engine 1426 reports all available information about the failure to the FM 1434 so that the FM 1434 can take the necessary actions (i.e., reconfigure or swap in a hot spare) to return the failed memory device 1306, 1406 to an "usable state."

[0288] In another failure scenario embodiment, when the CXL connections 1316a / 1316b, 1416a-1416f between the leaf switches 1314a / 1314b, 1414a / 1414b and the memory devices 1306, 1406a-1406c are broken, it may be difficult to diagnose the failure by the fault tolerance engine 1426. In response to the failure, the fault tolerance engine 1426 sends a poison response to the host device 1302, 1402 and reports all available information about the failure to the FM 1434 so that the FM 1434 can take appropriate actions (i.e., reconfigure or swap in a hot spare) to return the failed component to an "usable state".

[0289] Examples of these types of failure scenarios that may produce a poisoning response include: a system without redundant fabric (i.e., a non-redundant system); a system in which a failure occurs at an inter-leaf switch connection 1340, 3440 and renders it inoperable; or a system that experiences multiple failures between CXL connections 1316, 1416, leaf switches 1314, 1414, and memory devices 1306, 1406.

[0290] like Fig.16, in some implementations of the present disclosure, the RAID architecture extends the CXL.mem protocol and the CXL.UIO protocol to provide more responsiveness to read and write requests. Fig.16 Examples of these responses illustrated in Tables 3, 4, and 5 of enable the fault tolerance engine to improve cluster availability. The MPUs 1342, 1442 keep track of all CXL.mem read and write requests that the processor makes active in the CXL fabric. The active write request tracker contains a list of all active requests in the process of modifying the memory device. The RAID memory pipeline ensures that no two requests modify the same parity group at the same time. If the MPU runs out of resources in the active memory request tracker, it can control the flow of incoming read and / or write requests because in some embodiments, the RAID architecture can be configured to support only software consistency.

[0291] Table 3 shows an example of a RAID CXL response. For example, during operation, the fault tolerance engine can be requested to update metadata bits when reading and writing (Figure 3), and the extracted data will be returned to the host for a full-featured read request process ( Figure 4 ) and full-featured write request flow ( Figure 5 ). However, if RAID reconstruction is required, the fault-tolerant engine will return a DRS packet with (MetaField==No-Op) indicating that there is no useful information in the MetaValue field, as shown in Table 3.

[0292] exist Fig.16 , Table 4 shows an example of a RAID read response code. Table 5 shows an example of a RAID write response code.

[0293] like Fig.17 , in some embodiments, the RAID architecture may extend the CXL.mem protocol and the CXL.UIO protocol by providing CXL-specific extensions beyond the standard CXL protocol. According to the standard CXL protocol, the memory devices 1306, 1406 can only provide two types of responses: a valid data response or a poison response. According to the present disclosure, similarly, as described above with respect to the lock meta-state feature, in the completion state encoding scheme ( Fig.17 ) provides an additional 2-digit code. Therefore, the systems and methods of the present disclosure are able to send back both meta-status information and completion code responses.

[0294] Fig.17 Table 6 illustrates the CXL-proprietary extension of the CXL.mem / UIO response completion codes for two scenarios: when poison is not set and when poison is set. Fig.17As shown in Table 6 of , when poison is not set, poison value = 0 and the three CXL.mem / UIO response completion codes are "locked" (if the system is retrying to lock the row), "invalid" (the value of the "invalid" meta-status), and "valid".

[0295] When poison is set and poison value = 1, the three CXL.mem / UIO response completion codes are "meta state poisoned" (previously set by the host device), "uncorrectable memory", and "link down". When a poison response is sent back, the fault tolerance engine makes further inquiries to determine why the poison response was sent back. Under the CXL protocol, poison is stored in memory so that subsequent read operations know that the data is corrupted.

[0296] The system determines if the poison bit was previously set by the host or because there is uncorrectable memory. If there is uncorrectable memory, then the fault-tolerant engine can get the correct value by rebuilding the parity group and sending the value back. If the host previously sent the poison bit, then the fault-tolerant engine cannot correct the value because the last time it was written, it was poisonous. Sending the poison response back is the correct action to take. If the link is down, then the fault-tolerant engine needs to determine whether to send the request along a different path or if there is bad data in the actual spare.

[0297] Information about meta-state poisoning, uncorrectable memory, and link down status all needs to be sent back to the fault tolerance engine so that appropriate action can be taken. It is important to know the difference between the completion status message types because it helps determine the corrective action to take. The completion status encoding scheme shows how the system encodes the information and response packets so that it works properly, although these codes are not defined in the CXL standard.

[0298] like Fig.16 Table 3 and Fig.17 As shown in Table 6 of , using CXL.mem, the response can be encoded using the meta field and meta value bits, so that there are two 2-bit fields in the response packet. Using CXL.uio, the response can be encoded in the attribute field which is a 2-bit field. And we can use this to encode error responses. If the original request was a CXL.mem request, then the system sends back a CXL.mem response. If the original request was a CXL.UIO request, then the system sends back a CXL.uio response. Therefore, the type of the original request indicates whether the response sent back is a CXL.mem response or a CXL.uio response.

[0299] Preemptive maintenance

[0300] like Figures 18A to 191434, 234, 1434 can perform preventive / preemptive maintenance to replace "ready to be replaced" components by preemptively adding new devices and rebuilding data to the new devices before an impending failure can occur. As discussed above, in various examples, the FM 134, 234, 1434 can perform maintenance of fabric-wide switch configurations. The FM 134, 234, 1434 can add or remove memory devices 106, 206, 1406 from the system. In an embodiment, a new memory device, such as a new CXL device, can be added to replace a memory device within a cluster already included in the system.

[0301] In an embodiment, the memory device 106, 206, 306 that may be added or removed may be a hot spare device 148, 248, 348, 1438 and / or a new memory device. The example systems and methods disclosed herein utilize values ​​representing the state of the memory device 106, 206, 1406 measured by the FM 134, 234, 1434 to actively monitor and / or analyze the memory device 106, 206, 1406 to automatically trigger replacement or maintenance of the memory device based on this analysis.

[0302] The FM 134, 234, 1434 used as a management system can be configured to perform "health status" monitoring on the components of the cluster. All major components of the cluster can be configured as self-diagnostic components that will notify the management system (e.g., FM 134, 234, 1434) when a component is "ready to be replaced." If any memory device in the system begins to experience multimedia errors such that it needs to be replaced, the memory device can transmit a warning message to the mailbox of the FM 134, 234, 1434.

[0303] like Fig.18A As shown in FIG. 1 , a memory device may self-announce that it is ready to be replaced. The FM 134, 234, 1434 may detect health indicators of the memory device to trigger a "hardware replacement required" event. If a switch fails (e.g., a CXL switch), the FM 134, 234, 1434 may monitor the switch based on the frequency of link failure rates which may serve as an early indicator of an impending switch failure.

[0304] Figures 18A to 18C A preemptive maintenance process for adding a new storage device to replace a storage device within a cluster already included in a system is described. Fig.19 exhibit Figures 18A to 18C An example flow chart 1900 of a preemptive maintenance process is depicted in FIG.

[0305] exist Figures 18A to 18CIn an example of , the system may include a system operator 1808 and a system tenant 1810 in a distributed edge computing environment. The memory devices 106, 206, 1406 of the memory device cluster may be coupled to and shared by multiple tenants 1810 in the distributed edge computing environment. In an edge cloud in a multi-tenant setting, services and applications may be requested and shared by many users or multiple tenants. In various embodiments, multiple tenants may belong to the same or different types of systems.

[0306] For example, a GFAM device may contain both volatile and non-volatile memory, such as DRAM and flash memory. The GFAM device effectively acts as a shared memory pool. It should be understood that in some scenarios, the hardware platform or configuration may operate on a modifiable platform to, for example, enable the use of exchangeable or interchangeable hardware components at different layers of the edge cloud. In this way, Fig. 18C A system operator 1808 at a tenant 1810 shown in FIG. 1 is able to upgrade the deployment of edge systems with newer hardware and specify rules for sharing data provided from the tenant 1810 to the cluster.

[0307] In block 1905, the FM 134, 234, 1434 may monitor the operational status of all data devices 1806 by receiving health status information. One of the data devices may determine that it is at a point of failure, such as detected by, for example, a media error count reaching a predetermined threshold indicating an impending failure. In block 1910 and as Fig.18A As shown in FIG. 1 , data device 1806 can announce itself as ready to be replaced by transmitting a health status alert containing information that hardware replacement is required to the FM's mailbox while the system remains operational.

[0308] In block 1915, the FM 134, 234, 1434 detects a health status warning from the data device 1802. In block 1920 and as Fig.18B In response, the FM selects a data device and the system operator 1808 adds the selected data device as a new data device 1804 to replace the "ready to replace" device 1802 to improve the continuous availability of the system.

[0309] In block 1925, all fault-tolerant engine CSRs are updated. The CSRs define the interleaving of the devices of the parity group 1812 to determine the specific destination fault-tolerant engine for the request. Each fault-tolerant engine may be configured to maintain a record of the locations of the spare devices of the parity group.

[0310] In block 1930, all tenants 1810 perform spare device data reconstruction by having each tenant 1810 read each location of the edge computing device (e.g., GFAM device) used by the tenant 1810. The FM 134, 234, 1434 configures the new data device 1804. At each tenant, the encryption key is unlocked. The MPU records all read requests and write requests active in the fabric. Each tenant 1810 rebuilds the spare (i.e., the new data device 1804). Each tenant 1810 switches to using the new data device 1804 and reads the new data device 1804 so that subsequent reads (or writes) regenerate the data previously stored on the replaced data device 1802 to the new data device 1804.

[0311] In block 1935, the fault-tolerant engine CSR is updated. In block 1940, the old data device 1802 is removed from the parity group.

[0312] Fig. 20 Description and Fig.14 Compared to the alternative embodiment of the fabric fault-tolerant RAID design 2000. Fig. 20 Describes alternative topologies with redundant connections with failover paths. For brevity, only the Fig.14 The different aspects of the fault-tolerant RAID design 1400 and the fault-tolerant RAID design 2000 are discussed in detail. Fig.14 To the extent described in Fig. 20 Similar components in the drawings are designated by similar reference numerals.

[0313] and Fig.14 compared to, Fig. 20 An embodiment of an alternative fabric fault-tolerant RAID design 2000 in includes a single spine switch 2012, but does not include an MPU, RAID routing blocks, leaf switch connections, and lock controller 1438. The single spine switch 2012 is connected to hosts 2002a, 2002b via connections 2010a, 2010b, respectively. The single spine switch 2012 is connected to a first fabric switch 2008a via connections 2018a, 2018b and to a second fabric switch 2008b via connections 2018c, 2018d.

[0314] A redundant connection with failover paths 2050a, 2050b is provided between the first fabric switch 2008a and the second fabric switch 2008b. For example, the redundant connection can be configured to transfer a failure from the first fabric switch 2008a to an alternate active second fabric switch 2008b when a failure is detected at the first fabric switch 2008a. The redundant connection provides multiple alternate paths between the hosts 2002a, 2002b and the memory devices 2006a, 2006b, and 2006c.

[0315] A first redundant connection having a failover path 2050a extends from the first switch core 2022a to a USP 2052a in the first fabric switch 2008a and from the USP 2052a to a DSP 2054b in the second fabric switch 2008b and from the DSP 2054b to the second switch core 2022b. A second redundant connection having a failover path 2050b extends from the second switch core 2022b to a USP 2052b in the second fabric switch 2008b and from the USP 2052b to a DSP 2054a in the first fabric switch 2008a and from the DSP 2054a to the first switch core 2022a.

[0316] The switch core 2022a, 2022b may be configured to change the routing of data packets so that all requests are sent to the correct fault-tolerant engine 2026a, 2026b, 2026c, 2026d, 2026e, 2026f. When a failure occurs in one fabric switch, the switch core 2022a, 2022b may use one or more of the redundant connections to change the routing of the data packets to the active fabric switch. In an embodiment, all data packets may be transmitted via a single redundant connection. In an embodiment, one or both of the redundant connections may be configured to carry bidirectional traffic. In such an embodiment, a portion of the data packet may be transmitted via a first redundant connection and the remainder of the data packet may be transmitted via a second redundant connection.

[0317] exist Fig. 20 In the embodiment of the present invention, the fault tolerance engines 2026a, 2026b, 2026c, 2026d, 2026e, 2026f are provided downstream of the DSPs 2030a, 2030b, 2030c, 2030d, 2030e, 2030f respectively coupled to the switch cores 2022a, 2022b. The FM 2034 is connected to the fault tolerance engines 2026a, 2026b, 2026c, 2026d, 2026e, 2026f. As discussed above, the FM 2034 is configured as a management service to support network management of the fabric switches 2008a, 2008b. The memory devices 2006a, 2006b and 2006c can be configured by the FM 2034 via a command set defined in the FM API.

[0318] During normal operation, the redundant connection with failover paths 2050a and 2050b is disabled to not provide traffic through failover paths 2050a and 2050b. Specifically, during normal operation, failover paths 2050a, 2050b can be set to an idle state. Thus, traffic from hosts 2002a, 2002b is routed directly to storage devices 2006a, 2006b, 2006c through corresponding fabric switches 2008a, 2008b.

[0319] However, after a failover event, such as when one of the fabric switches 2008a, 2008b fails or a component and / or connection in one of the fabric switches 2008a, 2008b fails, the flow paths are reconfigured by the system so that the failover paths 2050a, 2050b are reconfigured to enable traffic to flow between the fabric switch 2008a and the fabric switch 2008b. Specifically, when one of the fabric switches fails, the cluster of devices 2006a, 2006b, 2006c can be restarted. In addition, when a device fails or a device switching link fails, the redundant connections with the failover paths 2050a, 2050b enable the system to function to keep the cluster running.

[0320] Fig.21 Flowchart illustrating an exemplary method 2100 for practicing an embodiment of the present disclosure. In block 2102, the method 2100 includes at least one group of memory devices, wherein for a destination address in a memory request, one of the devices is a target device and another of the devices is a parity device. In block 2104, the method 2100 includes providing at least one fabric that (i) is coupled to the at least one group of memory devices and at least one host and (ii) includes a memory switch including at least one fault tolerance engine, the at least one fault tolerance engine coupled to each memory device in the at least one group of memory devices.

[0321] At decision node 2106, if one of the memory devices is corrupted during a read access, the fault-tolerant engine performs (i) a read access and (ii) a read recovery in block 2108. If decision node 2106 determines that the operation is not a read access (e.g., one of the memory devices is corrupted during a write access), the fault-tolerant engine performs (i) a write access including atomically updating the data of the target device and the parity data of the parity device and (ii) a write recovery in block 2110.

[0322] Block 2112 of method 2100 includes providing at least one routing processor that is (i) coupled to the memory switch and (ii) configured to determine a path of a memory request received from at least one host to a target device and direct the memory request to the target device.

[0323] As discussed above, various aspects of the embodiments herein may be facilitated by corresponding software and / or firmware components and applications, such as software and / or firmware executed by an embedded processor or the like. Thus, embodiments of the present invention may be used as or in support of software programs, software modules, firmware, and / or distributed software executed on some form of processor, processing core, or embedded logic, on a virtual machine running on a processor or core, or otherwise implemented or realized on or within a computer-readable or machine-readable non-transitory storage medium. A computer-readable or machine-readable non-transitory storage medium includes any mechanism for storing or transmitting information in a form that can be read by a machine, such as a computer.

[0324] For example, a computer-readable or machine-readable non-transitory storage medium includes any mechanism that provides (i.e., stores and / or transmits) information in a form accessible by a computer or computing machine (e.g., a computing device, an electronic system, etc.), such as recordable / non-recordable media (e.g., read-only memory (ROM), random access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices, etc.). The content may be directly executable ("object" or "executable" form), source code, or difference code ("delta" or "patch" code).

[0325] A computer-readable or machine-readable non-transitory storage medium may also include a storage device or database from which content may be downloaded. A computer-readable or machine-readable non-transitory storage medium may also include a device or product on which content is stored when sold or delivered. Thus, delivering a device with stored content or providing content for download via a communication medium may be understood as providing an article of manufacture including a computer-readable or machine-readable non-transitory storage medium with such content described herein.

[0326] The various components described herein, referred to above as processes, servers, or tools, may be components for performing the described functions. The operations and functions performed by the various components described herein may be implemented by software running on a processing element, via embedded hardware or the like, or any combination of hardware and software. Such components may be implemented as software modules, hardware modules, dedicated hardware (e.g., dedicated hardware, ASIC, DSP, etc.), embedded controllers, hard-wired circuit systems, hardware logic, etc. Software content (e.g., data, instructions, configuration information, etc.) may be provided via an article of manufacture, which includes a computer-readable or machine-readable non-transitory storage medium that provides content representing executable instructions. The content may cause a computer to perform the various functions / operations described herein.

[0327] For example, all numerical values ​​used for the system are exemplary and may be other numerical values ​​and are not limited to the examples provided in this disclosure.

[0328] It should be understood that the elements shown in the various embodiments herein can be added, exchanged and / or eliminated to provide several additional embodiments of the present disclosure. In addition, the proportions and relative scales of the elements provided in the figures are intended to illustrate certain embodiments of the present disclosure and should not be considered as intended to be limiting.

[0329] Although specific embodiments have been illustrated and described herein, it will be appreciated by those of ordinary skill in the art that arrangements calculated to achieve the same results may be substituted for the specific embodiments shown. The present disclosure is intended to cover adaptations or variations of one or more embodiments of the present disclosure. It should be understood that the above description has been made in an illustrative manner and not in a restrictive manner.

[0330] Combinations of the above embodiments and other embodiments not explicitly described herein will be apparent to those skilled in the art after reviewing the above description. The scope of one or more embodiments of the present disclosure includes other applications in which the above structures and processes are used. Therefore, the scope of one or more embodiments of the present disclosure should be determined with reference to the attached claims and the full range of equivalents to which such claims are entitled.

[0331] In the Detailed Description, some features are grouped together in a single embodiment for the purpose of simplifying the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the disclosed embodiments of the disclosure must use more features than are expressly recited in each claim. Rather, as the appended claims reflect, inventive subject matter lies in less than all features of a single disclosed embodiment. Therefore, the appended claims are hereby incorporated into the Detailed Description, with each claim standing on its own as a separate embodiment.

Claims

1. A system comprising: at least one group of memory devices, wherein for a destination address in a memory request, one of the devices is a target device and another of the devices is a parity device; At least one group of memory devices and at least one host and comprising: A memory switch comprising: at least one fault-tolerance engine coupled to the at least one group of memory devices, configured to perform: (i) read access and (ii) read recovery when one of the memory devices fails during the read access; or (i) a write access including atomically updating data of the target device and parity data of the parity device and (ii) a write recovery when one of the memory devices fails during the write access; and At least one routing processor (i) is coupled to the memory switch and (ii) is configured to determine a path for the memory request received from the at least one host to the target device and direct the memory request to the target device.

2. The system of claim 1 , wherein the read recovery is performed when one of the memory devices is corrupted, the at least one fault-tolerant engine atomically reconstructing the data by: locking a location in the memory device associated with the parity data to provide fault tolerance to prevent subsequent read or write requests from accessing the parity data; and The lock location in the memory device is unlocked when the rebuild operation is complete, enabling the subsequent read or write request to be performed.

3. The system of claim 2, wherein the rebuild operation performs an atomic read access operation comprising: If the target device responds to the read access request from the requesting host with a valid data response, forwarding the requested data to the requesting host; or If the target device responds to the read access request with a poison response or a timeout response, then performing an atomic read access operation via the fault tolerance engine, the performing comprising: locking a cache line of the parity device; reading existing data from data rows of other data devices of the at least one group; calculating a regeneration target row value by performing an exclusive OR (XOR) operation on the check data of the parity check device and the data of the data row from the other data device; writing the regenerated target row value to the target device; sending the regeneration target row value to the requesting host; and The check data is written to the cache line and the cache line is unlocked.

4. The system of claim 1 , wherein when one of the memory devices fails during the write access, the fault-tolerant engine is configured to perform an atomic write access by: locking the entire parity row group containing the target device to prevent subsequent write access requests to the target device from accessing the target device while the fault-tolerant engine is writing to the target device; and The entire parity row group is unlocked when the write access is completed, enabling the subsequent write access request to be performed.

5. The system of claim 4, wherein the fault-tolerant engine is configured to generate new data by performing the atomic write access and partial cache line write operations, Wherein when the new data is written to the target device, the partial cache line write operation prevents a portion of existing data from being overwritten by the new data.

6. The system according to claim 5, further comprising: a hot spare device that is pre-configured such that metadata marks the entire row of the hot spare device as invalid; software means coupled to the memory containing instructions for controlling the operation of at least one fabric; wherein the software device receives notification of a single device failure from the fault-tolerant engine in response to the single device failure being detected by the fault-tolerant engine; and Wherein the software means replaces a single failed device with the hot spare device so that subsequent read and write accesses reproduce data previously stored on the single failed device to the hot spare device.

7. The system of claim 6, wherein the fault tolerance engine is configured to perform a rebuild operation to reconstruct data in response to detecting (i) a failure of a single memory device per device group or (ii) a failure of one or more switch links connected to the memory device.

8. The system of claim 1, further comprising: a hot spare device that is pre-configured such that metadata marks the entire row of the hot spare device as invalid; a fabric manager coupled to the at least one fabric and configured to manage operations of the at least one fabric; and The fabric manager is configured to decide to replace a failed device with the hot spare device.

9. The system of claim 1, further comprising: a fabric manager coupled to the at least one fabric and configured to manage operations of the at least one fabric; at least one device configured as a self-diagnostic data device that sends a ready-to-be-replaced warning to the fabric manager when one or more detected errors exceed a predetermined threshold indicating an impending failure; In response to the alert, the fabric manager is configured to: Preemptively add hot spare devices as new data devices; updating the fault-tolerance engines, wherein each fault-tolerance engine maintains a record of the hot spare device locations of the device group; instructing all tenants in the device group to perform hot spare device data reconstruction to rebuild the data to the new data device before the impending failure occurs by each tenant reading the location of its corresponding shared memory device; Updating the fault-tolerant engine; and Remove the device ready to be replaced.

10. A method using one or more processors and one or more memories, the method comprising: providing at least one group of memory devices; wherein for a destination address in a memory request, one of the devices is a target device and another of the devices is a parity device; and At least one fabric is provided that (i) is coupled to the at least one group of memory devices and at least one host and (ii) includes a memory switch including at least one fault tolerance engine coupled to each of the memory devices in the at least one group of memory devices, configured to perform: (i) read access and (ii) read recovery when one of the memory devices fails during the read access; or (i) a write access including atomically updating data of the target device and parity data of the parity device and (ii) a write recovery when one of the memory devices fails during the write access; and At least one routing processor is provided that is (i) coupled to the memory switch and (ii) configured to determine a path for the memory request received from the at least one host to the target device and direct the memory request to the target device.

Images