Consistency with nodes storing relevant data

Abstract

The present application relates to node consistency related to storage-related data. A system includes: a plurality of nodes; a first memory device including a plurality of volumes each visible to at least one of the plurality of nodes within a visibility configuration; and a processing device operatively coupled to the plurality of nodes and the first memory device. The processing device performs operations including: identifying a system failure that affects visibility of at least one of the plurality of volumes of the first memory device; and modifying the visibility configuration to address the system failure.

Description

Technical Field

[0001] Embodiments of the present disclosure relate generally to memory subsystems, and more particularly, to node coherency of storage related data. Background Art

[0002] The memory subsystem may include one or more memory devices that store data. The memory devices may be, for example, non-volatile memory devices and volatile memory devices. In general, the host system may utilize the memory subsystem to store data at the memory devices and retrieve data from the memory devices. Summary of the invention

[0003] One aspect of the present disclosure relates to a system comprising: a plurality of nodes; a first storage device comprising a plurality of volumes, each volume being visible to at least one of the plurality of nodes within a visibility configuration, each of the plurality of volumes comprising a persistent memory region (PMR) and a namespace; and a processing device operatively coupled to the first storage device to perform operations comprising: identifying a system failure that affects visibility of at least one of the plurality of volumes of the first storage device to at least one of the plurality of nodes; and modifying the visibility configuration to address the system failure.

[0004] Another aspect of the present disclosure relates to a method, comprising: identifying, by a processing device, a system failure that affects visibility of at least one volume of a plurality of volumes of a first storage device, wherein each volume of the plurality of volumes is visible to at least one of a plurality of nodes within a visibility configuration and includes a persistent memory region (PMR) and a namespace; and modifying the visibility configuration to resolve the system failure.

[0005] Yet another aspect of the present disclosure relates to a non-transitory computer-readable storage medium comprising instructions that, when executed by a processing device, cause the processing device to perform operations comprising: initializing a visibility configuration to enable visibility of each of a plurality of volumes of a first storage device to at least one of a plurality of nodes, wherein the plurality of volumes include a first volume and a second volume that each include a persistent memory region (PMR) and a namespace, and wherein the plurality of nodes include a first node and a second node; and resolving a system failure that affects the visibility of at least one of the plurality of volumes of the first storage device by modifying the visibility configuration to obtain a modified visibility configuration. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The present disclosure will be more fully understood from the detailed description given below and the accompanying drawings of various embodiments of the present disclosure. However, the drawings should not be considered to limit the present disclosure to specific embodiments, but are only for explanation and understanding.

[0007] Figure 1 An example computing system including a memory subsystem according to some embodiments of the present disclosure is described.

[0008] Figure 2 is a flow chart of an example method of implementing node consistency with storage-related data according to some embodiments of the present disclosure.

[0009] Figure 3 is a block diagram of a high-level overview of an example system that may implement node consistency with stored related data according to some embodiments of the present disclosure.

[0010] Figure 4 is a block diagram of an example visibility configuration of a system that may implement node consistency with stored related data according to some embodiments of the present disclosure.

[0011] Figure 5 is the implementation of some embodiments of the present disclosure and the node consistency of the storage related data Figure 4 Block diagram of the system.

[0012] Figure 6 is the implementation of some embodiments of the present disclosure and the node consistency of the storage related data Figure 4 Block diagram of the system.

[0013] Figure 7 is a block diagram of an example computer system in which embodiments of the present disclosure may operate. DETAILED DESCRIPTION

[0014] Various aspects of the present disclosure relate to node consistency related to storage-related data. The memory subsystem can be a storage device, a memory module, or a combination of a storage device and a memory module. Figure 1 Examples of storage devices and memory modules are described. Typically, a host system may utilize a memory subsystem that includes one or more components, such as memory devices, that store data. The host system may provide data for storage at the memory subsystem and may request retrieval of data from the memory subsystem.

[0015] The memory subsystem may include a high density non-volatile memory device where it is desired that data be retained when no power is supplied to the memory device. An example of a non-volatile memory device is a NAND memory device. Figure 1Other examples of nonvolatile memory devices are described. A nonvolatile memory device is a package of one or more dies. Each die may be composed of one or more planes. For some types of nonvolatile memory devices (e.g., NAND devices), each plane is composed of a set of physical blocks. Each block is composed of a set of pages. Each page is composed of a set of memory cells ("cells"). A cell is an electronic circuit that stores information. Depending on the cell type, a cell can store one or more bits of binary information and have various logical states related to the number of bits stored. The logical state can be represented as a binary value, such as "0" and "1", or a combination of such values.

[0016] A memory device (e.g., a solid-state drive (SSD)) in a memory subsystem may be accessed via a logical device interface attached via a communication bus. An example of such a logical device interface for accessing a memory device is Non-Volatile Memory Express (NVMe). A node (e.g., a host system) may access a memory subsystem via, for example, a Peripheral Component Interconnect Express (PCIe) bus, and access to the memory devices of the memory subsystem may be referred to herein as PCIe NVMe. A memory subsystem (e.g., an SSD) may include memory resources corresponding to an area of ​​readable persistent memory referred to as a persistent memory region (PMR). More specifically, a PMR may be a byte-addressable memory region accessible via a base address register (BAR), and may be implemented as a power-protected volatile memory device (e.g., a dynamic random access memory (DRAM)). For example, an NVMe-connected memory device may include a PMR, and such a device is referred to as an NVMe PMR device.

[0017] Storage-related data, such as storage metadata, logs, check data, etc., may be of small size (e.g., a few bytes), may be written very frequently (e.g., every time data is written or every few data writes), and is extremely important (i.e., required to support other data). Such storage-related data is typically not stored in storage devices, at least because it takes a long time to traverse the entire storage stack, and it may not be practical to load into blocks (e.g., loading a few bytes into a 4KB block is a waste of space and efficiency). In practice, such storage-related data is typically stored in volatile memory (e.g., DRAM). Therefore, if a node dies, the storage-related data on the memory subsystem associated with the node may become inaccessible to any other node, and thus cause potential data loss or consistency vulnerabilities.

[0018] The above problems are solved by introducing a high availability / failover system, in which the memory subsystem is connected to multiple host systems at the same time (e.g., a dual-port method), so that the failure of one node can be overcome by at least one other node (e.g., via a failover process) picking up the data that the failed node was using at the time. For example, in the dual-port method, the second node can pick up the data that the first node was operating at the time. However, with the emergence of fast storage (e.g., SSD storage), such methods, although effective, have become impractical for the above storage-related data.

[0019] One solution to this impracticality is to use non-volatile dual in-line memory modules (NVDIMMs). More specifically, NVDIMMs can be mirrored so that the NVDIMMs are split between local nodes and remote (mirrored) nodes. For example, a 32GB NVDIMM can be split, with 16GB allocated to the local node and 16GB allocated to the remote node. To improve performance, each node can write its own portion using a network interface card (NIC) that supports remote direct memory access (RDMA). For example, data can be stored on an SSD and accessed from a port, and metadata can be replicated on the NVDIMM via NIC mirroring. In the event of a failure, the system can use the metadata in the NVDIMM to pick up data from the storage device by just picking up the remote node portion as well.

[0020] However, the NVDIMM failover model has some limitations. For example, NVDIMMs use dynamic random access memory (DRAM) slots, which are expensive assets. If a DRAM slot is used for NVDIMMs, the slot cannot be used for regular DRAM. In addition, while cross-node mirroring works well for two-node failover, larger systems may not be able to use mirroring efficiently (for example, a four-node cluster would require cross-replicas on three additional nodes, which is impractical). In addition, using RDMA-capable NICs can be expensive and may require a dedicated network that consumes multiple interface slots (such as PCIe slots). For example, a 1 rack unit (1U) server can only have one interface slot.

[0021] Aspects of the present disclosure address the above and other deficiencies by providing a memory subsystem that implements node consistency for storage-related data (e.g., storage metadata, logs, and / or check data). More specifically, aspects of the present disclosure provide a mechanism for exposing PMRs in the event of a system fault takeover. Node consistency can provide a fault takeover model that can replace other fault takeover models (e.g., NVDIMM mirroring). The memory device described herein (e.g., a memory device of an SSD) may include several volumes, each volume including a namespace and a PMR for storing storage-related data. In the initial configuration and in response to a system failure (e.g., a node failure and / or a switch domain failure), each volume may be visible to at least one node of the host system. The visibility of each volume can be provided to any number of nodes by modifying the initial visibility configuration without the need for replication.

[0022] The node consistency described herein may be an interface standard that is independent of the interface standard associated with the PMR. For example, the PMR may be associated with a first interface standard or a second interface standard. The second interface standard may be an interface standard that implements one or more alternative protocols supported by the first interface standard. In general, a processor port may be used to negotiate (e.g., auto-negotiate) to a protocol of the first interface standard or one or more alternative protocols. In one embodiment, the first interface standard is PCIe (e.g., PCIe NVMe), and the second interface standard is Compute Express Link (CXL). Typically, CXL is an interface standard that supports several protocols including CXL.io protocol, CXL.mem protocol, and CXL.cache protocol. PCIe services may run over the CXL.io protocol, and the CXL.mem and CXL.cache protocols may share a common link layer and transaction layer, so that the CXL protocol may be multiplexed and transmitted via the PCIe physical layer. One advantage of using the PCIe interface standard is that PCIe is known for clear rules and implementation details. However, disadvantages of PCIe embodiments include non-cacheability. For example, the Extensible Firmware Interface (EFI) or Basic Input / Output System (BIOS) and Operating System (OS) are not designed to map large amounts of content on uncacheable memory. As another example, data structures are more difficult to handle by being uncacheable, and the file system may not be able to map memory efficiently (and may still require special code). The advantage of using a more cacheable interface standard such as CXL is that mapping and sizing are not an issue.

[0023] Advantages of the present disclosure include, but are not limited to: reduced costs due to the lack of a dedicated NIC for cross traffic; since the memory does not belong to a node, a node failure will not include the memory, so no mirror copy is required; and reduced size required for RDMA due to the lack of mirrored PMR memory (e.g., if current RDMA can use 32GB, only 16GB will be required). In addition, the embodiments described herein can be linearly extended beyond the 2-node limit of mirroring at almost no additional cost. In contrast, for larger node clusters (e.g., 16 nodes), NIC mirroring may be extremely limited or impossible.

[0024] Figure 1 An example computing system 100 is illustrated that includes a memory subsystem 110 in accordance with some embodiments of the present disclosure. Memory subsystem 110 may include media such as one or more volatile memory devices (e.g., memory device 140), one or more non-volatile memory devices (e.g., memory device 130), or a combination of such devices.

[0025] The memory subsystem 110 may be a storage device, a memory module, or a combination of a storage device and a memory module. Examples of storage devices include solid-state drives (SSDs), flash drives, universal serial bus (USB) flash drives, embedded multimedia controllers (eMMC) drives, universal flash storage (UFS) drives, secure digital (SD) cards, and hard disk drives (HDDs). Examples of memory modules include dual in-line memory modules (DIMMs), small outline DIMMs (SO-DIMMs), and various types of non-volatile dual in-line memory modules (NVDIMMs).

[0026] The computing system 100 may be a computing device such as a desktop computer, a laptop computer, a network server, a mobile device, a vehicle (e.g., an airplane, drone, train, automobile, or other transportation vehicle), a device with Internet of Things (IoT) capabilities, an embedded computer (e.g., an embedded computer included in a vehicle, industrial equipment, or a networked commercial device), or such a computing device including a memory and a processing device.

[0027] The computing system 100 may include a host system 120 coupled to one or more memory subsystems 110. In some embodiments, the host system 120 is coupled to multiple memory subsystems 110 of different types. Figure 1An example of a host system 120 coupled to one memory subsystem 110 is illustrated. As used herein, "coupled to" or "coupled with" generally refers to a connection between components, which may be an indirect communication connection or a direct communication connection (e.g., without intervening components), whether wired or wireless, including connections such as electrical connections, optical connections, magnetic connections, etc.

[0028] The host system 120 may include a processor chipset and a software stack executed by the processor chipset. The processor chipset may include one or more cores, one or more caches, a memory controller (e.g., an NVDIMM controller), and a storage protocol controller (e.g., a PCIe controller, a SATA controller). The host system 120 uses the memory subsystem 110, for example, to write data to the memory subsystem 110 and read data from the memory subsystem 110.

[0029] The host system 120 may be coupled to the memory subsystem 110 through a physical node interface. Examples of the physical node interface include, but are not limited to, a serial advanced technology attachment (SATA) interface, a peripheral component interconnect express (PCIe) interface, a universal serial bus (USB) interface, a fiber channel, a serial attached SCSI (SAS), a double data rate (DDR) memory bus, a small computer system interface (SCSI), a dual in-line memory module (DIMM) interface (e.g., a DIMM socket interface supporting double data rate (DDR)), etc. The physical node interface may be used to transfer data between the host system 120 and the memory subsystem 110. When the memory subsystem 110 is coupled to the host system 120 through a physical node interface (e.g., a PCIe bus), the host system 120 may further utilize an NVM express (NVMe) interface to access components (e.g., the memory device 130). The physical node interface may provide an interface for transferring control, address, data, and other signals between the memory subsystem 110 and the host system 120. Figure 1 Memory subsystem 110 is illustrated as an example. In general, host system 120 can access multiple memory subsystems via the same communication connection, multiple separate communication connections, and / or a combination of communication connections.

[0030] The memory devices 130, 140 may include any combination of different types of non-volatile memory devices and / or volatile memory devices. Volatile memory devices (e.g., memory device 140) may be, but are not limited to, random access memory (RAM), such as dynamic random access memory (DRAM) and synchronous dynamic random access memory (SDRAM).

[0031] Some examples of non-volatile memory devices (e.g., memory device 130) include NAND-type flash memory and write-in-place memory, such as a three-dimensional cross-point ("3D cross-point") memory device, which is a cross-point array of non-volatile memory cells. The cross-point array of non-volatile memory cells can be combined with a stackable cross-grid data access array to perform bit storage based on changes in body resistance. In addition, compared to many flash-based memories, cross-point non-volatile memories can perform write-in-place operations, where non-volatile memory cells can be programmed without pre-erasing the non-volatile memory cells. NAND-type flash memory includes, for example, two-dimensional NAND (2D NAND) and three-dimensional NAND (3D NAND).

[0032] Each of the memory devices 130 may include one or more memory cell arrays. One type of memory cell, such as a single-level cell (SLC), may store one bit per cell. Other types of memory cells, such as multi-level cells (MLC), three-level cells (TLC), four-level cells (QLC), and five-level cells (PLC) may store multiple bits per cell. In some embodiments, each of the memory devices 130 may include one or more memory cell arrays, such as SLC, MLC, TLC, QLC, PLC, or any combination of these. In some embodiments, a particular memory device may include an SLC portion of a memory cell, as well as an MLC portion, a TLC portion, a QLC portion, or a PLC portion. The memory cells of the memory device 130 may be grouped into pages, which may refer to a logical unit of a memory device for storing data. For some types of memory (e.g., NAND), pages may be grouped to form blocks.

[0033] Although non-volatile memory components such as a 3D cross-point array of non-volatile memory cells and NAND-type flash memory (e.g., 2D NAND, 3D NAND) are described, the memory device 130 may be based on any other type of non-volatile memory, such as read-only memory (ROM), phase-change memory (PCM), self-selected memory, other chalcogenide-based memory, ferroelectric transistor random access memory (FeTRAM), ferroelectric random access memory (FeRAM), magnetic random access memory (MRAM), spin transfer torque (STT)-MRAM, conductive bridging RAM (CBRAM), resistive random access memory (RRAM), oxide-based RRAM (OxRAM), or non-(NOR) flash memory, or electrically erasable programmable read-only memory (EEPROM).

[0034] The memory subsystem controller 115 (or, for simplicity, controller 115) may communicate with the memory device 130 to perform operations, such as reading data, writing data, or erasing data at the memory device 130, as well as other such operations. The memory subsystem controller 115 may include hardware, such as one or more integrated circuits and / or discrete components, buffer memory, or a combination thereof. The hardware may include digital circuitry with dedicated (i.e., hard-coded) logic to perform the operations described herein. The memory subsystem controller 115 may be a microcontroller, dedicated logic circuitry (e.g., a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), etc.), or other suitable processor.

[0035] The memory subsystem controller 115 may include a processing device including one or more processors (e.g., processor 117) configured to execute instructions stored in a local memory 119. In the illustrated example, the local memory 119 of the memory subsystem controller 115 includes an embedded memory configured to store instructions for performing various processes, operations, logic flows, and routines to control the operation of the memory subsystem 110, including handling communications between the memory subsystem 110 and the host system 120.

[0036] In some embodiments, local memory 119 may include memory registers that store memory pointers, fetched data, etc. Local memory 119 may also include read-only memory (ROM) for storing microcode. Figure 1 The example memory subsystem 110 in FIG. 1 is illustrated as including a memory subsystem controller 115, but in another embodiment of the present disclosure, the memory subsystem 110 does not include a memory subsystem controller 115 and may instead rely on external control (e.g., provided by an external node or by a processor or controller separate from the memory subsystem).

[0037] In general, the memory subsystem controller 115 may receive commands or operations from the host system 120, and may convert the commands or operations into instructions or appropriate commands to achieve the desired access to the memory device 130. The memory subsystem controller 115 may be responsible for other operations such as wear leveling operations, garbage collection operations, error detection and correction code (ECC) operations, encryption operations, cache operations, and address translation between logical addresses (e.g., logical block addresses (LBA), name space) and physical addresses (e.g., physical block addresses) associated with the memory device 130. The memory subsystem controller 115 may also include node interface circuitry to communicate with the host system 120 via a physical node interface. The node interface circuitry may convert commands received from the host system into command instructions to access the memory device 130, and convert responses associated with the memory device 130 into information for the host system 120.

[0038] The memory subsystem 110 may also include additional circuitry or components not illustrated. In some embodiments, the memory subsystem 110 may include a cache or buffer (e.g., DRAM) and address circuitry (e.g., row decoders and column decoders) that may receive addresses from the memory subsystem controller 115 and decode the addresses to access the memory device 130.

[0039] In some embodiments, the memory device 130 includes a local media controller 135 that operates in conjunction with the memory subsystem controller 115 to perform operations on one or more memory cells of the memory device 130. An external controller (e.g., the memory subsystem controller 115) can manage the memory device 130 externally (e.g., perform media management operations on the memory device 130). In some embodiments, the memory subsystem 110 is a managed memory device, which is a raw memory device 130 with control logic (e.g., the local controller 135) on the die and a controller (e.g., the memory subsystem controller 115) for media management within the same memory device package. An example of a managed memory device is a managed NAND (MNAND) device.

[0040] The node consistency (NC) component 113 can achieve node consistency with storage-related data (e.g., storage metadata, logs, and / or check data) by providing a mechanism to expose the PMR in a failover situation. More specifically, the NC component 113 can initialize a visibility configuration to achieve visibility of each of the plurality of volumes of the first storage device to at least one of the plurality of nodes (e.g., a host system), identify a system failure that affects the visibility of at least one of the plurality of volumes of the first storage device, and modify the visibility configuration to address the system failure. For example, identifying a system failure can include identifying a failure of a node in the plurality of nodes. As another example, identifying a system failure can include identifying a failure of a switch domain. The operations performed by the NC component 113 can be performed in an interface standard agnostic manner. For example, the interface standard can be a first interface standard or a second interface standard. The second interface standard can be an interface standard that implements one or more alternative protocols supported by the first interface standard. In one embodiment, the first interface standard is PCIe (e.g., PCIeNVMe) and the second interface standard is CXL. For example, the namespace of the volume may be accessed via PCIe or CXL.io, and the PMR of the volume may be accessed via, for example, CXL.mem. In addition, the two ports may each be connected to a different switch domain, with each port running PCIe and / or CXL according to the initial visibility configuration. In some embodiments, the memory subsystem controller 115 includes at least a portion of the NC component 113. In some embodiments, the NC component 113 is part of the host system 120, an application, or an operating system. In other embodiments, the local media controller 135 includes at least a portion of the NC component 113 and is configured to perform the functions described herein. Additional details regarding the operation of the NC component 113 are described below.

[0041] Figure 2 2 is a flow chart of an example method 200 for implementing node consistency with storage-related data according to some embodiments of the present disclosure. The method 200 may be performed by processing logic, which may include hardware (e.g., a processing device, a circuit system, a dedicated logic, a programmable logic, a microcode, hardware of a device, an integrated circuit, etc.), software (e.g., instructions running or executed on a processing device), or a combination thereof. In some embodiments, the method 200 is performed by Figure 1 113 of the NC component 113. Although shown in a particular order or sequence, the order of the processes may be modified unless otherwise specified. Therefore, the illustrated embodiments should be understood as examples only, and the illustrated processes may be performed in a different order, and some processes may be performed in parallel. In addition, one or more processes may be omitted in various embodiments. Therefore, not all processes are required in every embodiment. Other process flows are also possible.

[0042] At operation 210, processing logic initializes a visibility configuration to enable visibility of each of a plurality of volumes of a first storage device of the system to at least one of a plurality of nodes, wherein each of the plurality of volumes of the first storage device includes a namespace and a PMR. For example, the node may include a host system. That is, a visibility configuration is an allocation that exposes a volume to a corresponding node. The visibility configuration may be initialized by a configuration command. For example, in one visibility configuration, certain volumes of the first storage device are exposed to certain nodes (i.e., visible and accessible to those nodes for the purpose of programming and / or reading data), while in another visibility configuration, different volumes may be exposed to the same or different nodes, or the same volume may be exposed to different nodes. The PMR may be configured to store storage-related data (e.g., store metadata, logs, and / or check data). The PMR and namespace of a given volume are processed as atomic, meaning that the PMR and namespace of a given volume are not individually reallocated to another node. Therefore, the logic that manages the failover process may move or switch the namespace and PMR in the same command. In one embodiment, the first memory device is an SSD device (e.g., an NVMe PMR device). The PMR may be associated with any suitable interface standard. For example, the PMR may be associated with a first interface standard or a second interface standard. The second interface standard may be an interface standard that implements one or more alternative protocols supported by the first interface standard. In one embodiment, the first interface standard is PCIe (e.g., PCIe NVMe) and the second interface standard is CXL.

[0043] At operation 220, processing logic identifies a system failure that affects visibility of at least one of the plurality of volumes of the first storage device. As will be described in further detail below, one example of a system failure is a failed switch domain, and another example of a system failure is a failed node. The system failure may be identified using any suitable method for system failure detection. For example, if communications from a system component (e.g., a switch domain, a node) stop being received, the system may receive a signal that the system component has failed (e.g., using a heartbeat monitoring technique).

[0044] At operation 230, processing logic modifies the visibility configuration to account for the system failure. More specifically, processing logic may modify the visibility of the at least one volume of the plurality of volumes of the first storage device through other ports within the system. For example, modifying the visibility configuration may include: in response to identifying a node failure to which visibility is assigned to the at least one volume, determining that another node should take over, and using an administrator configuration command to tell the first storage device to expose the namespace and PMR of the at least one volume to the other node (e.g., two commands). The other node may then detect the hot-added storage to run the hot-swap stack, and may resume transactions and I / O after completion.

[0045] In one embodiment, the plurality of nodes include a first node and a second node, the plurality of volumes of the first storage device include a first volume and a second volume, and the visibility configuration includes visibility of the first volume to the first node and visibility of the second volume to the second node. At operation 222, identifying the system failure may include identifying a failure of the first node. Modifying the visibility configuration to address the system failure at operation 230 may include making the first volume visible to the second node.

[0046] In one embodiment, the system further includes a switch domain, the plurality of nodes includes a first node and a second node, and the visibility configuration includes visibility of at least a given volume of the plurality of volumes of the first storage device to the first node through the switch domain. At operation 222, identifying the system failure may include identifying a failure of the first node. Modifying the visibility configuration to address the system failure at operation 230 may include making the given volume visible to the second node via the switch domain.

[0047] In one embodiment, the system further includes a first switch domain and a second switch domain. The plurality of nodes includes a node having a first port and a second port, and the plurality of volumes of the first memory device includes a first volume and a second volume. The visibility configuration includes visibility of the first volume to the node through the first port of the node via the first switch domain and visibility of the second volume to the node through the second port of the node via the second switch domain. At operation 224, identifying a system failure may include identifying a failure of the first switch domain. Modifying the visibility configuration to resolve the system failure at operation 230 may include making the first volume visible to the node through the second port of the node via the second switch domain.

[0048] In some embodiments, the system further includes a first switch domain and a second switch domain and a second storage device, the second storage device including a plurality of volumes each having a PMR and a namespace. The plurality of nodes include a first node, a second node, and a third node each having a first port and a second port, the plurality of volumes of the first storage device include a first volume and a second volume, and the plurality of volumes of the second storage device include a third volume and a fourth volume. The visibility configuration includes visibility of the first volume and the second volume to the first node via corresponding switch domains in the first and second switch domains through corresponding ports of the first and second ports of the first node, visibility of the third volume to the second node via the first switch domain through the first port of the second node, and visibility of the fourth volume to the third node via the second switch domain through the second port of the third node.

[0049] In one embodiment, identifying the system failure may then include identifying a failure of the first node at operation 222. Modifying the visibility configuration to address the system failure at operation 230 includes making the first volume visible to the second node through the second port of the second node via the second switch domain, and making the second volume visible to the third node through the first port of the third node via the first switch domain.

[0050] In another embodiment, identifying the system failure may then include identifying a failure of the first switch domain at operation 224. Modifying the visibility configuration to address the system failure at operation 230 includes making the first volume visible to the first node through the second port of the first node via the second switch domain, and making the third volume visible to the second node through the second port of the second node via the second switch domain.

[0051] Figure 3 is a block diagram of a high-level overview of a system 300 that may implement node consistency with stored related data according to some embodiments of the present disclosure.

[0052] System 300 may include several nodes (e.g., host systems), including node 310-1 and node 310-2, and memory device 320. For example, memory device 320 may be an SSD. Node 310-1 and node 310-2 are attached to memory device 320 at respective ports (not shown). Memory device 320 includes several volumes, including volume 322-1 and volume 322-2. Each volume includes a corresponding PMR and namespace. For example, volume 322-1 includes PMR 324-1 and namespace 326-1, and volume 322-2 includes PMR 324-2 and namespace 326-2. The PMR and namespace of a given volume are processed as atomic, meaning that the PMR and namespace of a given volume are not individually reallocated to another node. Therefore, the logic managing the failover process can move or switch namespaces and PMRs in the same command.

[0053] Figure 3The "1 PMR per namespace" arrangement illustratively shown in provides various advantages over the conventional "1 NVDIMM per node" arrangement. For example, each of PMRs 324-1 and 324-2 is expressed as an entity owned by memory device 320 and associated with each corresponding namespace 326-1 and 326-2. This avoids the following situation in the NVDIMM approach: the node owns the memory, which forces the mirror to go to each owner node. Since the memory is not owned by either node 310-1 or 310-2, a node failure will not damage the memory (i.e., no mirrored copy is required). Since there is only one PMR per namespace, exposure to any number of nodes can be achieved without replication. This is in contrast to the problem of NVDIMM mirroring across multiple nodes and extending out to many nodes. In addition, not having to mirror the PMR memory can reduce the required size by half (for example, if RDMA uses 32GB, only 16GB is now required). If data protection of the entire PMR is required for additional attention, PMR data protection can adopt a similar model as storage data protection.

[0054] For simplicity, assume that there is no sharing (e.g., volumes 322-1 and 322-2 are not visible) and that only nodes 310-1 and 310-2 exist. In the non-sharing case, node 310-1 will only see PMR 324-1 and namespace 326-1, and node 310-2 will only see PMR 324-2 and 326-2. Each of node 310-1 and node 310-2 can operate on the visible volumes without having to coordinate or mirror copies on the other of node 310-2 and node 310-1, respectively.

[0055] Assume that node 310-1 fails. Node 310-2 will have to pick up namespaces 326-1 and 326-2 and the contents of PMR 324-1. In NVDIMM implementations, this means referencing the internal NVDIMM mirror copy of node 310-2. According to the embodiments described herein, as mentioned above, the same process that would allow namespace failover through dual ports can then expose the same PMR 324-1 through the second port. No mirroring is required, and there is the same data integrity as the reference data.

[0056] Extending to the shared volume case, there is no failover at the storage level since all volumes are visible through all ports. A node may then pick up a node failure in response to detecting, for example, a "lack of heartbeat" from another node.

[0057] Assume that there are N nodes or systems exposed through N ports, which are connected to storage devices (e.g., SSDs) with N volumes each containing a PMR and a namespace (e.g., through PCIe or NVMe over Fabrics (NVMe-oF) switches). In the optimal case, the i-th node (node i ) can use any of the above mechanisms to connect to the corresponding i-th PMR and i-th namespace component. i If a failure occurs, any other mth node (node m , where m≠i) can pick up the i-th PMR and the i-th namespace by taking over the i-th namespace and the i-th PMR through the existing failover as described above.

[0058] Reference will now be made to Figure 4-6 Describes additional details about a system that can implement node consistency.

[0059] Figure 4 is a block diagram of a visibility configuration of a system 400 that may implement node consistency with stored related data according to some embodiments of the present disclosure.

[0060] As shown, system 400 includes several nodes (eg, host systems), including nodes 410-1, 410-2, and 410-3. Figure 4 Three nodes are shown in FIG. , but the system may include any suitable number of nodes according to the embodiments described herein.

[0061] Each node includes a pair of ports. For example, node 410-1 includes port 412-1 and port 413-1, node 410-2 includes port 412-2 and port 413-2, and node 410-3 includes port 412-3 and port 414-3. More specifically, each port may correspond to an interface standard. In one embodiment, the interface standard is PCIe. In another embodiment, the interface standard is CXL. However, such embodiments should not be considered limiting.

[0062] System 400 also includes a number of memory devices ("devices"), including device 420-1 and device 420-2. In one embodiment, the memory devices may include SSDs. For example, the memory devices may include NVMe SSD devices. Figure 4 Two memory devices are shown in FIG. 1 , but a system may include any suitable number of memory devices according to embodiments described herein.

[0063] Each of the memory devices includes a number of volumes, wherein each volume includes a port operatively coupled to a PMR and a name space (NS). For example, device 420-1 includes: volume 421-1, which includes a port 422-1 operatively coupled to PMR 423-1 and NS 424-1; and volume 425-1, which includes a port 426-1 operatively coupled to PMR 427-1 and NS 428-1. Device 420-2 includes: volume 421-2, which includes a port 422-2 operatively coupled to PMR 423-2 and NS 424-2; and volume 425-2, which includes a port 426-2 operatively coupled to PMR 427-2 and NS 428-2. More specifically, each port may correspond to an interface standard (e.g., PCIe or CXL). However, such embodiments should not be considered limiting.

[0064] The system 400 also includes a number of switch domains associated with the interface standards, including switch domains 430-1 and 430-2. More specifically, the switch domains may include PCIe switch domains and / or CXL switch domains. However, such embodiments should not be considered limiting. Figure 4 As further shown in FIG. 4 , ports 412-1 through 412-3, 422-1, and 422-2 are operatively coupled to switch domain 430-2, and ports 413-1 through 413-3, 426-1, and 426-2 are operatively coupled to switch domain 430-1.

[0065] Figure 5 is a block diagram of a system 500 for implementing node consistency with stored related data according to some embodiments of the present disclosure. More specifically, in this illustrative example, assume that Figure 4 430-1 of switch domain 430-2 has failed, resulting in failed switch domain 510. The paths from ports 413-1 to 414-3, 423-1, and 423-2 to failed switch domain 510 have correspondingly failed, as indicated by bold line 520. As a result, volumes 421-1, 421-2, 424-1, and 424-2 are visible through the remaining path 415 of switch domain 430-2. More specifically, volume 424-2 has been reassigned to port 412-2, such that port 412-2 now has two volume assignments (i.e., volume 424-2 and previously assigned volume 421-2, as described above with reference to FIG. Figure 4 described).

[0066] Figure 6 is a block diagram of a system 600 for implementing node consistency with stored related data according to some embodiments of the present disclosure. More specifically, in this illustrative example, assume that Figure 4Node 410-2 of the failed node 610 fails, resulting in a failed node 610. The path from ports 412-2 and 413-2 of failed node 610 is marked by a bold line 620. Figure 4 As mentioned, volumes 421-2 and 424-2 are visible to node 410-2 when it is functioning properly. Now that node 410-2 has failed and has become failed node 610, a reconfiguration occurs to make volumes 421-2 and 424-2 visible to an alternative node. For example, volume 421-2 may be reassigned to port 412-1 to be visible to node 410-1, and volume 424-2 may be reassigned to port 413-3 to be visible to node 410-3. However, such reconfiguration is purely exemplary and should not be considered limiting.

[0067] Figure 7 An example machine illustrating a computer system 700 in which a set of instructions may be executed for causing the machine to perform any one or more of the methods discussed herein. In some embodiments, the computer system 700 may correspond to a host system (e.g., Figure 1 120) that includes, is coupled to, or utilizes a memory subsystem (e.g., Figure 1 110), or can be used to perform operations of the controller (for example, execute an operating system to execute corresponding Figure 1 In some embodiments, the machine may be connected (e.g., using a network) to other machines in a LAN, an intranet, an extranet, and / or the Internet. The machine may operate in the capacity of a server or a client machine in a client-server network environment, or in the capacity of a peer machine in a peer-to-peer (or distributed) network environment, or as a server or a client machine in a cloud computing infrastructure or environment.

[0068] The machine may be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular phone, a network appliance, a server, a network router, a switch or a bridge, or any machine capable of executing (sequentially or otherwise) a set of instructions that specify actions to be taken by the machine. Further, while a single machine is described, the term "machine" should also be construed to include any collection of machines that individually or collectively execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.

[0069] The example computer system 700 includes a processing device 702, a main memory 704 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM), such as synchronous DRAM (SDRAM) or RDRAM, etc.), a static memory 706 (e.g., flash memory, static random access memory (SRAM), etc.), and a data storage system 718, which communicate with each other via a bus 730.

[0070] The processing device 702 represents one or more general-purpose processing devices, such as a microprocessor, a central processing unit, etc. More specifically, the processing device can be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor that implements other instruction sets, or a processor that implements a combination of instruction sets. The processing device 702 can also be one or more special-purpose processing devices, such as an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a network processor, etc. The processing device 702 is configured to execute instructions 726 for performing the operations and steps discussed herein. The computer system 700 can also include a network interface device 708 to communicate through a network 720.

[0071] The data storage system 718 may include a machine-readable storage medium 724 (also referred to as a computer-readable medium) on which is stored one or more sets of instructions 726 or software embodying any one or more of the methodologies or functions described herein. The instructions 726 may also reside, completely or at least partially, within the main memory 704 and / or within the processing device 702 during execution thereof by the computer system 700, the main memory 704 and the processing device 702 also constituting machine-readable storage media. The machine-readable storage medium 724, the data storage system 718, and / or the main memory 704 may correspond to Figure 1 Memory subsystem 110.

[0072] In one embodiment, instructions 726 include instructions for implementing a node consistency component (e.g. Figure 1 Although the machine-readable storage medium 724 is shown as a single medium in the example embodiment, the term "machine-readable storage medium" should be considered to include a single medium or multiple media storing one or more sets of instructions. The term "machine-readable storage medium" should also be considered to include any medium capable of storing or encoding a set of instructions for execution by a machine and causing the machine to perform any one or more methods of the present disclosure. Therefore, the term "machine-readable storage medium" should be considered to include, but not limited to, solid-state memory, optical media, and magnetic media.

[0073] Some portions of the previous detailed description have been presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means by which those skilled in the art of data processing most effectively convey the substance of their work to others skilled in the art. An algorithm is considered here and generally to be a self-consistent sequence of operations that produce a desired result. The operations are those that require physical manipulation of physical quantities. These quantities are typically, but not necessarily, in the form of electrical or magnetic signals that can be stored, combined, compared, and otherwise manipulated. It has proven to be convenient at times, primarily for general reasons, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, etc.

[0074] It should be borne in mind, however, that all of these and similar terms should be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. The present disclosure may refer to the actions and processes of a computer system or similar electronic computing device that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system's memories or registers or other such information storage systems.

[0075] The present disclosure also relates to an apparatus for performing the operations described herein. This apparatus may be specially constructed for the intended purpose, or it may comprise a general purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a computer-readable storage medium, such as, but not limited to, any type of disk (including floppy disks, optical disks, CD-ROMs, and magnetic optical disks), read-only memory (ROM), random access memory (RAM), EPROM, EEPROM, magnetic or optical cards, or any type of medium suitable for storing electronic instructions, each medium coupled to a computer system bus.

[0076] The algorithms and displays presented herein are not inherently related to any particular computer or other device. Various general purpose systems may be used with programs according to the teachings herein, or it may prove convenient to build more specialized equipment to perform the methods. The structures of a variety of these systems will be presented as set forth in the description below. In addition, the present disclosure is not described with reference to any particular programming language. It should be appreciated that the teachings of the present disclosure described herein may be implemented using a variety of programming languages.

[0077] The present disclosure may be provided as a computer program product or software, which may include a machine-readable medium having instructions stored thereon, which may be used to program a computer system (or other electronic device) to perform a process according to the present disclosure. A machine-readable medium includes any mechanism for storing information in a form readable by a machine (e.g., a computer). In some embodiments, a machine-readable (e.g., computer-readable) medium includes a machine (e.g., computer) readable storage medium, such as a read-only memory ("ROM"), a random access memory ("RAM"), a magnetic disk storage medium, an optical storage medium, a flash memory component, etc.

[0078] In the foregoing description, embodiments of the present disclosure have been described with reference to specific example embodiments thereof. It should be apparent that various modifications may be made to the present disclosure without departing from the broader spirit and scope of embodiments of the present disclosure as set forth in the appended claims. Accordingly, the specification and drawings should be viewed in an illustrative rather than a restrictive sense.

Claims

1. A memory system, comprising: A plurality of dual-port nodes, each of the plurality of dual-port nodes comprising a first port and a second port; a plurality of switch domains, including a first switch domain and a second switch domain; a first memory device comprising a first plurality of volumes, each volume of the first plurality of volumes being visible to at least one of the plurality of dual-port nodes within a visibility configuration, and each volume of the first plurality of volumes comprising a persistent memory region PMR and a namespace, wherein the first plurality of volumes comprises a first volume and a second volume, and wherein the visibility configuration comprises visibility of at least one of the first volume and the second volume to at least one of the plurality of dual-port nodes via at least one of the plurality of switch domains; as well as a processing device operatively coupled to the first memory device to perform operations comprising: identifying a system failure that affects visibility of at least one of the first volume and the second volume of the first memory device to the at least one dual-port node of the plurality of dual-port nodes, wherein identifying the system failure comprises identifying at least one of the at least one dual-port node of the plurality of dual-port nodes or the at least one switch domain; as well as and modifying the visibility configuration to address the system failure, wherein modifying the visibility configuration comprises modifying the visibility of at least one of the first volume and the second volume to the at least one dual-port node through a first port or a second port of the at least one dual-port node of the plurality of dual-port nodes via the at least one switch domain of the plurality of switch domains.

2. The memory system of claim 1, wherein: The visibility configuration includes visibility of the first volume to a first dual-port node of the plurality of dual-port nodes and visibility of the second volume to a second dual-port node of the plurality of dual-port nodes; Identifying the system fault comprises identifying a fault of the first dual-port node; as well as Modifying the visibility configuration to account for the system failure includes making the first volume visible to the second dual-port node.

3. The memory system of claim 1, wherein: The visibility configuration includes visibility of a given volume of the first plurality of volumes to a first dual-port node of the plurality of dual-port nodes via the at least one switch domain; Identifying the system fault comprises identifying a fault of the first dual-port node; as well as Modifying the visibility configuration to account for the system failure includes making the given volume visible to a second dual-port node of the plurality of dual-port nodes via the at least one switch domain.

4. The memory system of claim 1, wherein: the visibility configuration comprising visibility of the first volume to a given dual-port node of the plurality of dual-port nodes through a first port of the given dual-port node via the first switch domain and visibility of the second volume to the given dual-port node through a second port of the given dual-port node via the second switch domain; Identifying the system failure includes identifying a failure of the first switch domain; as well as Modifying the visibility configuration to account for the system failure includes making the first volume visible to the given dual-port node through a second port of the given dual-port node via the second switch domain.

5. The memory system according to claim 1, further comprising: a second storage device comprising a second plurality of volumes, each volume in the second plurality of volumes having a PMR and a namespace; in: The plurality of dual-port nodes include a first dual-port node, a second dual-port node, and a third dual-port node; the second plurality of volumes includes a third volume and a fourth volume; and The visibility configuration includes visibility of the first volume and the second volume to the first dual-port node through corresponding ones of the first switch domain and the second switch domain through a first port of the first dual-port node and a corresponding one of the second port of the first dual-port node, visibility of the third volume to the second dual-port node through the first port of the second dual-port node via the first switch domain, and visibility of the fourth volume to the third dual-port node through the second port of the third dual-port node via the second switch domain.

6. The memory system of claim 5, wherein: Identifying the system fault comprises identifying a fault of the first dual-port node; and Modifying the visibility configuration to address the system failure includes making the first volume visible to the second dual-port node through the second port of the second dual-port node via the second switch domain, and making the second volume visible to the third dual-port node through the first port of the third dual-port node via the first switch domain.

7. The memory system of claim 5, wherein: Identifying the system failure includes identifying a failure of the first switch domain; as well as Modifying the visibility configuration to address the system failure includes making the first volume visible to the first dual-port node through a second port of the first dual-port node via the second switch domain, and making the third volume visible to the second dual-port node through a second port of the second dual-port node via the second switch domain.

8. A method for operating a memory system, comprising: identifying, by a processing device, a system failure that affects visibility of at least one volume of a first plurality of volumes of a first memory device of the memory system to at least one dual-port node of a plurality of dual-port nodes of the memory system, wherein the first plurality of volumes includes a first volume and a second volume, wherein each dual-port node of the plurality of dual-port nodes includes a first port and a second port, wherein each volume of the plurality of volumes of the first memory device is visible to the at least one dual-port node of the plurality of dual-port nodes within a visibility configuration, wherein the visibility configuration includes visibility of at least one of the first volume and the second volume to the at least one dual-port node of the plurality of dual-port nodes via at least one switch domain of a plurality of switch domains of the memory system, wherein the plurality of switch domains includes a first switch domain and a second switch domain, and wherein each volume of the first plurality of volumes includes a persistent memory region (PMR) and a namespace; and The visibility configuration is modified, by the processing device, to address the system failure, wherein modifying the visibility configuration includes modifying visibility of at least one of the first volume and the second volume to the at least one dual-port node through the at least one switch domain of the plurality of switch domains via a first port or a second port of the at least one dual-port node of the plurality of dual-port nodes.

9. The method according to claim 8, wherein: The visibility configuration includes visibility of the first volume to a first dual-port node of the plurality of dual-port nodes and visibility of the second volume to a second dual-port node of the plurality of dual-port nodes; Identifying the system fault comprises identifying a fault of the first dual-port node; as well as Modifying the visibility configuration to account for the system failure includes making the first volume visible to the second dual-port node.

10. The method according to claim 8, wherein: the visibility configuration comprising visibility of a given volume of the first plurality of volumes of the first memory device to a first dual-port node of the plurality of dual-port nodes via the at least one switch domain; Identifying the system fault comprises identifying a fault of the first dual-port node; as well as Modifying the visibility configuration to account for the system failure includes making the given volume visible to a second dual-port node of the plurality of dual-port nodes via the at least one switch domain.

11. The method according to claim 8, wherein: the visibility configuration comprising visibility of the first volume to a given dual-port node of the plurality of dual-port nodes through a first port of the given dual-port node via the first switch domain and visibility of the second volume to the given dual-port node through a second port of the given dual-port node via the second switch domain; Identifying the system failure includes identifying a failure of the first switch domain; as well as Modifying the visibility configuration to account for the system failure includes making the first volume visible to the given dual-port node through the second port of the given dual-port node via the second switch domain.

12. The method according to claim 8, wherein: The plurality of dual-port nodes include a first dual-port node, a second dual-port node, and a third dual-port node; The storage system further includes a second storage device, the second storage device including a second plurality of volumes, each volume in the second plurality of volumes having a PMR and a namespace, and the second plurality of volumes including a third volume and a fourth volume; as well as The visibility configuration includes visibility of the first volume and the second volume to the first dual-port node through corresponding ones of the first switch domain and the second switch domain through corresponding ones of the first port and the second port of the first dual-port node, visibility of the third volume to the second dual-port node through the first port of the second dual-port node via the first switch domain, and visibility of the fourth volume to the third dual-port node through the second port of the third dual-port node via the second switch domain.

13. The method according to claim 12, wherein: Identifying the system fault comprises identifying a fault of the first dual-port node; and Modifying the visibility configuration to address the system failure includes making the first volume visible to the second dual-port node through the second port of the second dual-port node via the second switch domain, and making the second volume visible to the third dual-port node through the first port of the third dual-port node via the first switch domain.

14. The method of claim 12, wherein: Identifying the system failure includes identifying a failure of the first switch domain; and Modifying the visibility configuration to address the system failure includes making the first volume visible to the first dual-port node through a second port of the first dual-port node via the second switch domain, and making the third volume visible to the second dual-port node through a second port of the second dual-port node via the second switch domain.

15. A non-transitory computer-readable storage medium comprising instructions that, when executed by a processing device, cause the processing device to perform operations comprising: Initializing a visibility configuration to enable visibility of each of a first plurality of volumes of a first memory device of a system to at least one of a plurality of dual-port nodes of the system, wherein each of the plurality of dual-port nodes comprises a first port and a second port, wherein the first plurality of volumes comprises a first volume and a second volume, each of the first plurality of volumes of the first memory device comprises a persistent memory region PMR and a namespace, wherein the visibility configuration comprises visibility of at least one of the first volume and the second volume to the at least one of the plurality of dual-port nodes via at least one of a plurality of switch domains of the system, and wherein the plurality of switch domains comprises a first switch domain and a second switch domain; and addressing a system failure affecting the visibility of at least one volume of the first plurality of volumes by modifying the visibility configuration to obtain a modified visibility configuration, The modified visibility configuration includes modified visibility of at least one of the first volume and the second volume to the at least one dual-port node through a first port or a second port of the at least one dual-port node of the plurality of dual-port nodes via at least one switch domain of the plurality of switch domains.

16. The non-transitory computer-readable storage medium of claim 15, wherein: The visibility configuration includes visibility of the first volume to a first dual-port node of the plurality of dual-port nodes and visibility of the second volume to a second dual-port node of the plurality of dual-port nodes; The system failure comprises a failure of the first dual-port node; as well as The modified visibility configuration includes visibility of the first volume to the second dual-port node.

17. The non-transitory computer-readable storage medium of claim 15, wherein: The visibility configuration includes visibility of a given volume of the first plurality of volumes to a first dual-port node of the plurality of dual-port nodes via the at least one switch domain; The system failure comprises a failure of the first dual-port node; as well as The modified visibility configuration includes the given volume being visible to a second dual-port node of the plurality of dual-port nodes via the at least one switch domain.

18. The non-transitory computer-readable storage medium of claim 15, wherein: the visibility configuration comprising visibility of the first volume to a given dual-port node of the plurality of dual-port nodes through a first port of the given dual-port node via the first switch domain and visibility of the second volume to the given dual-port node through a second port of the given dual-port node via the second switch domain; The system failure includes a failure of the first switch domain; as well as The modified visibility configuration includes visibility of the first volume to the given dual-port node through the second port of the given dual-port node via the second switch domain.

19. The non-transitory computer-readable storage medium of claim 15, wherein: The plurality of dual-port nodes include a first dual-port node, a second dual-port node, and a third dual-port node; The system further includes a second storage device, the second storage device including a second plurality of volumes, each volume of the second plurality of volumes having a PMR and a namespace, and the second plurality of volumes including a third volume and a fourth volume; the visibility configuration comprising visibility of the first volume and the second volume to the first dual-port node through respective ones of the first switch domain and the second switch domain through respective ones of the first port and the second port of the first dual-port node, visibility of the third volume to the second dual-port node through the first port of the second dual-port node through the first switch domain, and visibility of the fourth volume to the third dual-port node through the second port of the third dual-port node through the second switch domain; The system failure comprises a failure of the first dual-port node; as well as The modified visibility configuration includes visibility of the first volume to the second dual-port node through the second port of the second dual-port node via the second switch domain and visibility of the second volume to the third dual-port node through the first port of the third dual-port node via the first switch domain.

20. The non-transitory computer-readable storage medium of claim 15, wherein: The plurality of dual-port nodes include a first dual-port node, a second dual-port node, and a third dual-port node; The system further includes a second storage device, the second storage device including a second plurality of volumes, each volume in the second plurality of volumes having a PMR and a namespace, and the second plurality of volumes including a third volume and a fourth volume; the visibility configuration comprising visibility of the first volume and the second volume to the first dual-port node through respective ones of the first switch domain and the second switch domain through respective ones of the first port and the second port of the first dual-port node, visibility of the third volume to the second dual-port node through the first port of the second dual-port node through the first switch domain, and visibility of the fourth volume to the third dual-port node through the second port of the third dual-port node through the second switch domain; The system failure includes a failure of the first switch domain; as well as The modified visibility configuration includes visibility of the first volume to the first dual-port node through a second port of the first dual-port node via the second switch domain and visibility of the third volume to the second dual-port node through a second port of the second dual-port node via the second switch domain.

Images