Management of non-volatile memory fast nodes

Abstract

A programmable switch includes a port configured to communicate with a non-volatile memory express (NVMe) node. The programmable switch is configured to store a mapping of NVMe namespaces to physical storage locations located in the NVMe nodes. An NVMe node is determined by the programmable switch to have become inactive, and one or more NVMe namespaces associated with one or more physical storage locations in the inactive NVMe node are removed from the mapping. A notification of one or more removed NVMe namespaces is sent to a network controller. According to one aspect, the network controller stores a global mapping of NVMe namespaces to physical storage locations in NVMe nodes. The network controller sends at least one notification of an update to at least one other programmable switch to update at least one mapping stored at the at least one other programmable switch.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 63 / 127,423, filed on December 18, 2020, entitled “MANAGEMENT OF NON-VOLATILE MEMORY EXPRESS NODES” (Attorney Docket No. WDA-5254P-US), which is incorporated herein by reference in its entirety. This application also claims priority to U.S. Application No. 17 / 175,449, filed on February 12, 2021, entitled “MANAGEMENT OF NON-VOLATILE MEMORY EXPRESS NODES” (Attorney Docket No. WDA-5254-US), which is incorporated herein by reference in its entirety. This application is related to U.S. application No. 16 / 548,116 (Agent Case No. WDA-4307-US), entitled “DISTRIBUTED CACHE WITH IN-NETWORK PREFETCH,” filed on August 22, 2019 and published on November 5, 2020 as U.S. Patent Application Publication No. 2020 / 0349080, which is incorporated herein by reference in its entirety. This application is also related to U.S. Application No. 16 / 697,019, entitled “FAULT TOLERANT DATA COHERENCE IN LARGE-SCALE DISTRIBUTED CACHE SYSTEMS” (Attorney Docket No. WDA-4500-US), filed on November 26, 2019 and published on November 5, 2020 as U.S. Patent Application Publication No. 2020 / 0351370, which is incorporated herein by reference in its entirety. This application is also related to U.S. Application No. 16 / 914,206, entitled “DEVICES AND METHODS FOR MANAGING NETWORK TRAFFIC FOR A DISTRIBUTED CACHE” (Attorney Docket No. WDA-4952-US), filed on June 26, 2020, which is incorporated herein by reference in its entirety.This application is also related to U.S. Application No. 16 / 916,730 (Attorney Docket No. WDA-4977-US), filed on June 30, 2020, entitled “DEVICES AND METHODS FOR FAILURE DETECTION AND RECOVERY FOR A DISTRIBUTED CACHE”, which is incorporated herein by reference in its entirety. This application is also related to U.S. Application No. 17 / 174,681 (Attorney Docket No. WDA-5072-US), filed on February 12, 2021, entitled “DEVICES AND METHODS FOR NETWORK MESSAGE SEQUENCING”, which is incorporated herein by reference in its entirety. Background Art

[0003] The current trend of cloud computing, big data, and input / output (I / O) intensive applications has led to a greater demand for high-performance distributed shared memory systems in data centers with low latency, high throughput, and bandwidth. Emerging high-performance non-volatile memory express (NVMe) storage devices such as solid-state drives (SSDs) have become more common due to their improved performance in input / output operations per second (IOPS). Although NVMe was originally designed to be used with a peripheral component interconnect express (PCIe) interface within a computer system, recent NVMe extensions such as NVMe over network fabric (NVMeoF) can support remote access to NVMe devices, for example, via remote direct memory access (RDMA) (e.g., RDMA over aggregated Ethernet (RoCE) and iWarp) and Fibre Channel (FC) transport structures.

[0004] A recent trend related to NVMe enables NVMe controllers to support hot insertion and hot removal of locally connected storage devices such as SSDs during runtime. This means that NVMe devices can be added to or removed from the host's PCIe bus during runtime without interrupting host operations or requiring a restart of the NVMe controller. Since fault tolerance is an important part of developing Ethernet-based distributed systems, hot insertion and hot removal of NVMe devices need to be handled in Ethernet-based distributed systems, such as in today's data centers using NVMeoF / Transmission Control Protocol (TCP). BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The features and advantages of the embodiments of the present disclosure will become more apparent through the detailed description described below and in conjunction with the accompanying drawings. The drawings and associated descriptions are provided to illustrate the embodiments of the present disclosure, rather than to limit the scope of the claimed protection.

[0006] Figure 1 An exemplary network including non-volatile memory express (NVMe) nodes, programmable switches, and a network controller according to one or more embodiments is shown.

[0007] Figure 2 According to one or more embodiments Figure 1 A block diagram of exemplary components included in a network.

[0008] Figure 3 An exemplary NVMe mapping is shown according to one or more embodiments.

[0009] Figure 4 is a flow chart of an NVMe namespace removal process according to one or more embodiments.

[0010] Figure 5 is a flow chart of an NVMe namespace creation process according to one or more embodiments.

[0011] Figure 6 is a flow diagram of a message failure handling process according to one or more embodiments.

[0012] Figure 7 is a flow chart of a global NVMe mapping update process according to one or more embodiments. DETAILED DESCRIPTION

[0013] In the following detailed description, many specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent to one of ordinary skill in the art that the various embodiments disclosed may be practiced without some of these specific details. In other cases, well-known structures and techniques are not shown in detail to avoid unnecessarily obscuring the various embodiments.

[0014] System Example

[0015] Figure 1 An exemplary network 100 for implementing non-volatile memory express (NVMe) hot plugging, hot unplugging, and fault detection is shown according to one or more embodiments. As used herein, hot plugging refers to connecting an NVMe node during runtime without interrupting host operation or requiring a reboot of the NVMe controller, and hot unplugging refers to disconnecting an NVMe node during runtime without interrupting host operation or requiring a reboot of the NVMe controller.

[0016] like Figure 1 As shown, server racks 101A, 101B, 101C, and 101D communicate with other devices in network 100 using top-of-rack (ToR) switches 102A, 102B, 102C, and 102D, respectively. Each server rack 101 includes one or more NVMe nodes, such as Figure 2 The NVMe nodes 108 in the server rack 101 can access memory or storage locations in other NVMe nodes using NVMe messages such as NVMe read and write commands. The NVMe nodes in the server rack 101 can include, for example, servers or processing nodes, such as reduced instruction set computer (RISC)-V cores, and memory devices, such as solid-state drives (SSDs) or hard disk drives (HDDs). In some implementations, Figure 1 The network 100 can be used as at least part of a data center and / or for distributed processing, such as for distributed machine learning or big data analysis.

[0017] The network 100 may include, for example, a storage area network (SAN), a local area network (LAN), and / or a wide area network (WAN), such as the Internet. In this regard, one or more of the server racks 101, the ToR switches 102, the aggregation switches 104, the core switches 106, and / or the network controllers 120 may not be physically located at the same location. The server racks 101, the ToR switches 102, the aggregation switches 104, the core switches 106, and / or the network controllers 120 may communicate using one or more standards such as Ethernet.

[0018] like Figure 1As shown in the example of , each of the server racks 101A, 101B, 101C, and 101D is connected to a ToR or edge switch 102. In other implementations, each rack 101 can communicate with multiple ToR or edge switches 102 for redundancy. The ToR switch 102 may include a programmable switch, such as a 64-port ToR P4 programmable switch, which routes messages to or from NVMe nodes located in the server rack 101. Such a programmable switch may include, for example, a Barefoot Networks Tofino application-specific integrated circuit (ASIC) having a port configured to provide a 40 Gigabit Ethernet (GE) frame rate. Other types of programmable switches that can be used as ToR switches 102 may include, for example, a Cavium Xpliant programmable switch or a Broadcom Trident 3 programmable switch. In some implementations, and as discussed in Application Nos. 17 / 174,681, 16 / 697,019, 16 / 914,206, and 16 / 916,730, incorporated by reference above, each ToR switch 102 can provide centralized data consistency management for data stored in the end node devices of its corresponding server rack 101, such as for distributed caching.

[0019] For messages that remain in cluster 1101, aggregation switches 104A and 104B route messages between ToR switches 102A and 102B. Additionally, for messages that propagate between clusters 1101 and 1102, aggregation switches 104A and 104B route messages from ToR switches 102A and 102B to core switches 106A and 106B. Figure 1 In the example of FIG. 1 , racks 101A and 101B, ToR switches 102A and 102B, and aggregation switches 104A and 104B are grouped together into cluster 1101. Racks 101C and 101D, ToR switches 102C and 102D, and aggregation switches 104C and 104D are grouped together into cluster 1102.

[0020] Aggregation switches 104A, 104B, 104C, and 104D may include programmable switches, such as a 64-port ToR P4 programmable switch, that routes messages to or from ToR switches 102 in its cluster 110. Such programmable switches may include, for example, a Barefoot Networks Tofino application-specific integrated circuit (ASIC) having ports configured to provide 40 Gigabit Ethernet (GE) frame rates. Other types of programmable switches that may be used as aggregation switches 104 may include, for example, a Cavium Xpliant programmable switch or a Broadcom Trident 3 programmable switch.

[0021] The core switches 106A and 106B provide communication between the clusters 110 and with the network controller 120. In some implementations, the network controller 120 may include a software defined network (SDN) controller. As discussed in more detail below, the network controller 120 may store a global mapping of active NVMe namespaces or logical blocks to physical blocks located in the NVMe nodes of the server rack 101 (e.g., Figure 2 20). An NVMe namespace may include a collection of logical addresses (e.g., logical block addresses (LBAs)) accessible to other NVMe nodes in the network 100. The network controller 120 aggregates available NVMe namespaces to create a global or network-wide view of available memory or storage capacity.

[0022] The network controller 120 updates its global map to add NVMe namespaces corresponding to new NVMe nodes that may have been hot-plugged into the server rack 101, or remove NVMe namespaces corresponding to NVMe nodes that may have been hot-plugged from the server rack 101 or otherwise determined by the ToR switch 102 to be unavailable. In this regard, the ToR switch 102 is configured to determine that an NVMe node in its rack 101 has become inactive and remove the NVMe namespace from the NVMe map stored at the ToR switch 102 (e.g., Figure 2 and Figure 3In the embodiment of the present invention, the NVMe map 14 in the rack 101 is used to remove one or more namespaces associated with the physical storage location in the inactive NVMe node. Similarly, the ToR switch 102 is configured to determine that a new NVMe node has been connected to the port and add one or more new NVMe namespaces associated with the physical storage location in the new NVMe node to its NVMe map. Each ToR switch can be configured to send a notification to the network controller 120 in response to determining that an NVMe node within its rack 101 has been added, removed, or otherwise become unavailable, so that the network controller 120 can update its global NVMe map.

[0023] In some implementations, the ToR switches 102, the aggregation switches 104, and the core switches 106 may include programmable switches that are, for example, programmable to handle different custom protocols. The programmable switches 102, 104, and 106 may include programmable matching pipelines to provide configurable data planes and customized packet processing capabilities. Examples of such programmable switches may be found in the co-pending U.S. applications Nos. 17 / 174,681, 16 / 548,116, 16 / 697,019, 16 / 914,206, and 16 / 916,730, which are incorporated herein by reference above.

[0024] Figure 1 The data plane of the programmable switches 102, 104, and 106 in the example can control the point-to-point packet forwarding behavior of the programmable switches, such as using L1 / L2 Ethernet packet switching, packet admission control, and scheduling or queuing. The data plane of the programmable switches 102, 104, and 106 is programmable and separate from the high-level control plane that determines the end-to-end routing of packets or messages between devices in the network 100.

[0025] exist Figure 1 In the example of , the ToR switch 102 acts as an NVMe controller for the NVMe nodes in its corresponding server rack 101. In this regard, the NVMe management module 10 is responsible for determining a set of active NVMe nodes in its server rack 101 by identifying newly attached, failed, or recovered NVMe nodes and updating the available namespaces in its NVMe map. In addition, the NVMe management module 10 can notify other NVMe nodes and the network controller 120 about updates in the NVMe map. In more detail, the ToR switch 102 monitors the status of its ports in order to activate or deactivate ports. Activation of a port triggers the NVMe management module 10 to perform an NVMe discovery process via the activated port to determine whether there are one or more newly available namespaces. Deactivation of a port triggers the NVMe management module 10 to cancel the use of any NVMe nodes connected to the deactivated port and all outgoing messages for such NVMe nodes.

[0026] In addition, using a programmable switch as an NVMe controller can provide configuration of a high-performance and scalable memory-centric architecture by defining customized packet formats and processing behaviors. The programmable switches 102, 104, and 106 support the use of protocol-independent switch architectures and off-the-shelf switches, as opposed to specially designed networks on a chip (NoC). The processing resources of the programmable switches 102, 104, and 106, such as using content-addressable memory (CAM) or Ternary CAM (TCAM) tables or other types of matching action tables, can generally provide faster NVMe message processing than NVMe message processing that can occur at the terminal NVMe node.

[0027] A person of ordinary skill in the art will understand with reference to this disclosure that Figure 1 Other implementations may include different numbers or arrangements of server racks 101, ToR switches 102, aggregation switches 104, or core switches 106 than those shown in the examples. In this regard, Figure 1 The network 100 shown in FIG. 1 is for illustration purposes, and one of ordinary skill in the art will appreciate that the network 100 may include more than Figure 1 The example shown has many more server racks 101 and switches or routers. Other implementations may include additional levels in the network 100, which may include additional servers and / or programmable switches. In some variations, the aggregation switch 104 and / or the core switch 106 may be omitted.

[0028] Figure 2 According to one or more embodiments Figure 1 A block diagram of exemplary components included in a network 100 is shown. Figure 2 As shown, the NVMe node 108 includes a processor 116, a memory 118, a storage device 121, and an interface 122 for communicating on the network 100. The NVMe node 108 may be included as, for example Figure 1 A portion of server rack 101A in FIG. Figure 2 Only NVMe node 108 is shown in the example, but other NVMe nodes in network 100 may have similar or different components as NVMe node 108.

[0029] The processor 116 can execute instructions, such as instructions from the NVMe module 16 and the application 18, which may include an operating system (OS) and / or other applications used by the NVMe node 108. The processor 116 may include circuits such as a central processing unit (CPU), one or more RISC-V cores, a graphics processing unit (GPU), a microcontroller, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), hardwired logic, analog circuits, and / or combinations thereof. In some implementations, the processor 116 may include a system on chip (SoC), which may be combined with one or both of the memory 118 and the interface 122.

[0030] The memory 118 may include, for example, volatile random access memory (RAM), such as static RAM (SRAM), dynamic RAM (DRAM), non-volatile RAM, or other solid-state memory, which is used by the processor 116 as internal main memory to store data. The data stored in the memory 118 may include data read from the storage device 121, data to be stored in the storage device 121, instructions loaded from the NVMe module 16 or the application 18 for execution by the processor 116, and / or data for executing such applications. In addition to loading data from the memory 118, the processor 116 may also load data from the NVMe namespace of other NVMe nodes as external memory or distributed cache. Such data may also be refreshed by the processor 116 after modification, or driven back to the memory 118 or external NVMe node via the programmable switch 102 without modification.

[0031] like Figure 2 As shown, the memory 118 stores an NVMe module 16, which can provide instructions for accessing data in the NVMe namespace. Such instructions may include protocols for maintaining the consistency of data (e.g., cache lines) stored on different NVMe nodes of the network 100 and handling communications with the programmable switch 102, such as protocols for maintaining the proper sequence of NVMe messages. In some implementations, the NVMe module 16 may include a driver used by the OS of the NVMe node 108.

[0032] The storage device 121 is used as an auxiliary storage device, which may include, for example, one or more rotating disks or non-volatile solid-state memory, such as flash memory. Although the description herein generally refers to solid-state memory, it should be understood that the solid-state memory may include one or more of various types of memory devices, such as flash integrated circuits, NAND memory (e.g., single-level cell (SLC) memory, multi-level cell (MLC) memory (i.e., two or more layers), or any combination thereof), NOR memory, EEPROM, other discrete non-volatile memory (NVM) chips, or any combination thereof.

[0033] The interface 122 is configured to connect the NVMe node 108 to the programmable switch 102. The interface 122 can communicate using standards such as Ethernet. In this regard, the NVMe node 108, the programmable switch 102, and the network controller 120 may not be physically located in the same location and can communicate through a network such as a LAN or WAN. As will be understood by those of ordinary skill in the art, the interface 122 may be included as part of the processor 116.

[0034] As mentioned above Figure 1 As discussed, the programmable switch 102 may be a ToR switch for a server rack 101 including NVMe nodes 108. Figure 2 In the example of , the programmable switch 102 includes a port 130, a circuit 132, and a memory 134. The port 130 provides a connection and is configured to communicate with devices such as NVMe nodes, network controllers 120, aggregation switches 104, and core switches 106. For example, the port 130 may include an Ethernet port. As described above, the port 130 may be monitored to detect activation or deactivation of the port, thereby triggering the NVMe management module 10 to perform an NVMe discovery process or remove the NVMe namespace of the deactivated port.

[0035] The memory 134 of the programmable switch 102 may include, for example, volatile RAM, such as DRAM, or non-volatile RAM or other solid-state memory, such as a register array used by the circuit 132 to execute instructions loaded from the NVMe management module 10 or the firmware of the programmable switch 102 and / or data for executing such instructions, such as the NVMe map 14. The circuit 132 may include, for example, an ASIC, a microcontroller, a DSP, an FPGA, hard-wired logic, an analog circuit, and / or a combination thereof. In some implementations, the circuit 132 may include a SoC, which may be combined with the memory 134.

[0036] As discussed in more detail below, the NVMe management module 10 may include a method for implementing the Figures 4 to 6The process discussed is to implement hot insertion and hot removal of NVMe nodes and instructions for improving message failure handling. In addition, the use of NVMe mapping 14 enables the programmable switch 102 to track the active NVMe namespaces available at the NVMe nodes in its server rack.

[0037] Figure 2 The network controller 120 in the example of maintains a global NVMe map 20, which may include a mapping table or other type of data structure. The controller 120 receives an NVMe map update or notification from the programmable switch 102 via the interface 128, which indicates an update or change to an NVMe map such as the NVMe map 14 maintained by the programmable switch 102. Such updates may include removing one or more NVMe namespaces associated with physical storage locations located in an NVMe node that is determined to be inactive or otherwise unavailable, for example due to hot unplugging. Similarly, the programmable switch 102 can update its NVMe map to add one or more new NVMe namespaces that become available, for example due to hot insertion of an NVMe node.

[0038] In addition, the network controller 120 may send one or more notifications to the NVMe nodes in the network 100 to notify such nodes that a new NVMe namespace is available or that an NVMe namespace has become unavailable. In some implementations, the global NVMe map 20 maintained by the controller 120 may indicate which NVMe nodes have accessed a particular namespace. In such implementations, the network controller 120 may identify an NVMe node that has accessed the removed NVMe namespace that is not in the same server rack as the programmable switch 102 that sent the update. The controller 120 may then send a notification to such NVMe nodes to notify such NVMe nodes of the unavailable NVMe namespace.

[0039] The processor 124 of the controller 120 executes the network controller module 22 to maintain the global NVMe map 20 and notify the NVMe nodes of updates as needed. The processor 124 may include circuits such as CPUs, GPUs, microcontrollers, DSPs, ASICs, FPGAs, hardwired logic, analog circuits, and / or combinations thereof. In some implementations, the processor 124 may include a SoC that may be combined with one or both of the memory 126 and the interface 128. For example, the memory 126 may include volatile RAM such as DRAM, non-volatile RAM, or other solid-state memory used by the processor 124 to store data. The controller 120 communicates with the programmable switch 102 via an interface 128, which is configured to connect to a port of the programmable switch 102 and may be connected according to standards such as Ethernet.

[0040] As will be understood by those skilled in the art with reference to this disclosure, Figure 2 Other implementations may include different arrangements or numbers of components or modules than those shown in the examples. For example, in some implementations, the NVMe node 108 may not include a storage device 121, or two programmable switches 102 may be used in a single server rack to achieve redundancy.

[0041] Figure 3 An exemplary NVMe map 14 is shown according to one or more embodiments. Figure 3 In the example of , the NVMe map 14 can be a mapping table or other type of data structure. For example, the NVMe map can include a single data structure or can be formed by multiple data structures stored at the programmable switch 102, which can include a memory directly connected to the programmable switch 102 and used by the programmable switch (e.g., Figure 2 Memory 134 in.

[0042] like Figure 3 As shown, NVMe mapping 14 includes mapping logical blocks to physical storage locations (i.e., Figure 3 NVMe namespace identifier (i.e., Figure 3 14). The physical blocks may each represent a series of physical storage locations in an NVMe node included in the same server rack as the programmable switch 102 storing the NVMe map 14. In some implementations, each NVMe node 108 in a server rack may include one or more NVMe namespaces. In other cases, less than all nodes or servers in a server rack may have an available NVMe namespace.

[0043] exist Figure 3 In the example of NVMe, NVMe map 14 includes identifiers of NVMe nodes that may be in the same server rack and / or in other server racks that have access to a particular NVMe namespace. Figure 4 and Figure 5 As discussed in more detail in the process of, the programmable switch 102 can send a notification to the NVMe nodes that have accessed the NVMe namespace after the namespace becomes unavailable, for example, due to the NVMe node being hot-plugged. In other implementations, the access node of each NVMe namespace can be omitted.

[0044] The programmable switch 102 can expose or otherwise make the NVMe namespace in its server rack available and provide storage capacity for the NVMe namespace. In some implementations, the NVMe node can request a list of available NVMe namespaces from the programmable switch 102.

[0045] The NVMe map 14 may be updated by the programmable switch 102 in response to a new NVMe node (e.g., Figure 2 108 in the NVMe node 108). Similarly, the NVMe map 14 can be updated by the programmable switch 102 to remove an NVMe namespace that is unavailable, for example, due to hot unplugging an NVMe node from a server rack. In this regard, the programmable switch 102 can detect hot insertion or hot unplugging of the NVMe node 108 via monitoring the port 130 connected to the NVMe node 108. In some implementations, the programmable switch 102 can also remove one or more NVMe namespaces in response to determining that the NVMe node 108 is unavailable, such as removing one or more NVMe namespaces after a threshold number of messages are resent to the NVMe node without receiving an acknowledgment of the message.

[0046] A person of ordinary skill in the art will understand from this disclosure that the NVMe map 14 may include Figure 3 For example, some implementations of the NVMe map 14 may include a logical address range for each namespace or may omit the access node column.

[0047] Example Process

[0048] Figure 4 is a flow chart of an NVMe namespace removal process according to one or more embodiments. Figure 4 The process can be done by, for example Figure 2 The circuit 132 of the programmable switch 102 that executes the NVMe management module 10 is executed.

[0049] In block 402, the programmable switch determines that the NVMe node has become inactive. This determination may result from detecting the deactivation of a port of the programmable switch or the removal of a connection to the NVMe node via a port. In other cases, the determination that the NVMe node has become inactive may result from reaching a predetermined number of consecutive unconfirmed messages sent to the inactive NVMe node. Using a programmable switch (e.g., a programmable ToR switch) to determine that an NVMe node has become inactive can provide faster detection of hot unplugging and failure detection of NVMe nodes than relying on a terminal NVMe node to determine that an inactive NVMe node is no longer available.

[0050] In box 404, the programmable switch removes one or more NVMe namespaces associated with the physical storage location in the inactive NVMe node from the NVMe mapping. In this regard, each NVMe node can have one or more NVMe namespaces, so that when the NVMe node becomes inactive, the corresponding one or more NVMe namespaces are removed from the NVMe mapping of the unavailable physical storage location. In some implementations, one or more NVMe namespaces can be removed from the NVMe mapping by deleting one or more entries of the NVMe namespace. In other implementations, one or more NVMe namespaces can be removed by adding an inactive tag or other indicator that one or more NVMe namespaces have been removed. Such removed NVMe namespaces can be ultimately deleted in response to reaching the storage limit of the NVMe mapping. The use of such inactivity indications can help to notify NVMe nodes that previously accessed the inactive NVMe namespace in response to the NVMe namespace subsequently becoming active.

[0051] At block 406, the programmable switch sends a notification to the network controller 120 indicating that one or more NVMe namespaces were removed from the NVMe mapping at block 404. Figure 7 As discussed in more detail, the network controller 120 may use the notification sent at block 408 to update the global NVMe map (i.e., Figure 2 The global NVMe map 20 in the flow chart may be used to map the NVMe nodes 108 and may notify other programmable switches 102 and / or NVMe nodes 108 of updates.

[0052] In block 408, the programmable switch sends one or more notifications indicating the removed NVMe namespace to one or more NVMe nodes. The NVMe nodes may be in the same server rack or in different server racks. As described above, the programmable switch 102 may use a list of NVMe nodes that have previously accessed the removed NVMe namespace to send notifications to only those NVMe nodes indicating the removal of the NVMe namespace. This may typically provide faster adjustments by the NVMe nodes to replace the removed NVMe namespace with a different available NVMe namespace, rather than relying on the terminal node to determine that the NVMe namespace is no longer available.

[0053] A person of ordinary skill in the art will understand with reference to this disclosure that Figure 4The order of the boxes of the NVMe namespace removal process may be different in other implementations. For example, in other implementations, the order of boxes 406 and 408 may be switched so that the NVMe namespace is notified to the NVMe node before the NVMe namespace is notified to the network controller 120 that the NVMe namespace is removed. In yet other implementations, one of boxes 406 or 408 may be omitted so that only the NVMe node that previously accessed the removed NVMe namespace may be notified by the network controller 120 or by the programmable switch.

[0054] Figure 5 is a flow chart of an NVMe namespace creation process according to one or more embodiments. Figure 5 The process can be done by, for example Figure 2 The circuit 132 of the programmable switch 102 that executes the NVMe management module 10 is executed.

[0055] At block 502, the programmable switch determines that a new NVMe node has been connected to a port, for example in response to an NVMe node being connected to the programmable switch via hot plugging during runtime. The determination of block 502 may also occur during a boot process of the programmable switch, where the programmable switch performs an NVMe discovery process via all ports to identify available NVMe namespaces at NVMe nodes connected to the programmable switch via their ports.

[0056] In block 504, the programmable switch adds one or more NVMe namespaces associated with one or more physical storage locations in the new NVMe node. In some implementations, the programmable switch may send an NVMe request for an available namespace to the new NVMe node detected in block 502.

[0057] At block 506, the programmable switch sends a notification of the added one or more NVMe namespaces to the network controller 120. The network controller 120 can then update its global NVMe map 20 and can notify other programmable switches 102 and / or NVMe nodes 108 of the update.

[0058] In block 508, the programmable switch sends one or more notifications indicating the added NVMe namespace to one or more NVMe nodes. The NVMe nodes may be in the same server rack or in different server racks. In some implementations, the programmable switch may send a notification of the added NVMe namespace to each NVMe node in its server rack. In other implementations, the programmable switch may also send a notification to other NVMe nodes that have previously accessed the NVMe namespace in the server rack.

[0059] A person of ordinary skill in the art will understand with reference to this disclosure that Figure 5 The order of the boxes of the NVMe namespace addition process may be different in other implementations. For example, in other implementations, the order of boxes 506 and 508 may be switched so that the NVMe node is notified of the addition of one or more NVMe namespaces before the network controller 120 is notified of the addition of the NVMe namespace. In yet other implementations, one of boxes 506 or 508 may be omitted so that only the programmable switch 102 or the network controller 120 may notify the NVMe node of the addition of one or more NVMe namespaces.

[0060] Figure 6 is a flow diagram of a message failure handling process according to one or more embodiments. Figure 6 The process can be done by, for example Figure 2 The circuit 132 of the programmable switch 102 that executes the NVMe management module 10 is executed.

[0061] In block 602, the programmable switch receives data for a message from an NVMe node in its server rack. The data may include, for example, data for executing an NVMe write command at a target NVMe node in the same server rack or in a different server rack. As another example, the data may include data read from an NVMe node in a server rack in response to a read command received from another NVMe node.

[0062] At block 604, the programmable switch stores the data received for the message locally in, for example, Figure 2 In the memory 134 in the programmable switch 102 in the programmable switch 102. In some implementations, the programmable switch can parse the data from the message received from the NVMe node and buffer the NVMe capsule contained in the message, or store a copy of the data representing the payload in the local memory. The data stored for the message can be generated by one or more matching action stages or tables of the programmable switch, for example.

[0063] In block 606, the programmable switch sends a message including the data to the target node. In this regard, the programmable switch can parse one or more headers of the message to identify the target node and use one or more matching action stages or tables to route the message to the appropriate port to send the message to the target NVMe node.

[0064] In box 608, the programmable switch temporarily retains the data stored in box 604 if no confirmation is received and the message needs to be resent. By performing such message failure processing at the programmable switch, the time required to respond to the failure message can generally be reduced and the workload of the terminal NVMe node in processing such failure messages can be reduced. In some implementations, the programmable switch can retain the data until a confirmation of the data is received. In other implementations, the data can be temporarily stored in a circular buffer so that when the buffer becomes full, the oldest data received by the programmable switch is replaced by the latest data received by the programmable switch.

[0065] At block 610, the programmable switch determines whether a timeout value for receiving an acknowledgment (i.e., an ACK message) from a target node for the message sent at block 606 has expired. In some implementations, the programmable switch may set a timeout register for receiving a corresponding acknowledgment, for example, through an egress pipeline of the programmable switch, in response to sending the message. The timeout value may be based on, for example, an average round-trip packet transmission duration. In some implementations, the timeout value may be adjusted in a field based on, for example, an exponential moving average of sampled round-trip times.

[0066] In response to receiving an acknowledgement at block 610 before the timeout value is reached, Figure 6 The message failure handling process ends at block 612. On the other hand, in response to not receiving an acknowledgment at block 610 before reaching the timeout value, the process proceeds to block 614 to recreate the message and resend it using the data temporarily retained by the programmable switch at block 608. By using locally stored data, the programmable switch can respond quickly to reaching the timeout value without having to notify the NVMe node that issued the message and wait for a response from the NVMe node.

[0067] In some cases, the programmable switch may receive a negative acknowledgement (i.e., a NACK message) or other error message indicating that an error occurred when receiving a message sent by the programmable switch for reasons such as an incorrect sequence number. In this case, the programmable switch may attempt to correct the error when recreating the message. In some implementations, the programmable switch may also notify the NVMe node that issued the message of an error that the programmable switch cannot resolve.

[0068] At block 616, the programmable switch increments a message retry counter. The counter may be set, for example, in a register of the programmable switch, such as by an egress pipeline of the programmable switch, in response to resending the message.

[0069] At block 618, the programmable switch determines whether the number of retries has reached a threshold. For example, as part of an NVMe negotiation process between a sending and receiving NVMe node, a threshold number of retries may be set. In response to the number of retries reaching the threshold, Figure 6 The process proceeds to block 620 to begin the NVMe namespace removal process, as discussed above. Figure 4 In the process of removing the NVMe namespace of the unresponsive NVMe node from the NVMe mapping of the programmable switch. On the other hand, in response to not reaching the threshold number of retries at block 618, the process returns to block 610 to determine whether the timeout value for receiving the confirmation of the message resent at block 614 has been reached.

[0070] A person of ordinary skill in the art will understand with reference to this disclosure that Figure 6 The order of the blocks of the message failure handling process may be different in other implementations. For example, in some implementations, incrementing the retry counter at block 616 may occur before recreating the message and resending it at block 614.

[0071] Figure 7 is a flow chart of a global NVMe mapping update process according to one or more embodiments. Figure 7 The process can be done by, for example Figure 2 The processor 124 of the network controller 120 that executes the network controller module 22 is executed.

[0072] At block 702, a network controller receives a request from a programmable switch (e.g., Figure 1 A ToR switch 102A, 102B, 102C, or 102D in the NVMe node receives a notification that the NVMe node has become inactive. The notification may be sent as part of the NVMe namespace removal process, such as discussed above. Figure 5 In other implementations, the network controller may also receive updates to the newly added NVMe namespace from the programmable switch.

[0073] At block 704, the network controller updates the global map (e.g., Figure 2 The network controller may remove one or more NVMe namespaces corresponding to inactive NVMe nodes from its global map. In some implementations, one or more NVMe namespaces may be removed by deleting one or more entries of the NVMe namespace. In other implementations, one or more NVMe namespaces may be removed by adding an inactive tag or other indicator that one or more NVMe namespaces have been removed.

[0074] At block 706, the network controller sends at least one notification of an update to the global map to at least one NVMe node. The global map may include entries or indications of NVMe nodes that have accessed one or more NVMe namespaces removed from the global map. This information may be provided by a programmable switch in the network periodically or in response to, for example, a new NVMe node accessing the NVMe namespace. The network controller may then send a notification to the NVMe node that has previously accessed the data, read the data from the namespace, and / or write the data to the namespace, notifying the NVMe node that the one or more namespaces removed from the global map are no longer available.

[0075] In some implementations, in response to the network controller receiving an update from the programmable switch that the NVMe node has become active again, the network controller may then send a new notification about the reactivated NVMe node to the NVMe nodes that previously accessed one or more NVMe namespaces. In this regard, removing the NVMe namespace from the global map may simply include adding an inactive tag or indicator to one or more entries of the inactive NVMe namespace in the global map. Such removed NVMe namespaces may be eventually deleted in response to reaching the storage limit of the global map. Using such an inactivity indication may help notify the NVMe nodes that previously accessed the removed NVMe namespace in response to the NVMe namespace subsequently becoming active.

[0076] A person of ordinary skill in the art will understand with reference to this disclosure that Figure 7 The order of the boxes of the global NVMe map update process may be different in other implementations. For example, in some implementations, sending at least one notification of the update in box 706 may be before the global map is updated in box 704.

[0077] As described above, the use of an embedded programmable switch to maintain NVMe mappings typically enables hot insertion and hot removal of NVMe nodes in the network. In addition, the aforementioned message processing can provide faster fault detection and improved fault tolerance, which is particularly useful in inherently lossy Ethernet networks. Using a programmable switch for NVMe mapping can facilitate faster mapping updates than relying on communication between NVMe end nodes, while eliminating the need for such processing from the NVMe nodes. As described above, faster NVMe mapping updates may be attributed to port monitoring performed by a programmable switch, a programmable matching action stage or table, and online or network processing.

[0078] Other Implementations

[0079] Those skilled in the art will appreciate that the various illustrative logic blocks, modules, and processes described in conjunction with the examples disclosed herein may be implemented as electronic hardware, computer software, or a combination of both. In addition, the aforementioned processes may be embodied in a computer-readable medium that enables a processor or controller circuit to perform or implement certain functions.

[0080] To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, and modules have been generally described above in terms of their functionality. Whether this functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. One of ordinary skill in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

[0081] The various illustrative logical blocks, units, modules, processor circuits, and controller circuits described in conjunction with the examples disclosed herein may be implemented or executed with a general purpose processor, GPU, DSP, ASIC, FPGA, or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor or controller circuit may also be implemented as a combination of computing devices, such as a DSP and a microprocessor, multiple microprocessors, SoC, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0082] The activities of the methods or processes described in conjunction with the examples disclosed herein may be directly embodied in hardware, in software modules executed by a processor or controller circuit, or in a combination of the two. The steps of the method or algorithm may also be performed in an order alternative to the order provided in the example. The software module may reside in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable medium, an optical medium, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor or controller circuit so that the processor or controller circuit can read information from the storage medium and write information to the storage medium. In an alternative, the storage medium may be a component of a processor or controller circuit. The processor or controller circuit and the storage medium may reside in an ASIC or SoC.

[0083] The above description of the exemplary embodiments of the present disclosure is provided to enable any person of ordinary skill in the art to make or use the embodiments of the present disclosure. Various modifications to these examples will be apparent to those of ordinary skill in the art, and the principles disclosed herein may be applied to other examples without departing from the spirit or scope of the present disclosure. The embodiments described are to be considered in all respects to be merely exemplary and non-restrictive. In addition, the language used in the form of "at least one of A and B" in the following claims should be understood to mean "only A, only B, or both A and B".

Claims

1. A programmable switch, comprising: a plurality of ports configured to communicate with a plurality of non-volatile memory express (NVMe) nodes over a network, wherein each NVMe node of the plurality of NVMe nodes is configured to communicate over the network using NVMe over fiber (NVMeOF); at least one memory configured to store a mapping of an NVMe namespace to physical storage locations located in the plurality of NVMe nodes; and A circuit, the circuit being configured to: determining that an NVMe node among the plurality of NVMe nodes has become inactive by detecting removal of a connection to the NVMe node via one of the plurality of ports; removing from the mapping one or more NVMe namespaces associated with one or more physical storage locations located in the inactive NVMe node; as well as At least one notification is sent, via the network, to at least one other NVMe node among the plurality of NVMe nodes, the notification indicating that the one or more NVMe namespaces have been removed from the mapping.

2. The programmable switch of claim 1 , wherein the circuit is further configured to determine that a different NVMe node has become inactive based on at least one of: detecting inactivity of a port of the plurality of ports connected to the different NVMe node, and reaching a predetermined number of consecutive unacknowledged messages sent to the different NVMe node.

3. The programmable switch according to claim 1, wherein the circuit is further configured to: Determining that a new NVMe node has been connected to a port of the plurality of ports; and One or more new NVMe namespaces associated with one or more new physical storage locations in the new NVMe node are added to the mapping.

4. The programmable switch of claim 1 , wherein the circuit is further configured to temporarily retain the data sent to the target NVMe node after sending the data to the target NVMe node in the form of a message.

5. The programmable switch according to claim 1, wherein the circuit is further configured to: Sending a message to a target NVMe node among the multiple NVMe nodes; determining whether a timeout value for receiving an acknowledgment of a sent message from the target NVMe node has been reached; and In response to reaching the timeout value without receiving the acknowledgment from the target NVMe node, resending the message to a different NVMe node.

6. The programmable switch of claim 1 , wherein the circuit is further configured to send a notification to a network controller of the removed one or more NVMe namespaces to update a global mapping of active NVMe namespaces to NVMe nodes in the network.

7. The programmable switch of claim 1 , wherein the mapping further indicates other NVMe nodes that have accessed data in a particular NVMe namespace.

8. The programmable switch of claim 1, wherein the programmable switch is a top-of-rack (ToR) switch for the plurality of NVMe nodes.

9. The programmable switch of claim 1, wherein the circuit is further configured to: detecting activation of a port of the plurality of ports; and In response to detecting the activation of the port, an NVMe discovery service is performed to determine whether one or more new NVMe namespaces are available via the activated port.

10. A method performed by a programmable switch, the method comprising: In the programmable switch, storing a mapping of a non-volatile memory express (NVMe) namespace to physical storage locations located in a plurality of NVMe nodes; Each of the plurality of NVMe nodes is a separate physical device configured to communicate over a network using NVMe over fiber, i.e., NVMeOF; Determining that an NVMe node among the plurality of NVMe nodes has become inactive; removing, from a stored mapping, one or more NVMe namespaces associated with one or more physical storage locations located in the inactive NVMe node; as well as Send at least one notification to at least one other NVMe node of the plurality of NVMe nodes indicating that the one or more NVMe namespaces have been removed from the mapping.

11. The method of claim 10, further comprising determining that the NVMe node has become inactive based on at least one of: detecting inactivity of a port of the programmable switch connected to the inactive NVMe node, and reaching a predetermined number of consecutive unacknowledged messages sent to the inactive NVMe node.

12. The method according to claim 10, further comprising: Determine that the new NVMe node is connected to the port of the programmable switch; as well as One or more new NVMe namespaces associated with one or more new physical storage locations in the new NVMe node are added to the mapping.

13. The method of claim 10, further comprising temporarily retaining the data sent to the target NVMe node after sending the data to the target NVMe node in the form of a message.

14. The method according to claim 10, further comprising: Sending a message to a target NVMe node among the multiple NVMe nodes; Determining whether a timeout value for receiving an acknowledgment of a sent message from the target NVMe node has been reached; as well as In response to reaching the timeout value without receiving the confirmation from the target NVMe node, resending the message to the target NVMe node.

15. The method of claim 10, further comprising sending a notification to a network controller of the removed one or more NVMe namespaces to update a global mapping of active NVMe namespaces to NVMe nodes in the network.

16. The method of claim 10, wherein the mapping indicates NVMe nodes among the plurality of NVMe nodes that have accessed data in a particular NVMe namespace.

17. The method of claim 10, wherein the programmable switch is a top-of-rack (ToR) switch for the plurality of NVMe nodes.

18. The method according to claim 10, further comprising: detecting activation of a port of the programmable switch; as well as In response to detecting the activation of the port, an NVMe discovery service is performed to determine whether one or more new NVMe namespaces are available via the activated port.

19. A network controller for a network including a plurality of non-volatile memory express (NVMe) nodes, the network controller comprising: at least one memory storing a global mapping of active NVMe namespaces to physical blocks located in the plurality of NVMe nodes, wherein each NVMe node in the plurality of NVMe nodes is a separate physical device configured to communicate over the network; an interface configured to communicate with a plurality of programmable switches; and Devices for: Receiving a notification from a programmable switch among the plurality of programmable switches that an NVMe node has become inactive; Updating the global map based on the received notification to remove one or more NVMe namespaces corresponding to the inactive NVMe nodes; and At least one notification is sent, via the network, to at least one NVMe node among the plurality of NVMe nodes, the notification indicating that the one or more NVMe namespaces have been removed from the global mapping.

20. The network controller of claim 19, further comprising means for identifying the at least one NVMe node using the global map, wherein the global map indicates which NVMe nodes have access to the one or more NVMe namespaces.

Images