System and method for supporting inter-chassis manageability of NVMe-oF based systems
By using switched and switchless Ethernet SSD chassis in NVMe-oF data storage system, combined with the method of extending IPMI protocol and electing the dominant BMC, the problem of inter-chassis communication and management in NVMe-oF system is solved, achieving low Cost, highly manageable NVMe-oF groups/domains.
Patent Information
- Application Number
- CN201811471984.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-05-02
- Filing Date
- 2018-12-04
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2038-12-04
AI Technical Summary
Existing NVMe-oF-based data storage systems lack standard protocols for connecting multiple NVMe-oF chassis, resulting in configuration, control, and management difficulties and high costs.
By introducing switched Ethernet SSD chassis and switchless Ethernet SSD chassis in the data storage system, communication between chassis is achieved using Ethernet switches and repeaters, and the extended IPMI protocol supports inter-BMC communication within the domain, electing the dominant BMC to manage device information within the domain.
It realizes low-cost management of NVMe-oF groups/domains, reduces system costs, provides manageability and high availability between chassis, and supports configuration and status monitoring of devices within the domain.
Smart Images

Figure CN110032334B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates generally to data storage systems and management of data storage systems, and more specifically, to systems and methods for supporting inter-chassis manageability of data storage systems based on non-volatile memory express over fabrics (NVMe-oF). Background Art
[0002] A data storage system based on non-volatile memory transport (NVMe) over fabrics (NVMe-oF) may have an Ethernet switch connected to multiple NVMe-oF devices within an NVMe-oF chassis. The Ethernet switch included in the NVMe-oF chassis may have a sufficient number of Ethernet ports to support additional NVMe-oF chassis that lack an Ethernet switch. Such an NVMe-oF chassis without an Ethernet switch is often referred to as just a bunch of offlash (JBoF).
[0003] Each NVMe-oF chassis may have at least one motherboard, and each motherboard has a baseboard management controller (BMC). The BMC may be a low-power controller embedded in the motherboard of the NVMe-oF chassis. In addition to the BMC, the motherboard of the NVMe-oF chassis also includes an Ethernet switch, a local central processing unit (CPU), memory, and a peripheral component interconnect express (PCIe) switch. The BMC may use various sensors embedded in the NVMe-oF chassis and an Ethernet SSD connected to the NVMe-oF chassis to read the environment and operating conditions of the corresponding NVMe-oF chassis, and control the NVMe-oF chassis and the Ethernet SSD based on commands from the system administrator or the status of the sensor. The BMC can access and control various components of the NVMe-oF chassis through local system buses such as the system management bus (SMBus) and the PCIe bus.
[0004] For NVMe-oF-based data storage systems, there is a need to connect multiple NVMe-oF chassis with an Ethernet switch or an Ethernet switchless chassis. The Ethernet switchless chassis may be referred to as a just a bunch of flash (JBoF) chassis. In some examples, the JBoF chassis may have an Ethernet repeater or retimer instead of an Ethernet switch to reduce the cost of the data storage system. Currently, there is no standard protocol for connecting multiple NVMe-oF chassis and using inter-chassis communication to facilitate configuration, control, and management. Summary of the invention
[0005] An object of the present invention is to provide a system and method for NVMe-oF group / domain with low cost.
[0006] According to one embodiment, a data storage system includes: a plurality of Ethernet solid-state drive (SSD) chassis, including at least one switched Ethernet SSD chassis and one or more switchless Ethernet SSD chassis. At least one switched Ethernet SSD chassis includes an Ethernet switch, a first baseboard management controller (BMC), and a first management local area network (LAN) port. At least one of the one or more switchless Ethernet SSD chassis includes an Ethernet repeater, a second BMC, and a second management LAN port. The first management LAN port and the second management LAN port of at least one switched Ethernet SSD chassis are connected. The first BMC collects the status of at least one of the one or more switched Ethernet SSD chassis from the second BMC via the connection between the first management LAN port and the second management LAN port, and provides information of at least one of the one or more switched Ethernet SSD chassis and at least one switched Ethernet SSD chassis to a system administrator.
[0007] According to another embodiment, a data storage system includes: a switched Ethernet SSD chassis including an Ethernet switch, a baseboard management controller (BMC), and a management LAN port; and a first switchless Ethernet SSD chassis and a second switchless Ethernet SSD chassis. Each of the first switchless Ethernet SSD chassis and the second switchless Ethernet SSD chassis includes an Ethernet repeater, a BMC, and a management LAN port, which are connected to each other and to the management LAN port of the switched Ethernet SSD. The BMC of the second switchless Ethernet SSD chassis provides device information of the second switchless Ethernet SSD chassis to the first switchless Ethernet SSD chassis via the management LAN port. The BMC of the first switchless Ethernet SSD chassis provides device information of the first switchless Ethernet SSD chassis and the second switchless Ethernet SSD chassis to the BMC of the switched Ethernet SSD chassis via the management LAN port. The BMC of the switched Ethernet SSD chassis provides device information of the switched Ethernet SSD chassis, the first switchless Ethernet SSD chassis, and the second switchless Ethernet SSD chassis to a system administrator connected through a fabric network.
[0008] According to another embodiment, a method includes: selecting a candidate BMC from among a plurality of BMCs in a domain, wherein the domain includes a plurality of Ethernet solid state drive (SSD) chassis including at least one switched Ethernet SSD chassis and one or more switchless Ethernet SSD chassis; broadcasting to a plurality of BMCs in the domain to assert domain presidency; checking eligibility of the candidate BMC based on responses received from the plurality of BMCs; and electing the candidate BMC as the president BMC of the domain name based on the eligibility. The president BMC is included in a first switched Ethernet SSD chassis including a first Ethernet switch. The president BMC collects device information of a plurality of Ethernet SSD chassis in the domain from a system administrator via a fabric network.
[0009] The above and other preferred features including various novel implementation details and event combinations will now be described in more detail with reference to the accompanying drawings, and will be pointed out in the claims. It should be understood that the specific systems and methods described herein are shown only by way of illustration and not as a limitation. As will be appreciated by those skilled in the art, the principles and features described herein may be employed in various and multiple embodiments without departing from the scope of the present disclosure.
[0010] According to the present invention, the NVMe-oF group / domain includes an Ethernet switching board and a switchless board. The switchless board shares the switch of the Ethernet switching board. Therefore, a system and method for an NVMe-oF group / domain are provided. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] The accompanying drawings, which are incorporated as a part of this specification, illustrate presently preferred embodiments, and together with the general description given above and the detailed description of the preferred embodiments given below, serve to explain and teach the principles described herein.
[0012] Figure 1 shows an example data structure of an IPMI message in an Ethernet frame;
[0013] Figure 2A illustrates an architecture of an example NVMe-oF domain including multiple boards according to one embodiment;
[0014] Figure 2B illustrates an architecture of an example NVMe-oF domain including multiple boards according to another embodiment;
[0015] Figure 3 is an example flow chart for electing a dominant BMC in a domain according to one embodiment;
[0016] Figure 4 is an example flow chart of replacing a dominant BMC in a domain according to one embodiment;
[0017] Figure 5 illustrates a domain of an example NVMe-oF domain without a domain Ethernet switch according to one embodiment;
[0018] Figure 6 illustrates example data flows within a domain of an example NVMe-oF domain according to one embodiment; and
[0019] Figure 7 A flow chart for processing a device information request according to one embodiment is shown.
[0020] The figures are not necessarily drawn to scale, and throughout the figures, for illustrative purposes, elements of similar structure or function are generally represented by the same reference numerals. The figures are intended only to facilitate description of the various embodiments described herein. The figures do not describe every aspect of the teachings disclosed herein and do not limit the scope of the claims. DETAILED DESCRIPTION
[0021] Each of the features and teachings disclosed herein can be used alone or in combination with other features and teachings to provide systems and methods for supporting inter-chassis manageability of NVMe-oF-based data storage systems. Many representative examples utilizing these additional features and teachings (both individually and in combination) are further described in detail with reference to the accompanying drawings. This detailed description is intended only to teach those skilled in the art further details for practicing various aspects of the present teachings and is not intended to limit the scope of the claims. Therefore, the combination of features disclosed below in the detailed description may not be necessary to practice the present teachings in the broadest sense, but rather merely teaches a combination of features to describe specific representative examples of the present teachings.
[0022] In the following description, for purposes of explanation only, specific terms are set forth to provide a thorough understanding of the present disclosure. However, it will be apparent to one skilled in the art that these specific details are not required to practice the teachings of the present disclosure.
[0023] Some portions of the detailed description herein are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are used by those skilled in the art of data processing to effectively communicate the essence of their work to others skilled in the art. An algorithm here is generally considered to be a self-consistent sequence of steps leading to a desired result. These steps are steps that require physical manipulation of physical quantities. Typically, although not necessarily, these quantities take the form of electrical or magnetic signals that can be stored, transmitted, combined, compared, and otherwise manipulated. Sometimes, primarily for general reasons, it has proven convenient to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, etc.
[0024] It should be remembered, however, that all of these and similar terms are associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise clearly indicated from the following discussion, it should be understood that throughout the specification, discussions utilizing terms such as "processing," "computing," "calculating," "determining," "displaying," etc. refer to the actions and processes of a computer system or similar electronic computing device that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system's memories or registers or other such information storage, transmission, or display devices.
[0025] Furthermore, the various features of the representative examples and the dependent claims may be combined in ways not specifically and explicitly listed in order to provide additional useful embodiments of the present teachings. It is also expressly noted that for the purposes of the original disclosure, and for the purposes of limiting the claimed subject matter, all value ranges or indications of groups of entities disclose every possible intermediate value or intermediate entity. It is also expressly noted that the sizes and shapes of the components shown in the figures are designed to aid in understanding how to implement the present teachings, but are not intended to limit the sizes and shapes shown in the examples.
[0026] The present disclosure relates to systems and methods for supporting inter-chassis manageability of NVMe-oF-based systems. The NVMe-oF protocol provides a transport mapping mechanism for exchanging commands and responses between a host computer and a target storage device over a Fabric network such as Ethernet, Fibre Channel, and InfiniBand using a message-based model. The present system allows a system administrator to manage a group or a domain of BMCs without directly managing the BMCs of each NVMe-oF domain. In each group / domain, one BMC in the group / domain is designated to function as the "president" of the group / domain. The president can provide discovery information for other BMCs within the group / domain. The president can also manage the status of all BMCs in the group / domain and report to the system administrator. The system administrator can contact the president to obtain the status of all member BMCs and use the president BMC as an agent to perform certain actions on a specific member BMC or all member BMCs of the group / domain.
[0027] In order to achieve manageability of domains / groups, the system requires a connection topology to connect multiple BMCs. According to one embodiment, the system and method provide an external management switch that provides connections between BMCs within a group / domain. The management LAN port of each NVMe-oF chassis can be connected to a management switch (e.g., a 1Gb switch). In some embodiments, some of the management LAN ports of the NVMe-oF chassis can be connected in a daisy chain.
[0028] According to one embodiment, the present system and method provide a communication protocol between BMCs. For example, new IPMI commands can be added to extend the standard IPMI-over-LAN protocol to facilitate manageability between chassis. The extended IPMI protocol over UDP / IP can provide functions that the standard IPMI-over-LAN protocol is not suitable for, such as domain communication, discovery, etc. In addition to existing system information, the present system and method can also support the exchange of new system information, including but not limited to the configuration of Ethernet SSD boards within the domain, the network configuration of the switch board within the domain, the allocation of static IPs of Ethernet SSDs (eSSDs) connected to the board, and restarting the Dynamic Host Configuration Protocol (DHCP) client to obtain the IP address of the eSSD.
[0029] The first BMC to appear can be selected as the domain ruler, or a specific BMC within a domain / group can be designated as the ruler. In some embodiments, the system administrator maintains a list and queue of BMCs that can be selected as rulers. In some embodiments, the election of rulers can be completed through arbitration. When the ruler BMC is out of service, the next ruler can be selected from the remaining active member BMCs.
[0030] Typically, the BMC of an NVMe-oF chassis can be connected to an administrator through a management local area network (LAN). System administrators can monitor multiple NVMe-oF chassis directly through the management LAN via the Intelligent Platform Management Interface (IPMI) protocol. The IPMI protocol allows communication between system administrators and BMCs through the management LAN using IPMI messages. IPMI messages are encapsulated in Remote Management Control Protocol (RMCP / RMCP+) packets defined by the Distributed Management Task Force (DMTF).
[0031] Figure 1 An example data structure of an IPMI message in an Ethernet frame is shown. An IPMI message 105 includes a network function (NetFn), a logical unit number (LUN), a sequence number (Seq#), a command (CMD), and data. An IPMI message 105 is encapsulated in an Ethernet frame 101. The Ethernet frame 101 includes a MAC address and encapsulates an IP / UDP data packet 102. The IP / UDP data packet 102 includes an IP address and an RMCP port number and encapsulates an RMCP message 103. The RMCP message 103 includes a class of a message (e.g., IPMI) and an RMCP sequence number and encapsulates an IPMI packet 104. The IPMI packet 104 includes a session encapsulator and includes an IPMI message 105.
[0032] According to one embodiment, the present system and method enable inter-chassis communication between different NVMe-oF chassis to minimize system cost. To save costs, one NVMe-oF chassis in a domain / group may include an Ethernet switch while the other chassis do not. In this case, the chassis lacking an Ethernet switch will include a switchless board that is similar to the chassis that includes an Ethernet switch board, except that they do not include an expensive Ethernet switch. The following description is based on Ethernet connections between multiple BMCs. However, it should be understood that the present system and method may use other types of network-based connections and protocols. The present system and method may not require additional cables other than network cables to achieve inter-chassis communication.
[0033] According to one embodiment, the present disclosure provides inter-chassis communication between multiple BMCs through an external Ethernet switch and provides cost-effective manageability of multi-chassis NVMe-oF domains. Inter-chassis communication can be implemented using a standard interface with an extended IPMI protocol.
[0034] Figure 2A The architecture of an example NVMe-oF domain including multiple boards according to one embodiment is shown. The NVMe-oF domain 200A includes two NVMe-oF chassis 250A and 250B, and each of the NVMe-oF chassis 250A and 250B includes two NVMe-oF boards 201 of the same type, i.e., Ethernet switching boards or switchless boards. In this example, the first NVMe-oF chassis 250A includes two switching boards 201A and 201B, and the second NVMe-oF chassis 250B includes two switchless boards 201C and 201D. The NVMe-oF domain 200A may also be referred to herein as an NVMe-oF cluster or an eSSD cluster. In some embodiments, an NVMe-oF chassis including one or more Ethernet switching boards may be referred to as an Ethernet switching chassis or an Ethernet switching SSD chassis.
[0035] Both the switch boards 201A and 201B include an Ethernet switch 205, while the switchless boards 201C and 201D include repeaters 207 (or retimers) instead of the Ethernet switch 205. Note that the NVMe-oF domain 200A is configured with two switch boards and two switchless boards as an example, and it can be understood that the NVMe-oF domain 200A can have a different configuration, including a greater or lesser number and different types of boards in multiple NVMe-oF chassis, without departing from the scope of the present disclosure.
[0036] Each of the NVMe-oF boards 201 may include other components and modules, such as a local CPU 202, a BMC 203, a PCIe switch 206, an uplink Ethernet port 211, a downlink Ethernet port 212, and a management LAN port 215. Several Ethernet solid-state drives (eSSDs) may be plugged into the device port of the NVMe-oF board 201 via the midplane 261. For example, each of the eSSDs is connected to a U.2 connector (not shown) on the midplane 261. An eSSD plugged into a drive bay and mated with the midplane 261 is also referred to herein as an NVMe-oF device or Ethernet SSD (eSSD). NVMe-oF chassis boards 201C and 201D that lack their own internal Ethernet switches are also referred to herein as NVMe-oF disk clusters (JBOF, just a bunch of flash).
[0037] The management LAN (not shown) includes a management Ethernet switch 260 connected to the management LAN ports 215 of all NVMe-oF boards 201 in the NVMe-oF domain 200A. The management LAN port 215 may be an Ethernet port. The BMC 203 of the switch or switchless board 201 is connected to the management Ethernet switch 260 via the management LAN port 215. The management Ethernet switch 260 provides a connection between multiple NVMe-oF chassis 250 and a system administrator to allow the system administrator to monitor the NVMe-oF chassis on the management LAN port 215 using the Intelligent Platform Management Interface (IPMI) protocol. In addition, the BMC 203 may report errors of the NVMe-oF chassis 250 to the system administrator via the IPMI protocol. In one embodiment, the management Ethernet switch 260 may be included in a chassis separate from the NVMe-oF chassis 250A or 250B, but in the same rack. The uplink Ethernet port 211 of the switchless board 201C or 201D can be connected to the internal Ethernet switch 205 of the coupled switch board 201A or 201B to route Ethernet traffic between the host (or initiator) and the target eSSD attached to the switchless board 201C or 201D.
[0038] The NVMe-oF domain 200A may have at least one dominant BMC 203. The dominant BMC of the NVMe-oF domain 200A may be selected in several ways. In a domain having only one switch board including an Ethernet switch, the BMC of the switch board is selected as the dominant BMC by default. The remaining switchless boards are JBOFs without embedded Ethernet switches. In this case, the JBOFs without switch boards are connected to the Ethernet switches 205 of the switch boards, and they function through the switch boards with the Ethernet switches 205.
[0039] In a group / domain with multiple switch boards including multiple BMCs, the uptime of the BMC (i.e., the period of time during which the BMC is continuously operational, without power outages or failures) can be used to determine the dominant BMC by comparing the uptime of all qualified candidate BMCs in the domain. Some BMCs in the group / domain may or may not be identified as dominant BMCs. For example, the BMC with the longest uptime is selected as the dominant BMC. In another example, the BMC with the lowest or highest IP address among the candidate BMCs may be selected as the dominant BMC.
[0040] Figure 2B FIG. 200B shows an example NVMe-oF domain architecture including multiple boards according to another embodiment. NVMe-oF domain 200B is substantially similar to Figure 2ANVMe-oF domain 200A, except that there is no management Ethernet switch. In this case, BMCs 203C and 203D report to the dominant BMC, for example, BMC 203A of switch board 201A, via the corresponding management LAN port 215. When two switch boards (e.g., NVMe-oF chassis 250A) are present in the NVMe-oF chassis to support high availability (HA) mode, one of the BMCs (e.g., BMC 203A) is active and the other BMC (e.g., BMC 203B) may be inactive. Any non-dominant BMC (e.g., BMCs 203C and 203D) can collect information of other BMCs in the domain and report the collected information to the dominant BMC 203A in a daisy chain form. For example, BMC 203C can report the status of one or more other NVMe-oF chassis (not shown) through communication between BMCs. In the event of a failure or power outage of the governor BMC 203A, the BMC 203B of the switch board 201B may be selected as the governor BMC and report the status of the NVMe-oF chassis in the domain to the system administrator.
[0041] Figure 3 is an example flow chart for electing a dominant BMC in a domain according to one embodiment. After the initialization process begins (301), the BMC in the domain successfully completes booting and is ready (302). For example, a domain may include one or more chassis, including Figure 2A and Figure 2BThe switched or switchless Ethernet SSD chassis shown. In another example, the domain may include more than one NVMe-oF chassis in the same rack or on multiple racks within a data center. A candidate BMC is selected based on a default selection criteria (303) and broadcast to other peer BMCs to claim dominance (304). For example, the candidate BMC may be the BMC of the switch board with the longest uptime. In a domain with only one candidate BMC, the only candidate BMC may declare its dominance without broadcasting to other peer BMCs. In another example, the candidate BMC may be selected based on different selection criteria other than uptime, such as an IP address, a service set identifier (SSID), a MAC address, or other unique identifier. Determine whether there is an objection raised by a peer BMC (305). If the peer BMC does not raise an objection, confirm that the candidate BMC is selected as the dominant BMC (311), and the election process is completed (312). If the peer BMC raises any objection, the next candidate BMC of the switch board is selected (306). For example, the BMC of the switch board with the second longest uptime is selected. If the selected candidate BMC has the same qualifications as the previous candidate BMC that has been rejected (307), the candidate BMC may be selected as the dominant BMC (311). If the candidate BMC's qualifications are different from the previously rejected candidate BMC, the candidate BMC broadcasts to other peer BMCs to claim dominance (304). This process continues until the dominant BMC is elected. If a dominant BMC is not elected, an error is reported to the system administrator.
[0042] Figure 4 is an example flow chart of replacing a dominant BMC in a domain according to one embodiment. The failover process begins when the current dominant BMC fails and a system administrator receives a report about a problem with the dominant BMC (401). First, a check is made to see if the failed dominant BMC is located in an HA chassis that includes two or more switch boards (402). If so, a standby BMC in the same HA chassis takes over as the dominant (405), and the process is completed (406). If it is confirmed that no heartbeats are being sent from the failed dominant BMC to other peer BMCs (403), then Figure 3 The dominance election process is shown restarting (404).
[0043] Figure 5A domain of an example NVMe-oF domain without a domain Ethernet switch according to one embodiment is shown. Domain 520 includes a switch board 501 and a plurality of switchless boards (JBoFs). Each of switch board 501 and switchless board 502 has two Ethernet ports eth[0] and eth[1] connected to each other in a daisy chain. Ethernet ports eth[0] and eth[1] represent Figure 2A and Figure 2B The management LAN port 215 of the JBoF 502A is connected to the management LAN port 215 of the JBoF 502A. For example, the first Ethernet port eth[0] of the JBoF 502A is connected to the first Ethernet port eth[0] of the switch board 501, and the second Ethernet port eth[1] of the JBoF 502A is connected to the second Ethernet port eth[1] of the next JBoF 502B. The daisy chain connection of the Ethernet ports allows the ruler BMC of the switch board 501 to communicate with the peer BMC of the JBoF 502. The ruler BMC can manage and report device information of the JBoF 502 in the domain 520 to the domain management server via the network 560 (e.g., Ethernet). Although this example shows one switch board and three switchless boards in the domain 520, it should be understood that at least one switch board and any number of switchless boards may be included in the domain 520 without departing from the scope of the present disclosure.
[0044] Figure 6 An example data flow in a domain of an example NVMe-oF domain according to one embodiment is shown. The device information 601a of a switch board or a switchless board includes a BMC ID, device specific information, and a next BMC ID. The next BMC ID points to another device information 601b, and so on. The dominator BMC can collect and summarize the device information of the Ethernet SSD boards in the domain and report to the system administrator. The dominator BMC can also receive commands from the system administrator to operate a specific board (e.g., change configuration or parameters) through peer-to-peer communication between BMCs in the domain.
[0045] refer to Figure 5 , the current NVMe-oF domain may not include a domain Ethernet switch to reduce costs and simplify the configuration of the system. The current NVMe-oF domain provides peer-to-peer communication and management. Once the dominant BMC is elected, the dominant BMC can send a request, and the request can be passed to the target BMC via a direct connection or a daisy chain connection through one or more middle boards. The dominant BMC can collect and summarize device information from each BMC in the domain and report to the system administrator via the network.
[0046] According to one embodiment, the present system and method provide a recursive request processing mechanism to collect all BMC device information in the same domain. Each BMC has its own BMC ID and two management LAN ports, including an upstream port and a downstream port. Each of the upstream port and the downstream port may have a unique IP address and a MAC address. Each BMC is responsible for managing its own device information. The BMC may also be responsible for discovering the downstream BMC ID and passing the device information received via the downstream port from the downstream BMC to the upstream BMC via the upstream port. The dominant BMC may not have an upstream port report. Instead, the dominant BMC may trigger BMC discovery to a peer BMC, process the device information from the peer BMC to identify the addition of a newly added BMC in the domain or the removal of an existing BMC, and perform necessary management tasks. The end BMC at the end of the daisy chain may not have a downstream BMC. In this case, when the upstream BMC queries, the end BMC reports its device information to the upstream BMC.
[0047] Figure 7 A flow chart for processing a device information request according to one embodiment is shown. A BMC in a domain starts / receives a request from an upstream BMC or a dominant BMC in the domain (701). In response to the request, the BMC processes its local device information (702) and updates the device information (703) to report to the requesting BMC. If the next BMC ID is valid (704), in other words, if the BMC has a downstream BMC in a daisy chain, the BMC sends a request to the next BMC to send its device information (707), receives the requested device information from the next BMC (708), and updates the device information from the downstream BMC (703). If there is no valid next BMC, the BMC sends the collected device information to the requesting BMC (705) and terminates the process (706).
[0048] According to one embodiment, a data storage system includes: a plurality of Ethernet solid-state drive (SSD) chassis, including at least one switched Ethernet SSD chassis and one or more switchless Ethernet SSD chassis. At least one switched Ethernet SSD chassis includes an Ethernet switch, a first baseboard management controller (BMC), and a first management local area network (LAN) port. At least one of the one or more switchless Ethernet SSD chassis includes an Ethernet repeater, a second BMC, and a second management LAN port. The first management LAN port and the second management LAN port of at least one switched Ethernet SSD chassis are connected. The first BMC collects the status of at least one of the one or more switch Ethernet SSD chassis from the second BMC via the connection between the first management LAN port and the second management LAN port, and provides device information of at least one of the one or more switch Ethernet SSD chassis and at least one switched Ethernet SSD chassis to a system administrator.
[0049] The data storage system may further include a management Ethernet switch. The first BMC may be connected to the management Ethernet switch via a first management LAN port, and the second BMC may be connected to the management Ethernet switch via a second management LAN port. The first BMC may provide device information of at least one of the one or more switch Ethernet SSD chassis and at least one switched Ethernet SSD chassis to a system administrator via the management Ethernet switch.
[0050] The at least one switched Ethernet SSD enclosure may support message transmission between a host computer and a data storage system via a fabric network.
[0051] The system administrator may send a request or command to one of the first BMC and the second BMC in the data storage system using an Intelligent Platform Management Interface (IPMI) message.
[0052] The request or command may enable discovery of newly added Ethernet SSDs in a domain, and rebooting and configuring one or more Ethernet SSDs connected to one of a plurality of Ethernet SSD enclosures using a static IP or via Dynamic Host Configuration Protocol (DHCP).
[0053] At least one of the one or more switchless Ethernet SSD enclosures may also include an Ethernet SSD (eSSD).
[0054] According to another embodiment, a data storage system includes: a switched Ethernet SSD chassis including an Ethernet switch, a baseboard management controller (BMC), and a management LAN port; and a first switchless Ethernet SSD chassis and a second switchless Ethernet SSD chassis. Each of the first switchless Ethernet SSD chassis and the second switchless Ethernet SSD chassis includes an Ethernet repeater, a BMC, and a management LAN port, which are connected to each other and to the management LAN port of the switched Ethernet SSD. The BMC of the second switchless Ethernet SSD chassis provides device information of the second switchless Ethernet SSD chassis to the BMC of the first switchless Ethernet SSD chassis via the management LAN port. The BMC of the first switchless Ethernet SSD chassis provides device information of the first switchless Ethernet SSD chassis and the second switchless Ethernet SSD chassis to the BMC of the switched Ethernet SSD chassis via the management LAN port. The BMC of the switched Ethernet SSD chassis provides device information of the switched Ethernet SSD chassis, the first switchless Ethernet SSD chassis, and the second switchless Ethernet SSD chassis to a system administrator connected through a fabric network.
[0055] The fabric network can be one of Ethernet, Fibre Channel, and InfiniBand.
[0056] Switched Ethernet SSD enclosures can support the transmission of messages between the host computer and the data storage system over a Fabric network.
[0057] System administrators can use Intelligent Platform Management Interface (IPMI) messages to send requests or commands to the BMC of the switched Ethernet SSD enclosure.
[0058] The request or command may support discovery of newly added Ethernet SSDs in a domain, and restarting and configuring one or more Ethernet SSDs connected to one of a plurality of Ethernet SSD chassis using a static IP or via Dynamic Host Configuration Protocol (DHCP).
[0059] The first and second switchless Ethernet SSD enclosures may also include one or more Ethernet SSDs (eSSDs).
[0060] According to another embodiment, a method includes: selecting a candidate BMC from among a plurality of BMCs in a domain, wherein the domain includes a plurality of Ethernet solid state drive (SSD) chassis, including at least one switched Ethernet SSD chassis and one or more switchless Ethernet SSD chassis; broadcasting to a plurality of BMCs in the domain to assert dominance of the domain; checking the qualifications of the candidate BMCs based on responses received from the plurality of BMCs; and electing the candidate BMC as the dominator BMC of the domain based on the qualifications. The dominator BMC is included in a first switched Ethernet SSD chassis including a first Ethernet switch. The dominator BMC collects device information of a plurality of Ethernet SSD chassis in the domain and provides it to a system administrator via a fabric network.
[0061] Device information for multiple Ethernet SSD enclosures can be collected via daisy chaining through peer-to-peer communication between multiple BMCs in the domain.
[0062] The one or more switchless Ethernet SSD chassis may include a first switchless Ethernet SSD chassis and a second switchless Ethernet SSD chassis. The second switchless Ethernet SSD chassis may have a management LAN port connected to the management LAN port of the first switchless Ethernet SSD chassis, and the BMC of the second switchless Ethernet SSD chassis may send device information of the second switchless Ethernet SSD chassis to the BMC of the first switchless Ethernet SSD chassis.
[0063] The BMC of the first switchless Ethernet SSD chassis may send device information of the first switchless Ethernet SSD chassis and the second switchless Ethernet SSD chassis to the controller BMC.
[0064] The first and second switchless Ethernet SSD enclosures may also include one or more Ethernet solid-state drives (eSSDs).
[0065] The first Ethernet switch may have the highest uptime in the domain.
[0066] The method may also include: determining that the dominant BMC is shut down or out of service; selecting a second candidate BMC from among the plurality of BMCs in the domain, wherein the second candidate BMC is included in a second switched Ethernet SSD chassis having a second Ethernet switch; and electing a new dominant BMC.
[0067] The second Ethernet switch may have the second highest uptime in the domain.
[0068] The above example embodiments have been described above to illustrate various embodiments of implementing systems and methods for supporting inter-chassis manageability of NVMe-oF-based data storage systems. Various modifications and deviations from the disclosed example embodiments will occur to those of ordinary skill in the art. Subject matter intended to fall within the scope of the invention is set forth in the claims.
Claims
1. A data storage system, include: a plurality of Ethernet solid state drive (SSD) chassis, including at least one switched Ethernet SSD chassis and one or more switchless Ethernet SSD chassis, Wherein, the at least one switched Ethernet SSD chassis comprises an Ethernet switch, a first baseboard management controller (BMC) and a first management local area network (LAN) port, wherein at least one of the one or more switchless Ethernet SSD chassis comprises an Ethernet repeater, a second BMC, and a second management LAN port, wherein the first management LAN port and the second management LAN port of the at least one switched Ethernet SSD chassis are connected, and The first BMC collects the status of at least one of the one or more switchless Ethernet SSD chassis from the second BMC via the connection between the first management LAN port and the second management LAN port, and provides the device information of at least one of the one or more switchless Ethernet SSD chassis and the device information of the at least one switched Ethernet SSD chassis to the system administrator.
2. The data storage system according to claim 1, in, The data storage system also includes a management Ethernet switch, wherein the first BMC is connected to the management Ethernet switch via the first management LAN port, the second BMC is connected to the management Ethernet switch via the second management LAN port, and wherein the first BMC provides device information of at least one of the one or more switchless Ethernet SSD chassis and device information of the at least one switched Ethernet SSD chassis to the system administrator via the management Ethernet switch.
3. The data storage system according to claim 1, in, The at least one switched Ethernet SSD enclosure supports transmission of messages between a host computer and the data storage system via a Fabric network.
4. The data storage system according to claim 3, in, The system administrator sends a request or command to one of the first BMC and the second BMC in the data storage system using an Intelligent Platform Management Interface (IPMI) message.
5. The data storage system according to claim 4, in, The request or the command supports discovery of a newly added Ethernet SSD in a domain, and restarting and configuring one or more Ethernet SSDs attached to one of the plurality of Ethernet SSD enclosures using a static IP or via a Dynamic Host Configuration Protocol (DHCP).
6. The data storage system according to claim 1, in, At least one of the one or more switchless Ethernet SSD enclosures also includes an Ethernet SSD (eSSD).
7. A data storage system, include: Switched Ethernet SSD enclosure, including Ethernet switch, baseboard management controller (BMC), and management LAN port; as well as a first switchless Ethernet SSD chassis and a second switchless Ethernet SSD chassis, wherein each of the first switchless Ethernet SSD chassis and the second switchless Ethernet SSD chassis comprises an Ethernet repeater, a BMC, and a management LAN port, the management LAN ports being connected to each other and to a management LAN port of the switched Ethernet SSD, wherein the BMC of the second switchless Ethernet SSD chassis provides the device information of the second switchless Ethernet SSD chassis to the BMC of the first switchless Ethernet SSD chassis via the management LAN port, wherein the BMC of the first switchless Ethernet SSD chassis provides device information of the first switchless Ethernet SSD chassis and the second switchless Ethernet SSD chassis to the BMC of the switched Ethernet SSD chassis via the management LAN port, The BMC of the switched Ethernet SSD chassis provides device information of the switched Ethernet SSD chassis, the first switchless Ethernet SSD chassis, and the second switchless Ethernet SSD chassis to a system administrator connected through a Fabric network.
8. The data storage system according to claim 7, in, The Fabric network is one of Ethernet, Fibre Channel, and InfiniBand.
9. The data storage system according to claim 8, in, The switched Ethernet SSD enclosure supports transmission of messages between a host computer and the data storage system over the Fabric network.
10. The data storage system according to claim 7, in, The system administrator sends a request or command to the BMC of the switched Ethernet SSD enclosure using Intelligent Platform Management Interface (IPMI) messages.
11. The data storage system according to claim 10, in, The request or the command supports discovery of a newly added Ethernet SSD in the domain and restarting and configuring one or more Ethernet SSD chassis attached to the switched Ethernet SSD chassis and one of the first switchless Ethernet SSD and the second switchless Ethernet SSD using a static IP or via Dynamic Host Configuration Protocol (DHCP).
12. The data storage system according to claim 7, in, The first switchless Ethernet SSD chassis and the second switchless Ethernet SSD chassis also include one or more Ethernet SSDs (eSSDs).
13. A method for supporting inter-chassis manageability, include: selecting a candidate BMC from among a plurality of BMCs in a domain, wherein the domain includes a plurality of Ethernet solid-state drive (SSD) chassis including at least one switched Ethernet SSD chassis and one or more switchless Ethernet SSD chassis; broadcasting to multiple BMCs in the domain to declare dominance of the domain; checking eligibility of the candidate BMC based on responses received from the plurality of BMCs; and Based on the qualifications, electing the candidate BMC as the dominant BMC of the domain, The dominator BMC is included in a first switched Ethernet SSD chassis including a first Ethernet switch, The controller BMC collects device information of the multiple Ethernet SSD chassis in the domain and provides it to the system administrator through the Fabric network.
14. The method according to claim 13, in, Device information of the plurality of Ethernet SSD chassis is collected through peer-to-peer communication between the plurality of BMCs in the domain via a daisy chain.
15. The method of claim 13, in, The one or more switchless Ethernet SSD chassis include a first switchless Ethernet SSD chassis and a second switchless Ethernet SSD chassis, wherein the second switchless Ethernet SSD chassis has a management LAN port connected to the management LAN port of the first switchless Ethernet SSD chassis, and the BMC of the second switchless Ethernet SSD chassis sends device information of the second switchless Ethernet SSD chassis to the BMC of the first switchless Ethernet SSD chassis.
16. The method of claim 15, in, The BMC of the first switchless Ethernet SSD chassis sends device information of the first switchless Ethernet SSD chassis and the second switchless Ethernet SSD chassis to the controller BMC.
17. The method of claim 15, in, The first switchless Ethernet SSD enclosure and the second switchless Ethernet SSD enclosure also include one or more Ethernet solid-state drives (eSSDs).
18. The method of claim 13, in, The first Ethernet switch has the longest uptime in the domain.
19. The method of claim 13, further comprising: include: Determine that the controller BMC is shut down or stops service; selecting a second candidate BMC among the plurality of BMCs in the domain, wherein the second candidate BMC is included in a second switched Ethernet SSD chassis having a second Ethernet switch; as well as Elect a new ruler, the BMC.
20. The method of claim 19, in, The second Ethernet switch has a second longest uptime in the domain.
