Transferring data between solid state drives (SSDs) via connections between SSDs

Abstract

A first solid state drive (SSD) includes a first built-in network interface device configured to communicate via a network fabric, and a second SSD includes a second built-in network interface device configured to communicate via the network fabric. A connection is opened between the first SSD and the second SSD over the network fabric. Based on a non-volatile memory (NVMe-oF) communication protocol on a structure, NVMe commands for transferring data between a first SSD and a second SSD over a connection are encapsulated in an encapsulation. The package is sent over the connection from the first SSD to the second SSD via the network fabric. The second SSD executes the NVMe command in the package to transfer data between the first SSD and the second SSD over the connection.

Description

[0001] This application is a divisional application of an application with an application date of March 16, 2020, an application number of 202080032451.7, and an invention title of "Transferring Data between Solid State Drives (SSDs) via a Connection between SSDs". Technical Field

[0002] The present disclosure relates to storage services, namely, transferring data between solid state drives (SSDs) having built-in network interface devices via a connection between the SSDs on a network fabric. Background Art

[0003] The background description provided herein is to present the context of the present disclosure generally. The work of the currently named inventors, to the extent it is described in this background section, and aspects of the work that, as of the time of filing the application, may not constitute prior art as described, are neither expressly nor impliedly admitted to be prior art to the present disclosure.

[0004] A host computer ("host") communicates with a solid state drive (SSD) to obtain data services provided by the SSD, such as reading data, storing data, and erasing data. A solid state drive (SSD) is a data storage device that uses non-volatile memory, such as NAND (Not-And) non-volatile memory, to store persistent digitally encoded data. The SSD can be configured to emulate a hard disk drive, i.e., a device that stores persistent digitally encoded data on the magnetic surface of a rapidly rotating disk platter and replaces a hard disk drive (HDD) in many applications. The SSD includes an interface, a computing unit, an aggregator, and one or more non-volatile memories. The interface is capable of communicating with the host computer via a network, which may include one or more intermediate devices, such as one or more interface switches and / or routers, coupled via wired and / or wireless interconnections. The computing unit is a processor that coordinates operations on the SSD, and the aggregator provides a Peripheral Component Interconnect Express (PCIe) connection to the one or more non-volatile memories.

[0005] The host sends commands over the network to the SSD to obtain data services. The commands are received at the interface, and based on the commands, access to the non-volatile memory of the SSD is computed over the PCIe connection of the aggregator. The commands from the host conform to the Non-Volatile Memory Express over Fabric (NVME-oF) protocol associated with accessing the SSD over the network. The access to the non-volatile memory is computed via the NVMe protocol, which defines a set of commands for accessing the non-volatile memory over the PCIe connection. The host also facilitates the transfer of data from one SSD to another. To transfer data between SSDs, the host sends one or more commands over the network to the SSDs to cause the computation to provide the data stored on one SSD to the host. Then, the host sends one or more commands to the SSDs to cause the data on the host to be stored in another SSD. The transfer of data from one SSD to another via the host generates data traffic on the network between the host and the SSDs. Summary of the Invention

[0006] The present disclosure relates to solid state drive (SSD) devices, specifically to transferring data between SSDs having respective built-in network interface devices. The SSDs having respective built-in network interface devices are coupled together via a network fabric, and data is transferred between the SSDs over the connection between the SSDs and via the network fabric, rather than via a host as an intermediary.

[0007] In one embodiment, a method for transferring data between a first solid state drive (SSD) and a second SSD, where the first SSD has a first built-in network interface device configured to communicate via a network fabric, and the second SSD has a second built-in network interface device configured to communicate via a network fabric. The method includes: opening a connection between the first SSD and the second SSD over the network fabric; encapsulating NVMe commands in a packet based on the Non-Volatile Memory (NVMe-oF) communication protocol to transfer data between the first SSD and the second SSD over the connection; sending the packet from the first SSD to the second SSD over the connection via the network fabric; and executing the NVMe commands in the packet by the second SSD to transfer data between the first SSD and the second SSD over the connection. Brief Description of the Drawings

[0008] Figure 1 is a diagram of an example storage system according to one embodiment, which facilitates transferring data between solid state drives (SSDs) having respective built-in network interface devices, and the transfer of data occurs over the connection between the SSDs and via the connection of the network fabric.

[0009] Figure 2 It is a diagram of another example storage system according to another embodiment.

[0010] Figure 3 It is a diagram of yet another example storage system according to yet another embodiment.

[0011] Figure 4 It is a diagram of an example Ethernet frame according to some embodiments, which is used to transfer data between SSDs on the connection between SSDs in the Figures 1 to 3 example storage system.

[0012] Figure 5 It is a flowchart of an example method according to one embodiment, which is used to transfer data between SSDs with corresponding built-in network interface devices on the connection between SSDs.

[0013] The accompanying drawings are for illustrative purposes of example embodiments, but it should be understood that the embodiments are not limited to the arrangements and means shown in the drawings. Detailed Description

[0014] This disclosure relates to storage services and, in particular, to transferring data between solid state drives (SSDs) via a connection between the SSDs that does not include a host processor. For example, data is transferred between SSDs without having to transfer the data from one SSD to the host processor and then from the host processor to another SSD. In one embodiment, an SSD includes a respective built-in network interface device (e.g., an Ethernet network interface device, a Fibre Channel network interface device, an InfiniBand network interface device, etc.). An SSD having a respective built-in network interface device is a storage device, e.g., the SSDs are capable of peer-to-peer communication and communicate via a communication protocol (e.g., the Non-Volatile Memory over Fabrics (NVMe-oF) protocol or another suitable communication protocol) that is designed to control (or communicate with) non-volatile storage devices such as SSDs. According to various embodiments, the SSDs described herein communicate over a connection between the SSDs. According to various embodiments, the SSDs described herein communicate according to one or more suitable communication protocols, such as one or more of the following: Fibre Channel, InfiniBand, Fibre Channel over Ethernet (FCoE), Remote Direct Memory Access (RDMA), RDMA over Converged Ethernet (RoCE), User Datagram Protocol (UDP) as RDMA over the transport layer, iWARP (a network protocol that implements RDMA over Transmission Control Protocol (TCP) and Internet Protocol (IP) (TCP / IP)), TCP as Non-Volatile Memory Express (NVMe) over the transport layer, etc., or other suitable communication protocols for exchanging data with a storage subsystem over a network fabric (such as an Ethernet fabric, a Fibre Channel fabric, an InfiniBand fabric, etc.). Examples of the systems, apparatus, methods, and techniques described herein transfer data between SSDs having respective built-in network interface devices via a connection between the SSDs without having to transfer the data from one SSD to the host and then from the host to another SSD, thereby reducing the amount of data transferred between the SSDs and the host and / or reducing the processing performed by the host.

[0015] The NVMe-oF protocol is related to the NVM Express (NVMe) protocol. NVMe defines a register-level interface for communication between a host and a non-volatile memory subsystem over a PCI Express (PCIe) bus. NVMe-oF is built on the architecture, command set, and queue interface defined by NVMe and is a protocol that enables NVMe access to SSDs over a network fabric. Although both NVMe and NVMe-oF have similar characteristics, such as using a command set to control operations on a storage device as requested by a host, one of the main differences between NVMe and NVMe-oF is the transport mapping mechanism for sending and receiving commands and responses that communicate between devices. NVMe-oF uses a message-based model for communication between a host and an SSD, where local NVMe communication involves mapping commands and responses to shared memory in the host over a PCIe interface protocol. NVMe-oF enables the use of alternative transports to PCIe, where these alternative transports extend the distance across which a host and an SSD are connected. Greater storage system throughput is achieved by using NVMe-oF technology.

[0016] The embodiments described below utilize SSDs with built-in Ethernet interface devices (sometimes referred to herein as Ethernet-enabled SSDs). Embodiments with Ethernet-enabled SSDs are merely illustrative examples of SSDs with built-in network interface devices. However, other embodiments employ SSDs with other types of built-in network interface devices, such as Fibre Channel network interface devices, InfiniBand network interface devices, etc.

[0017] In some embodiments, an SSD with a built-in network interface device is (at least partially) encapsulated in a housing having an appropriate form factor, such as a standard hard disk drive (HDD) / SSD form factor, such as a 3.5-inch form factor, a 2.5-inch form factor, or a 1.8-inch form factor. The housing may include a physical network interface connector (or define an aperture for receiving it) suitable for the network interface device. For example, according to one embodiment, if the built-in network interface device includes an Ethernet interface device, the housing may include an appropriate physical Ethernet connector (e.g., an RJ45 connector) (or define an aperture for receiving it). As another example, according to another embodiment, if the built-in network interface device includes a Fibre Channel (FC) interface device, the housing may include an appropriate physical FC connector (or define an aperture for receiving it). As another example, according to another embodiment, if the built-in network interface device includes an InfiniBand interface device, the housing may include an appropriate physical InfiniBand connector (e.g., a QSFP connector, a CXP connector, an MPO connector, etc.) (or define an aperture for receiving it).

[0018] In other embodiments, an SSD with a built-in network interface device is not encapsulated in a housing.

[0019] In some embodiments, the SSD with a built-in network interface device described herein has a structure such as that disclosed in the U.S. patent application Ser. No. filed on the same day as this application and titled "Ethernet Enabled Solid State Drive (SSD)", which is incorporated herein by reference in its entirety. In other embodiments, the SSD with a built-in network interface device has a different suitable structure.

[0020] Figure 1 FIG. is a diagram of an example storage system 100 according to one embodiment, the storage system 100 enabling data transfer between Ethernet-enabled SSDs via a connection between the Ethernet-enabled SSDs that does not include a host. The example storage system 100 includes a host 102 coupled to an interconnect 114, which in turn is coupled to a storage subsystem 104. The storage subsystem 104 has a network fabric 106 and two Ethernet-enabled SSDs, shown as Ethernet-enabled SSD 108 and Ethernet-enabled SSD 110. The interconnect 114 can be a network between the host 102 and the storage subsystem 104, such as a local area network (LAN) or a wide area network (WAN). The interconnect 114 can include one or more intermediate devices, such as one or more interface switches and / or routers, which are coupled by wired and / or wireless interconnections. In the example, according to one embodiment, the storage subsystem 104 can provide connectivity among the network fabric 106, the Ethernet-enabled SSD 108, and the Ethernet-enabled SSD 110, and one or more of the storage subsystem 104 and the host 102 can be located in a data center rack. The rack can have a plurality of mounting slots called bays, each bay being designed to hold a hardware unit such as the host 102 or the storage subsystem 104. In some examples, the rack can include a top-of-rack switch, which provides connectivity between the hardware units and a remote network. In other examples, the interconnect 114 includes the backplane of the rack. Additionally, more than one storage subsystem 104 and more than one host 102 can be coupled to the interconnect 114. Other storage systems can be part of a first rack, where the storage subsystem 104 is located or in a second rack. Similarly, a second host can be part of the first rack, and the host 102 is located or in a second rack.

[0021] The host 102 includes any type of host, such as a computer processor or a network of computers and processors. Additionally, the host 102 need not be limited to a single host device and can represent multiple host devices. In one embodiment, the host 102 includes a memory 122 in the form of dynamic random access memory (DRAM), a processor 124 such as a central processing unit (CPU), and a network interface card (NIC) 126. The memory 122 can store data only when powered on, and the processor 124 can be implemented on one or more integrated circuits and is configured to execute machine-readable instructions stored in the memory 122 (or another memory (not shown)) to perform arithmetic, logical, input / output (I / O), and other operations. The host 102 may or may not have a limited memory (e.g., non-volatile memory) to store data when the memory is not powered on. To facilitate storing data in the non-volatile memory of the storage subsystem 104, the host 102 utilizes the NIC 126 to access the non-volatile memory of the storage subsystem 104. The NIC 126 can facilitate the transfer of data over the interconnect 114 between the host 102 and the storage subsystem 104. In one embodiment, the NIC 126 includes an Ethernet network interface device.

[0022] The storage subsystem 104 includes an Ethernet-enabled SSD 108 and an Ethernet-enabled SSD 110. The Ethernet-enabled SSDs 108 / 110 include data storage devices that use non-volatile memory, such as NAND (Not-And) non-volatile memory, flash memory, etc., to persistently store digitally encoded data. The Ethernet-enabled SSDs 108 / 110 can be configured to emulate a hard disk drive (HDD). The Ethernet-enabled SSD 108 and the Ethernet-enabled SSD 110 can store data in respective non-volatile memories (NVMs) 116, 118. In an example, the Ethernet-enabled SSD 108 can be configured with a processor 128, an aggregator such as a PCI Express (PCIe) cluster 130, and one or more non-volatile memories 116. In one embodiment, the processor 128 is configured to execute machine-readable instructions stored in a memory (not shown) to perform arithmetic, logical, I / O, and other operations. In another embodiment, the processor 128 additionally or alternatively includes a hardware processor (e.g., including one or more hardware state machines) that includes hardware circuits configured to perform arithmetic, logical, I / O, and other operations. The PCIe cluster 130 provides a PCIe connection to one or more non-volatile memories 116. The processor 128 accesses the non-volatile memory 116 over the PCIe connection of the PCIe cluster 130 using, for example, the NVMe protocol or other suitable protocol. The NVMe protocol defines a command set for accessing data stored on the non-volatile memory 116 over the PCIe connection of the PCIe cluster 130 to the non-volatile memory 116. In other embodiments, the NVM 116 of the Ethernet-enabled SSD 108 is accessed via another suitable bus / connection, such as a Fibre Channel network, a Serial AT Attachment (SATA) bus, a Serial Attached Small Computer System Interface (SCSI) (SAS) connection, etc.

[0023] The Ethernet-enabled SSD 108 can also be configured to implement the NVMe-oF protocol. In various embodiments, NVMe-oF is built on the architecture, command set, and queuing interfaces defined by NVMe and supports using FCoE, RDMA, InfiniBand, iWARP, RoCEv2, NVMeTCP (which specifies the NVMe protocol over TCP), etc., to perform NVMe access to the Ethernet-enabled SSD 108 over the network fabric 106. To support such access, the Ethernet-enabled SSD 108 includes an Ethernet interface device 136, which is configured to communicate over the network fabric 106 according to the Ethernet protocol. Additionally, according to various embodiments, the processor 128 is configured to implement one or more protocols, such as NVMe, SATA, SAS, FCoE, RDMA, InfiniBand, iWARP, RoCE, NVMeTCP, etc. In one embodiment, the Ethernet-enabled SSD 108 includes a submission queue 132 and a completion queue 134. The submission queue 132 and the completion queue 134 are shown as separate from the processor 128, but in other examples, the submission queue 132 and the completion queue 134 can be integrated into the processor 128. Both the submission queue 132 and the completion queue 134 can be associated with unidirectional communication channels. The submission queue 132 can facilitate sending information units, referred to herein as "capsules," to another Ethernet-enabled SSD, such as the Ethernet-enabled SSD 110, and the completion queue 134 can facilitate receiving capsules from other Ethernet-enabled SSDs, such as the Ethernet-enabled SSD 110. A capsule is an information exchange unit associated with NVMe-oF and can include an NVMe command, a response to an NVMe command, data, and / or a scatter-gather list (SGL).

[0024] The Ethernet-enabled SSD 110 can have components that are the same as or similar to the components shown and described with respect to the Ethernet-enabled SSD 108. For example, the Ethernet-enabled SSD 110 can have a corresponding Ethernet interface device 150, a processor 152, a PCIe cluster 154, NVM 118, a submission queue 156, and a completion queue 158. In other examples, compared to the Ethernet-enabled SSD 108, the Ethernet-enabled SSD 110 can include other components and / or can omit components. Additionally, in other examples, the storage subsystem 104 can have more or fewer Ethernet-enabled SSDs than Figure 1 shown.

[0025] The connection between the host 102 and the Ethernet-enabled SSD can allow the host 102 to access the data stored on the Ethernet-enabled SSD. This connection can also support data transfer from one SSD to the memory 122 of the host 102 and then from the memory 122 of the host 102 to the second SSD. Instead of the host 102 facilitating data transfer between the SSDs, the Ethernet-enabled SSDs 108, 110 are configured to perform data transfer between each other without first transferring the data to the host 102. The data transfer is performed via the connection 138 between the Ethernet-enabled SSD 108 and the Ethernet-enabled SSD 110, rather than the connection between the Ethernet-enabled SSD 108 or the Ethernet-enabled SSD 110 and the host 102. The transfer of data includes transferring data from the Ethernet-enabled SSD 108 to the Ethernet-enabled SSD 110 on the connection 138 (which connects the Ethernet-enabled SSD 108 and the Ethernet-enabled SSD 110) without transferring the data to the memory 122 of the host 102 as an intermediate operation, thereby reducing the amount of data transferred between the Ethernet-enabled SSDs 108 / 110 and the host 102 on the interconnect 114 and / or the amount of data processed by the host 102.

[0026] As an illustrative example, the first Ethernet-enabled SSD 108 can copy data to the second Ethernet-enabled SSD 110. The non-volatile memory 116 can be an addressable memory, and the data to be copied can be located within the address range of the non-volatile memory 116 of the first Ethernet-enabled SSD 108. The first Ethernet-enabled SSD 108 can open a connection 138 to the second Ethernet-enabled SSD 110 on the network fabric 106. As an illustrative example, the connection 138 can be a Fibre Channel-based NVMe-oF transport layer connection. Alternatively, the connection 138 can be based on RDMA, InfiniBand, RoCEv2, iWARP, NVMeTCP, or other suitable protocols. The connection 138 can be opened before data is transferred between the Ethernet-enabled SSDs. The connection 138 can establish one or more queues, such as queues 132, 134 and queues 156, 158, to facilitate subsequent transfer of data between the Ethernet-enabled SSD 108 and the Ethernet-enabled SSD 110 on the connection 138.

[0027] The processor 128 of the first Ethernet-enabled SSD 108 can send the packet to the second Ethernet-enabled SSD 110 over the connection 138 in the network fabric 106 by placing the packet in the submission queue 132. The packet to be sent to the second Ethernet-enabled SSD 110 can include commands (such as NVMe write commands), data from the NVM 116 to be copied to the second Ethernet-enabled SSD 110, SGLs indicating the network addresses of the data to be copied in the non-volatile memory 116, etc. The processor 128 generates an Ethernet packet including the packet and provides the Ethernet packet to the Ethernet interface device 136. The Ethernet interface device 136 transmits the Ethernet packet to the network fabric 106.

[0028] The Ethernet interface device 150 of the second Ethernet-enabled SSD 110 receives the Ethernet packet from the network fabric 106 and provides the Ethernet packet to the processor 152. The processor 152 retrieves the packet from the Ethernet packet and stores the packet in the submission queue 156. Subsequently, the processor 158 retrieves the packet from the submission queue 156 and, in response to the command in the packet, copies the data from the first Ethernet-enabled SSD 108 to the NVM 118 of the second Ethernet-enabled SSD 110. The data to be copied can be included in the packet, or the second Ethernet-enabled SSD 110 can use the SGL information in the packet to subsequently request the data identified by the SGL from the first Ethernet-enabled SSD 110 over the connection 138 and copy the requested data to the NVM 118 when the requested data is received over the connection 138. The data can be copied without the data retrieved from the first Ethernet-enabled SSD 108 being copied to the host 102 over the interconnect 114 and without the host 102 copying the data to the second Ethernet-enabled SSD 110 over the interconnect 114. In some examples, the second Ethernet-enabled SSD 110 can send an NVMe response to the NVMe command over the connection 138 by placing a packet with the NVMe response in the completion queue 158 to indicate the completion of the copy. The processor 152 generates an Ethernet packet including the packet and provides the Ethernet packet to the Ethernet interface device 150. The Ethernet interface device 150 transmits the Ethernet packet to the network fabric 106.

[0029] The Ethernet interface device 136 of the second Ethernet-enabled SSD 110 receives Ethernet packets from the network fabric 106 and provides the Ethernet packets to the processor 128. The processor 128 retrieves the encapsulation from the Ethernet packets and stores the encapsulation in the completion queue 134. Subsequently, the processor 128 retrieves the encapsulation from the completion queue 134 and determines, in response to the NVMe response in the encapsulation, that the copy of data from the first Ethernet-enabled SSD 108 to the NVM 118 of the second Ethernet-enabled SSD 110 has been completed.

[0030] In an example, the first Ethernet-enabled SSD 108 and the second Ethernet-enabled SSD 110 may have addressable local random access memories (RAMs), such as dynamic RAM (DRAM) 160 and DRAM 162, to facilitate the copy of data. For example, as part of a copy command, the first Ethernet-enabled SSD 108 may copy data from the NVM 116 into its DRAM 160. The local address in the DRAM 160 into which the data is copied may be stored in the encapsulation along with the write command. The encapsulation may be placed in the submission queue 132. Then, when the encapsulation is received by the second Ethernet-enabled SSD 110, the second Ethernet-enabled SSD 110 may obtain the data from the DRAM 160 of the first Ethernet-enabled SSD 108 based on the local address for storage in the DRAM 162. The second Ethernet-enabled SSD 110 may then store the data in the DRAM 162 to the NVM 118 via the PCIe connection according to NVMe to complete the copy of the data. The DRAM 160 and DRAM 162 may facilitate the copy when the NVM 116 cannot be directly accessed by the second Ethernet-enabled SSD 110.

[0031] According to one embodiment, a copy operation may be performed to form a redundant array of independent disks (RAID), where each Ethernet-enabled SSD corresponds to a "disk" of the RAID and each Ethernet-enabled SSD has a copy of the same data. For example, the host 102 may first copy data to one Ethernet-enabled SSD. Then, that one Ethernet-enabled SSD may copy the data to another Ethernet-enabled SSD over the connection 138 to form the RAID. In the example, the copy may be repeated one or more times further between additional Ethernet-enabled SSDs to form additional Ethernet-enabled SSDs with copies of the data.

[0032] Other data operations may include retrieving data from one Ethernet-enabled SSD and storing it in another Ethernet-enabled SSD via the connection 138 between the Ethernet-enabled SSDs in the storage subsystem 104. For example, the first Ethernet-enabled SSD 108 may send the capsule by placing it in the submission queue 132 and sending the capsule over the fabric 106 on the connection 138 to the second Ethernet-enabled SSD 110. The capsule sent to the second Ethernet-enabled SSD 110 may contain an NVMe read command and may contain an indication of the data to be read from the non-volatile memory 118 of the second Ethernet-enabled SSD 110, such as an SGL that indicates the network address of the data to be retrieved in the non-volatile memory 118. The processor 128 generates an Ethernet packet including the capsule and provides the Ethernet packet to the Ethernet interface device 136. The Ethernet interface device 136 transmits the Ethernet packet to the fabric 106.

[0033] The Ethernet interface device 150 of the second Ethernet-enabled SSD 110 receives the Ethernet packet from the fabric 106 and provides the Ethernet packet to the processor 152. The processor 152 retrieves the capsule from the Ethernet packet and stores the capsule in the submission queue 156. In response to the capsule in the submission queue 156, the second Ethernet-enabled SSD 110 also provides data to the first Ethernet-enabled SSD 108 on the connection 138. The second Ethernet-enabled SSD 110 may send an NVMe response with the data to the first Ethernet-enabled SSD 108 by placing the NVMe response in the capsule and placing the capsule with the NVMe response in the completion queue 158. The processor 152 generates an Ethernet packet including the capsule and provides the Ethernet packet to the Ethernet interface device 150. The Ethernet interface device 150 transmits the Ethernet packet to the fabric 106. The NVMe-oF response may be sent to the completion queue 134 of the Ethernet-enabled SSD 108 on the connection 138.

[0034] In the example, DRAM 160 and DRAM 162 can facilitate the first Ethernet-enabled SSD 108 to retrieve data from the NVM 118 of the second Ethernet-enabled SSD 110. The local address in DRAM 160 where the second Ethernet-enabled SSD 110 is to write the retrieved data can be stored in the encapsulation body as a read command. The encapsulation body can be placed in the submission queue 132. Then, when the encapsulation body is received by the second Ethernet-enabled SSD 110, the second Ethernet-enabled SSD 110 can read the data to be retrieved from the NVM 118 based on the SGL in the encapsulation body and store the data in DRAM162. Then, the second Ethernet-enabled SSD 110 can write the data to be retrieved to DRAM 160 based on the local address in the encapsulation body. Then, the first Ethernet-enabled SSD 108 can write the data in DRAM160 to the NVM 116 via the PCIe connection according to NVMe to complete the read command. When the NVM 118 cannot be directly accessed by the first Ethernet-enabled SSD 108, DRAM 160 and DRAM 162 can facilitate the retrieval.

[0035] The retrieval operation can be used to build data in one Ethernet-enabled SSD based on the data in another Ethernet-enabled SSD. For example, if the data on the "disk" of a RAID is corrupted. The Ethernet-enabled SSD can retrieve data from another Ethernet-enabled SSD so that it has an identical data copy on the other Ethernet-enabled SSD.

[0036] Figure 2 is a block diagram of another example of a storage system 200 according to another embodiment. The storage system 200 includes Figure 1 the Ethernet-enabled SSDs 108 and 110. The processor 128 of the Ethernet-enabled SSD 108 executes communication protocol layers 214, 216, 218 to facilitate the transfer of data between the Ethernet-enabled SSDs over the connection between the Ethernet-enabled SSDs. In one embodiment, the communication protocol layers 214, 216, 218 are implemented using software executed by the processor 128. In another embodiment, at least a portion of the communication protocol layers 214, 216, 218 is implemented using hardware circuitry, such as one or more hardware state machines.

[0037] The storage system 200 includes Figure 1The components of the example storage system 100 are not shown in the storage system 200 for simplicity. The example storage system 200 includes a host 102, an interconnect 114, storage systems 104, 250, Ethernet switches 106, 260, and Ethernet-enabled SSDs shown as Ethernet-enabled SSD 108, Ethernet-enabled SSD 110, Ethernet-enabled SSD 256, and Ethernet-enabled SSD 258.

[0038] The Ethernet switch 106 includes one or more network ports 210. One or more of the network ports 210 can be physical interfaces that enable communication between the Ethernet-enabled SSDs or between an Ethernet-enabled SSD and the host 102. Additionally, the Ethernet-enabled SSDs 108, 110, 256, and 258 can each have one or more network ports 212 to support communication with the Ethernet switch 106.

[0039] The Ethernet-enabled SSD 108 can also be configured with one or more communication protocol layers associated with accessing data stored on the Ethernet-enabled SSD. For example, the Ethernet-enabled SSD 108 can have an NVMe layer 214, an NVMe-oF layer 216, and an RDMA layer 218. The NVMe layer 214 can be a protocol layer that defines an architecture, command set, and queuing interface for accessing data stored in the non-volatile memory 116 on the PCIe cluster 130. The NVMe-oF layer 216 can be a protocol layer that defines an extension to the NVMe layer 214 for accessing the SSD over the Ethernet switch 106. The RDMA layer 218 can be a transport protocol layer that provides reliable delivery of data, NVMe commands, and NVMe responses over the Ethernet switch 106. The RDMA layer 218 facilitates direct memory access (DMA) operations for the data, SGL, commands, and responses to be delivered, reducing delivery latency compared to delivery via the application stack. According to various embodiments, the RDMA layer 218 can take the form of RDMA over Converged Ethernet v2 (e.g., RoCEv2) over aggregated Ethernet to support RDMA over Ethernet, InfiniBand, or iWARP. In some embodiments, the functions of the NVMe-oF layer 216 and the RDMA layer 218 can be configured as a single layer. In some embodiments, the RDMA layer 218 is integrated within the NVMe-oF layer 216.

[0040] The Ethernet-enabled SSDs 110, 256, and 258 can have Figure 1 andFigure 2 Components similar to those shown and described for the Ethernet-enabled SSD 108. For example, the Ethernet-enabled SSD 110, the Ethernet-enabled SSD 256, and the Ethernet-enabled SSD 258 may have corresponding NVMe layers, NVMe-oF layers, and RDMA layers. In other examples, compared to the Ethernet-enabled SSD 108, the Ethernet-enabled SSD 110, the Ethernet-enabled SSD 256, and the Ethernet-enabled SSD 258 may have additional layers.

[0041] The various software layers in the Ethernet-enabled SSDs can be used to perform data operations between the first Ethernet-enabled SSD 108 and the second Ethernet-enabled SSD 110, such as copying data in the NVM 116 of the first Ethernet-enabled SSD 108 to the NVM 118 of the second Ethernet-enabled SSD 110 via the connection 138 between the first Ethernet-enabled SSD and the second Ethernet-enabled SSD, which reduces the need to transfer data to the host 102 on the interconnect 114 as an intermediate step.

[0042] For example, the RDMA layer of the first Ethernet-enabled SSD 108 and the RDMA layer of the second Ethernet-enabled SSD 110 can open the connection 138 between the first Ethernet-enabled SSD 108 and the second Ethernet-enabled SSD 110. The corresponding RDMA layers can open the connection 138 based on the port identifier (ID) associated with the first Ethernet-enabled SSD 108 and the port ID associated with the second Ethernet-enabled SSD 110, which uniquely identifies the port of the connection 138. An Ethernet-enabled SSD port is a protocol interface between the Ethernet-enabled SSD and the Ethernet switch 106 and is a collection of one or more physical fabric interface(s) that together act as a single protocol interface. An Ethernet-enabled SSD can have one or more Ethernet-enabled SSD ports. Each Ethernet-enabled SSD port can have a port ID, which can be a 16-bit identifier. The Ethernet-enabled SSD ports of an Ethernet-enabled SSD can support different NVMe-oF transport connections. If there is more than one NVMe-oF transport binding specification in the underlying fabric (e.g., the Ethernet-enabled SSD port identified by the port ID can support both iWARP and RoCEv2), the Ethernet-enabled SSD port can support multiple NVMe-oF transport connections.

[0043] Each of the Ethernet-enabled SSDs 108 and 110 may also have a network address that uniquely identifies the Ethernet-enabled SSD in the network. The network address can be any logical or physical address, such as a Media Access Control (MAC) address or an Internet Protocol (IP) address. The connection 138 can also be based on the network address of the first Ethernet-enabled SSD 108 and the network address of the second Ethernet-enabled SSD 110. The NVMe layer of the first Ethernet-enabled SSD 108 may generate a command associated with a data operation, which is then encapsulated in a packet by the NVMe-oF layer 216 and placed in the submission queue of the Ethernet-enabled SSD 108. The RDMA layer of the first Ethernet-enabled SSD 108 may transmit the packet through its network port 212 and over the connection 138 of the Ethernet switch 106 to the second Ethernet-enabled SSD 110. The RDMA layer of the second Ethernet-enabled SSD 110 may receive the packet over the connection 138 via its network port 212. The NVMe-oF layer of the second Ethernet-enabled SSD 110 may recover the NVMe command and provide the NVMe command to its NVMe layer to perform the data operation. Additionally, as described above, the functions of the NVMe-oF layer 216 and the RDMA layer 218 may be configured as a single layer. In various examples, the data operation may be to copy data in the NVM 116 of the first Ethernet-enabled SSD 108 to the NVM 118 of the second Ethernet-enabled SSD 110, or to retrieve data in the NVM 118 of the second Ethernet-enabled SSD 110 by the first Ethernet-enabled SSD 108 and store the data in the NVM 116 of the first Ethernet-enabled SSD 108. In some examples, the RDMA layer of the second Ethernet-enabled SSD 110 may additionally send an NVMe response, which is also in the packet, to the first Ethernet-enabled SSD 108 over the connection 138. The sending of the response may be based on the type and content of the NVMe command as described above. The response may have been generated by the NVMe layer of the second Ethernet-enabled SSD 110 and encapsulated in a packet by the NVMe-oF layer of the second Ethernet-enabled SSD 110. The packet may then be placed in the completion queue of the second Ethernet-enabled SSD 110 for transmission to the first Ethernet-enabled SSD 108 by the RDMA layer of the second Ethernet-enabled SSD 110.

[0044] The connection 138 described as being opened between the Ethernet-enabled SSDs is exemplary in nature. The connections opened to perform data operations can take various forms, depending on which Ethernet-enabled SSDs will transfer data. For example, the Ethernet-enabled SSD 256 and the Ethernet-enabled SSD 258 on different storage systems 250, 104 can open a connection 254. The connection 254 can be on the Ethernet switch 106, the Ethernet switch 260, and the interconnect 114. Other variations are possible.

[0045] Figure 3 is a block diagram of yet another example of a storage system 300 according to another embodiment. The storage system 300 can include Figure 1 the components of the example storage system 100 in, which for simplicity are not shown in the storage system 300. The storage system 200 includes Figure 1 the SSDs 108 and 110 of, where the SSDs 108 and 110 include corresponding Fibre Channel (FC) network interface devices instead of the Ethernet network interface devices 136, 150. In conjunction with Figure 3 , the SSDs 108 and 110 are sometimes referred to as FC-enabled SSDs.

[0046] In an example of the storage system 300, the network fabric 106 of the storage subsystem 104 includes an FC switch 304, and the storage subsystem 250 includes an FC switch 306.

[0047] The processor 128 of the FC-enabled SSD 108 executes communication protocol layers 214, 216, 302 to facilitate the transfer of data between the FC-enabled SSDs over a connection between the FC-enabled SSDs. In one embodiment, the communication protocol layers 214, 216, 302 are implemented using software executed by the processor 128. In another embodiment, at least a portion of the communication protocol layers 214, 216, 302 is implemented using hardware circuitry, such as one or more hardware state machines.

[0048] The example storage system 300 includes a host 102, an interconnect 114, a storage subsystem 104, and FC-enabled SSDs, which are shown as FC-enabled SSD 108, FC-enabled SSD 110, FC-enabled SSD 256, and FC-enabled SSD 258.

[0049] FC-enabled SSDs 108, as well as similar FC-enabled SSDs 110, FC-enabled SSDs 256, and FC-enabled SSDs 258 implement various communication protocol layers, including NVMe layer 214 and NVMe-oF layer 216. FC-enabled SSDs 108, as well as similar FC-enabled SSDs 110, FC-enabled SSDs 256, and FC-enabled SSDs 258 also implement the NVMe-over-Fabric (FC-NVMe) layer over Fibre Channel, which includes a transport protocol layer that provides reliable delivery of data, NVMe commands, NVMe responses, SGLs, and / or data on FC switches 304 and / or FC switches 306. In some embodiments, the functions of NVMe-oF layer 216 and FC-NVMe layer 302 may be configured as a single layer. In some embodiments, FC-NVMe layer 302 is integrated within NVMe-oF layer 216. FC switches 304, 306 facilitate FC communication from one FC-enabled SSD to another FC-enabled SSD. Each of network ports 210, 212 may be an FC port, and each of the FC-enabled SSDs may have a unique network address, such as a unique FC address. FC-NVMe layer 302 (or another suitable communication protocol layer) may establish a logical connection between a first FC-enabled SSD 108 and a second FC-enabled SSD 110 based on the respective port ID and / or network address to perform the data operations described above, reducing the need to transfer data from the first FC-enabled SSD to host 102 and then from the host to the second FC-enabled SSD as an intermediate step. Connections 138 and connection 254 are illustrated, but the connections may also take other forms.

[0050] Figure 4 is a diagram of an example Ethernet frame 400 according to an embodiment, which is associated with the transfer of data between a first Ethernet-enabled SSD and a second Ethernet-enabled SSD over an Ethernet connection and an Ethernet-enabled network fabric. Frame 400 includes an Ethernet layer 2 header 402 and an Ethernet type field 404, which indicates that frame 400 is associated with Ethernet. In one embodiment, header 402 includes a transmitter MAC address (e.g., the MAC address of the Ethernet-enabled SSD transmitting Ethernet frame 400) and a receiver MAC address (e.g., the MAC address of the Ethernet-enabled SSD that will receive Ethernet frame 400). Example frame 400 may include an IP header 406, which indicates a source IP network address and a destination IP network address. The source IP network address may be the IP address of the Ethernet-enabled SSD transmitting frame 400, and the destination IP network address may be the IP address of the Ethernet-enabled SSD to which frame 400 will be sent. The MAC address and / or IP network address may facilitate transfer via fabric 106 / 260 (Figure 1 and Figure 2 ) boot the frame from the first Ethernet - enabled SSD to the second Ethernet - enabled SSD. According to one embodiment, the MAC address and / or IP network address in the Ethernet frame 400 do not include the network address of the host 102. The payload 408 of the Ethernet frame 400 includes an NVMe - oF encapsulation 410 that includes NVMe commands, responses, SGLs, etc. associated with the transfer of data between the first Ethernet - enabled SSD and the second Ethernet - enabled SSD. The frame 400 may also include other data, such as port IDs associated with the connection between the Ethernet - enabled SSDs, to facilitate routing of the frame from the first Ethernet - enabled SSD to the second Ethernet - enabled SSD over the connection.

[0051] Figure 5 is a flowchart of an example method 500 according to one embodiment for transferring data between a first SSD and a second SSD via a network fabric. According to one embodiment, the first SSD includes a first built - in network interface device configured to communicate via the network fabric, and the second SSD includes a second built - in network interface device configured to communicate via the network fabric. In various embodiments, method 500 involves transferring data between the first SSD and the second SSD via the network fabric without first transferring any of the data to a host computer. For example, in some embodiments, the first SSD and the second SSD are communicatively coupled to a host computer via the network fabric, and method 500 involves transferring data between the first SSD and the second SSD via the network fabric without first transferring any of the data to a host computer.

[0052] In various embodiments, method 500 is implemented by storage system 100( Figure 1 ) or storage system 200( Figure 2 ) or storage system 300( Figure 3 ), and for the sake of illustration, reference is made to Figures 1 - 3 one or more of. In other embodiments, method 500 is implemented by another suitable storage system that includes SSDs having respective built - in network interface devices configured to communicate via the network fabric and configured to transfer data between the SSDs via the network fabric without first transferring any of the data to a host computer.

[0053] At block 504, the first SSD opens a connection with the second SSD via the network fabric. In one embodiment, opening the connection at block 504 includes: a first processor of the first SSD (e.g., Figure 1The processor 128) opens a connection. In one embodiment, opening the connection at block 504 includes: opening the connection according to a communication protocol for communicating with a non-volatile memory device. In one embodiment, opening the connection at block 504 includes: opening a connection that does not include a host computer (e.g., host computer 102).

[0054] In one embodiment, opening the connection at block 504 includes: opening the connection according to the NVMe-oF communication protocol. In various other embodiments, opening the connection at block 504 additionally or alternatively includes opening the connection according to one or more of the following: i) Fibre Channel communication protocol, ii) RDMA communication protocol, iii) InfiniBand communication protocol, iv) NVMeTCP communication protocol, etc.

[0055] In some embodiments, opening the connection at block 504 includes: a first built-in network interface device of a first SSD transmitting, via a network fabric, one or more first packets to a second built-in network interface device of a second SSD, the first packets including information corresponding to opening the connection according to a communication protocol for communicating with a non-volatile memory device. In some embodiments, opening the connection at block 504 additionally includes: the first built-in network interface device of the first SSD receiving, via the network fabric, one or more second packets from the second built-in network interface device of the second SSD, the second packets including information corresponding to opening the connection according to a communication protocol for communicating with a non-volatile memory device. In some embodiments, the information in the second packets is a response to the information in the first packets.

[0056] In one embodiment, the host computer is communicatively coupled to the first SSD and the second SSD via an interconnect (e.g., interconnect 114), and opening the connection at block 504 includes opening the connection via the interconnect, where the connection does not include the host computer.

[0057] At block 508, the first SSD encapsulates, according to a communication protocol for communicating with a non-volatile memory device, commands for transferring data between the first SSD and the second SSD over the network fabric in an encapsulation. In one embodiment, encapsulating the commands at block 508 includes: encapsulating NVMe commands for transferring data to or from the SSD. In one embodiment, encapsulating the commands at block 508 includes: encapsulating the commands according to the NVMe-oF communication protocol. In various other embodiments, encapsulating the commands at block 508 additionally or alternatively includes encapsulating the commands according to one or more of the following: i) Fibre Channel communication protocol, ii) RDMA communication protocol, iii) InfiniBand communication protocol, iv) NVMeTCP communication protocol, etc.

[0058] At block 512, the first SSD sends the package over the connection to the second SSD via the network fabric according to the communication protocol. In one embodiment, the network fabric includes an Ethernet switch, and sending the package at block 512 includes: sending the package in an Ethernet packet over the connection via the Ethernet switch. For example, the first network interface device of the first SSD includes a first Ethernet network interface device (e.g., Ethernet network interface device 136); the first Ethernet network interface device encapsulates the package in an Ethernet packet and transmits the Ethernet packet to the Ethernet switch; and the Ethernet switch forwards the Ethernet packet to the second SSD. In one embodiment, the second network interface device of the second SSD includes a second Ethernet network interface device (e.g., Ethernet network interface device 150), and the second Ethernet network interface device decapsulates the package from the Ethernet packet and provides the package to the processor (e.g., processor 152) of the second SSD to execute the commands in the package.

[0059] In another embodiment, the network fabric includes a Fibre Channel (FC) switch, and sending the package at block 512 includes: sending the package in an FC frame over the connection via the FC switch. For example, the first network interface device of the first SSD includes a first FC network interface device; the first FC network interface device encapsulates the package in an FC frame and transmits the FC frame to the FC switch; and the FC switch forwards the FC frame to the second SSD. In one embodiment, the second network interface device of the second SSD includes a second FC network interface device, and the second FC network interface device decapsulates the package from the FC frame and provides the package to the processor (e.g., processor 152) of the second SSD to execute the commands in the package.

[0060] In another embodiment, the network fabric includes an InfiniBand switch, and sending the package at block 512 includes: sending the package in an InfiniBand packet over the connection via the InfiniBand switch. For example, the first network interface device of the first SSD includes a first InfiniBand network interface device; the first InfiniBand network interface device encapsulates the package in an InfiniBand packet and transmits the InfiniBand packet to the InfiniBand switch; and the InfiniBand switch forwards the InfiniBand packet to the second SSD. In one embodiment, the second network interface device of the second SSD includes a second InfiniBand network interface device, and the second InfiniBand network interface device decapsulates the package from the InfiniBand packet and provides the package to the processor (e.g., processor 152) of the second SSD to execute the commands in the package.

[0061] At block 516, the second SSD executes a command to transfer data between the first SSD and the second SSD over a network fabric according to a communication protocol. Executing the command at block 516 to transfer data includes: executing NVMe commands in the encapsulation to transfer data between the first SSD and the second SSD. In one embodiment, executing the command at block 516 to transfer data includes: transferring data between the first SSD and the second SSD without transferring any of the data to a host computer.

[0062] In one embodiment, executing the command at block 516 to transfer data includes: copying from the first SSD to the second SSD via the network fabric. In one embodiment, executing the command at block 516 to transfer data includes: retrieving from the second SSD and transferring the retrieved data to the first SSD via the network fabric.

[0063] Additionally or alternatively, the connection between the above SSDs can be used for other purposes in addition to transferring data between the SSDs as described above. For example, the connection can allow an Ethernet - enabled SSD, an FC - enabled SSD, an InfiniBand - enabled SSD, etc. to manage the operation of the SSDs, such as the data rate to be used by each SSD when communicating with the host 102. In an example, one SSD can act as the master device and send a corresponding encapsulation specifying the corresponding data rate to be used by other SSDs (slave devices) when communicating with the host. As an illustrative example, the host can be capable of communicating at a data rate such as 25 Gbps. The master SSD can specify a percentage (or fraction) of the host data rate (e.g., a percentage or fraction specified by a number in the range from 0 to 1). To divide the host data rate fairly or appropriately, the host can specify the corresponding percentage or fraction of the host data rate at which the corresponding SSD communicates with the host 102. The master device can indicate the percentage / fraction of the data rate via an encapsulation sent over the connection between the master device and the slave device. Additionally, when the number of slave SSDs serving the host 102 changes or a particular slave SSD is required to change, the master device can adjust the percentage / fraction. The connection between the SSDs can also be used to exchange other information for other suitable purposes.

[0064] Example 1: A method for transferring data between a first solid state drive (SSD) and a second SSD, the first SSD having a first built-in network interface device configured to communicate via a network fabric, and the second SSD having a second built-in network interface device configured to communicate via the network fabric, the method comprising: opening a connection between the first SSD and the second SSD on the network fabric; encapsulating an NVMe command in an encapsulation body based on a non-volatile memory over fabric (NVMe-oF) communication protocol to transfer data between the first SSD and the second SSD on the connection; sending the encapsulation body from the first SSD to the second SSD on the connection via the network fabric; and executing the NVMe command in the encapsulation body by the second SSD to transfer data between the first SSD and the second SSD on the connection.

[0065] Example 2: The method according to Example 1, wherein: the network fabric includes an Ethernet switch; the method further comprises: encapsulating the encapsulation body in an Ethernet packet at the first built-in network interface device of the first SSD, and de-encapsulating the encapsulation body from the Ethernet packet at the second built-in network interface device of the second SSD; and sending the encapsulation body from the first SSD to the second SSD includes: sending the encapsulation body in the Ethernet packet via the Ethernet switch.

[0066] Example 3: The method according to Example 1, wherein opening the connection includes opening the connection according to one of the following: i) Fibre Channel communication protocol, ii) Remote Direct Memory Access (RDMA) communication protocol, iii) InfiniBand communication protocol, or iv) NVMe over Transmission Control Protocol (NVMeTCP) communication protocol.

[0067] Example 4: The method according to any one of Examples 1 to 3, wherein: opening a connection between the first SSD and the second SSD further includes: further opening a connection on an interconnect associated with a host computer; opening a connection between the first SSD and the second SSD includes: opening a connection that does not include the host computer; and executing the NVMe command in the encapsulation body to transfer data between the first SSD and the second SSD includes: transferring data between the first SSD and the second SSD without transferring any of the data to the host computer.

[0068] Example 5: The method according to any one of Examples 1 to 4, wherein the transfer of data includes: copying or retrieving data on a connection between the first SSD and the second SSD.

[0069] Example 6: The method according to any one of Examples 1 to 5, wherein: opening a connection between a first SSD and a second SSD includes: opening a connection that does not include a host computer communicatively coupled to the first SSD and the second SSD; and performing NVMe commands in the enclosure to transfer data between the first SSD and the second SSD includes: transferring data between the first SSD and the second SSD without transferring any of the data in the data to the host computer.

[0070] Example 7: The method according to any one of Examples 1 to 6, wherein opening the connection includes: the RDMA layer of a first Ethernet-enabled SSD opening a connection with the RDMA layer of a second Ethernet-enabled SSD in a network fabric.

[0071] Example 8: The method according to any one of Examples 1 to 7, wherein the host communicates with the first Ethernet-enabled SSD and the second Ethernet-enabled SSD at a certain data rate, and the method further includes the first Ethernet-enabled SSD sending an indication of a fraction of the data rate on the connection, at which fraction of the data rate the second Ethernet-enabled SSD communicates with the host.

[0072] Example 9: The method according to any one of Examples 1 to 8, wherein the network fabric is configured with TCP or Ethernet.

[0073] Example 10: The method according to any one of Examples 1 to 9, wherein sending the enclosure from the first Ethernet-enabled SSD to the second Ethernet-enabled SSD on the connection includes: sending the enclosure based on an IP address in the enclosure of the second NVMe-oF SSD, the IP address indicating the destination of the enclosure.

[0074] Embodiment 11: A storage system includes: a first solid state drive (SSD) having i) a first built-in network interface device configured to communicate via a network fabric, and ii) a first processor; a second SSD having i) a second built-in network interface device configured to communicate via the network fabric, and ii) a second processor; and a network fabric; wherein the first processor of the first SSD is configured to: i) structurally open a connection between the first SSD and the second SSD over the network fabric, ii) encapsulate a Non-Volatile Memory Express (NVMe) command in a packet based on the Networked Non-Volatile Memory Express (NVMe-oF) communication protocol for transferring data between the first SSD and the second SSD over the connection, and iii) provide the packet to the first built-in network interface device to send the packet over the connection to the second built-in network interface device of the second SSD; and wherein the second processor of the second SSD is configured to: i) receive the packet from the second built-in network interface device, ii) de-encapsulate the NVMe command from the packet according to the NVMe-oF communication protocol, and iii) execute the NVMe command to transfer data between the first SSD and the second SSD over the connection.

[0075] Embodiment 12: The storage system according to Embodiment 11, wherein: the first processor of the first SSD is configured to encapsulate the packet in an Ethernet packet; the first built-in network interface device includes a first Ethernet interface device configured to transmit the Ethernet packet; the network fabric includes an Ethernet switch configured to: forward the Ethernet packet to the second built-in network interface device of the second SSD; the second built-in network interface device includes a second Ethernet interface device configured to provide the Ethernet packet to the second processor of the second SSD; and the second processor is configured to de-encapsulate the packet from the Ethernet packet.

[0076] Embodiment 13: The storage system according to Embodiment 12, wherein: the first Ethernet interface device is further communicatively coupled to the second Ethernet interface device via an interconnect associated with a host computer; the first processor of the first SSD is configured to: further open a connection between the first SSD and the second SSD over the interconnect; and the first processor is configured to open a connection between the first SSD and the second SSD that does not include the host computer; and the Ethernet switch is configured to: direct the Ethernet packet between the first SSD and the second SSD without transmitting a packet to the host computer.

[0077] Example 14: The storage system according to any one of Examples 11 to 13, wherein the first processor of the first SSD is configured to open a connection according to one of the following: i) Fibre Channel communication protocol, ii) Remote Direct Memory Access (RDMA) communication protocol, iii) InfiniBand communication protocol, or iv) NVMe over Transmission Control Protocol (NVMeTCP) communication protocol.

[0078] Example 15: The storage system according to any one of Examples 11 to 14, wherein the network fabric is configured to: transfer data between the first SSD and the second SSD without transferring any of the data to a host computer communicatively coupled to the first SSD and the second SSD.

[0079] Example 16: The storage system according to Example 15, wherein: the host computer is configured to communicate with the first SSD and the second SSD at a first data rate via the network fabric; the first processor of the first SSD is configured to: send an indication of a second data rate on the connection, at which the second SSD communicates with the host computer on the network fabric; and the second data rate is a fraction of the first data rate.

[0080] Example 17: The storage system according to any one of Examples 11 to 16, wherein the network fabric includes an Ethernet switch.

[0081] Example 18: The storage system according to Example 17, wherein: the first processor of the first SSD is configured to: encapsulate the payload in an Ethernet packet having the network address of the second SSD; the first built-in network interface device includes a first Ethernet interface device configured to: convey the Ethernet packet to the Ethernet switch; the Ethernet switch is configured to: forward the Ethernet packet to the second built-in network interface device using the network address of the second SSD in the Ethernet packet; the second built-in network interface device includes a second Ethernet interface device configured to: provide the Ethernet packet to the second processor of the second SSD.

[0082] Example 19: The storage system according to any one of Examples 11 to 18, wherein the transfer of data includes: copying or retrieving data on the connection between the first SSD and the second SSD.

[0083] Embodiment 20: The storage system according to any one of Embodiments 11 to 19, wherein: the first processor of the first SSD is configured to implement a first Remote Direct Memory Access (RDMA) communication protocol layer; the second processor of the second SSD is configured to implement a second RDMA communication protocol layer; and the first RDMA communication protocol layer is configured to open a connection with the second RDMA communication protocol layer.

[0084] Although aspects of the present disclosure have been described in connection with specific embodiments of the present disclosure presented as examples, alternatives, modifications, and variations of the examples may be made. Accordingly, the embodiments set forth herein are intended to be illustrative and not restrictive. Changes may be made without departing from the scope of the claims set forth.

Claims

1. A method for transferring data between a first solid state drive (SSD) and a second SSD, the first SSD having a first built-in network interface device configured to communicate via a network fabric, and the second SSD having a second built-in network interface device configured to communicate via the network fabric, the method comprises: opening a connection between the first SSD and the second SSD on the network fabric, wherein the first SSD is further communicatively coupled to the second SSD via an interconnect associated with a host computer; encapsulating an NVMe command in a packet based on the structural non-volatile memory NVMe-oF communication protocol for transferring data between the first SSD and the second SSD over the connection; sending the packet from the first SSD to the second SSD over the connection via the network fabric; and executing the NVMe command in the packet by the second SSD to transfer the data between the first SSD and the second SSD over the connection according to the NVMe-oF communication protocol, including transferring the data without transferring any of the data to the host computer.

2. The method according to claim 1, wherein: the network fabric includes an Ethernet switch; the method further comprises: encapsulating the packet in an Ethernet packet at the first built-in network interface device of the first SSD, and decapsulating the packet from the Ethernet packet at the second built-in network interface device of the second SSD; and sending the packet from the first SSD to the second SSD includes: sending the packet in the Ethernet packet via the Ethernet switch.

3. The method according to claim 1, wherein opening the connection includes opening the connection according to one of the following: i) Fibre Channel communication protocol, ii) Remote Direct Memory Access (RDMA) communication protocol, iii) Wireless Bandwidth communication protocol, or iv) NVMe over Transmission Control Protocol (NVMeTCP) communication protocol.

4. The method according to claim 1, wherein: opening the connection between the first SSD and the second SSD includes: opening a connection that does not include a host computer communicatively coupled to the first SSD and the second SSD.

5. The method according to claim 1, wherein the network fabric is configured with TCP or Ethernet.

6. The method according to any one of claims 1-5, wherein transferring the data comprises: copying or retrieving the data through the connection between the first SSD and the second SSD.

7. The method according to any one of claims 1-5, wherein the first SSD is a leader SSD and the second SSD is a follower SSD, and the method further comprises: sending one or more commands from the leader SSD to the follower SSD via the network fabric to manage the operation of the follower SSD on the network fabric.

8. The method according to claim 7, wherein sending the one or more commands to manage the operation of the follower SSD on the network fabric comprises: sending, from the leader SSD to the follower SSD, an indication of a data rate to be used by the follower SSD when communicating with the host computer.

9. The method according to claim 8, wherein the host computer is capable of communicating at a host data rate, and sending the indication of the data rate to be used by the follower SSD when communicating with the host computer comprises sending an indication of a percentage of the host data rate to be used by the follower SSD when communicating with the host computer.

10. The method according to claim 7, wherein the follower SSD is one of a plurality of follower SSDs each having a respective built-in network interface device configured to communicate via the network fabric, the plurality of follower SSDs being communicatively coupled via the interconnect associated with the host computer, and the method further comprises: sending, via the network fabric, from the leader SSD to a respective one of the plurality of follower SSDs, a respective indication of a data rate to be used by the respective follower SSD when communicating with the host computer.

11. The method according to claim 10, wherein the host computer is capable of communicating at a host data rate, and sending the respective indication of the data rate to be used by the respective follower SSD when communicating with the host computer comprises sending a respective indication of a respective percentage of the host data rate to be used by the respective follower SSD when communicating with the host computer.

12. The method according to claim 11, further comprises: adjusting, by the leader SSD, a particular percentage of the host data rate to be used by a particular follower SSD when communicating with the host computer based on one or both of: i) a change in the requirements for the particular SSD to communicate with the host computer and ii) a change in the number of SSDs servicing the host computer.

13. A storage system, comprising: a first solid state drive (SSD) having: i) a first built-in network interface device configured to communicate via a network fabric, and ii) a first processor; a second SSD having: i) a second built-in network interface device configured to communicate via the network fabric, and ii) a second processor; and the network fabric; The first processor of the first SSD is configured to: i) open a connection between the first SSD and the second SSD on the network fabric, wherein the first SSD is further communicatively coupled to the second SSD via an interconnect associated with a host computer; ii) encapsulate Non-Volatile Memory Express (NVMe) commands in a capsule based on the NVMe over Fabrics (NVMe-oF) communication protocol for transferring data between the first SSD and the second SSD over the connection; and iii) provide the capsule to the first built-in network interface device to send the capsule over the connection to the second built-in network interface device of the second SSD; The second processor of the second SSD is configured to: i) receive the capsule from the second built-in network interface device; ii) decapsulate the NVMe commands from the capsule according to the NVMe-oF communication protocol; and iii) execute the NVMe commands to transfer the data between the first SSD and the second SSD over the connection, wherein the data is transferred according to the NVMe-oF communication protocol and no data among the data is transferred to the host computer.

14. The storage system according to claim 13, wherein: the first processor of the first SSD is configured to encapsulate the capsule in an Ethernet packet; the first built-in network interface device includes a first Ethernet interface device configured to transmit the Ethernet packet; the network fabric includes an Ethernet switch configured to forward the Ethernet packet to the second built-in network interface device of the second SSD; the second built-in network interface device includes a second Ethernet interface device configured to provide the Ethernet packet to the second processor of the second SSD; and the second processor is configured to decapsulate the capsule from the Ethernet packet.

15. The storage system according to claim 13, wherein the first processor of the first SSD is configured to open the connection according to one of the following: i) Fibre Channel communication protocol; ii) Remote Direct Memory Access (RDMA) communication protocol; iii) Wireless Bandwidth communication protocol; or iv) NVMe over Transmission Control Protocol (NVMeTCP) communication protocol.

16. The storage system according to claim 13, wherein the network fabric includes an Ethernet switch.

17. The storage system according to any one of claims 13-16, wherein the transfer of the data comprises: copying or retrieving the data over the connection between the first SSD and the second SSD.

18. The storage system according to any one of claims 13-16, wherein the first SSD is configured to operate as a leader SSD and the second SSD is configured to operate as a follower SSD, and The first processor of the lead SSD sends one or more commands from the lead SSD to the follower SSDs via the network fabric to manage the operation of the follower SSDs on the network fabric.

19. The storage system of claim 18, wherein the first processor of the lead SSD is configured to send an indication of a data rate to be used by the follower SSD when communicating with the host computer from the lead SSD to the follower SSD.

20. The storage system of claim 19, wherein the host computer is capable of communicating at a host data rate, and the first processor of the lead SSD is configured to send an indication of a percentage of the host data rate to be used by the follower SSD when communicating with the host computer.

21. The storage system of claim 18, wherein the follower SSD is one of a plurality of follower SSDs each having a respective built-in network interface device configured to communicate via the network fabric, the plurality of follower SSDs being communicatively coupled via the interconnect associated with the host computer, and the first processor of the lead SSD is configured to send a respective indication of a data rate to be used by a respective follower SSD of the plurality of follower SSDs when communicating with the host computer from the lead SSD to the respective follower SSD via the network fabric.

22. The storage system of claim 21, wherein the host computer is capable of communicating at a host data rate, and the first processor of the lead SSD is configured to send a respective indication of a respective percentage of the host data rate to be used by the respective follower SSD when communicating with the host computer.

23. The storage system of claim 22, wherein the first processor of the lead SSD is configured to adjust a particular percentage of the host data rate to be used by a particular follower SSD when communicating with the host computer based on one or both of: i) a change in the needs of the particular SSD to communicate with the host computer and ii) a change in the number of SSDs servicing the host computer.

Images