Transferring data between solid-state drives (SSDs) via connections between SSDs

Abstract

A first solid-state drive (SSD) includes a first built-in network interface device configured to communicate via a network structure, and a second SSD includes a second built-in network interface device configured to communicate via the network structure. A connection is opened between the first SSD and the second SSD via the network structure. An NVMe command for transferring data between the first SSD and the second SSD over the connection based on a Non-Volatile Memory over Fabric (NVMe‑oF) communication protocol is encapsulated in an encapsulation. The encapsulation is sent from the first SSD to the second SSD over the connection via the network structure. The second SSD executes the NVMe command in the encapsulation to transfer data between the first SSD and the second SSD over the connection.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] The present disclosure claims priority to U.S. Provisional Application No. 62 / 818,566, filed on March 14, 2019, entitled “PEER-TO-PEER NVMe-oF SSD COMMUNICATION OVER FABRIC WITHOUT HOST / COMPUTE,” the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure relates to storage services, namely, transferring data between solid-state drives (SSDs) having built-in network interface devices via connections between SSDs over a network fabric. Background Art

[0004] The background description provided herein is intended to generally present the context of the present disclosure. The work of the presently named inventors is neither explicitly nor implicitly admitted to be prior art to the present disclosure within the scope of the work described in this background section nor to the extent that it may not qualify as prior art at the time the application was filed.

[0005] A host computer ("host") communicates with a solid-state drive (SSD) to obtain data services provided by the SSD, such as reading data, storing data, and erasing data. A solid-state drive (SSD) is a data storage device that uses non-volatile memory, such as NAND (Not-And) non-volatile memory, to store persistent digitally encoded data. The SSD can be configured to emulate a hard drive, a device that stores persistent digitally encoded data on the magnetic surface of a rapidly rotating disk and replaces a hard disk drive (HDD) in many applications. The SSD includes an interface, a compute, an aggregator, and one or more non-volatile memories. The interface is capable of communicating with the host computer over a network, and the network may include one or more intermediate devices coupled by wired and / or wireless interconnects, such as one or more interface switches and / or routers. The compute is a processor that coordinates operations on the SSD, and the aggregator provides a peripheral component interconnect express (PCIe) connection to one or more non-volatile memories.

[0006] The host sends a command to the SSD over the network to obtain data services. The command is received at the interface, and based on the command, the computing accesses the non-volatile memory of the SSD over the PCIe connection of the aggregator. The command from the host complies with the Non-Volatile Memory Express over Fabric (NVME-oF) protocol associated with accessing the SSD over the network. The computing accesses the non-volatile memory via the NVMe protocol, which defines a command set for accessing non-volatile memory via a PCIe connection. The host also facilitates the transfer of data from one SSD to another. In order to transfer data between SSDs, the host sends one or more commands to the SSD over the network to cause the computing to provide the host with data stored on one SSD. The host then sends one or more commands to the SSD to cause the data on the host to be stored in another SSD. The transmission of data from one SSD to another via the host generates data traffic on the network between the host and the SSD. Summary of the invention

[0007] The present disclosure relates to solid state storage devices (SSDs), namely, to transmitting data between SSDs having corresponding built-in network interface devices. The SSDs having corresponding built-in network interface devices are coupled together by a network structure, and data is transmitted between the SSDs over the connection between the SSDs and via the network structure, rather than transmitting the data via a host as an intermediary.

[0008] In one embodiment, a method is used to transfer data between a first solid-state drive (SSD) and a second SSD, the first SSD having a first built-in network interface device configured to communicate via a network structure, and the second SSD having a second built-in network interface device configured to communicate via the network structure. The method includes: opening a connection between the first SSD and the second SSD on the network structure; encapsulating an NVMe command in an encapsulation based on a Non-Volatile Memory over Fabric (NVMe-oF) communication protocol to transfer data between the first SSD and the second SSD on the connection; sending the encapsulation from the first SSD to the second SSD on the connection via the network structure; and executing the NVMe command in the encapsulation by the second SSD to transfer data between the first SSD and the second SSD on the connection. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a diagram of an example storage system that facilitates transferring data between solid-state drives (SSDs) having corresponding built-in network interface devices, the transfer of data occurring over connections between the SSDs, over connections via a network fabric, according to one embodiment.

[0010] Figure 2 is a diagram of another example storage system according to another embodiment.

[0011] Figure 3 is a diagram of yet another example storage system according to yet another embodiment.

[0012] Figure 4 is a diagram of an example Ethernet frame for use in transmitting data in accordance with some embodiments. Figures 1 to 3 In the example of a storage system, data is transferred between the SSDs over a connection between the SSDs.

[0013] Figure 5 is a flow chart of an example method for transferring data between SSDs having corresponding built-in network interface devices over a connection between the SSDs, according to one embodiment.

[0014] The drawings are for the purpose of illustrating example embodiments, but it should be understood that the embodiments are not limited to the arrangements and instrumentality shown in the drawings. DETAILED DESCRIPTION

[0015] The present disclosure relates to storage services, and specifically to transferring data between solid-state drives (SSDs) via a connection between SSDs, the connection not including a host processor, for example, data is transferred between SSDs without having to transfer the data from one SSD to a host processor and then transfer the data from the host processor to another SSD. In one embodiment, the SSD includes a corresponding built-in network interface device (e.g., an Ethernet network interface device, a Fibre Channel network interface device, a wireless bandwidth (InfiniBand) network interface device, etc.). The SSD with the corresponding built-in network interface device is a storage device, for example, the SSD is capable of peer-to-peer communication and communicates via a communication protocol (e.g., a Non-Volatile Memory over Fabric (NVMe-oF) protocol or another appropriate communication protocol) that is designed to control (or communicate with) a non-volatile storage device such as an SSD. According to various embodiments, the SSDs described herein communicate over a connection between the SSDs. According to various embodiments, the SSDs described herein communicate according to one or more appropriate communication protocols, such as one or more of the following: Fibre Channel, InfiniBand, Fibre Channel over Ethernet (FCoE), Remote Direct Memory Access (RDMA), RDMA over Converged Ethernet (RoCE), RDMA over User Datagram Protocol (UDP) as a transport layer, iWARP (a network protocol that implements RDMA over Transmission Control Protocol (TCP) and Internet Protocol (IP) (TCP / IP)), Non-Volatile Memory Express (NVMe) over TCP as a transport layer, etc., or other appropriate communication protocols for exchanging data with a storage subsystem over a network fabric (such as an Ethernet fabric, a Fibre Channel fabric, an InfiniBand fabric, etc.) Examples of systems, devices, methods, and techniques as described herein transmit data between SSDs having corresponding built-in network interface devices via connections between SSDs, without having to transmit data from one SSD to a host and then transmitting the data from the host to another SSD, thereby reducing the amount of data transmitted between the SSD and the host and / or reducing the processing performed by the host.

[0016] The NVMe-oF protocol is related to the NVM Express (NVMe) protocol. NVMe defines a register-level interface for communicating between a host and a non-volatile memory subsystem on a PCI Express (PCIe) bus. NVMe-oF is built on the architecture, command set, and queue interface defined by NVMe, and is a protocol that supports NVMe to access SSDs on a network structure. Although both NVMe and NVMe-oF have similar features, such as using command sets to control operations on storage devices as requested by the host, one of the main differences between NVMe and NVMe-oF is the transport mapping mechanism for sending and receiving commands and responses for communication between devices. NVMe-oF uses a message-based model for communication between the host and the SSD, where local NVMe communication includes mapping commands and responses to shared memory in the host on the PCIe interface protocol. NVMe-oF implements the use of alternate transmissions to PCIe, where these alternate transmissions extend the distance over which the host and SSD are connected. Greater storage system throughput is achieved by using NVMe-oF technology.

[0017] The embodiments described below utilize an SSD with a built-in Ethernet interface device (sometimes referred to herein as an Ethernet-enabled SSD). The embodiments with an Ethernet-enabled SSD are merely illustrative examples of an SSD with a built-in network interface device. However, other embodiments employ the SSD with other types of built-in network interface devices, such as a Fibre Channel network interface device, an InfiniBand network interface device, etc.

[0018] In some embodiments, an SSD with a built-in network interface device is (at least partially) encapsulated in a housing having an appropriate form factor, such as a standard hard disk drive (HDD) / SSD form factor, such as a 3.5-inch form factor, a 2.5-inch form factor, or a 1.8-inch form factor. The housing may include a physical network interface connector suitable for the network interface device (or define an aperture to accommodate it). For example, according to one embodiment, if the built-in network interface device includes an Ethernet interface device, the housing may include an appropriate physical Ethernet connector (e.g., an RJ45 connector) (or define an aperture to accommodate it). As another example, according to another embodiment, if the built-in network interface device includes a Fibre Channel (FC) interface device, the housing may include an appropriate physical FC connector (or define an aperture to accommodate it). As another example, according to another embodiment, if the built-in network interface device includes an InfiniBand interface device, the housing may include an appropriate physical InfiniBand connector (e.g., a QSFP connector, a CXP connector, an MPO connector, etc.) (or define an aperture to accommodate it).

[0019] In other embodiments, the SSD with a built-in network interface device is not enclosed in a housing.

[0020] In some embodiments, the SSD with built-in network interface device described herein has a structure such as disclosed in US patent application No. _____, entitled "Ethernet Enabled Solid State Drive (SSD)" filed on the same date as this application, which is incorporated herein by reference in its entirety. In other embodiments, the SSD with built-in network interface device has a different suitable structure.

[0021] Figure 1 is a diagram of an example storage system 100 that enables data to be transferred between Ethernet-enabled SSDs via a connection between the Ethernet-enabled SSDs that does not include a host, according to one embodiment. The example storage system 100 includes a host 102 coupled to an interconnect 114, which in turn is coupled to a storage subsystem 104 having a network fabric 106 and two Ethernet-enabled SSDs, shown as Ethernet-enabled SSD 108 and Ethernet-enabled SSD 110. The interconnect 114 may be a network between the host 102 and the storage subsystem 104, such as a local area network (LAN) or a wide area network (WAN). The interconnect 114 may include one or more intermediate devices, such as one or more interface switches and / or routers, coupled via a wired and / or wireless interconnect. In an example, according to one embodiment, the storage subsystem 104 can provide connectivity between the network structure 106, the Ethernet-enabled SSD 108, and the Ethernet-enabled SSD 110, and one or more of the storage subsystem 104 and the host 102 can be located in a data center rack. The rack can have a plurality of mounting slots, referred to as brackets, each of which is designed to hold a hardware unit such as a host 102 or a storage subsystem 104. In some examples, the rack can include a rack top switch that provides connectivity between the hardware unit and a remote network. In other examples, the interconnect 114 includes a backplane of the rack. In addition, more than one storage subsystem 104 and more than one host 102 can be coupled to the interconnect 114. Other storage systems can be part of a first rack, where the storage subsystem 104 is located or in a second rack. Similarly, a second host can be part of a first rack, and the host 102 is located or in a second rack.

[0022] The host 102 includes any type of host, such as a computer processor or a network of computers and processors. In addition, the host 102 is not necessarily limited to a single host device, and can represent multiple host devices. In one embodiment, the host 102 includes a memory 122 in the form of a dynamic random access memory (DRAM), a processor 124 such as a central processing unit (CPU), and a network interface card (NIC) 126. The memory 122 can store data only when powered on, and the processor 124 can be implemented on one or more integrated circuits, and the processor 124 is configured to execute machine-readable instructions stored in the memory 122 (or another memory (not shown)) to perform arithmetic, logic, input / output (I / O) and other operations. The host 102 may or may not have limited memory (e.g., non-volatile memory) to store data when the memory is not powered on. In order to facilitate the storage of data in the non-volatile memory of the storage subsystem 104, the host 102 uses the NIC 126 to access the non-volatile memory of the storage subsystem 104. The NIC 126 may facilitate the transfer of data over the interconnect 114 between the host 102 and the storage subsystem 104. In one embodiment, the NIC 126 comprises an Ethernet network interface device.

[0023] The storage subsystem 104 includes an Ethernet-enabled SSD 108 and an Ethernet-enabled SSD 110. The Ethernet-enabled SSD 108 / 110 includes a data storage device that uses a non-volatile memory, such as a NAND (Not-And) non-volatile memory, a flash memory, etc., to persistently store digitally encoded data. The Ethernet-enabled SSD 108 / 110 can be configured to emulate a hard disk drive (HDD). The Ethernet-enabled SSD 108 and the Ethernet-enabled SSD 110 can store data in corresponding non-volatile memories (NVM) 116, 118. In an example, the Ethernet-enabled SSD 108 can be configured with a processor 128, an aggregator such as a PCI Express (PCIe) cluster 130, and one or more non-volatile memories 116. In one embodiment, the processor 128 is configured to execute machine-readable instructions stored in a memory (not shown) to perform arithmetic, logic, I / O, and other operations. In another embodiment, the processor 128 additionally or alternatively includes a hardware processor (e.g., including one or more hardware state machines) that includes hardware circuits configured to perform arithmetic, logic, I / O, and other operations. The PCIe cluster 130 provides a PCIe connection to one or more non-volatile memories 116. The processor 128 accesses the non-volatile memory 116 over the PCIe connection of the PCIe cluster 130 using, for example, the NVMe protocol or other appropriate protocols. The NVMe protocol defines a command set for accessing data stored on the non-volatile memory 116 over the PCIe connection of the PCIe cluster 130 to the non-volatile memory 116. In other embodiments, the NVM 116 of the Ethernet-enabled SSD 108 is accessed via another appropriate bus / connection, such as a Fibre Channel network, a Serial AT Attachment (SATA) bus, a Serial Attached Small Computer System Interface (SCSI) (SAS) connection, etc.

[0024] The Ethernet-enabled SSD 108 may also be configured to implement the NVMe-oF protocol. In various embodiments, NVMe-oF is built on the architecture, command set, and queuing interface defined by NVMe, and supports NVMe access to the Ethernet-enabled SSD 108 over the network fabric 106 using FCoE, RDMA, InfiniBand, iWARP, RoCEv2, NVMeTCP (which specifies the NVMe protocol over TCP), etc. To support such access, the Ethernet-enabled SSD 108 includes an Ethernet interface device 136 that is configured to communicate over the network fabric 106 according to the Ethernet protocol. In addition, according to various embodiments, the processor 128 is configured to implement one or more protocols, such as NVMe, SATA, SAS, FCoE, RDMA, InfiniBand, iWARP, RoCE, NVMeTCP, etc. In one embodiment, the Ethernet-enabled SSD 108 includes a submission queue 132 and a completion queue 134. Submission queue 132 and completion queue 134 are shown as separate from processor 128, but in other examples, submission queue 132 and completion queue 134 can be integrated into processor 128. Submission queue 132 and completion queue 134 can each be associated with a unidirectional communication channel. Submission queue 132 can facilitate sending a unit of information referred to herein as a "capsule" to another Ethernet-enabled SSD, such as Ethernet-enabled SSD 110, and completion queue 134 can facilitate receiving capsules from other Ethernet-enabled SSDs, such as Ethernet-enabled SSD 110. Capsules are information exchange units associated with NVMe-oF and can include NVMe commands, responses to NVMe commands, data, and / or scatter-gather lists (SGLs).

[0025] The Ethernet-enabled SSD 110 may have the same or similar components as those shown and described with respect to the Ethernet-enabled SSD 108. For example, the Ethernet-enabled SSD 110 may have a corresponding Ethernet interface device 150, a processor 152, a PCIe cluster 154, an NVM 118, a submission queue 156, and a completion queue 158. In other examples, the Ethernet-enabled SSD 110 may include other components and / or may omit components as compared to the Ethernet-enabled SSD 108. Furthermore, in other examples, the storage subsystem 104 may have a plurality of CPUs and / or a plurality of CPUs. Figure 1 More or less Ethernet-enabled SSDs are shown.

[0026] The connection between the host 102 and the Ethernet-enabled SSD may allow the host 102 to access data stored on the Ethernet-enabled SSD. The connection may also support data transfers from one SSD to the memory 122 of the host 102, and then from the memory 122 of the host 102 to a second SSD. Instead of the host 102 facilitating data transfers between the SSDs, the Ethernet-enabled SSDs 108, 110 are configured to perform data transfers between each other without first transferring the data to the host 102. The data transfers are performed via the connection 138 between the Ethernet-enabled SSD 108 and the Ethernet-enabled SSD 110, rather than a connection between the Ethernet-enabled SSD 108 or the Ethernet-enabled SSD 110 and the host 102. The transfer of data includes transferring data from Ethernet-enabled SSD 108 to Ethernet-enabled SSD 110 over connection 138 (which connects Ethernet-enabled SSD 108 and Ethernet-enabled SSD 110) without transferring the data to memory 122 of host 102 as an intermediate operation, thereby reducing the amount of data transferred between Ethernet-enabled SSD 108 / 110 and host 102 over interconnect 114 and / or the amount of data processed by host 102.

[0027] As an illustrative example, a first Ethernet-enabled SSD 108 may copy data to a second Ethernet-enabled SSD 110. The non-volatile memory 116 may be an addressable memory, and the data to be copied may be located in an address range of the non-volatile memory 116 of the first Ethernet-enabled SSD 108. The first Ethernet-enabled SSD 108 may open a connection 138 to the second Ethernet-enabled SSD 110 over the network fabric 106. As an illustrative example, the connection 138 may be an NVMe-oF transport layer connection based on Fibre Channel. Alternatively, the connection 138 may be based on RDMA, InfiniBand, RoCEv2, iWARP, NVMeTCP, or other appropriate protocols. The connection 138 may be opened before data is transferred between the Ethernet-enabled SSDs. Connection 138 may establish one or more queues, such as queues 132 , 134 and queues 156 , 158 , to facilitate subsequent transfer of data between Ethernet enabled SSD 108 and Ethernet enabled SSD 110 over connection 138 .

[0028] The processor 128 of the first Ethernet-enabled SSD 108 may send the capsule to the second Ethernet-enabled SSD 110 over the connection 138 in the network fabric 106 by placing the capsule in the submission queue 132. The capsule to be sent to the second Ethernet-enabled SSD 110 may contain a command (such as an NVMe write command), data from the NVM 116 to be copied to the second Ethernet-enabled SSD 110, an SGL indicating a network address of the data to be copied in the non-volatile memory 116, etc. The processor 128 generates an Ethernet packet including the capsule and provides the Ethernet packet to the Ethernet interface device 136. The Ethernet interface device 136 transmits the Ethernet packet to the network fabric 106.

[0029] The Ethernet interface device 150 of the second Ethernet-capable SSD 110 receives the Ethernet packet from the network fabric 106 and provides the Ethernet packet to the processor 152. The processor 152 retrieves the capsule from the Ethernet packet and stores the capsule in the submission queue 156. Subsequently, the processor 158 retrieves the capsule from the submission queue 156 and, in response to the command in the capsule, copies the data from the first Ethernet-capable SSD 108 to the NVM 118 of the second Ethernet-capable SSD 110. The data to be copied may be included in the capsule, or the second Ethernet-capable SSD 110 may use the SGL information in the capsule to subsequently request the data identified by the SGL from the first Ethernet-capable SSD 110 on the connection 138 and, upon receiving the requested data via the connection 138, copy the requested data to the NVM 118. Data may be copied without the data retrieved from the first Ethernet-enabled SSD 108 having to be copied to the host 102 over the interconnect 114, and without the host 102 having to copy the data to the second Ethernet-enabled SSD 110 over the interconnect 114. In some examples, the second Ethernet-enabled SSD 110 may send an NVMe response to the NVMe command over the connection 138, indicating completion of the copy by placing a capsule with the NVMe response in the completion queue 158. The processor 152 generates an Ethernet packet including the capsule, and provides the Ethernet packet to the Ethernet interface device 150. The Ethernet interface device 150 transmits the Ethernet packet to the network fabric 106.

[0030] The Ethernet interface device 136 of the second Ethernet-capable SSD 110 receives the Ethernet packet from the network fabric 106 and provides the Ethernet packet to the processor 128. The processor 128 retrieves the capsule from the Ethernet packet and stores the capsule in the completion queue 134. Subsequently, the processor 128 retrieves the capsule from the completion queue 134 and, in response to the NVMe response in the capsule, determines that the copying of the data from the first Ethernet-capable SSD 108 to the NVM 118 of the second Ethernet-capable SSD 110 has been completed.

[0031] In an example, the first Ethernet-enabled SSD 108 and the second Ethernet-enabled SSD 110 may have addressable local random access memory (RAM) such as dynamic RAM (DRAM) 160 and DRAM 162 to facilitate the replication of data. For example, as part of a replication command, the first Ethernet-enabled SSD 108 may copy data from the NVM 116 to its DRAM 160. The local address in the DRAM 160 to which the data is copied may be stored in the package along with the write command. The package may be placed in the submission queue 132. Then, when the package is received by the second Ethernet-enabled SSD 110, the second Ethernet-enabled SSD 110 may obtain data from the DRAM 160 of the first Ethernet-enabled SSD 108 based on the local address for storage in the DRAM 160. The second Ethernet-enabled SSD 110 may then store the data in the DRAM 162 to the NVM 118 via a PCIe connection according to NVMe to complete the replication of the data. When NVM 116 cannot be directly accessed by second Ethernet enabled SSD 110 , DRAM 160 and DRAM 162 may facilitate replication.

[0032] According to one embodiment, the copy operation may be performed to form a redundant array of independent disks (RAID), where each Ethernet-enabled SSD corresponds to a "disk" of the RAID, and each Ethernet-enabled SSD has a copy of the same data. For example, the host 102 may first copy the data to one Ethernet-enabled SSD. The one Ethernet-enabled SSD may then copy the data to another Ethernet-enabled SSD on connection 138 to form a RAID. In an example, the copy may be further repeated one or more times between additional Ethernet-enabled SSDs to form additional Ethernet-enabled SSDs having copies of the data.

[0033] Other data operations may include retrieving data from one Ethernet-enabled SSD and storing it in another Ethernet-enabled SSD via a connection 138 between Ethernet-enabled SSDs in the storage subsystem 104. For example, the first Ethernet-enabled SSD 108 may send a capsule to the second Ethernet-enabled SSD 110 over the connection 138 through the network fabric 106 by placing the capsule in the submission queue 132. The capsule sent to the second Ethernet-enabled SSD 110 may include an NVMe read command and may include an indication of data to be read from the non-volatile memory 118 of the second Ethernet-enabled SSD 110, such as an SGL indicating a network address of the data to be retrieved in the non-volatile memory 118. The processor 128 generates an Ethernet packet including the capsule and provides the Ethernet packet to the Ethernet interface device 136. The Ethernet interface device 136 transmits the Ethernet packet to the network fabric 106.

[0034] The Ethernet interface device 150 of the second Ethernet-capable SSD 110 receives the Ethernet packet from the network fabric 106 and provides the Ethernet packet to the processor 152. The processor 152 retrieves the capsule from the Ethernet packet and stores the capsule in the submission queue 156. In response to the capsule in the submission queue 156, the second Ethernet-capable SSD 110 also provides data to the first Ethernet-capable SSD 108 on the connection 138. The second Ethernet-capable SSD 110 can send an NVMe response with data to the first Ethernet-capable SSD 108 by placing the NVMe response in the capsule and placing the capsule with the NVMe response in the completion queue 158. The processor 152 generates an Ethernet packet including the capsule and provides the Ethernet packet to the Ethernet interface device 150. The Ethernet interface device 150 transmits the Ethernet packet to the network fabric 106. The NVMe-oF response can be sent to the completion queue 134 of the Ethernet-capable SSD 108 on the connection 138.

[0035] In an example, DRAM 160 and DRAM 162 may facilitate the first Ethernet-enabled SSD 108 to retrieve data in NVM 118 of the second Ethernet-enabled SSD 110. The local address in DRAM 160 to which the second Ethernet-enabled SSD 110 is to write the retrieved data may be stored in a package with a read command. The package may be placed in a submission queue 132. Then, when the package is received by the second Ethernet-enabled SSD 110, the second Ethernet-enabled SSD 110 may read the data to be retrieved from NVM 118 based on the SGL in the package and store the data in DRAM 162. Then, the second Ethernet-enabled SSD 110 may write the data to be retrieved to DRAM 160 based on the local address in the package. Then, the first Ethernet-enabled SSD 108 may write the data in DRAM 160 to NVM 116 via a PCIe connection according to NVMe to complete the read command. When NVM 118 is not directly accessible by first Ethernet-enabled SSD 108 , DRAM 160 and DRAM 162 may facilitate retrieval.

[0036] The fetch operation can be used to build data in one Ethernet-enabled SSD based on data in another Ethernet-enabled SSD, for example, if data on a "disk" of a RAID is corrupted. An Ethernet-enabled SSD can fetch data from another Ethernet-enabled SSD so that it has an identical copy of the data on the other Ethernet-enabled SSD.

[0037] Figure 2 is a block diagram of another example of a storage system 200 according to another embodiment. The storage system 200 includes Figure 1 The processor 128 of the Ethernet-enabled SSD 108 executes the communication protocol layers 214, 216, 218 to facilitate the transmission of data between the Ethernet-enabled SSDs over the connection between the Ethernet-enabled SSDs. In one embodiment, the communication protocol layers 214, 216, 218 are implemented using software executed by the processor 128. In another embodiment, at least a portion of the communication protocol layers 214, 216, 218 are implemented using hardware circuitry, such as one or more hardware state machines.

[0038] The storage system 200 includes Figure 1Components of the example storage system 100 in FIG. 1 are not shown in the storage system 200 for simplicity. The example storage system 200 includes a host 102, an interconnect 114, storage systems 104, 250, Ethernet switches 106, 260, and Ethernet-enabled SSDs, which are shown as Ethernet-enabled SSD 108, Ethernet-enabled SSD 110, Ethernet-enabled SSD 256, and Ethernet-enabled SSD 258.

[0039] The Ethernet switch 106 includes one or more network ports 210. One or more of the network ports 210 may be physical interfaces that enable communication between Ethernet-enabled SSDs or between the Ethernet-enabled SSDs and the host 102. In addition, the Ethernet-enabled SSD 108, the Ethernet-enabled SSD 110, the Ethernet-enabled SSD 256, and the Ethernet-enabled SSD 258 may have one or more network ports 212 to support communication with the Ethernet switch 106.

[0040] The Ethernet-enabled SSD 108 may also be configured with one or more communication protocol layers associated with accessing data stored on the Ethernet-enabled SSD. For example, the Ethernet-enabled SSD 108 may have an NVMe layer 214, an NVMe-oF layer 216, and an RDMA layer 218. The NVMe layer 214 may be a protocol layer that defines an architecture, a command set, and a queuing interface to access data stored in the non-volatile memory 116 on the PCIe cluster 130. The NVMe-oF layer 216 may be a protocol layer that defines an extension to the NVMe layer 214 for accessing the SSD on the Ethernet switch 106. The RDMA layer 218 may be a transport protocol layer that provides reliable delivery of data, NVMe commands, and NVMe responses on the Ethernet switch 106. The RDMA layer 218 facilitates direct memory access (DMA) operations on data, SGL, commands, and responses to be delivered, reducing delivery latency compared to delivery via an application stack. According to various embodiments, the RDMA layer 218 can take the form of RDMA over Converged Ethernet v2 (e.g., RoCEv2) to support RDMA over Ethernet, InfiniBand, or iWARP. In some embodiments, the functions of the NVMe-oF layer 216 and the RDMA layer 218 can be configured as a single layer. In some embodiments, the RDMA layer 218 is integrated within the NVMe-oF layer 216.

[0041] Ethernet-enabled SSD 110, Ethernet-enabled SSD 256, and Ethernet-enabled SSD 258 may have the same Figure 1 and Figure 2 108. For example, Ethernet-enabled SSD 110, Ethernet-enabled SSD 256, and Ethernet-enabled SSD 258 may have corresponding NVMe layers, NVMe-oF layers, and RDMA layers. In other examples, Ethernet-enabled SSD 110, Ethernet-enabled SSD 256, and Ethernet-enabled SSD 258 may have additional layers compared to Ethernet-enabled SSD 108.

[0042] Various software layers in the Ethernet-enabled SSD may be used to perform data operations between the first Ethernet-enabled SSD 108 and the second Ethernet-enabled SSD 110, such as copying data in the NVM 116 of the first Ethernet-enabled SSD 108 to the NVM 118 of the second Ethernet-enabled SSD 110 via the connection 138 between the first Ethernet-enabled SSD and the second Ethernet-enabled SSD, which reduces the need to transfer data to the host 102 over the interconnect 114 as an intermediate step.

[0043] For example, the RDMA layer of the first Ethernet-enabled SSD 108 and the RDMA layer of the second Ethernet-enabled SSD 110 can open a connection 138 between the first Ethernet-enabled SSD 108 and the second Ethernet-enabled SSD 110. The corresponding RDMA layer can open the connection 138 based on the port identifier (ID) associated with the first Ethernet-enabled SSD 108 and the port ID associated with the second Ethernet-enabled SSD 110 (which uniquely identifies the port of the connection 138). The Ethernet-enabled SSD port is a protocol interface between the Ethernet-enabled SSD and the Ethernet switch 106, and is a collection of one or more physical structure interfaces that act together as a single protocol interface. The Ethernet-enabled SSD can have one or more Ethernet-enabled SSD ports. Each Ethernet-enabled SSD port can have a port ID, which can be a 16-bit identifier. The Ethernet-enabled SSD ports of the Ethernet-enabled SSD can support different NVMe-oF transport connections. If the underlying structure has more than one NVMe-oF transport binding specification (for example, an Ethernet-enabled SSD port identified by a port ID can support both iWARP and RoCEv2), the Ethernet-enabled SSD port can support multiple NVMe-oF transport connections.

[0044] Each of the Ethernet-enabled SSD 108 and the Ethernet-enabled SSD 110 may also have a network address that uniquely identifies the Ethernet-enabled SSD in the network. The network address may be any logical or physical address, such as a media access control (MAC) address or an Internet Protocol (IP) address. The connection 138 may also be based on the network address of the Ethernet-enabled SSD 108 and the network address of the Ethernet-enabled SSD 110. The NVMe layer of the first Ethernet-enabled SSD 108 may generate a command associated with a data operation, which is then encapsulated in a capsule by the NVMe-oF layer 216 and placed in the submission queue of the Ethernet-enabled SSD 108. The RDMA layer of the first Ethernet-enabled SSD 108 may transmit the capsule to the second Ethernet-enabled SSD 110 through its network port 212 and on the connection 138 of the Ethernet switch 106. The RDMA layer of the second Ethernet-enabled SSD 110 layer may receive the capsule on the connection 138 via its network port 212. The NVMe-oF layer of the second Ethernet-supporting SSD 110 can recover the NVMe command and provide the NVMe command to its NVMe layer to perform the data operation. In addition, as described above, the functions of the NVMe-oF layer 216 and the RDMA layer 218 can be configured as a single layer. In various examples, the data operation can be to copy the data in the NVM 116 of the first Ethernet-supporting SSD 108 to the NVM 118 of the second Ethernet-supporting SSD 110, or to retrieve the data in the NVM 118 of the second Ethernet-supporting SSD 110 by the first Ethernet-supporting SSD 108, and store the data in the NVM 116 of the first Ethernet-supporting SSD 108. In some examples, the RDMA layer of the second Ethernet-supporting SSD 110 can additionally send an NVMe response also in the encapsulation to the first Ethernet-supporting SSD 108 on the connection 138. The sending of the response can be based on the type and content of the NVMe command as described above. The response may have been generated by the NVMe layer of the second Ethernet-enabled SSD 110 and encapsulated into a capsule by the NVMe-oF layer of the second Ethernet-enabled SSD 110. The capsule may then be placed in the completion queue of the second Ethernet-enabled SSD 110 to be transferred to the first Ethernet-enabled SSD 108 by the RDMA layer of the second Ethernet-enabled SSD 110.

[0045] The connection 138 described as being opened between Ethernet-enabled SSDs is exemplary in nature. The connection opened to perform data operations may take a variety of forms, depending on which Ethernet-enabled SSDs will transfer data. For example, Ethernet-enabled SSD 256 and Ethernet-enabled SSD 258 on different storage systems 250, 104 may open connection 254. Connection 254 may be on Ethernet switch 106, Ethernet switch 260, and interconnect 114. Other variations are possible.

[0046] Figure 3 is a block diagram of another example of a storage system 300 according to another embodiment. The storage system 300 may include Figure 1 Components of the example storage system 100 in FIG. 2 are not shown in the storage system 300 for simplicity. The storage system 200 includes Figure 1 The SSD 108 and SSD 110 of the embodiment of the present invention may include corresponding Fibre Channel (FC) network interface devices instead of Ethernet network interface devices 136, 150. Figure 3 , SSD 108 and SSD 110 are sometimes referred to as FC-capable SSDs.

[0047] In the example of storage system 300 , the network fabric 106 of the storage subsystem 104 includes a FC switch 304 , and the storage subsystem 250 includes a FC switch 306 .

[0048] The processor 128 of the FC-enabled SSD 108 executes the communication protocol layers 214, 216, 302 to facilitate the transfer of data between the FC-enabled SSDs over the connection between the FC-enabled SSDs. In one embodiment, the communication protocol layers 214, 216, 302 are implemented using software executed by the processor 128. In another embodiment, at least a portion of the communication protocol layers 214, 216, 302 are implemented using hardware circuitry, such as one or more hardware state machines.

[0049] Example storage system 300 includes host 102 , interconnect 114 , storage subsystem 104 , and FC-enabled SSDs, which are illustrated as FC-enabled SSD 108 , FC-enabled SSD 110 , FC-enabled SSD 256 , and FC-enabled SSD 258 .

[0050] The FC-enabled SSD 108 and similar FC-enabled SSDs 110, FC-enabled SSDs 256, and FC-enabled SSDs 258 implement various communication protocol layers, including an NVMe layer 214 and an NVMe-oF layer 216. The FC-enabled SSD 108 and similar FC-enabled SSDs 110, FC-enabled SSDs 256, and FC-enabled SSDs 258 also implement an NVMe-oF (FC-NVMe) layer over Fibre Channel, which includes a transport protocol layer that provides reliable delivery of data, NVMe commands, NVMe responses, SGLs, and / or data over FC switches 304 and / or FC switches 306. In some embodiments, the functionality of the NVMe-oF layer 216 and the FC-NVMe layer 302 can be configured as a single layer. In some embodiments, the FC-NVMe layer 302 is integrated within the NVMe-oF layer 216. The FC switches 304, 306 facilitate FC communications from one FC-enabled SSD to another FC-enabled SSD. Each of the network ports 210, 212 may be a FC port, and each of the FC-enabled SSDs may have a unique network address, such as a unique FC address. The FC-NVMe layer 302 (or another appropriate communication protocol layer) may establish a logical connection between the first FC-enabled SSD 108 and the second FC-enabled SSD 110 based on the corresponding port ID and / or network address to perform data operations as described above, reducing the need to transfer data from the first FC-enabled SSD to the host 102 and then from the host to the second FC-enabled SSD as an intermediate step. Connection 138 and connection 254 are illustrated, but the connection may also take other forms.

[0051] Figure 4 400 is a diagram of an example Ethernet frame 400 associated with transmission of data between a first Ethernet-enabled SSD and a second Ethernet-enabled SSD over an Ethernet connection and an Ethernet-enabled network fabric, according to an embodiment. Frame 400 includes an Ethernet Layer 2 header 402 and an Ethernet type field 404, which indicates that frame 400 is associated with Ethernet. In one embodiment, header 402 includes a transmitter MAC address (e.g., a MAC address of the Ethernet-enabled SSD that is transmitting Ethernet frame 400) and a receiver MAC address (e.g., a MAC address of the Ethernet-enabled SSD that is to receive Ethernet frame 400). Example frame 400 may include IP header 406, which indicates a source IP network address and a destination IP network address. The source IP network address may be the IP address of the Ethernet-enabled SSD that is transmitting frame 400, and the destination IP network address may be the IP address of the Ethernet-enabled SSD to which frame 400 is to be sent. The MAC address and / or IP network address may facilitate transmission of data via fabric 106 / 260( Figure 1 and Figure 2 ) directs a frame from the first Ethernet-enabled SSD to the second Ethernet-enabled SSD. According to one embodiment, the MAC address and / or IP network address in the Ethernet frame 400 does not include the network address of the host 102. The payload 408 of the Ethernet frame 400 includes an NVMe-oF encapsulation 410, which includes NVMe commands, responses, SGLs, etc. associated with the transmission of data between the first Ethernet-enabled SSD and the second Ethernet-enabled SSD. The frame 400 may also include other data, such as a port ID associated with the connection between the Ethernet-enabled SSDs to facilitate routing of the frame from the first Ethernet-enabled SSD to the second Ethernet-enabled SSD over the connection.

[0052] Figure 5 is a flow diagram of an example method 500 for transferring data between a first SSD and a second SSD via a network structure, according to one embodiment. According to one embodiment, the first SSD includes a first built-in network interface device configured to communicate via the network structure, and the second SSD includes a second built-in network interface device configured to communicate via the network structure. In various embodiments, the method 500 involves transferring data between the first SSD and the second SSD via the network structure without first transferring any of the data to a host computer. For example, in some embodiments, the first SSD and the second SSD are communicatively coupled to a host computer via the network structure, and the method 500 involves transferring data between the first SSD and the second SSD via the network structure without first transferring any of the data to the host computer.

[0053] In various embodiments, the method 500 is performed by the storage system 100 ( Figure 1 )、Storage system 200( Figure 2 ) or storage system 300( Figure 3 ) is implemented, and for ease of explanation, reference is made to Figure 1-Figure 3 In other embodiments, the method 500 is implemented by another suitable storage system including an SSD having a corresponding built-in network interface device configured to communicate via a network structure, and configured to transfer data between the SSDs via the network structure without first transferring any of the data to a host computer.

[0054] At block 504, the first SSD opens a connection with the second SSD via the network structure. In one embodiment, opening the connection at block 504 includes: a first processor (e.g., Figure 1In one embodiment, opening the connection at block 504 includes opening the connection according to a communication protocol for communicating with the non-volatile memory device. In one embodiment, opening the connection at block 504 includes opening a connection that does not include a host computer (e.g., host computer 102).

[0055] In one embodiment, opening the connection at block 504 includes opening the connection according to the NVMe-oF communication protocol. In various other embodiments, opening the connection at block 504 additionally or alternatively includes opening the connection according to one or more of the following: i) Fibre Channel communication protocol, ii) RDMA communication protocol, iii) InfiniBand communication protocol, iv) NVMeTCP communication protocol, etc.

[0056] In some embodiments, opening the connection at box 504 includes: the first built-in network interface device of the first SSD transmits one or more first packets to the second built-in network interface device of the second SSD via the network structure, the first packets including information corresponding to opening the connection according to the communication protocol used to communicate with the non-volatile memory device. In some embodiments, opening the connection at box 504 additionally includes: the first built-in network interface device of the first SSD receives one or more second packets from the second built-in network interface device of the second SSD via the network structure, the second packets including information corresponding to opening the connection according to the communication protocol used to communicate with the non-volatile memory device. In some embodiments, the information in the second packet responds to the information in the first packet.

[0057] In one embodiment, a host computer is communicatively coupled to the first SSD and the second SSD via an interconnect (eg, interconnect 114), and opening the connection at block 504 includes opening a connection via the interconnect, wherein the connection does not include the host computer.

[0058] At box 508, the first SSD encapsulates a command for transferring data between the first SSD and the second SSD on the network structure in an encapsulation according to a communication protocol for communicating with a non-volatile memory device. In one embodiment, encapsulating the command at box 508 includes: encapsulating an NVMe command for transferring data to or from the SSD. In one embodiment, encapsulating the command at box 508 includes: encapsulating the command according to the NVMe-oF communication protocol. In various other embodiments, encapsulating the command at box 508 additionally or alternatively includes encapsulating the command according to one or more of the following: i) Fibre Channel communication protocol, ii) RDMA communication protocol, iii) InfiniBand communication protocol, iv) NVMeTCP communication protocol, etc.

[0059] At box 512, the first SSD sends the capsule to the second SSD via the network structure over the connection according to the communication protocol. In one embodiment, the network structure includes an Ethernet switch, and sending the capsule at box 512 includes: sending the capsule in an Ethernet packet over the connection via the Ethernet switch. For example, the first network interface device of the first SSD includes a first Ethernet network interface device (e.g., Ethernet network interface device 136); the first Ethernet network interface device encapsulates the capsule in the Ethernet packet and transmits the Ethernet packet to the Ethernet switch; and the Ethernet switch forwards the Ethernet packet to the second SSD. In one embodiment, the second network interface device of the second SSD includes a second Ethernet network interface device (e.g., Ethernet network interface device 150), and the second Ethernet network interface device decapsulates the capsule from the Ethernet packet and provides the capsule to a processor of the second SSD (e.g., processor 152) to execute the command in the capsule.

[0060] In another embodiment, the network structure includes a Fibre Channel (FC) switch, and sending the capsule at block 512 includes: sending the capsule in a FC frame over the connection via the FC switch. For example, the first network interface device of the first SSD includes a first FC network interface device; the first FC network interface device encapsulates the capsule in a FC frame and transmits the FC frame to the FC switch; and the FC switch forwards the FC frame to the second SSD. In one embodiment, the second network interface device of the second SSD includes a second FC network interface device, the second FC network interface device decapsulates the capsule from the FC frame, and provides the capsule to a processor (e.g., processor 152) of the second SSD to execute the command in the capsule.

[0061] In another embodiment, the network structure includes an InfiniBand switch, and sending the capsule at box 512 includes: sending the capsule in an InfiniBand packet on the connection via the InfiniBand switch. For example, the first network interface device of the first SSD includes a first InfiniBand network interface device; the first InfiniBand network interface device encapsulates the capsule in an InfiniBand packet and transmits the InfiniBand packet to the InfiniBand switch; and the InfiniBand switch forwards the InfiniBand packet to the second SSD. In one embodiment, the second network interface device of the second SSD includes a second InfiniBand network interface device, and the second InfiniBand network interface device decapsulates the capsule from the InfiniBand packet and provides the capsule to a processor (e.g., processor 152) of the second SSD to execute the command in the capsule.

[0062] At block 516, the second SSD executes the command to transfer data between the first SSD and the second SSD over the network fabric according to the communication protocol. Executing the command to transfer data at block 516 includes executing an NVMe command in the enclosure to transfer data between the first SSD and the second SSD. In one embodiment, executing the command to transfer data at block 516 includes transferring data between the first SSD and the second SSD without transmitting any of the data to a host computer.

[0063] In one embodiment, executing the command to transfer data at block 516 includes copying from the first SSD to the second SSD via the network fabric. In one embodiment, executing the command to transfer data at block 516 includes retrieving from the second SSD and transmitting the retrieved data to the first SSD via the network fabric.

[0064] Additionally or alternatively, the connection between the above-mentioned SSDs can be used for other purposes besides transmitting data between SSDs as described above. For example, the connection can allow an Ethernet-supported SSD, an FC-supported SSD, an InfiniBand-supported SSD, etc. to manage the operation of the SSD, such as the data rate to be used by each SSD when communicating with the host 102. In the example, one SSD can act as a master device and send a corresponding package specifying the corresponding data rate to be used by other SSDs (slave devices) when communicating with the host. As an illustrative example, the host may be able to communicate at a data rate such as 25Gbps. The master SSD can specify a percentage (or fraction) of the host data rate (e.g., a percentage or fraction specified by a number in the range of 0 to 1). In order to divide the data rate of the host fairly or appropriately, the host can specify a corresponding percentage or fraction of the host data rate, and the corresponding SSD communicates with the host 102 at the corresponding percentage or fraction of the host data rate. The master device can indicate the percentage / fraction of the data rate via a package sent on the connection between the master device and the slave device. Furthermore, the master device may adjust the percentage / fraction when the number of slave SSDs serving the host 102 changes or when a particular slave SSD is required to change. The connection between the SSDs may also be used to exchange other information for other appropriate purposes.

[0065] Embodiment 1: A method for transmitting data between a first solid-state drive (SSD) and a second SSD, the first SSD having a first built-in network interface device configured to communicate via a network structure, and the second SSD having a second built-in network interface device configured to communicate via a network structure, the method comprising: opening a connection between the first SSD and the second SSD on the network structure; encapsulating an NVMe command in an encapsulation body based on a non-volatile memory over fabric (NVMe-oF) communication protocol to transmit data between the first SSD and the second SSD on the connection; sending the encapsulation body from the first SSD to the second SSD on the connection via the network structure; and executing the NVMe command in the encapsulation body by the second SSD to transmit data between the first SSD and the second SSD on the connection.

[0066] Embodiment 2: A method according to Embodiment 1, wherein: the network structure includes an Ethernet switch; the method further includes: encapsulating the capsule in an Ethernet packet at a first built-in network interface device of the first SSD, and decapsulating the capsule from the Ethernet packet at a second built-in network interface device of the second SSD; and sending the capsule from the first SSD to the second SSD includes: sending the capsule in the Ethernet packet via the Ethernet switch.

[0067] Embodiment 3: A method according to embodiment 1, wherein opening a connection comprises opening a connection according to one of the following items: i) Fibre Channel communication protocol, ii) Remote Data Memory Access (RDMA) communication protocol, iii) InfiniBand communication protocol, or iv) NVMe over Transmission Control Protocol (NVMeTCP) communication protocol.

[0068] Embodiment 4: A method according to any one of Embodiments 1 to 3, wherein: opening a connection between the first SSD and the second SSD further includes: further opening a connection on an interconnect associated with a host computer; opening a connection between the first SSD and the second SSD includes: opening a connection that does not include a host computer; and executing an NVMe command in the package to transfer data between the first SSD and the second SSD includes: transferring data between the first SSD and the second SSD without transferring any of the data to the host computer.

[0069] Embodiment 5: The method according to any one of embodiments 1 to 4, wherein the transmission of data comprises: copying or retrieving data over a connection between the first SSD and the second SSD.

[0070] Embodiment 6: A method according to any one of Embodiments 1 to 5, wherein: opening a connection between a first SSD and a second SSD comprises: opening a connection that does not include a host computer that is communicatively coupled to the first SSD and the second SSD; and executing an NVMe command in the package to transfer data between the first SSD and the second SSD comprises: transferring data between the first SSD and the second SSD without transferring any of the data to the host computer.

[0071] Embodiment 7: The method according to any one of Embodiments 1 to 6, wherein opening a connection comprises: the RDMA layer of the first Ethernet-enabled SSD opens a connection with the RDMA layer of the second Ethernet-enabled SSD in the network structure.

[0072] Embodiment 8: A method according to any one of Embodiments 1 to 7, wherein a host communicates with a first Ethernet-enabled SSD and a second Ethernet-enabled SSD at a certain data rate, and the method also includes the first Ethernet-enabled SSD sending an indication of a fraction of the data rate on the connection, at which the second Ethernet-enabled SSD communicates with the host.

[0073] Embodiment 9: The method according to any one of embodiments 1 to 8, wherein the network structure is configured with TCP or Ethernet.

[0074] Embodiment 10: A method according to any one of embodiments 1 to 9, wherein sending a capsule from a first Ethernet-enabled SSD to a second Ethernet-enabled SSD over a connection comprises: sending the capsule based on an IP address in the capsule of the second NVMe-oF SSD, the IP address indicating a destination of the capsule.

[0075] Embodiment 11: A storage system comprising: a first solid-state drive (SSD) having i) a first built-in network interface device configured to communicate via a network structure, and ii) a first processor; a second SSD having i) a second built-in network interface device configured to communicate via a network structure, and ii) a second processor; and a network structure; wherein the first processor of the first SSD is configured to: i) open a connection between the first SSD and the second SSD on the structure, connected to the network structure, ii) based on a non-volatile memory over fabric (NVMe-oF) communication protocol, transmit a non-volatile memory to the second SSD; The invention relates to a method of encapsulating a non-volatile memory express (NVMe) command in an encapsulation to transfer data between a first SSD and a second SSD over a connection, and iii) providing the encapsulation to a first built-in network interface device to send the encapsulation to a second built-in network interface device of the second SSD over the connection; and wherein a second processor of the second SSD is configured to: i) receive the encapsulation from the second built-in network interface device, ii) decapsulate the NVMe command from the encapsulation according to the NVMe-oF communication protocol, and iii) execute the NVMe command to transfer data between the first SSD and the second SSD over the connection.

[0076] Embodiment 12: A storage system according to Embodiment 11, wherein: the first processor of the first SSD is configured to encapsulate the capsule in an Ethernet packet; the first built-in network interface device includes a first Ethernet interface device, and the first Ethernet interface device is configured to transmit the Ethernet packet; the network structure includes an Ethernet switch, and the Ethernet switch is configured to: forward the Ethernet packet to the second built-in network interface device of the second SSD; the second built-in network interface device includes a second Ethernet interface device, and the second Ethernet interface device is configured to provide the Ethernet packet to the second processor of the second SSD; and the second processor is configured to decapsulate the capsule from the Ethernet packet.

[0077] Embodiment 13: A storage system according to Embodiment 12, wherein: the first Ethernet interface device is further coupled to the second Ethernet interface device via an interconnect associated with a host computer; the first processor of the first SSD is configured to: further open a connection between the first SSD and the second SSD on the interconnect; and the first processor is configured to open a connection between the first SSD and the second SSD, the connection not including the host computer; and the Ethernet switch is configured to: direct Ethernet packets between the first SSD and the second SSD without transmitting the packets to the host computer.

[0078] Embodiment 14: A storage system according to any one of Embodiments 11 to 13, wherein the first processor of the first SSD is configured to open a connection according to one of the following items: i) Fibre Channel communication protocol, ii) Remote Data Memory Access (RDMA) communication protocol, iii) InfiniBand communication protocol, or iv) NVMe over Transport Control Protocol (NVMeTCP) communication protocol.

[0079] Embodiment 15: A storage system according to any one of Embodiments 11 to 14, wherein the network structure is configured to: transfer data between the first SSD and the second SSD without transferring any of the data to a host computer that is communicatively coupled to the first SSD and the second SSD.

[0080] Embodiment 16: A storage system according to Embodiment 15, wherein: a host computer is configured to: communicate with a first SSD and a second SSD at a first data rate via a network structure; a first processor of the first SSD is configured to: send an indication of a second data rate on a connection, at which the second SSD communicates with the host computer on the network structure; and the second data rate is a fraction of the first data rate.

[0081] Embodiment 17: A storage system according to any one of Embodiments 11 to 16, wherein the network structure includes an Ethernet switch.

[0082] Embodiment 18: A storage system according to Embodiment 17, wherein: a first processor of the first SSD is configured to: encapsulate the package body in an Ethernet packet having a network address of the second SSD; the first built-in network interface device includes a first Ethernet interface device, and the first Ethernet interface device is configured to: transmit the Ethernet packet to an Ethernet switch; the Ethernet switch is configured to: use the network address of the second SSD in the Ethernet packet to forward the Ethernet packet to the second built-in network interface device; the second built-in network interface device includes a second Ethernet interface device, and the second Ethernet interface device is configured to: provide the Ethernet packet to the second processor of the second SSD.

[0083] Embodiment 19: The storage system according to any one of Embodiments 11 to 18, wherein the transmission of data comprises: copying or retrieving data on a connection between the first SSD and the second SSD.

[0084] Embodiment 20: A storage system according to any one of Embodiments 11 to 19, wherein: the first processor of the first SSD is configured to implement a first remote direct memory access (RDMA) communication protocol layer; the second processor of the second SSD is configured to implement a second RDMA communication protocol layer; and the first RDMA communication protocol layer is configured to open a connection with the second RDMA communication protocol layer.

[0085] Although aspects of the present disclosure have been described in conjunction with specific embodiments of the present disclosure proposed as examples, substitutions, modifications and variations of the examples may be made. Therefore, the embodiments set forth herein are intended to be illustrative rather than restrictive. Changes may be made without departing from the scope of the set forth claims.

Claims

1. A method for transferring data between a first solid-state drive (SSD) and a second SSD, the first SSD having a first built-in network interface device configured to communicate via a network structure, and the second SSD having a second built-in network interface device configured to communicate via the network structure, the method comprising: Opening a connection between the first SSD and the second SSD on the network structure; Based on a Non-Volatile Memory Express over Fabric (NVMe-oF) communication protocol, encapsulating an NVMe command in an encapsulation to transfer data between the first SSD and the second SSD over the connection; sending the capsule from the first SSD to the second SSD over the connection via the network structure; executing, by the second SSD, the NVMe command in the package to transfer the data between the first SSD and the second SSD over the connection; as well as Sending, by the first SSD to the second SSD, an indication of a data rate to be used by the second SSD when communicating with a host computer communicatively coupled to the network structure includes sending the indication of the data rate to the second SSD via the network structure.

2. The method according to claim 1, wherein: The network structure includes an Ethernet switch; The method further comprises: encapsulating the capsule in an Ethernet packet at the first built-in network interface device of the first SSD, and At the second built-in network interface device of the second SSD, decapsulating the capsule from the Ethernet packet; and Sending the capsule from the first SSD to the second SSD includes sending the capsule in the Ethernet packet via the Ethernet switch.

3. The method of claim 1, wherein opening the connection comprises opening the connection according to one of: i) a Fibre Channel communication protocol, ii) a Remote Data Memory Access (RDMA) communication protocol, iii) a wireless bandwidth communication protocol, or iv) an NVMe over Transport Control Protocol (NVMeTCP) communication protocol.

4. The method according to claim 1, wherein: Opening the connection between the first SSD and the second SSD further comprises: further opening the connection on an interconnect associated with a host computer; Opening the connection between the first SSD and the second SSD includes: opening a connection that does not include the host computer; and Executing the NVMe command in the enclosure to transfer the data between the first SSD and the second SSD includes transferring data between the first SSD and the second SSD without transferring any of the data to the host computer.

5. The method according to claim 1, wherein the transmitting of the data comprises: The data is copied or retrieved over the connection between the first SSD and the second SSD.

6. The method according to claim 1, wherein: Opening the connection between the first SSD and the second SSD includes: opening a connection that does not include a host computer communicatively coupled to the first SSD and the second SSD; and Executing the NVMe command in the enclosure to transfer the data between the first SSD and the second SSD includes transferring data between the first SSD and the second SSD without transferring any of the data to the host computer.

7. The method of claim 1 , wherein opening the connection comprises: The RDMA layer of the first Ethernet enabled SSD opens a connection with the RDMA layer of the second Ethernet enabled SSD in the network fabric.

8. The method of claim 1 , wherein sending the indication of the data rate to be used by the second SSD when communicating with the host computer comprises: An indication of a fraction of a host data rate that the host computer is capable of using is sent.

9. The method of claim 1 , wherein sending the indication of the data rate to be used by the second SSD when communicating with the host computer comprises: An indication of a percentage of a host data rate that the host computer is capable of using is sent.

10. The method of claim 1, wherein the network structure is configured with TCP or Ethernet.

11. The method of claim 1 , wherein sending the capsule from the first SSD to the second SSD over the connection comprises: The capsule within the Ethernet frame is sent based on an IP address of the Ethernet frame, the IP address indicating a destination of the capsule.

12. A storage system comprising: a first solid-state drive (SSD) having i) a first built-in network interface device configured to communicate via a network fabric, and ii) a first processor; a second SSD having i) a second built-in network interface device configured to communicate via the network structure, and ii) a second processor; as well as The network structure; wherein the first processor of the first SSD is configured to: i) open a connection between the first SSD and the second SSD on the fabric, the connection being on the network fabric, ii) encapsulate a non-volatile memory express (NVMe) command in an encapsulation based on a non-volatile memory over fabric (NVMe-oF) communication protocol to transfer data between the first SSD and the second SSD on the connection, iii) provide the encapsulation to the first built-in network interface device to send the encapsulation to the second built-in network interface device of the second SSD on the connection; and iv) sending to the second SSD an indication of a data rate to be used by the second SSD when communicating with a host computer communicatively coupled to the network fabric, comprising sending the indication of the data rate to the second SSD via the network fabric; The second processor of the second SSD is configured to: i) receive the capsule from the second built-in network interface device, ii) decapsulate the NVMe command from the capsule according to the NVMe-oF communication protocol, and iii) execute the NVMe command to transmit the data between the first SSD and the second SSD on the connection.

13. The storage system according to claim 12, wherein: The first processor of the first SSD is configured to encapsulate the capsule in an Ethernet packet; The first built-in network interface device comprises a first Ethernet interface device configured to transmit the Ethernet packet; The network structure includes an Ethernet switch, wherein the Ethernet switch is configured to: forward the Ethernet packet to the second built-in network interface device of the second SSD; The second built-in network interface device comprises a second Ethernet interface device configured to provide the Ethernet packets to the second processor of the second SSD; and The second processor is configured to decapsulate the encapsulation from the Ethernet packet.

14. The storage system according to claim 13, wherein: The first Ethernet interface device is further communicatively coupled to the second Ethernet interface device via an interconnect associated with a host computer; The first processor of the first SSD is configured to: further open the connection between the first SSD and the second SSD on the interconnect; and The first processor is configured to open a connection between the first SSD and the second SSD that does not include the host computer; and The Ethernet switch is configured to direct Ethernet packets between the first SSD and the second SSD without transmitting the packets to the host computer.

15. The storage system of claim 12, wherein the first processor of the first SSD is configured to open the connection according to one of: i) a Fibre Channel communication protocol, ii) a Remote Data Memory Access (RDMA) communication protocol, iii) a wireless bandwidth communication protocol, or iv) an NVMe over Transport Control Protocol (NVMeTCP) communication protocol.

16. The storage system of claim 12, wherein the network structure is configured to transfer the data between the first SSD and the second SSD without transferring any of the data to a host computer communicatively coupled to the first SSD and the second SSD.

17. The storage system according to claim 12, wherein: The host computer is configured to: communicate with the first SSD and the second SSD via the network structure at a host data rate; The first processor of the first SSD is configured to send the indication of the data rate to be used by the second SSD when communicating with the host computer as an indication of a fraction of the host data rate.

18. The storage system according to claim 12, wherein: The host computer is capable of communicating with the first SSD and the second SSD via the network structure at a host data rate; and The indication of the data rate to be used by the second SSD when communicating with the host computer is specified as a percentage of the host data rate.

19. The storage system of claim 12, wherein the network fabric comprises an Ethernet switch.

20. The storage system of claim 19, wherein: The first processor of the first SSD is configured to: encapsulate the capsule in an Ethernet packet having a network address of the second SSD; The first built-in network interface device includes a first Ethernet interface device, the first Ethernet interface device being configured to: transmit the Ethernet packet to the Ethernet switch; The Ethernet switch is configured to: forward the Ethernet packet to the second built-in network interface device using the network address of the second SSD in the Ethernet packet; and The second built-in network interface device includes a second Ethernet interface device configured to provide the Ethernet packet to the second processor of the second SSD.

21. The storage system according to claim 12, wherein the transmission of the data comprises: The data is copied or retrieved over the connection between the first SSD and the second SSD.

22. The storage system according to claim 12, wherein: The first processor of the first SSD is configured to implement a first remote direct memory access (RDMA) communication protocol layer; The second processor of the second SSD is configured to implement a second RDMA communication protocol layer; as well as The first RDMA communication protocol layer is configured to open a connection with the second RDMA communication protocol layer.

Images