Termination of non-volatile memory networking messages at the driver level

Abstract

A storage device, such as a solid-state drive, is configured to receive messages, such as non-volatile memory networking messages, issued by a host processor or computer, formatted using NVM Express over Fabric (NVMe‑oF) or NVMe / Transmission Control Protocol communications for transmitting commands over a network fabric. The storage device is configured to terminate the NVMe‑oF or NVMe / TCP formatted communications at the storage drive level. The storage device may also be configured to issue a reply message, the reply message including NVMe‑oF or NVMe / TCP formatting formatted for communications for delivering the reply message over a network fabric.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] The present disclosure claims priority to U.S. Provisional Application Serial No. 62 / 818,557, filed on March 14, 2019, and entitled “NVMe-oF TERMINATION AT THE DRIVE LEVEL,” the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure relates to the field of data storage for computer and electronic network systems, and more particularly, to termination of non-volatile memory networking messages formatted using various communication formats between storage devices coupled to a computer or electronic network system. Background Art

[0004] The background description provided herein is intended to generally present the context of the present disclosure. The work of the presently named inventors is neither explicitly nor implicitly admitted to be prior art to the present disclosure within the scope of the work described in this background section nor to the extent that it may not qualify as prior art at the time the application was filed.

[0005] A solid-state drive (SSD) is a data storage device that uses non-volatile memory, such as NAND (Not-And) or NOR (Not-Or) non-volatile memory, to store persistent digitally encoded data. An SSD can be configured to emulate a hard disk drive, a device that stores persistent digitally encoded data on the magnetic surface of a rapidly rotating platter and replaces a hard disk drive (HDD) in many applications.

[0006] A host computer ("host") is typically coupled to the SSD to request and control data services provided by the SSD, such as reading data from the SSD, writing data to the SSD, and erasing data from the SSD. The host may be coupled to the SSD via a network structure, which may include one or more intermediate devices coupled via wired and / or wireless interconnections, such as one or more interface switches and / or routers. To facilitate data operations, the SSD may include an SSD controller, also referred to as a "local controller," which may include an SSD interface for communicating with the host computer via the network structure, and may include a local processor. In some examples, the local controller may communicate directly with the host computer, where the host computer is then communicatively coupled to additional computer devices and / or networks, which allows communication between the SSD and these additional computer devices and / or networks.

[0007] Traditionally, the physical connection between the SSD and the local processor is a hardwired connection, and communication between the SSD and the local processor may be performed using Peripheral Component Interconnect Express (PCI Express or PCIe) as a transport layer, and NVM Express (NVMe) as a protocol layer for formatting communications between these devices. When an interface switch is included, this same PCI Express / NVMe communication format may be used to provide communications between the local processor and the interface switch.

[0008] Communications between the host computer and the interface switch may be provided over a network fabric, such as Ethernet, Fibre Channel, or InfiniBand. Communications between the host computer and the interface switch may utilize NVMe over Fabric (NVMe-oF) as a protocol layer. The transport layer for these communications may utilize Ethernet or Remote Direct Memory Access (RDMA). Summary of the invention

[0009] The present disclosure generally relates to storage devices, such as but not limited to solid-state drives (SSDs), configured to receive and terminate commands configured using NVMe-oF communication formatting.

[0010] Aspects of the present disclosure relate to a method for terminating a non-volatile memory networking message at a drive level, the method comprising: receiving, at a storage device, a message requesting a data storage service to be provided by the storage device, the message being provided over a network interconnect, the message having NVM Express over Fabric (NVMe-oF) communication formatting of the message. The method further comprises: terminating, at the storage device, the message received at the storage device having NVMe-oF communication formatting.

[0011] Aspects of the present disclosure relate to a method for terminating a non-volatile memory networking message at a driver level, the method comprising: receiving, at a storage device, a message requesting a data storage service to be provided by the storage device, the message being provided over a network interconnect using NVMe / Transmission Control Protocol (NVMe / TCP) communication formatting of the message. The method also comprises: terminating, at the storage device, the message received at the storage device having the NVMe / TCP communication formatting.

[0012] Aspects of the present disclosure relate to an apparatus for terminating non-volatile memory networking messages at a drive level, the apparatus comprising a storage device, the storage device comprising a local processor and a memory array, wherein the memory array is configured to: store persistent digitally encoded data on the non-volatile memory, and wherein the local processor is configured to: terminate at the local processor an NVM Express over Fabric (NVMe-oF) or NVMe / Transport Control Protocol (NVMe / TCP) message that has been issued by a host processor and communicated over a network fabric. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 An example configuration of a computer networking system including a storage device configured to terminate non-volatile memory networking messages at the storage device level is illustrated.

[0014] Figure 2A is a flow chart of a method for terminating non-volatile memory networking messages at the driver level.

[0015] Figure 2B is a flow chart of another method for terminating non-volatile memory networking messages at the driver level.

[0016] Figure 3 is a simplified example block diagram of a storage device that may be configured to terminate non-volatile memory networking messages at the driver level.

[0017] The drawings are for the purpose of illustrating example embodiments, but it should be noted that the embodiments are not limited to the arrangements and instrumentalities shown in the drawings. DETAILED DESCRIPTION

[0018] The present disclosure provides examples and details related to data storage devices, such as solid-state drives (SSDs), hard disk drives (HDDs), and / or hybrid SSD / HDD drives, which are configured to provide data storage, and to receive and terminate non-volatile memory networking messages at the drive level. Non-volatile memory networking messages may include commands that include a request for a data service issued by a host computer, which data service may be provided by a target storage device to which the communication is directed. In addition, the examples of storage devices described herein may be configured to generate and issue replies formatted using NVMe communication formatting, which are transmitted to one or more other devices (such as a host processor or a computer device) over a network structure (such as Ethernet).

[0019] NVM Express (NVMe) is a specification that defines how a host device (such as a host processor or computer) can communicate with a storage device (such as an SSD) over a PCI Express-compatible bus. NVMe over Fabric (NVMe-oF) is another technical specification designed to implement commands based on non-volatile memory express messages to transfer data between a host device (such as a host processor or computer) and a target storage device (such as an SSD) over a network fabric (such as Ethernet, Fibre Channel (FC), or InfiniBand). The addition of the NVMe Transport Control Protocol (NVMe / TCP) enables the communication of non-volatile memory networking messages with TCP as the transport layer for communication.

[0020] Although both NVMe and NVMe-oF have similar features, such as using a command set to control operations on a storage device as requested by a host device, one of the main differences between NVMe and NVMe-oF is the transport mapping mechanism used to send and receive commands and responses for communication between devices. NVMe-oF uses a message-based model for communication between a host device and a target storage device, where local NVMe communication includes mapping commands and responses to shared memory in the host over a PCIe interface protocol.

[0021] NVMe-oF implements the use of alternative transmissions to PCIe, where these alternative transmissions extend the distances across which NVMe host devices and NVMe storage drives or subsystems can be connected. The original design goal of NVMe-oF was to increase the latency of no more than 10 microseconds between the NVMe host and the remote NVMe target storage device connected on the appropriate network structure, relative to the latency of the NVMe storage device using the PCIe bus of the local server. A greater storage system throughput is achieved by using NVMe-oF technology. In addition, as described herein, the examples of systems, devices, methods, and techniques for terminating NVMe-oF transmissions at the drive (SSD / HDD) level rather than at an aggregator (such as a network adapter card-sNIC / NIC) allow for greater storage system throughput and better drive utilization. The examples of storage systems, devices, methods, and techniques described herein may no longer be limited to network adapter bottlenecks, or rely on computing as a method of connecting multiple drives to a network, but may utilize interface devices such as Ethernet switches in a less-computing architecture, where NVMe-oF formatted commands can be terminated at the drive level rather than at an intermediate device.

[0022] In addition, the NVMe / TCP standard defines the mapping of NVMe queues, NVMe-oF encapsulation, and data delivery over the IETF Transmission Control Protocol (TCP). The addition of NVMe over TCP makes it possible to use NVMe-oF across standard Ethernet networks.

[0023] Figure 1 An example configuration of a computer networking system 100 including a storage device configured to terminate non-volatile memory networking messages (hereinafter, "messages") at the storage device level is illustrated. The messages may be communicated over a fabric network using NVMe-oF as a communication format and terminated at a storage device (such as a solid-state drive). In other examples, the messages may be communicated over a fabric network using NVMe / TCP as a communication format and terminated at a storage device (again, such as a solid-state drive). Figure 1 As shown in , a computer networking system 100 (system 100) includes a host 101 coupled to a network 110. The network 110 is also coupled to a data storage site, which is generally represented by brackets 120 (hereinafter, "storage site 120"). The storage site 120 includes at least one backplane 130. In various examples, the backplane 130 can represent a storage rack including one or more backplanes 130, each backplane including multiple storage devices, such as solid-state drives (SSDs). Each backplane can also include a network structure that couples the SSDs included on the backplane through one or more switching devices and / or other networking devices.

[0024] Host 101 is not limited to any particular type of host or device, and may include any type of host, such as a computer processor or a network of computers and processors. Host 101 is not necessarily limited to including a single host device, and may represent multiple host devices communicatively coupled through one or more network interface cards, illustratively represented by network interface card (NIC) 102. NIC 102 is coupled to network 110 through interconnect 109. Network 110 is also communicatively coupled to storage site 120 through interconnect 111. For example, network 110 is communicatively coupled to backplane 130 through interconnect 111 of network fabric 134 coupled to backplane 130. Network fabric 134 is further coupled to each NVMe compatible storage device in NVMe compatible storage devices 136A-136X through corresponding interconnects 135A-135X. In various examples, each of the interconnect 133, the network fabric 134, and the interconnects 135A-135X is configured to communicate using NVMe-oF or NVMe / TCP formatted commands so that incoming messages received at the interconnect 133 are passed and not terminated until the command reaches one or more storage devices 136A-136X. Each of the storage devices 136a-136X includes one or more processors (PROCs) coupled to a non-volatile memory array (NVM). The one or more processors at each storage device are configured to terminate any NVMe-oF and / or NVMe / TCP formatted communications received from the network fabric 134 at the storage device.

[0025] In an illustrative, non-limiting example, the host 101 provides commands and / or data to the NIC 102 to communicate with the backplane 130, wherein the commands and / or data may include: data to be stored on one or more storage devices 136A-136X, (multiple) commands for reading data stored on one or more of the storage devices 136A-136X, or other commands, such as commands to erase data stored on one or more (multiple) storage devices 136A-136X, or other types of data housekeeping commands related to data stored in the storage devices of the backplane 130, such as defragmentation commands. The NIC 102 uses NVMe-oF as a protocol layer to provide commands and / or data for transmission to the backplane 130 using, for example, RDMA v02 over a converged Ethernet network as a transport layer for communication.

[0026] Communications output from the NIC 102 use the NVMe-oF protocol, are passed through the interconnect 109 to the network 110, through the network 110, and continue to be passed through the interconnect 111 to the network fabric 134 at the backplane 130. Communications received at the network fabric 134 are forwarded to one or more NVMe storage devices 136A-136X using the same communication formatting (such as NVMe-oF formatting or NVMe / TCP formatting). At one or more NVMe storage devices 136A-136X, the message communicated to and through the network fabric 134 is terminated at each storage device 136A-136X, and the operation requested by the message is initiated by the local processor and memory array, which is included in the appropriate storage device in the storage devices 136A-136X.

[0027] The lack of a need for a network interface card, such as an Ethernet network interface card (and in many cases, one or more processors included in the network interface card), which is typically required to couple a network to the storage devices 136A-136X, reduces the overall number of devices required to provide communications between the host 101 and the storage devices. In existing systems, NVMe communications originating from the NIC 102 by the host 101 and conveyed over the network 110 and interconnect 111 to a network interface card located at the backplane would be terminated, for example, at a network interface card at the storage box or backplane 130, and then forwarded to the underlying storage, for example, over a PCIe switch.

[0028] Termination as used in this disclosure refers to communication having a specific format, such as a transport layer or protocol layer relative to the communication, or reaching its final destination device, or reaching an intermediate device that is not its final destination device to which the command is directed, and wherein the intermediate device receives the communication transmitted using a first communication formatting, and reconfigures the communication using a second communication formatting that is different from the first communication formatting before forwarding the communication to another device.

[0029] In some examples, termination refers to ending the use of NVMe-oF as a protocol for further transmitting received communications and switching to a different protocol layer for further transmitting received communications. In other examples, termination refers to ending the use of NVMe / TCP as a protocol for further transmitting received communications and switching to a different communication format for further transmitting received communications. For example, in an existing system, NVMe-oF communications received at a network interface card of a storage box would be terminated, and the switch would be configured to forward the communications to one or more storage devices in the storage devices 136A-136X using a different protocol, such as using PCIe as a transport layer and using NVMe as a protocol layer. Terminating NVMe-oF communications at the network interface card requires that the network interface card include one or more processors and be programmed to terminate incoming NVMe-oF communications and reconfigure outgoing communications from the NIC to the storage devices 136A-136X using different transport layers and protocol layers.

[0030] An example of the backplane 130 is configured so that one or more of the storage devices 136A-136X are NVMe-oF compatible devices and are capable of directly receiving and terminating communications including NVMe-oF formatted communications. In other examples, the backplane 130 is configured so that one or more of the storage devices 136A-136X are NVMe / TCP compatible devices and are capable of directly receiving and terminating communications including NVMe / TCP formatted communications. In this way, communications received at the network fabric 134 are not terminated at the network fabric, but are passed through the network fabric as NVMe formatted communications to one or more of the storage devices 136A-136X. This allows the above-mentioned advantages related to NVMe devices and communications, including: increasing the distance over which NVMe host devices and NVMe storage devices can communicate with each other; the maximum cumulative bandwidth of the entire flash array that is not limited by NIC capacity; and reduced latency in communications compared to devices such as NICs, CPUs, and devices that require PCIe interface protocols and use PCIe switches along certain portions of the path that the communication must pass.

[0031] In addition to receiving commands having NVMe formatting, the storage devices 136A-136X can be configured to provide reply messages directed back to, for example, the host 101, the reply messages having an NVMe-oF or NVMe / TCP formatting configuration generated at the storage device. In this way, these outbound reply messages issued by the storage devices 136A-136X can also be transmitted across the network without terminating the reply message until the reply message reaches the target host (e.g., host 101), the network including the network structure 134, the interconnect 111, the network 110, and the interconnect 109. Because these reply messages can be transmitted across the network structure using NVMe-oF or NVMe / TCP formatting without terminating the reply message at any intermediate device, the above-mentioned advantages of extended distance and reduced latency for NVMe formatted communications can be achieved with respect to the transmission of the reply messages.

[0032] Examples of backplane 130 may include additional features, such as redundant blocks 140. Examples of redundant blocks 140 may include replication of circuitry, including network fabric 134 and storage devices 136A-136X as described above, or any equivalent thereof. Redundant blocks 140 may also be coupled to network 110 via interconnect 111, and may be configured to receive the same communications directed to storage devices 136A-136X. Redundant storage devices included in redundant blocks 140 may also be configured as NVMe-oF or NVMe / TCP compatible devices, and are configured to receive and terminate commands formatted via NVMe-oF or NVMe / TCP issued by host 101 over the network and via interconnect 111 at the storage devices included in block 140. By being configured to receive the same NVMe-oF or NVMe / TCP commands received by storage devices 136A-136X, redundant blocks 140 may provide replication and therefore backup for any or all data transactions and data operations performed / provided by storage devices 136A-136X. Thus, redundant blocks 140 provide an additional level of robustness to the data storage functionality provided by backplane 130 .

[0033] Examples of system 100 are not limited to having a particular number of storage devices (such as SSDs). In various examples, system 100 may include as few as two SSDs, and may include some positive integer "N" number of SSDs, such as Figure 1 130 is illustratively represented by a point generally indicated by bracket 137. As a non-limiting example, data storage may include multiple independent SSDs coupled to interconnects 135A-135X in an arrangement of system 100. Each of the SSDs included in backplane 130 may be communicatively linked to network 110 via network fabric 134 and connection 133.

[0034] Furthermore, storage site 120 is not limited to having a particular number of storage units. As a non-limiting example, Figure 1 As shown in FIG. 1 , the storage site 120 may include multiple storage backplanes, including Figure 1 The backplanes 130, 152, 154, and 156 shown in FIG. The additional storage units 152, 154, and 156 may include a number of NVMe-of compatible storage devices and may be configured to perform any functions and provide any features described above with respect to the backplane 130.

[0035] Example NVMe-oF termination at the drive level

[0036] In various examples, one or more storage devices (such as SSDs) are provided at a storage site or as part of a storage unit and are configured as NVMe-oF compatible devices that are configured to terminate NVMe-oF formatted communications at the drive level. The storage devices may also be configured to generate responses in the form of reply messages. The reply messages may be transmitted from the storage device to a host device that initiated a command received by the storage device, wherein the reply messages are formatted using NVMe-oF communication formatting so that the reply messages may be transmitted from the storage device to the host device over a network structure without terminating the NVMe-oF formatted communications before the reply messages arrive at the target host device.

[0037] Example NVMe / TCP termination at the driver level

[0038] In various examples, one or more storage devices (such as SSDs) are provided at a storage site or as part of a storage unit and are configured as NVMe compatible devices that are configured to terminate NVMe / TCP formatted communications at the drive level. For example, incoming NVMe / TCP formatted communications received at the backplane 130 can be passed through the network structure 134 and passed to one or more storage devices in the storage devices 136A-136X using NVMe / TCP formatting for the message. The NVMe / TCP communication is then terminated, for example, by a corresponding processor or multiple processors located at each storage device. The storage devices may also be configured to generate a response in the form of a reply message. The reply message may be transmitted from the storage device to the host device that initiated the command received by the storage device, wherein the reply message is formatted using NVMe / TCP communication formatting so that the reply message can be transmitted from the storage device to the host device through the network structure 134 without terminating the NVMe / TCP formatted communication before the reply message reaches the target host device.

[0039] Figure 2A1 is a flow chart of a method 200 for terminating non-volatile memory networking messages at a driver level. The method 200 includes various functions associated with one or more storage devices, such as SSDs, that are configured as NVMe-oF compatible devices and are further configured to terminate any NVMe-oF formatted messages directed to the storage device at the driver level. The storage device may also be configured to generate and issue one or more reply messages directed to the host and regarding a status provided in response to a received message, wherein the reply messages may also be generated and issued using NVMe-oF formatting.

[0040] At block 202, an example of method 200 includes receiving an NVMe message issued by a host at a storage device, the NVMe message being directed to a data storage service to be provided by the storage device, the message being provided over a network fabric using NVMe-oF communication formatting. In various examples, the storage device is a solid state drive (SSD) device. In various examples, the storage device is a hard disk drive (HDD). In various examples, the storage device is a hybrid SSD / HDD. In various examples, the network fabric includes an Ethernet device and an Ethernet compatible interconnect coupling devices of the network fabric.

[0041] At block 204, the example of method 200 includes terminating the NVMe-oF formatted communication including the message at the storage device. In various examples, terminating the NVMe-oF formatted communication includes: terminating the communication at a local processor of the storage device. In various examples, the local processor and the memory array of the storage device are provided on the same integrated circuit device or circuit device.

[0042] At box 206, an example of method 200 includes receiving an NVMe message at a redundant storage device, the NVMe message being directed to the storage device and received by the storage device as the same message, the same message being provided on a network structure using NVMe-oF communication formatting. In various examples, the redundant storage device is a solid-state drive (SSD) device. In various examples, the redundant storage device is a hard disk drive device (HDD). In various examples, the storage device is a hybrid SSD / HDD. In various examples, the redundant storage device is a device of the same type as the storage device receiving the message at box 202 of method 200 (e.g., an SSD). In various examples, the redundant storage device is a different type of device (e.g., an SSD and an HDD) relative to the storage device receiving the message at box 202 of method 200.

[0043] At block 208, an example of method 200 includes performing one or more data storage operations based on the received message. Performing the one or more data storage operations may include reading data stored in a storage array of a storage device, writing data to a storage array of a storage device, erasing data stored in a storage device, some combination of these operations, and / or other operations related to data stored in the storage device and / or an operating state of the storage device. For example, a message may be received that includes a request to provide information related to the total and / or remaining data storage capacity present in the storage device, and / or a request to report performance information about latency or other performance parameters associated with the operation of the storage device receiving the message.

[0044] In some examples, performing the data storage operation may include setting a memory location or flag in a request queue of the storage device to indicate that one or more operations requested as a result of receiving the message at the storage device have been received, and setting a memory location or flag in a completion queue of the storage device to indicate that one or more operations requested as part of receiving the message have been completed. In various examples, one or more memory locations or flags within the storage device may be set when the operation requested by the message cannot be fully and / or partially completed for any reason (e.g., by setting one or more error codes associated with the desired data storage operation associated with the message).

[0045] At box 210, an example of method 200 includes: generating a reply message using NVMe-oF formatted communications when one or more data storage operations associated with the received message are completed. In various examples, the reply message may include data requested to be read from a data array of the storage device receiving the message. In various examples, the reply message may include information or other indications indicating that the data storage operation requested by the received message has been successfully completed. In various examples, the reply message may include information or other indications, such as flag settings, indicating that the data storage operation requested by the received message was not successfully completed. The reply message indicating that the requested data storage service was not successfully completed may further include data, such as one or more error codes or text information, which further indicates or explains why the requested operation was not or could not be successfully completed.

[0046] At block 212, the example of method 200 includes transmitting the reply message generated at block 210 to the host over a network fabric using NVMe-oF formatted communications. In various examples, at least some portion of the network fabric used to transmit the NVMe-oF formatted reply message is the same portion of the network fabric used to transmit the message received at the storage device.

[0047] Figure 2B2 is a flow chart of another method 220 for terminating non-volatile memory networking messages at a driver level. The method 220 includes various functions associated with one or more storage devices, such as SSDs, that are configured as NVMe / TCP compatible devices and are further configured to terminate any NVMe / TCP formatted messages directed to the storage device at the driver level. The storage device may also be configured to generate and issue one or more reply messages directed to the host and regarding a status provided in response to a received message, wherein the reply messages may also be generated and issued using NVMe / TCP formatting.

[0048] At box 222, an example of method 220 includes: receiving a message sent by a host at a storage device, the message being directed to a data storage service to be provided by the storage device, the message being provided on a network structure using NVMe / TCP as a transport layer for communication formatting of the message. In various examples, the storage device is a solid state drive (SSD) device. In various examples, the storage device is a hard disk drive (HDD). In various examples, the storage device is a hybrid SSD / HDD. In various examples, the network structure includes a TCP compatible interconnect coupling devices of the network structure.

[0049] At block 224, an example of method 220 includes terminating, at the storage device, the NVMe / TCP formatted communication including the message. In various examples, terminating the NVMe / TCP formatted communication includes terminating the communication at a local processor or a local interface of the storage device. In various examples, the local processor and the memory array of the storage device are provided on the same integrated circuit device or circuit device.

[0050] At box 226, an example of method 220 includes: receiving a message at a redundant storage device, the message being directed to the storage device and received by the storage device as the same message, the same message being provided on a network structure using NVMe / TCP communication formatting. In various examples, the redundant storage device is a solid-state drive (SSD) device. In various examples, the redundant storage device is a hard disk drive device (HDD). In various examples, the storage device is a hybrid SSD / HDD. In various examples, the redundant storage device is a device of the same type as the storage device receiving the message at box 222 of method 220 (e.g., an SSD). In various examples, the redundant storage device is a different type of device (e.g., an SSD vs. an HDD) relative to the storage device receiving the message at box 222 of method 220.

[0051] At box 228, an example of method 220 includes: based on the received message, performing one or more data storage operations. Performing one or more data storage operations may include: reading data stored in a storage array of a storage device, writing data to a storage array of a storage device, erasing data stored in a storage device, some combination of these operations, and / or other operations related to data stored in the storage device and / or an operating state of the storage device. For example, a message may be received that includes a request for providing information related to the total and / or remaining data storage capacity present in the storage device, and / or a request for reporting performance information about latency or other performance parameters associated with the operation of the storage device receiving the message.

[0052] In some examples, performing the data storage operation may include setting a memory location or flag in a request queue of the storage device to indicate that one or more operations requested as a result of receiving the message at the storage device have been received, and setting a memory location or flag in a completion queue of the storage device to indicate that one or more operations requested as part of receiving the message have been completed. In various examples, one or more memory locations or flags within the storage device may be set when the operation requested by the message cannot be fully and / or partially completed for any reason (e.g., by setting one or more error codes associated with the desired data storage operation associated with the message).

[0053] At box 230, an example of method 220 includes: generating a reply message using NVMe / TCP formatted communications when one or more data storage operations associated with the received message are completed. In various examples, the reply message may include data requested to be read from a data array of the storage device receiving the message. In various examples, the reply message may include information or other indications indicating that the data storage operation requested by the received message has been successfully completed. In various examples, the reply message may include information or other indications, such as flag settings, indicating that the data storage operation requested by the received message was not successfully completed. The reply message indicating that the requested data storage service was not successfully completed may further include data, such as one or more error codes or text information, which further indicates or explains why the requested operation was not or could not be successfully completed.

[0054] At block 232, the example of method 220 includes transmitting the reply message generated at block 230 to the host over a network fabric using NVMe / TCP formatted communications. In various examples, at least some portions of the network fabric used to transmit the NVMe / TCP formatted reply message are the same portions of the network fabric used to transmit messages received at the storage device.

[0055] Example Device

[0056] Figure 3 300, which may be configured according to various examples described throughout this disclosure and any equivalents thereof. An example of a solid-state drive (SSD) 300 may include a local processor 302, a memory array 304, an interface 306, and one or more queue sets 308. A bus 303 may be coupled to various elements included in the SSD 300 and configured to communicatively couple these elements. An example of the SSD 300 may be as described with respect to Figure 1 Any one or more of the storage devices 136A-136X shown and described may be configured to perform some or all of the functionality described with respect to the method 200 (FIG. 2) and any equivalents thereof.

[0057] Examples of local processor 302 may include multiple processors, multiple cores, multiple nodes, and / or implement multithreading, etc., that coordinate operations on SSD 300. Examples of memory array 304 are not limited to any particular type of memory, and may be system memory (e.g., one or more of cache, random access memory (RAM), synchronous RAM (SRAM), dynamic RAM (DRAM), zero-capacitor RAM, two-transistor RAM, embedded DRAM (eDRAM), extended data output RAM (EDORAM), double data rate RAM (DDRRAM), electrically erasable programmable read-only memory (EEPROM), Nano-RAM (NRAM), resistive RAM (RRAM), silicon oxide-nitride-oxide memory (SONOS), parameter random access memory (PRAM), etc.) or any one or more other possible implementations of non-transitory machine-readable media / medium.

[0058] Examples of interface 306 are not limited to any particular type of interface and may include one or more interface devices or modules that can be controlled by local processor 302 to provide communication between SSD 300 and one or more other devices (such as host 101 ( Figure 1)). For example, the interface 306 is configured to receive and terminate an incoming message that has been formatted using NVMe-oF communication to be transmitted to the interface over the network structure. The interface can be configured to receive a message formatted with NVMe-oF and terminate the communication including the message at the interface. In other examples, the interface 306 is configured to receive and terminate an incoming message that has been formatted using NVMe / TCP communication to be transmitted to the interface over the network structure. The interface can be configured to receive a message formatted with NVMe / TCP and terminate the communication including the message at the interface.

[0059] The message portion of the terminated NVMe-oF or NVMe / TCP formatted communication received by the interface 306 can be passed to the local processor 302 for further processing and operation based on the content of the message. In addition, upon completion of the process indicated by the command portion of the incoming message, the local processor 302 can be configured to generate a reply message. The reply message may include information related to the completion or non-completion of the data storage operation as described above, such as data retrieved from the memory array 304, and / or other information related to the operation performed by the local processor 302 in response to the command. The local processor 302 can generate or control the interface 306 to generate a reply message in an NVMe-oF or NVMe / TCP compatible communication format, wherein the interface 306 is configured to transmit NVMe-oF or NVMe / TCP formatted communication on a network structure and directed to the host that initiated the command. The reply message can be transmitted along the entire route between the SSD 300 and the target host on a corresponding NVMe-oF or NVMe / TCP compatible network structure (such as an Ethernet or TCP compatible network structure) without terminating the reply message until the reply message arrives at the target host.

[0060] The bus 303 is not limited to any particular type of bus and may conform to, for example, the Peripheral Component Interconnect (PCI), Industry Standard Architecture (ISA), PCI-Express, New Bus (NuBus), Advanced Extensible Bus AXI bus, and Ethernet standards. The bus 303 may also facilitate communication between the local processor 302 and the memory array 304 and the queue 308. In various examples, the queue 308 includes a pair of queues 310, which include a submission queue (SQ) and a completion queue (CQ). The submission queue of the queue 310 may be configured to allocate memory space for storing status and commands related to the receipt of incoming commands, and the completion queue of the queue 310 may be configured to allocate memory space for storing status related to the completion of operations requested by the incoming commands received at the SSD 300. The queue 308 is not limited to having one submission / completion queue set and may have one or more additional queue sets, as shown by queue 311. In various examples, the respective queue sets included in the queue 308 may be reserved for storing commands received from corresponding specific host devices (such as Figure 1 Status information related to the command received by the host 101).

[0061] The functions attributed to SSD 300 may be implemented in a combination of hardware and / or software (e.g., computer code, program instructions, program code, computer instructions), which is stored on a non-transitory machine-readable medium / media. In some cases, local processor 302 and memory array 304 may implement or facilitate implementation of these functions instead of or in addition to management processor modules, data processor modules, and / or command processor modules. Additionally, examples of SSD 300 may include Figure 3 Fewer components or additional components not shown (e.g., video cards, audio cards, additional network interfaces, peripheral devices, etc.).

[0062] Although this specification contains many details, these details should not be interpreted as limitations on the scope of possible protection, but rather as descriptions of features that may be specific to a particular embodiment. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable sub-combination. In addition, although the above-mentioned features may be described as working in certain combinations, even initially claimed as such, in some cases, features in one or more claimed combinations may be deleted from the combination, and the claimed combination may be directed to a sub-combination or a variation of the sub-combination.

[0063] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that such operations be performed in the particular order or sequential order shown, or that all illustrated operations be performed, to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system components in the above-described embodiments should not be understood as requiring such separation in all embodiments.

Claims

1. A method for performing a data storage service, comprising: At a first storage device, receiving a message requesting a data storage service to be provided by the first storage device, the message being provided over a network interconnect and formatted according to a Non-Volatile Memory Express over Fabric (NVMe-oF) protocol; as well as At the first storage device, terminating the message formatted according to the NVMe-oF protocol; receiving, at a spare second storage device, a copy of the message formatted according to the NVMe-oF protocol; as well as At the backup second storage device, the copy of the message formatted according to the NVMe-oF protocol is terminated.

2. The method of claim 1 , wherein receiving the message at the first storage device comprises: receiving the message at a first solid-state drive; as well as Wherein receiving the copy of the message at the backup second storage device comprises: receiving the message at one of i) a second solid state drive and ii) a hard disk drive.

3. The method of claim 1 , wherein receiving the message at the first storage device comprises: The message is received according to an Ethernet transport layer protocol.

4. The method of claim 1 , wherein receiving the message at the first storage device comprises: The message is received according to the RoCEv2 transport layer protocol.

5. The method of claim 1 , wherein receiving the message at the first storage device comprises: The message is received in accordance with a wireless bandwidth communication protocol.

6. The method of claim 1, wherein terminating the message formatted according to the NVMe-oF protocol at the first storage device comprises: The message is terminated at a local processor of the first storage device.

7. The method according to claim 1, further comprising: When the requested data storage service is completed, a reply message is sent from the first storage device to the host processor, and the reply message is formatted according to the NVMe-oF protocol at the first storage device.

8. The method according to claim 7, wherein the reply message includes an indication of completion of the request for the data storage service sent by the host to the first storage device.

9. The method of claim 7, wherein the reply message includes data retrieved from the first storage device at the request of the host.

10. A method for performing a data storage service, comprising: At a first storage device, receiving a message requesting a data storage service to be provided by the first storage device, the message being provided over a network interconnect and formatted according to a Non-Volatile Memory Express over TCP (NVMe / TCP) protocol; as well as At the first storage device, terminating the message formatted according to the NVMe / TCP protocol; At a backup second storage device, receiving a copy of the message formatted according to the NVMe / TCP protocol; as well as At the backup second storage device, the copy of the message formatted according to the NVMe / TCP protocol is terminated.

11. The method of claim 10, wherein receiving the message at the first storage device comprises: receiving the message at a first solid-state drive; as well as Wherein receiving the copy of the message at the backup second storage device comprises: receiving the copy of the message at one of i) a second solid state drive and ii) a hard disk drive.

12. The method of claim 10, wherein terminating the message formatted according to the NVMe / TCP protocol comprises: The message is terminated at a local processor of the storage device.

13. The method of claim 10, wherein receiving the message at the first storage device comprises: The message is received from a network structure according to a Transmission Control Protocol (TCP) transport layer protocol.

14. The method according to claim 10, further comprising: When the requested data storage service is completed, a reply message is sent from the first storage device to the host processor, and the reply message is formatted according to the NVMe / TCP protocol at the first storage device.

15. The method of claim 14, wherein the reply message includes an indication of completion of the request for the data storage service issued by the host to the first storage device.

16. A storage system for performing data storage services, comprising: a first storage device, the first storage device comprising a first local processor and a first memory array, wherein the first memory array is configured to: store persistent digitally encoded data in non-volatile memory, and wherein the first local processor is configured to: terminate at the first local processor a message formatted according to a Non-Volatile Memory Express over Fabric (NVMe-oF) protocol or a NVMe / Transmission Control Protocol (NVMe / TCP) protocol, the message having been issued by a host processor and communicated over a network fabric; as well as a second storage device, the second storage device comprising a second local processor and a second memory array, wherein the second memory array is configured to: store persistent digitally encoded data in non-volatile memory, and The second local processor is configured to terminate, at the second local processor, the copy of the message terminated at the first local processor of the first storage device.

17. The storage system of claim 16, wherein the first storage device is a solid state drive.

18. The storage system of claim 16, wherein the network structure is configured to use RoCEv2 or TCP as a transport layer for communicating the message formatted according to the NVMe-oF protocol or the NVMe / TCP protocol.

19. The storage system of claim 16, wherein the network structure is configured to operate according to a wireless bandwidth transport layer protocol.

Images