System and apparatus for message tunneling communication

Abstract

Systems and apparatuses for message tunneling communications are provided. The apparatus for message tunneling communications can include a host interface circuit configured to communicate with a host device via a data protocol employing data messages. The apparatus can include a storage element configured to store data in response to the data messages. The host interface circuit can be configured to: detect whether a tunneling communication command is embedded within the data messages; extract message address information for a tunneling communication from the data messages; retrieve a tunneling communication message stored in a memory of the host device via the message address information for the tunneling communication; and route the tunneling communication message to an on-board processor and / or data processing logic. The on-board processor and / or data processing logic can be configured to execute one or more instructions in response to the tunneling communication message.

Description

TECHNICAL FIELD

[0001] The present description relates to computer communications, and more specifically, to systems and methods for message tunneling communications using a computer protocol, such as Non-Volatile Memory Express (NVMe). BACKGROUND

[0002] In modern information technology (IT) infrastructures, a relatively large amount of data can be generated by various devices and processes. Some examples of these data generators include, but are not limited to, smart machines, autonomous vehicles, social networks, and Internet of Things (IoT) devices. In addition, various artificial intelligence (AI) and machine learning techniques can be used to effectively analyze the collected data and use it to achieve higher efficiency and productivity of applications.

[0003] Moving a relatively large amount of raw data to a processor for processing and analysis can be expensive in terms of energy consumed and computer and network resources deployed. As a result, moving a large amount of raw data can increase the burden on resources, such as network bandwidth, processor cycles, and memory usage. In addition, moving a large amount of raw data to a server for processing can also increase the latency experienced by applications. SUMMARY

[0004] According to one general aspect, an apparatus can include a host interface circuit configured to communicate with a host device via a data protocol that employs data messages. The apparatus can include a storage element configured to store data in response to the data messages. The host interface circuit can be configured to: detect whether a tunneling communication command is embedded within a data message; extract message address information for a tunneling communication from the data message; retrieve a message for the tunneling communication that is stored in a memory of the host device via the message address information for the tunneling communication; and route the message for the tunneling communication to an on-board processor. The on-board processor can be configured to execute one or more instructions in response to the message for the tunneling communication.

[0005] According to another general aspect, a system can include a host computing device. The host computing device can include a processor configured to read and write data from and to a storage device via data messages, and offload commands to the storage device. The host computing device can include a memory configured to be at least partially allocated for use by the storage device. The system can include the storage device. The storage device can include a host interface circuit configured to communicate with the host computing device via a data protocol that employs data messages. The storage device can include a storage element configured to store data in response to the data messages. The host interface circuit can be configured to detect whether a tunneling communication command is embedded within a data message, extract message address information for the tunneling communication from the data message, retrieve a message for the tunneling communication stored in the memory of the host device via the message address information for the tunneling communication, and route the message for the tunneling communication to an on-board processor and / or data processing logic. The on-board processor and / or data processing logic can be configured to execute one or more instructions in response to the message for the tunneling communication, or perform specific data processing and other operations.

[0006] According to another general aspect, a method of tunneling a remote procedure call over a data protocol can include allocating, for communication between a host computing device and an enhanced storage device, at least a portion of a host memory buffer included by the host computing device. The method can include creating a data message, the data message including an indication that a message for tunneling is stored within the portion of the host memory buffer. The method can include sending the data message to the enhanced storage device. The method can include, upon receiving the data message, the enhanced storage device reading the message for tunneling from the host computing device. The method can include, in response to the message for tunneling, executing one or more instructions by the enhanced storage device.

[0007] The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims.

[0008] A system and / or method for computer communication, and more specifically, a method and system for message tunneling using a computer protocol such as Non-Volatile Memory Express (NVMe) is set forth in the claims, which are incorporated into this description, and are more fully described in the description and drawings. BRIEF DESCRIPTION OF DRAWINGS

[0009] Figure 1 is a block diagram of an example embodiment of a system that utilizes tunneling data messages in accordance with the disclosed subject matter.

[0010] Figure 2is a block diagram of an example embodiment of a storage device utilizing tunneling of data messages according to the disclosed subject matter.

[0011] Figure 3 is a diagram of an example embodiment of a data structure usable with tunneling of data messages sent from a host computing device according to the disclosed subject matter.

[0012] Figure 4 is a diagram of more example embodiments of a data structure usable with tunneling of data messages sent from an enhanced storage device according to the disclosed subject matter.

[0013] Figure 5 is a block diagram of an information handling system that can include devices formed according to principles of the disclosed subject matter for computer communication and more specifically for methods and systems for message tunneling using NVMe.

[0014] Figure 6 is a diagram of an example embodiment of a technique for processing tunneling of data messages according to the disclosed subject matter.

[0015] In the various drawings, like reference numerals refer to like elements. DETAILED DESCRIPTION

[0016] Various example embodiments will be described more fully hereinafter with reference to the accompanying drawings, in which some, but not all example embodiments are

[0017] It will be understood that, although the terms first, second, third, etc. can be used herein to describe various elements, components, regions, layers and / or sections, these elements, components, regions, layers and / or sections should not be limited by these terms. These terms are only used to distinguish one element, component, region, layer and / or section from another element, component, region, layer and / or section. Thus, a first element, component, region, layer or section discussed below could be termed a second element, component, region, layer or section without departing from the teachings of the present disclosure.

[0018] The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting of the subject matter of the present disclosure. As used herein, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0019] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this subject matter belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0020] In the following, example embodiments will be explained in detail with reference to the accompanying drawings.

[0021] Peripheral Component Interconnect (PCI) Express (PCIe or PCI-e) can refer to a high-speed serial computer expansion bus standard. PCIe has features such as a relatively high maximum system bus throughput, a relatively low number of input / output (I / O) pins, and a small physical footprint, performance scaling for bus devices, error detection and reporting mechanisms (e.g., advanced error reporting), and local hot-plug functionality. The PCIe standard can provide hardware support for I / O virtualization. PCIe is based on a point-to-point topology with a separate serial link connecting each device to a root complex (e.g., a host). In terms of bus protocol, PCIe communication is typically encapsulated in packets.

[0022] Non-Volatile Memory (NVM) Express (NVMe) or NVM Host Controller Interface Specification (NVMHCI) can refer to a logical device interface specification for accessing non-volatile storage media attached via a PCIe bus. NVM can include flash memory that can be used in solid state drives. NVMe, as a logical device interface, has a function of relatively low latency and internal parallelism of flash-based storage devices by reflecting parallelism of central processors, platforms, and applications.

[0023] NVMe can be used as a replacement for Small Computer System Interface (SCSI) (or more specifically, Serial Attached SCSI (a standard for connecting and transferring data between a host and a peripheral target storage device or system)). NVMe can be used with media such as SSDs and post-flash-based technologies.

[0024] In distributed computing, a remote procedure call (RPC) can represent a computer program that causes a procedure (subroutine) to execute in a different address space (usually on another computer on a shared network) which is coded as if it was a local routine call. This can represent a form of client-server interaction (the caller is the client and the performer is the server), usually implemented via a request-response messaging system. In object-oriented programming paradigms, an RPC call can be represented by a remote method invocation (RMI).

[0025] Figure 1 is a block diagram of an example embodiment of a system 100 in accordance with the disclosed subject matter. In various embodiments, the system 100 can include a storage device or storage medium (e.g., a hard disk drive, an SSD, a flash drive, etc.). In various embodiments, the system 100 can be used by or included in a computing device such as a laptop computer, a desktop computer, a workstation, a data center, a cloud, a personal digital assistant, a smart phone, a tablet computer, and other appropriate computers or virtual machines or virtual computing devices thereof.

[0026] In some information technology (IT) infrastructures, a relatively large amount of data can be generated by various devices and processes. Some examples of these data generators include, but are not limited to, smart machines, autonomous vehicles, social networks, and Internet of Things (IoT) devices. In various embodiments, artificial intelligence and / or machine learning techniques can be employed to efficiently analyze the collected data and utilize them.

[0027] In various embodiments, systems are disclosed herein that efficiently process and analyze data. In some system architectures, data can be extracted from persistent storage devices to relatively high performance servers that can be connected to the storage devices using relatively high performance networks. Moving large amounts of raw data to processors for analysis can consume large amounts of energy and deployed computer and network resources. Such data movement can put a burden on resources such as network bandwidth, CPU cycles, and CPU memory. Moving relatively large amounts of raw data to servers for processing can also increase the latency associated with the application. That is, the increased latency can cause the processing of the application to be delayed while the data is extracted to the server before the processing and analysis of the application can be completed.

[0028] In the illustrated embodiment, the system 100 can attempt to reduce the movement of data by processing the raw data within the storage device 108. In various embodiments, this can be a more efficient solution that is suitable for data analysis use cases that compute increasing amounts of raw data. Additionally, by eliminating data movement, the response latency of the application can be improved. Furthermore, data analysis tasks that typically read large amounts of data, process it, and reduce the data through filtering and other reduction operations can be improved.

[0029] In various embodiments, system 100 can include a host computing device 102. As described above, in various embodiments, system 100 can include a computing device such as a laptop computer, a desktop, a workstation, a data center, a cloud, a personal digital assistant, a smart phone, a tablet computer, and other appropriate computer or virtual machine or virtual computing device thereof. In various embodiments, host computing device 102 can be a portion of system 100 that includes a processor 110 (e.g., a motherboard, a system on a chip, etc.).

[0030] In various embodiments, host computing device 102 can include a processor 110. Processor 110 can be configured to execute instructions or commands, and in doing so, read data from and / or write data to memory 116 and / or storage 108. In various embodiments, processor 110 can execute software such as programs or applications 112 and drivers 114. Further, processor 110 can offload commands to storage 108.

[0031] In some embodiments, host computing device 102 can include memory 116. In such embodiments, memory 116 can include volatile memory (e.g., random access memory (RAM)). In another embodiment, memory 116 can include NVM (e.g., NAND memory or flash memory). In various embodiments, memory 116 can be configured to store data in a temporary or semi-permanent form. In some embodiments, portions of memory 116 can be partitioned or include circuitry that acts as a host memory buffer 126 and / or a queue 128 (e.g., a submission queue). Host memory buffer 126 can be reserved for storage.

[0032] In the illustrated embodiment, system 100 can include a communication path 106. In various embodiments, communication path 106 can include a bus or set of wires that connects host computing device 102 and storage 108. In various embodiments, communication path 106 can include a PCIe high-speed serial computer expansion bus. It should be understood that the above is merely one illustrative example, and the disclosed subject matter is not limited thereto. Some other examples of communication paths between a host and storage can include, but are not limited to, Ethernet, remote direct memory access (RDMA), Fibre Channel, SCSI, Serial Advanced Technology Attachment (SATA), Transmission Control Protocol / Internet Protocol (TCP / IP), Infiniband, etc.

[0033] In various embodiments, the communication path 106 can transport data, and the host computing device 102 and the storage device 108 can communicate data using a data protocol that employs data messages. In various embodiments, the data protocol can include the Non-Volatile Memory Host Controller Interface Specification (NVMHCIS) or NVMe protocol or interface. It should be understood that the above is merely one illustrative example, and the disclosed subject matter is not limited in this regard. The NVMe protocol can use different transport protocols as the communication path, some examples of which are mentioned above. The example embodiments disclosed herein can be independent of the transport protocol used to carry the NVMe protocol from the host device to the storage device.

[0034] In the illustrated embodiment, the system 100 can include a storage device 108. In another embodiment, the storage device 108 can include NVM or storage elements or memory 146. In various embodiments, the storage elements or memory 146 can be configured to store data in a permanent, semi-permanent, or substantially permanent form.

[0035] In various embodiments, the storage device 108 can include a host interface circuit 142. The host interface circuit 142 can be configured to communicate with the host computing device 102 via a data protocol that employs data messages.

[0036] In the illustrated embodiment, the storage device 108 can include an on-board processor 144 and / or data processing discrete logic. In such an embodiment, because the storage device 108 can not only store data but also perform operations or process data (at least partially), the storage device 108 can be referred to as an enhanced storage device or a smart storage device.

[0037] As mentioned above, an RPC is a form of procedure, subroutine, or method call in which a first device requests a second device to perform a particular procedure remotely on the second device. The procedure, when run remotely, can access data local to the second device, and by doing so, work more efficiently with the latency of the first device. An RPC can include a mechanism to typically package input parameters to the called procedure on the first device side.

[0038] In the illustrated embodiment, the remote procedure call can occur locally (e.g., within the same system 100 or chassis). Additionally, in the illustrated embodiment, the calling and serving devices can be separated by a physical address space and a virtual address space.

[0039] In various embodiments, the technology described herein can be implemented remotely over a network. For example, NVMe over local and NVMe over Fibre Channel (NVMe-oF) are two such embodiments that provide a block device interface to a host operating system (OS) or device without differentiating its transport backend. Some example NVMe transport protocols can include, but are not limited to, Ethernet, RDMA, Fibre Channel, TCP / IP, InfiniBand, or a proprietary transport method that carries the NVMe protocol.

[0040] In some examples, the RPC protocol can be implemented using a network protocol stack, such as the Transmission Control Protocol (TCP) and Internet Protocol (IP), to communicate. In the illustrated embodiment, the RPC messages can be communicated over a local communication path 106, such as over PCIe. As noted above, PCIe can not necessarily be a network protocol, but can be a bus interface within a host server or chassis for connecting end devices and peripheral devices to a host. In various embodiments, an RPC that tunnels over PCIe can be employed to connect a software stack running on a host computing device 102 to a service managed on a storage device 108 directly attached to the host computing device 102. This mechanism can allow certain application functionality to be offloaded and accelerated onto the storage device 108 (e.g., an NVMe SSD) without requiring configuration changes to the host stack.

[0041] The disclosed system for message tunneling using NVMe can allow any other application level protocol to be implemented between the host and the SSD in addition to RPC. RPC is merely one example, however, and the tunneling mechanism described herein can be independent of the particular implementation used between the host and the SSD. There can be various standard and / or custom and / or proprietary protocols implemented in the system 100 that can use the bidirectional, full-duplex, and high performance message tunneling technology described herein.

[0042] In the illustrated embodiment, the system 100 can be configured to perform a remote procedure call between the host computing device 102 and the storage device 108. In such embodiments, the host computing device 102 can be configured to request the storage device 108 to perform an operation on data 182 stored by the storage device 108. In such embodiments, the transfer of the data 182 between the storage device 108 and the host computing device 102 can be avoided or at least reduced. As noted above, such reduction can save power and bandwidth, while reducing latency or other computer resource usage.

[0043] In the illustrated embodiment, the host computing device 102 and the storage device 108 can encode, embed, or tunnel their communications or messages within a data protocol, and more specifically, encode, embed, or tunnel data messages of the protocol. In such embodiments, the RPCs can follow the protocol and conventions of the data messages. For example, the RPCs can appear transparently or ostensibly (but not actually) as NVMe data messages, but are embedded within the data messages, appearing as data messages rather than RPCs. In various embodiments, the format of the data messages can be described in conjunction with the following illustrated Figure 3 and Figure 4 It should be understood that the above are merely some illustrative examples, and the disclosed subject matter is not limited thereto.

[0044] In such embodiments, the communication path 106 can include both the transmission of data messages 194 (which are actually tunnelled RPCs) and actual data messages 192 or data.

[0045] In such embodiments, the message (e.g., PCIe) tunnel can provide a way to transmit messages between the host computing device 102 and the storage device 108. In various embodiments, the message tunnel can be agnostic to the actual message content, as it cannot interpret the data messages (whether actually or just superficially). In such embodiments, the tunnelled messages can merely appear as another block of data to the message tunnel being transmitted to the other side of the communication path 106. In various embodiments, a variety of forms or techniques of performing tunnelled communications are contemplated.

[0046] Figure 2 is a block diagram of an example embodiment of a system 200 in accordance with the disclosed subject matter. In various embodiments, the system 200 can include a storage device, which in some embodiments can be referred to as a smart storage device or an enhanced storage device. In various embodiments, the system 200 can communicate with a host computing device via a tunnelled communication protocol, and perform operations on data at the request of the host computing device.

[0047] In various embodiments, the system 200 can include a transport layer interface circuit 210. In various embodiments, the transport layer interface circuit 210 can be configured to communicate with a host computing device via a communication path or transport layer (e.g., Ethernet, RDMA, TCP / IP).

[0048] In the illustrated embodiment, system 200 can include one or more command submission queues (SQs) 211. In various embodiments, each SQ 221 can be configured to store commands as they await execution by system 200. System 200 depicts an NVMe-oF based system in which the command SQs are located on the SSD device side. This system architecture can also be referred to as remote direct attached storage (RDAS). In a direct attached storage (DAS) system architecture, the SQs are held in the host system memory rather than on the SSD device side. In any case, embodiments of the present disclosure are independent of the location of the SQs and the underlying system architecture.

[0049] In the illustrated embodiment, system 200 can include host interface circuitry 212. In such embodiments, host interface (IF) circuitry 212 can be configured to communicate with a host computing device via a data protocol (e.g., NVMe) that employs data messages.

[0050] In such embodiments, host interface circuitry 212 can be configured to receive storage access commands (e.g., reads or writes) from a host device (not shown) and respond to the host device when those commands have been processed (successfully processed or unsuccessfully processed). Generally, storage access commands can include read / write request commands or messages and, once accepted, can be implemented by system 200 and / or responded to by system 200, e.g., by system 200 sending a read / write response command or message to the host.

[0051] In some embodiments, host interface circuitry 212 can be configured to detect whether a tunneling communication or command message or tunneling communication is embedded within a data message. In such embodiments, as described below, host interface circuitry 212 can be configured to extract information from a tunneling communication data message and route the tunneling communication data message to onboard processor and / or data processing logic 216.

[0052] In the illustrated embodiment, system 200 can include message interface circuitry 214. In various embodiments, message interface circuitry 214 can be configured to process tunneling communication data messages and provide desired information associated with the tunneling communication data messages to onboard processor and / or data processing logic 216.

[0053] In the illustrated embodiment, system 200 can include onboard processor 216. Instead of or in addition to an onboard processor, there can be discrete data processing logic such as logic gates and state machines. In various embodiments, onboard processor and / or data processing logic 216 can be configured to perform one or more operations in response to a command or tunneling communication data message of a tunneling communication.

[0054] In the illustrated embodiment, system 200 can include translation layer circuitry 218. In various embodiments, translation layer circuitry 218 can include a flash translation layer (FTL). However, it should be understood that the above is merely one illustrative example, and the disclosed subject matter is not limited in this regard.

[0055] In various embodiments, translation layer circuitry 218 can be configured to perform low-level management of persistent memory or storage elements 220. In various embodiments, low-level management can include writing (or, in the parlance of some technologies, “programming”) data to storage elements 220, performing wear leveling to ensure that writes occur sufficiently evenly throughout storage elements 220, generating metadata and / or error correction codes, and performing garbage collection to reclaim invalid storage locations within storage elements 220. In various embodiments, translation layer circuitry 218 can perform logical address block translation. It should be understood that the above are merely some illustrative examples, and the disclosed subject matter is not limited in this regard.

[0056] In the illustrated embodiment, system 200 can include storage elements 220. In such embodiments, storage elements 220 can be configured to store data. In various embodiments, storage elements 220 can include one or more storage technologies (such as NAND storage, ferroelectric storage, SSDs, etc.). In the illustrated embodiment, for ease of illustration, storage will be described as solid state storage. It should be understood that the above are merely some illustrative examples, and the disclosed subject matter is not limited in this regard.

[0057] In various embodiments, storage elements 220 can include portions having allocated capacity 222 or actual storage of data. Storage elements 220 can include portions having unallocated capacity 224 or free storage of data. In various embodiments, storage elements 220 can also include space or portions 226 that store applications (apps) or microservices (collectively, processes) that can be executed by on-board processor and / or data processing logic 216. In various embodiments, these processes, methods, and / or applications can be invoked remotely by a host computing device. In response, on-board processor and / or data processing logic 216 can perform operations using data stored in storage elements 220 and / or supplied by the host computing device, and then return output data to storage elements 220 or the host computing device. It should be understood that the above are merely some illustrative examples, and the disclosed subject matter is not limited in this regard.

[0058] In one embodiment, as part of the process of the host computing device sending a message (e.g., an RPC) to the storage device or system 200, the message can first be created in the host memory. In various embodiments, the host computing device can then pass a pointer to the message and the size of the message to the NVMe driver. The NVMe driver can then create a message for the tunneling communication (e.g., a "tnl_msg_send") command and place it in the command SQ 211. The host computing device can then notify the storage device or system 200 that a message is waiting for it by issuing a command (e.g., a "tnl_msg_send" NVMe command).

[0059] In various embodiments, the storage device or system 200 can select or read this particular SQ 211 and, depending on the protocol, can execute this particular tunneling communication command. In various embodiments, the system 200 can first fetch the message for the tunneling communication or the command for the tunneling communication from the host memory. As described above, the actual message for the tunneling communication can appear in or exist in the protocol of the data message as expected by the communication medium (e.g., NVMe).

[0060] In the illustrated embodiment, the host interface layer (HIL) or host interface circuit 212 can interpret the fetched (surface) data message. In such embodiments, the HIL or host interface circuit 212 can recognize that the message is a tunneling command (e.g., tnl_msg_send).

[0061] In various embodiments, the HIL 212 can extract and use the message address and message size values from the command to fetch the message data from the host memory. In some embodiments, the HIL 212 can also extract the message sequence number field from the command to pass the fetched message in an ordered fashion to the message interface circuit 214.

[0062] In various embodiments, once the message has been successfully fetched, the HIL 212 can place a command completion entry (CE) in the appropriate command completion queue (CQ) in the host memory. In various embodiments, the HIL 212 can place the message channel identifier (ID) and the message sequence number of the fetched message into the CE. In some embodiments, after the command CE is placed into the host, the HIL 212 can send a signal (such as an interrupt) to the host to indicate that the CE is in the CQ.

[0063] Figure 3 are diagrams of example embodiments of data structures 300 and 301 in accordance with the disclosed subject matter. In various embodiments, these data structures 300 and 301 can be used to tunnel messages from a host computing device to a storage device.

[0064] In the illustrated embodiment, data structure 300 can illustrate an example layout of a vendor-defined NVMe command, referred to as, for example, a tnl_msg_send command or tunneling messaging send command. As noted above, data structure 300 can follow the format of the NVMe protocol, but can include commands to the storage device.

[0065] In the illustrated embodiment, data structure 300 can include a command ID (CID) 310 that uniquely identifies the NVMe command during execution by the SSD device. In the illustrated embodiment, data structure 300 can include a physical region page (PRP) or scatter gather list (SGL) for data transfer (PRP or SGL for data transfer (PSDT)) field 312 that indicates whether a PRP or SGL is used for the transfer of data associated with the command. In the illustrated embodiment, data structure 300 can include an operation code (OPC) field 314 that indicates the type of command (e.g., tnl_msg_send).

[0066] In the illustrated embodiment, data structure 300 can include a PRP or PRP1 field 316 that includes a pointer to a physical memory page in which data is stored for use by storage device 200. In the illustrated embodiment, data structure 300 can include a message length (MSG LENGTH) field 320 that indicates the length of the message. Data structure 300 can also include a message channel ID (Msg Chan ID) field 322 and a message sequence number (Msg Seq Num) field 324.

[0067] As noted above, HIL 212 can retrieve the message from SQ 211. Via the OPC 314, HIL 212 can determine that this is a message tunneling command. HIL 212 can then use the message address (e.g., PRP1 316, PSDT 312) and message size values (e.g., fields 322 and 320) to fetch the message from host memory. HIL 212 can retrieve the message for tunneling from a host memory buffer reserved for the device. Message interface circuit 214 can use the message sequence number field 324 to pass the message to the firmware or hardware of the storage device that processes application layer messages in the proper order.

[0068] In the illustrated embodiment, data structure 301 can include an example layout of a CE associated with a tnl_msg_send command. In such embodiments, data structure 301 can include a message channel ID field 322 and a message sequence number field 324 to match the CE with the command (e.g., data structure 300). In various embodiments, data structure 301 can include a submission queue ID (SQID) field 332 indicating the SQ from which the associated command was issued. A SQ header pointer (SQHD) field 334 indicates the current pointer for the associated SQ. In various embodiments, data structure 301 can include a status and stage tag (STS) field 336 and a CID field 310.

[0069] As noted above, in various embodiments, once a message has been transmitted, HIL 212 creates a data structure 301 or CE. HIL 212 places the CE in the appropriate command CQ of the host memory. HIL 212 can then send an interrupt to the host to indicate that the CE has been placed.

[0070] As Figure 2 As illustrated in FIG. 3, a host (e.g., NVMe) driver can be configured to provide a mechanism for storage device 200 to send messages to the host computing device. In one embodiment, the host driver can pre-issue a number of receive commands (e.g., NVMe tnl_msg_recv commands) to storage device 200. In such embodiments, this can keep a certain number of receive commands outstanding. In such embodiments, when storage device 200 sends a message to the host computing device, one of those outstanding NVMe receive commands can be completed. It should be understood that the above is merely one illustrative example, and the disclosed subject matter is not limited in this regard.

[0071] In various embodiments, the driver of the host computing device can first allocate a buffer in the host memory. In such embodiments, the driver can then create a tunnel communication message receive command that includes a pointer to the allocated buffer and the size of the buffer. In some embodiments, the driver can allocate only one size of buffer or different sizes of buffers. In various embodiments, the driver can also add a message channel ID to the command. As noted above, in various embodiments, the host can then place the created command into one of the SQs (e.g., NVMe) and alert storage device 200 that this process has been completed. In some embodiments (e.g., in an NVMe-oF system), the host computing device can not provide this alert; rather, the host computing device can send the command directly to storage device 200.

[0072] In such embodiments, at some point in time, the HIL circuit 212 of the storage device 200 can select a command for execution. As part of command execution, the storage device 200 (via the transport layer interface 210) can acquire a tunneling communication command (e.g., tnl_msg_recv).

[0073] In various embodiments, the acquired command can indicate that the command can remain in a pending or open state until the storage device 200 sends a message to the host. The number of outstanding commands that are issued can depend on the available host buffer memory and / or the maximum number of messages that the host computing device is configured to receive in a given time. Such number of outstanding commands can also be used by the host computing device as a flow control mechanism to limit the number of messages sent by the storage device 200.

[0074] In various embodiments, when the storage device 200 sends a message to the host computing device, one of the outstanding receive commands can be utilized. In such embodiments, the storage device 200 can employ the host buffer memory pointer to transfer the message to the host memory. In various embodiments, the message interface circuit 214 can be configured to determine that the message length is within the message buffer size indicated in the command.

[0075] In such embodiments, after the message is transferred, the HIL circuit 212 can create a CE for the command. The HIL circuit 212 can place the message channel ID field 322, the message sequence number field 324, and the message length field 320 in the CE. The HIL circuit 212 can then place the CE into the appropriate command CQ in the host memory. In various embodiments, as described above, the storage device 200 can send an interrupt (such as a message signaled interrupt (MSI) or similar signal) to the host computing device.

[0076] In various embodiments, when the driver of the host computing device recognizes that a tunneling communication command CE has been received, the driver of the host computing device can extract the message channel ID 322, the message sequence number 324, and the message length 320 from the CE. In some embodiments, the host message tunneling communication software can then use this information to transfer the received message to upper layer software in the host system software stack in an ordered fashion.

[0077] In various embodiments, the host driver can periodically issue additional tunneling communication message receive (e.g., tnl_msg_recv) commands to increase the number of available open message buffers for the storage device 200 to send messages to the host computing device. It should be understood that the above is merely one illustrative example, and the disclosed subject matter is not limited in this regard.

[0078] Figure 4are diagrams of example embodiments of data structures 400 and 401 according to the disclosed subject matter. In various embodiments, these data structures 400 and 401 can be used to tunnel a message from a storage device 200 to a host computing device.

[0079] In the illustrated embodiment, the data structure 400 can illustrate an example layout of a vendor-defined NVMe command referred to as, for example, tnl_msg_recv or tunneling message receive. As described above, the data structure 400 can follow the format of the data message protocol, but can include a command to the storage device.

[0080] In the illustrated embodiment, the data structure 400 can include a CID 410 that uniquely identifies the NVMe command during execution by the SSD device. In the illustrated embodiment, the data structure 400 can include a PSDT field 412 that indicates whether a PRP or SGL is used for transfer of data associated with the command. In the illustrated embodiment, the data structure 400 can include an OPC field 414 that indicates the type of command (e.g., tnl_msg_recv).

[0081] In the illustrated embodiment, the data structure 400 can include a PRP or PRP1 field 416 that includes a pointer to a physical memory page in which data is stored for use by the storage device 200. The data structure 400 can also include a message channel ID field 422. In the illustrated embodiment, the data structure 400 can include a message buffer size (MSG BUFFER SIZE) field 426 that indicates the size of the allocation of buffers associated with the message.

[0082] As described above, a host driver can create the data structure 400 or tunneling receive message (tnl_msg_recv) command. The host driver can then place the created command in an SQ. The storage device 200 can fetch the tunneling receive message command from the SQ. When the storage device 200 is in communication with the host device, the storage device 200 can use the host buffer memory pointer in the data structure 400 to write a message to the host computing device. Once the message is transferred, the HIL 212 of the storage device 200 can create a completion entry (CE) or data structure 401 for the command. The HIL 212 can, in response to a request to send a tunneling message to the host device, extract a pointer to a portion of a host memory buffer that has been allocated on the host device from an open data message, asynchronously transfer the tunneling message to the portion of the host memory buffer, and create a completion entry associated with the data message, wherein receipt of the completion entry by the host device causes the host device to read the tunneling message.

[0083] In the illustrated embodiment, data structure 401 can include an example layout of a CE associated with a tunnel communication data message. In such embodiments, data structure 401 can include a message channel ID field 422 and a message sequence number field 424 to match the CE with a command (e.g., data structure 400). In various embodiments, data structure 401 can include a SQ ID field 432 indicating the SQ from which the associated command was issued and a SQ HD field 434 indicating the current pointer of the associated SQ. In various embodiments, data structure 401 can include a STS field 436 and a CID field 410. The CE can include a length field 420 (e.g., MSG LENGTH field) that can indicate the length of the message being transferred from the SSD device to the host device. The CE includes a message channel ID field 422 (e.g., MSG CHAN ID) and a message sequence number field 424 (e.g., MSG SEQ NUM field) indicating the tunnel communication channel ID and message sequence number so that the host driver can provide the messages received from the storage device 200 to the upper layer software in an ordered fashion.

[0084] As described above, in various embodiments, once a message has been transferred, HIL 212 can create a data structure 401 or CE. HIL (212) places the CE in the appropriate command CQ of the host memory. HIL (212) can then send an interrupt to the host to indicate that the CE has been placed.

[0085] In Figure 1 additional functionality of the disclosed subject matter is described. For example, the following techniques can be used for flow control between host computing device 102 and storage device 108. In various embodiments, a message tunnel can operate in a full duplex, high performance manner. In such embodiments, the message tunnel can employ at least some functionality of a data protocol (e.g., NVMe) to perform flow control. In various embodiments, flow control can include a back pressure functionality. Back pressure is a resistance or force opposite to the desired flow of data. In such embodiments, the flow of data can be reduced.

[0086] In one embodiment, for messages traveling from host computing device 102 to storage device 108, host computing device 102 can submit a tunnel communication message send command to one or more command SQs. In such embodiments, after the entry in the SQ is signaled, storage device 108 can fetch the command and execute the command.

[0087] In various embodiments, the storage device 108 can apply flow control to the host computing device 102 when the resources of the storage device 108 used to receive messages are running low, or for any other reason. In particular, the storage device 108 can reduce the rate at which tunnel communication message send (tnl_msg_send) commands are executed. In some embodiments, the storage device 108 can delay execution of tnl_msg_send commands, suspend execution of tnl_msg_send commands, or reduce the rate of execution of tnl_msg_send commands. In such embodiments, the storage device 108 can limit the number of messages that the host computing device 102 is sending to the storage device 108.

[0088] In contrast, for messages that travel from the storage device 108 to the host computing device 102, the host computing device 102 can apply flow control as needed. In one embodiment, to receive messages from the storage device 108, the host computing device 102 can submit tunnel communication message receive (tnl_msg_recv) commands to one or more command SQs. The host computing device 102 can then indicate to the storage device 108 that messages are in the SQs. The storage device 108 can then fetch those commands or can leave the commands outstanding until the storage device 108 sends a message to the host computing device 102. To apply back pressure, the host computing device 102 can reduce the number of outstanding or queued tunnel communication message receive commands. In some embodiments, the host computing device 102 can also abort some or all of the currently outstanding commands (e.g., tnl_msg_recv commands). By reducing the number of such commands, the host computing device 102 can throttle the storage device 108, thereby limiting its ability to send messages to the host.

[0089] In various embodiments, the system 100 can be configured to provide quality of service (QoS) functionality to the tunnel communication message protocol. In such embodiments, the QoS functionality can help the execution of operations. The QoS functionality can be applied to individual message channels and / or directions of flow.

[0090] In one embodiment, the disclosed implementation of a system for message tunneling as described herein can support multiple simultaneously active tunnels (also referred to as channels). In such embodiments, the tunnels can be identified by a substantially unique message channel ID. In some embodiments, the system 100 can provide differentiated QoS to different message channels. In such embodiments, some channels can be designated as higher priority than others.

[0091] In one embodiment, such differentiation can be achieved by using dedicated command SQs and CQs for different message channels. For example, in one embodiment, the NVMe protocol provides a method for SQ arbitration to provide different levels of QoS. Additionally, in various embodiments, the storage device 108 can implement a message layer that can assign appropriate resources and apply differentiated execution priority among message channels. In one embodiment, QoS configuration for message tunnels can be performed by the host computing device 102 using Set Feature / Get Feature commands of the NVMe protocol. It should be understood that the above is merely one illustrative example, and the disclosed subject matter is not limited thereto. In another embodiment, the SSD device can have a default QoS configuration. In yet another embodiment, QoS settings for tunnel communication channels can be provided to the SSD device by a baseboard management controller (BMC) device. Specifically, the BMC can provide QoS settings based on storage administrator guidance, direction, and policy settings of such BMC device. In one embodiment, the BMC can be located on the host device (such as the host computing device 102 in Figure 1

[0092] In various embodiments, the system 100 can be substantially compliant with the underlying data protocol. In such embodiments, the storage device 108 can provide a data protocol that is substantially compliant with a standard and based on a storage interface (e.g., NVMe or PCIe). In such embodiments, the host (NVMe) driver and storage stack can still perform input / output (IO or I / O) operations to the storage device 108. The message tunnel communication functionality can coexist with normal storage functionality, and can not interfere with normal storage functionality. The message tunnel communication functionality can employ dedicated command SQs and CQs. In various embodiments, the host interface circuit (HIL) 142 of the storage device 108 can apply programmable priority to tunnel communication commands relative to normal storage commands. In some embodiments, the message tunnel communication functionality can be enabled or disabled, or can be configured to the SSD device by the host software or by the BMC device (e.g., by adjusting storage administrator guidance, direction, and policy settings).

[0093] Figure 5 is an illustrative block diagram of an information handling system 500 that can include a semiconductor device formed in accordance with the principles of the disclosed subject matter.

[0094] As shown in Figure 5 , the information handling system 500 can include one or more devices constructed in accordance with the principles of the disclosed subject matter. In another embodiment, the information handling system 500 can employ or perform one or more techniques in accordance with the principles of the disclosed subject matter.

[0095] ​In various embodiments, information handling system 500 can include a computing device such as a laptop, a desktop, a workstation, a server, a blade server, a personal digital assistant, a smart phone, a tablet, and other appropriate computer or virtual machine or virtual computing device thereof. In various embodiments, information handling system 500 can be used by a user (not shown).

[0096] Information handling system 500 according to the disclosed subject matter can also include a CPU, logic, or processor 510. In some embodiments, processor 510 can include one or more functional unit blocks (FUBs) or combinational logic blocks (CLBs) 515. In such embodiments, CLBs can include various Boolean logic operations (e.g., NAND, NOR, NOT, XOR), stable logic devices (e.g., flip-flops, latches), other logic devices, or combinations thereof. These combinational logic operations can be configured in simple or complex ways to process input signals to achieve a desired result. It should be understood that while some illustrative examples of synchronous combinational logic operations are described, the disclosed subject matter is not so limited and can include asynchronous operations or mixtures thereof. In one embodiment, combinational logic operations can include a plurality of complementary metal-oxide-semiconductor (CMOS) transistors. In various embodiments, these CMOS transistors can be arranged into gates that perform logic operations, although it should be understood that other technologies can be used and are within the scope of the disclosed subject matter.

[0097] Information handling system 500 according to the disclosed subject matter can also include volatile memory 520 (e.g., RAM). Information handling system 500 according to the disclosed subject matter can also include non-volatile memory (NVM) 530 (e.g., a hard drive, optical storage, NAND, or flash memory). In some embodiments, volatile memory 520, NVM 530, or combinations or portions thereof can be referred to as a "storage medium." In various embodiments, volatile memory 520 and / or NVM 530 can be configured to store data in semi-permanent or substantially permanent form.

[0098] In various embodiments, the information handling system 500 can include one or more network interfaces 540 configured to allow the information handling system 500 to operate as part of a communications network and to communicate over the communications network. Examples of Wi-Fi protocols can include, but are not limited to, Institute of Electrical and Electronics Engineers (IEEE) 802.1 lg and 802.1 In. Examples of cellular protocols can include, but are not limited to, IEEE 802.16m (also known as Wireless MAN-Advanced), Long Term Evolution (LTE)-Advanced, Enhanced Data rates for GSM Evolution (EDGE), Evolved High Speed Packet Access (HSPA+), and International Mobile Telecommunications 2020 standard (also known as Cellular 5G). Examples of wired protocols can include, but are not limited to, IEEE 802.3 (also known as Ethernet), Fibre Channel, Powerline Communication (e.g., HomePlug, IEEE 1901). It should be understood that the above are merely some illustrative examples, and the disclosed subject matter is not limited in this regard.

[0099] The information handling system 500 according to the disclosed subject matter can also include a user interface unit 550 (e.g., a display adapter, a haptic interface, a human machine interface device). In various embodiments, the user interface unit 550 can be configured to receive input from a user and / or to provide output to the user. Other kinds of devices can be used to provide for interaction with a user as well, such as feedback provided to the user in any form of sensory feedback (such as visual feedback, auditory feedback, or tactile feedback), and input from the user in any form, including acoustic, speech, or tactile input.

[0100] In various embodiments, the information handling system 500 can include one or more other devices or hardware components 560 (e.g., a display or monitor, a keyboard, a mouse, a camera, a fingerprint reader, or a video processor). It should be understood that the above are merely some illustrative examples, and the disclosed subject matter is not limited in this regard.

[0101] The information handling system 500 according to the disclosed subject matter can also include one or more system buses 505. In such embodiments, the system bus 505 can be configured to communicatively connect the processor 510, the volatile memory 520, the NVM 530, the network interface 540, the user interface unit 550, and the one or more hardware components 560. Data processed by the processor 510 or data input externally from the NVM 530 can be stored in the NVM 530 or the volatile memory 520.

[0102] In various embodiments, information handling system 500 can include or execute one or more software components 570. In some embodiments, software components 570 can include an OS and / or an application (app). In some embodiments, the OS can be configured to provide one or more services to apps and manage intermediation between apps and various hardware components of information handling system 500 (e.g., processor(s) 510, network interface(s) 540), or act as an intermediary between apps and various hardware components of information handling system 500. In such embodiments, information handling system 500 can include one or more native apps, which can be installed locally (e.g., installed within NVM 530) and configured to be executed directly by processor(s) 510 and to interact directly with the OS. In such embodiments, native apps can include pre-compiled machine executable code. In some embodiments, native apps can include a script interpreter (e.g., C shell (csh), AppleScript, AutoHotkey) or a virtual execution machine (VM) (e.g., Java Virtual Machine, Microsoft Common Language Runtime) configured to translate source code or object code into executable code that is subsequently executed by processor(s) 510.

[0103] Figure 6 is a flow diagram of example embodiments of techniques in accordance with the disclosed subject matter. In various embodiments, techniques 600 can be used or produced by a system such as Figure 1 , Figure 2 or Figure 5 . Moreover, portions of techniques 600 can use data structures such as Figure 3 or Figure 4 . It should be understood, however, that the above are merely some illustrative examples, to which the disclosed subject matter is not limited. It should be understood that the disclosed subject matter is not limited to the order or number of acts illustrated by techniques 600.

[0104] Block 602 illustrates that in one embodiment, as described above, at least a portion of a host memory buffer included by the host computing device can be allocated for communication between the host computing device and the enhanced storage device. Block 604 illustrates that in one embodiment, as described above, a data message including an indication that a message of the tunnel communication is stored within the portion of the host memory buffer can be created.

[0105] Block 606 illustrates that in one embodiment, as described above, the data message can be sent to the enhanced storage device. Block 608 illustrates that in one embodiment, as described above, the enhanced storage device can read the message of the tunnel communication from the host computing device. Block 610 illustrates that in one embodiment, as described above, in response to the message of the tunnel communication, the enhanced storage device can execute one or more instructions.

[0106] Method steps can be performed by one or more programmable processors executing a computer program to perform functions by operating on input data and generating output. Method steps also can be performed by, and device can be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). In various embodiments, this circuitry can be included in a storage device and / or host computing device.

[0107] In various embodiments, a computer-readable medium can include instructions that, when executed, cause a device to perform at least a portion of the method steps. In some embodiments, a computer-readable medium can be included in a magnetic medium, optical medium, other medium, or combination thereof (e.g., a compact disk read-only memory (CD-ROM), a hard disk drive, a read-only memory, a flash drive). In such embodiments, a computer-readable medium can be an article of manufacture that is tangibly and non-transitorily embodied.

[0108] While the principles of the disclosed subject matter have been described above in connection with example implementations, it is to be understood that this disclosure is not limited to those implementations. On the contrary, it is contemplated that various modifications in addition to those described above can be made. Accordingly, it is intended that the scope of the disclosed subject matter extend to all such modifications and that the scope is to be interpreted in the broadest possible manner consistent with the spirit and scope of the concepts disclosed herein. Therefore, the above embodiments are merely illustrative and should not be considered restrictive in any way. Rather, the scope of the disclosed subject matter is to be determined only by the broadest interpretation of the following claims and their equivalents, and will not be restricted or limited to the abovementioned described embodiments. Accordingly, it is to be understood that the appended claims are intended to cover all such modifications and changes as fall within the scope of the embodiments.

Claims

1. An apparatus for message tunneling communication, comprising: a host interface circuit configured to communicate with a host device via a data protocol; a storage element configured to store data in response to a data message of the data protocol; and an on-board processor configured to execute one or more instructions in response to a message of the tunneling communication, wherein the host interface circuit is further configured to: detect whether a tunneling communication command is embedded within the data message of the data protocol, extract message address information of the message of the tunneling communication from the data message of the data protocol, retrieve the message of the tunneling communication stored in a memory of the host device via the message address information of the tunneling communication, and route the message of the tunneling communication stored in the memory of the host device to the on-board processor.

2. The apparatus of claim 1, wherein, An opcode field of the data message comprises a remote procedure call opcode.

3. The apparatus of claim 1, wherein, The host interface circuit is configured to retrieve the message of the tunneling communication from a host memory buffer reserved for the apparatus.

4. The apparatus of any one of claims 1-3, wherein, The host interface circuit is configured to indicate a completion of the data message in response to a successful retrieval of the message of the tunneling communication.

5. The apparatus of claim 4, wherein, The host interface circuit is configured to send an interrupt signal to the host device in response to a successful retrieval of the message of the tunneling communication.

6. The apparatus of any one of claims 1-3, wherein, The apparatus is configured to: keep one or more data messages to the host device in an open state, and close one of the plurality of data messages by sending a data message to the host device in response to a start of the message of the tunneling communication with respect to the host device.

7. The apparatus of claim 6, wherein, The open data message indicates that a portion of the host memory buffer has been allocated on the host device.

8. The apparatus of claim 6, wherein, The host interface circuit is further configured to, in response to a request to send the message of the tunneling communication to the host device: extract a pointer to the portion of the host memory buffer from the open data message that has been allocated on the host device; asynchronously transfer the message of the tunneling communication to the portion of the host memory buffer; and create a completion entry associated with the data message, wherein a reception of the completion entry by the host device causes the host device to read the message of the tunneling communication. The data message is associated with a quality of service indicator.

9. The apparatus of claim 1, wherein, The host interface circuit is configured to perform data message flow control via the data protocol for tunneling the data message.

10. The apparatus of claim 1, wherein, 11. A system for message tunneling communication, comprising: a host computing device; and a storage device configured to communicate with the host computing device, wherein the host computing device comprises: a processor configured to: read and write data from the storage device, and offload commands to the storage device; and the storage device comprises: a host interface circuit configured to communicate with the host computing device via a data protocol; a storage element configured to store data in response to a data message of the data protocol; and an on-board processor configured to execute one or more instructions in response to a message of the tunneling communication, the host interface circuit is further configured to: detect whether a tunneling communication command is embedded within the data message of the data protocol, extract message address information of the message of the tunneling communication from the data message of the data protocol, retrieve the message of the tunneling communication stored in a memory of the host computing device via the message address information of the tunneling communication, and ​ routing a tunneled communication message stored in a memory of a host device to an on-board processor.

12. The system of claim 11, wherein, The memory of the host computing device includes a host memory buffer reserved for the storage device, and wherein the host interface circuit is configured to retrieve the tunneled communication message from the host memory buffer.

13. The system of claim 11 or claim 12, wherein, The host interface circuit is configured to indicate a data message completion in response to successful retrieval of the tunneled communication message.

14. The system of claim 13, wherein, The host interface circuit is configured to send an interrupt signal to the host computing device in response to successful retrieval of the tunneled communication message.

15. The system of claim 11 or claim 12, wherein, The storage device is configured to: keep one or more data messages to the host computing device in an open state, and close one of the plurality of data messages by sending a data message to the host computing device in response to a start of the tunneled communication message to the host computing device.

16. The system of claim 15, wherein, The open data message indicates that a portion of the memory of the host computing device has been allocated on the host computing device.

17. The system of claim 15, wherein, The host interface circuit is further configured to, in response to a request to send the tunneled communication message to the host computing device: extract a pointer to the portion of the memory of the host computing device that has been allocated on the host computing device from the open data message; asynchronously transfer the tunneled communication message to the portion of the memory of the host computing device; and and create a completion entry associated with the data message, wherein receipt of the completion entry by the host computing device causes the host computing device to read the tunneled communication message.

18. The system of claim 11, wherein, The data message is associated with a quality of service metric.

19. The system of claim 11, wherein, The host interface circuit is configured to perform data message flow control via a data protocol used to tunnel the data message.

20. A method of tunneling a remote procedure call over a data protocol, the method comprising: allocating at least a portion of a host memory buffer included by a host computing device for communication between the host computing device and an enhanced storage device; creating a data message, the data message including an indication that a tunneled communication message is stored within the portion of the host memory buffer; sending the data message to the enhanced storage device; upon receipt of the data message, the enhanced storage device reading the tunneled communication message from the host computing device; and in response to the tunneled communication message, executing one or more instructions by the enhanced storage device.

Images