Systems, methods, and apparatus for remote data transfers to memory

Abstract

A method for remotely transferring data to memory may include: receiving commands, data identification information, and access information from a server at a target location to perform data transfer using a memory access protocol; and performing data transfer between the target and a client based on the commands, the access information, and the memory access protocol. The data identification information may include an object key, and the object key and access information may be at least partially encoded in the encoded object key. The method may further include sending a completion notification from the target to the server based on the data transfer. The method may further include sending a success indication from the server to the client based on the completion notification. The method may further include reconstructing the data based on parity data.

Description

Technical Field

[0001] This disclosure generally relates to data transmission, and more particularly, to systems, methods, and devices for remote data transmission to a memory. [Reference to Related Applications]

[0002] This application claims the priority and benefit of U.S. Provisional Patent Application No. 63 / 230,687, filed on August 6, 2021, which is incorporated herein by reference. Prior Art

[0003] A client can send a request to a server to read data from one or more storage resources that can be controlled by the server. The server can send a command to the one or more storage resources to retrieve the data. The one or more storage resources can transmit the data to the server, and the server can transmit the data to the client.

[0004] The above information disclosed in this prior art section is only for enhancing the understanding of the background of the present invention, and thus it may contain information that does not constitute prior art. Summary of the Invention

[0005] A method may include: receiving, at a target, a command, information for identifying data, and access information from a server to perform data transfer using a memory access protocol; and performing data transfer between the target and a client based on the command, based on the access information, using the memory access protocol. The information for identifying data may include an object key, and the object key and the access information may be at least partially encoded in an encoded object key. The method may further include sending a completion from the target to the server based on the data transfer. The method may further include sending an indication of success from the server to the client based on the completion. The target may be a first target, the command may be a first command, the information for identifying data may be first information for identifying first data, the access information may be first access information, and the data transfer may be a first data transfer, and the method may further include: receiving, at the target, a second command, second information for identifying second data, and second access information from the server to perform a second data transfer using the memory access protocol; obtaining the second data based on the second information for identifying the second data; and performing the second data transfer between the target and the client based on the second command, based on the second access information, using the memory access protocol. The method may further include: sending a first completion from the target to the server based on the first data transfer; sending a second completion from the target to the server based on the second data transfer; and sending an indication of success from the server to the client based on the first completion and the second completion. The data may be configured for error correction. The data may be erasure-encoded. The data transfer may be a first data transfer, and the target may be one of one or more targets, and the method may further include: determining a state of the first data transfer; and performing a second data transfer of parity data between the client and one of the one or more targets based on the state using the memory access protocol. The method may further include reconstructing the data based on the parity data. The reconstruction may be performed at the client. The method may further include initializing a connection between the client and the target for the memory access protocol. The method may further include sending, from the server to the client, configuration information for the target, wherein the connection may be based on the configuration information. Performing data transfer may include transferring data from the target to the client. Performing data transfer may include transferring data from the client to the target. The method may further include receiving, at the target, a request to access one or more resources controlled by the server, wherein the target may be configured to provide access to at least one of the one or more resources. The one or more resources may include one or more of a storage resource or a computing resource.The one or more resources may include a key-value (KV) storage. The server may include an object storage server.

[0006] A system may include: a target configured to: receive a command using a first network path; and perform data transfer using a second network path and a memory access protocol based on the command. The data transfer may include data transfer from the target to a client. The data transfer may include data transfer from the client to the target. The system may further include a server configured to: receive a request using a third network path; and send the command to the target using the first network path based on the request. The system may further include a client configured to: send the request to the server using the third network path; and perform data transfer using the second network path and a memory access protocol. The server may be configured to receive the request using a network stack of the third network path. The target may be configured to send a completion using the first network path to the server based on the transfer. The server may be configured to send a success indication to the client using the third network path based on the completion. The command may be a first command, the transfer may be a first transfer, the target may be one of one or more targets, the first network path may be one of one or more first network paths, the second network path may be one of one or more second network paths, and the system may further include a server configured to: determine a state of the first transfer; and send a second command to one of the one or more targets using one of the one or more first network paths based on the state, wherein the second command may be a second transfer of parity data between the client and one of the one or more targets using one of the second network paths and a memory access protocol.

[0007] A device may include: a target interface; and one or more network interfaces configured to connect the target interface to a first network path and a second network path, where the target interface may be configured to: receive a command using the first network path; and perform a transmission using the second network path and a memory access protocol based on the command. The target interface may be configured to send a completion using the first network path based on the transmission. The target interface may be configured to initialize a connection for the second network path and for the memory access protocol. The device may further include one or more resources, where the target interface may be configured to provide access to the one or more resources using the second network path and the memory access protocol. The one or more resources may include storage components. The one or more resources may include accelerators. The one or more resources may include key-value stores. The target interface may include a Nonvolatile Memory Express (NVMe) interface. Brief Description of the Drawings

[0008] Each drawing is not necessarily drawn to scale, and for illustrative purposes, components having similar structures or functions in all drawings may generally be represented by the same reference numeral or a portion thereof. Each drawing is only intended to facilitate the illustration of various embodiments described herein. Each drawing does not illustrate every aspect of the teachings disclosed herein and does not limit the scope of the patent application. To prevent the drawings from becoming blurred, not all components, connections, and similar forms may be shown, and not all components may have reference numerals. However, the pattern of component configuration may be readily apparent in the drawings. The accompanying drawings, together with the description, illustrate exemplary embodiments of the present disclosure and are used together with the description to explain the principles of the present disclosure. FIG. 1 shows an embodiment of a data transmission scheme according to an exemplary embodiment of the present disclosure. FIG. 2 shows an exemplary embodiment of a data transmission scheme for remote data transmission to a memory using a memory access protocol according to an exemplary embodiment of the present disclosure. FIG. 3A shows a first part of a sequence diagram of an embodiment of a data transmission scheme for remote data transmission to a memory using a memory access protocol according to an exemplary embodiment of the present disclosure. FIG. 3B shows a second part of a sequence diagram of an embodiment of a data transmission scheme for remote data transmission to a memory using a memory access protocol according to an exemplary embodiment of the present disclosure. FIG. 4 shows an exemplary embodiment of a method of a client-side process of an embodiment of a data transmission scheme for remote data transmission to a memory using a memory access protocol according to an exemplary embodiment of the present disclosure. FIG. 5A shows a first part of an exemplary embodiment of a method of a server-side process of a data transmission scheme for remote data transmission to a memory using a memory access protocol according to an exemplary embodiment of the present disclosure. FIG. 5B shows a second part of an exemplary embodiment of a method of a server-side process of a data transmission scheme for remote data transmission to a memory using a memory access protocol according to an exemplary embodiment of the present disclosure. FIG. 6 shows an exemplary embodiment of a client according to an exemplary embodiment of the present disclosure. FIG. 7 shows an exemplary embodiment of a server cluster according to an exemplary embodiment of the present disclosure. FIG. 8 shows an exemplary embodiment of a system for remote data transmission to a memory using a memory access protocol according to an exemplary embodiment of the present disclosure. FIG. 9 shows an exemplary embodiment of a method for remote data transmission to a memory using a memory access protocol according to an exemplary embodiment of the present disclosure. Implementation

[0009] A data processing system may include a client that may access one or more resources (such as storage elements, accelerators, and / or the like) that may be controlled by a server. The server may be connected to the one or more resources via a back-end network that may use a protocol with relatively low overhead and / or latency, such as Remote Direct Memory Access (RDMA). However, the server may be connected to the client via a front-end network that may use a network stack with one or more protocols such as Hypertext Transfer Protocol (HTTP) and / or Transmission Control Protocol (TCP), which have relatively high overhead and / or latency. Thus, when the client sends a request for data to the server, the data may be transmitted from the one or more resources to the server with relatively low overhead and / or latency, but the data may be transmitted from the server to the client with relatively high overhead and / or latency. Depending on the implementation details, this may create a bottleneck that may impair the client's ability to process the requested data.

[0010] In a data transfer solution according to an exemplary embodiment of the present disclosure, a client may request data for one or more resources controlled by a server by sending a request to the server. Based on the received request, the server may send a command to one or more network targets instructing the one or more network targets to retrieve the requested data from the one or more resources. However, the one or more network targets may send a data payload to the client instead of sending the data payload to the server. For example, in some embodiments, the one or more network targets may use one or more network paths and / or protocols that may have relatively low cost and / or latency to send the data payload to one or more memory locations of the client. Depending on the implementation details, this may improve the client's ability to process the requested data.

[0011] In some embodiments, the data transfer solution may implement one or more data protection techniques. For example, in some embodiments, upon completion of one or more data transfers to the client memory, the one or more network targets may send one or more corresponding completions to the server. Based on the completions, the server may send a success indication or an indication of failure to the client.

[0012] As another example of data protection, in some embodiments, error correction techniques (e.g., erasure coding (EC)) may be implemented, in which the data payload may be divided into one or more parts (which may also be referred to as chunks) that can be sent to the client from one or more network targets in separate transmissions. Parity information may be generated for and / or stored with the data chunks. If one or more of the data chunks are not successfully transmitted to the client memory (e.g., one or more of the data chunks are lost), then some or all of the parity information may be retrieved and / or transmitted to the client memory from the one or more network targets using, for example, the one or more network paths and / or protocols that may have relatively low cost and / or latency. The client may use some or all of the parity information to reconstruct the one or more lost data chunks.

[0013] In some embodiments, one or more of the features, functions, operations, and / or similar elements related to a data transfer scheme may be hidden from an application running on a client. For example, in some embodiments, a client interface may perform one or more operations, such as: initializing one or more network connections between the client and one or more network targets; sending a command to a server to request and / or send data to one or more resources controlled by the server; performing one or more data protection operations, such as checksum verification, data reconstruction, and / or similar operations. In some embodiments, the client interface may be implemented, for example, using one or more wrapper data, libraries (e.g., helper libraries), application programming interfaces (APIs), component drivers, function calls, system calls, and / or similar interfaces or combinations thereof.

[0014] The principles disclosed herein have independent utility and can be implemented separately, and not every embodiment can utilize every principle. However, the principles can also be implemented in various combinations, and some of these combinations can amplify the benefits of the individual principles in a synergistic manner.

[0015] FIG. 1 shows an embodiment of a data transfer scheme according to an exemplary embodiment of the present disclosure. The scheme shown in FIG. 1 may include a client 102, a server 104, and one or more targets 106. The server 104 may be connected to the client 102 via a first network path 108. The server 104 may be connected to the one or more targets 106 via one or more second network paths 110. The one or more targets 106 may be connected to the client 102 via one or more third network paths 112.

[0016] In some embodiments, the server 104 may be configured to receive a request 114 from the client 102 via the first network path 108 to access one or more resources controlled by the server 104. An example of a resource may include a storage resource (e.g., individual storage components), a collection of storage resources, an object (e.g., key-value) storage, and / or similar resources. In embodiments where the one or more resources include storage resources, the server 104 may be at least partially configured as a storage server, such as an object storage server. Another example of a resource may include a computing resource, such as an accelerator, a graphics processing unit (GPU), a neural processing unit (NPU), a tensor processing unit (TPU), and / or similar resources. In embodiments where the one or more resources include computing resources, the server 104 may be at least partially configured as a computing server.

[0017] In some embodiments, the server 104 may be configured to send one or more commands 116 to the one or more network targets 106 via the one or more second network paths 110 in response to the request 114. The one or more network targets 106 may be arranged to provide access to the one or more resources based on the one or more commands 116. For example, one or more of the network targets 106 may be connected to one or more resources (e.g., storage resources, computing resources, and / or similar resources) via one or more interconnections, network paths, and / or similar means.

[0018] In some embodiments, the client 102 may include a client memory 120 (which may also be referred to as a client buffer) and / or the one or more network targets 106 may include one or more target memories 122 (which may also be referred to as target buffers). The one or more network targets 106 may be configured to transfer one or more data payloads 118 between the client memory 120 and the one or more target memories 122 in response to the one or more commands 116. For example, if the request 114 is a put request, data may be transferred from the client memory 120 to the one or more target memories 122. As another example, if the request 114 is a get request, data may be transferred from the one or more target memories 122 to the client memory 120.

[0019] In some embodiments, the transfer of one or more data payloads from the client 102 to the memory of the one or more network targets 106 or from the one or more network targets 106 to the memory of the client 102 may be referred to as remote data transfer to memory. In some embodiments, a memory access protocol may be utilized to effectuate remote data transfer to memory. In some embodiments, a memory access protocol may refer to a protocol that may have one or more of the following characteristics: (1) the protocol may transfer data directly or nearly directly to and / or from the memory of a receiving device and / or a sending device; (2) the protocol may bypass at least a portion of the network stack of the network path used to transfer the data; (3) the protocol may operate with little or no involvement of one or more central processing units (CPUs) or other processors of the receiving device and / or the sending device; (4) the protocol may transfer data with little or no data copying, such as between layers of a network stack or other network software; (5) the protocol may transfer data to and / or from the user space of an operating system (OS) with little or no involvement of the kernel of the OS; (6) the protocol may transfer data with little or no use of context switches, little or no use of cache space, and / or the like; and / or (7) the protocol may have relatively low cost, latency, and / or the like. For example, for illustrative purposes, in some embodiments, a memory access protocol in accordance with an exemplary embodiment of the present disclosure may be implemented using RDMA, but the principles of the present invention are not limited to use with RDMA or any other memory access protocol.

[0020] In some embodiments, the client 102 may act as a server for one or more applications running on the client, one or more other devices, and / or the like. For example, in some embodiments, the client 102 may be implemented as a compute server that may run one or more applications that may operate on data retrievable by the client 102 and stored to one or more storage resources controlled by the server 104 and / or free storage resources controlled by the server 104.

[0021] In some embodiments, any one of the client 102, the server 104, and / or the one or more network targets 106 can be implemented using one or more processors (such as a complex instruction set computer (CISC) processor (such as an x86 processor) and / or a reduced instruction set computer (RISC) processor (such as an advanced RISC machine (ARM) processor, a RISC-V processor, and / or a similar processor)), a microcontroller, a complex programmable logic device (CPLD), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), and / or a similar processor, and any one of them can execute instructions stored in any type of memory. In some embodiments, any one of the client 102, the server 104, and / or the one or more network targets 106 can be implemented using any processing resource or combination of processing resources that can be disposed, for example, in one or more servers, and the one or more servers are configured, for example, in one or more server chassis, server racks, server rack groups, data rooms, data centers, edge data centers, mobile edge data centers, and / or similar locations and / or any combination thereof. In some embodiments, any one of the client 102, the server 104, and / or the one or more network targets 106 can be implemented using one or more compute server clusters. In some embodiments, any one of the client 102, the server 104, and / or the one or more network targets 106 can be deployed in a software implementation. Although the client 102, the server 104, and the one or more network targets 106 are shown as separate components, in some embodiments, the functions of any one of these components can be distributed among multiple components, and / or the functions of one or more of the components can be combined into a common component. In addition, some embodiments may include multiple clients 102 and / or servers 104.

[0022] In some embodiments, any one of the first network path 108, one or more second network paths 110, and / or one or more third network paths 112 may be implemented using any networking media, interfaces, protocols, and / or the like, including, for example: Ethernet, Fibre Channel, InfiniBand, Internet Wide Area RDMA Protocol (iWARP), Transmission Control Protocol / Internet Protocol (TCP / IP), User Datagram Protocol / IP (UDP / IP), Hypertext Transfer Protocol (HTTP), Hypertext Transfer Protocol Secure (HTTPS), RDMA, RDMA over Converged Ethernet (RoCE), any generation of wireless network including second generation (2G), third generation (3G), fourth generation (4G), fifth generation (5G), and / or the like, any generation of wireless fidelity (Wi-Fi), Bluetooth, near-field communication (NFC), and / or the like, or any combination thereof.In some embodiments, any one of the first network path 108, one or more second network paths 110, and / or one or more third network paths 112 may be implemented using any interconnect media, interface, protocol, and / or the like, including, for example: Peripheral Component Interconnect Express (PCIe), Nonvolatile Memory Express (NVMe), NVMe-over-fabric (NVMe-oF), Compute Express Link (CXL), and / or coherent protocol (such as CXL.mem, CXL.cache, CXL.IO, and / or similar protocols), Gen-Z, Open Coherent Accelerator Processor Interface (OpenCAPI), Cache Coherent Interconnect for Accelerators (CCIX), Advanced eXtensible Interface (AXI), and / or the like, or any combination thereof. In some embodiments, any one of the first network path 108, one or more second network paths 110, and / or one or more third network paths 112 may include one or more switches, hubs, routers, gateways network, network of networks (e.g., the Internet), and / or the like. In some embodiments, any one of the first network path 108, one or more second network paths 110, and / or one or more third network paths 112 may be implemented using one or more security features (such as encryption). For example, in some embodiments, network protocols, stacks, and / or the like may implement a link layer that may support encryption of one or more data transfers using a memory access protocol.

[0023] In some embodiments, the memory access protocol may be implemented using hardware, software, or a combination thereof. For example, in embodiments implementing RDMA (e.g., using RoCE), some (e.g., most) of the RDMA functions may be implemented using an RDMA-enabled channel adapter card or a network interface card (NIC). In such embodiments, the hardware RDMA functions may operate in conjunction with a relatively small kernel module and / or a user space API. As another example, in some embodiments, RDMA may be implemented using at least partially Soft-RoCE. Although implemented in software, Soft-RoCE may have significantly fewer costs, latencies, and / or similar factors compared to a network stack having separate layers for, e.g., socket, TCP, and / or UDP, IP, Ethernet driver, and / or similar technologies.

[0024] Although the embodiments shown in FIG. 1 are not limited to any specific implementation details, for illustrative purposes, in some embodiments, the first network path 108 may be implemented using one or more relatively high-cost and / or latency protocols (e.g., HTTP, TCP, and / or similar protocols) using a network stack, any type of object storage protocol, an object storage service, a remote procedure call for object storage, an API service for objects, and / or similar technologies (e.g., via HTTP, TCP, and / or similar protocols), while the one or more second network paths 110 and / or the one or more third network paths 112 may be implemented using one or more protocols having relatively low costs, latencies, and / or similar factors (e.g., a memory access protocol (e.g., RDMA)). In some embodiments, the first network path 108 may be a front-end network or may be a part of a front-end network, and the one or more second network paths 110 may be a back-end network or may be a part of a back-end network. Depending on the implementation details, the solution shown in FIG. 1 may reduce the costs, latencies, and / or similar factors associated with transferring data between the client 102 and one or more network targets 106.

[0025] In some embodiments, one or more of the network targets 106 may be configured to send one or more completions to the server 104 via one or more network paths 110, e.g., based on one or more successful transmissions of one or more corresponding data payloads 118. In some embodiments, in response to receiving any number of completions from the one or more network targets 106, the server 104 may be configured to send an indication of success, degraded data transfer (e.g., partial success), failure, and / or the like to the client 102 using the first network path 108. For example, if not all of the data payloads 118 associated with the request 114 are successfully transmitted, but enough data payloads 118 are successfully transmitted such that the client can, e.g., use error correction techniques (e.g., erasure coding) to reconstruct one or more lost data payloads, then an indication of degraded data transfer may be sent. In some embodiments, in the case of degraded data transfer, one or more parity data payloads may be transmitted from one or more of the network targets 106 to the client 102 via the one or more third network paths 112 and / or using a memory access protocol. In some embodiments, error correction may be performed, e.g., at the client 102 rather than the server 104, since the server 104 may not receive the one or more normal and / or parity data payloads 118 to perform error correction operations.

[0026] FIG. 2 illustrates an exemplary embodiment of a data transfer scheme for remote data transfer to a memory using a memory access protocol according to an exemplary embodiment of the present disclosure. The embodiment shown in FIG. 2 may be implemented using the embodiment shown in FIG. 1 or any other embodiment described herein, or may be used to implement the embodiment shown in FIG. 1 or any other embodiment described herein. The embodiment shown in FIG. 2 may include one or more components similar to the components shown in FIG. 1, and components having reference numbers ending with the same digit may have similar structures and / or functions. For illustrative purposes, the embodiment shown in FIG. 2 may be described in the context of some specific implementation details, such as network targets implemented as NVMe targets, a memory access protocol implemented using RDMA, data protection implemented using 8:2 erasure coding, and / or similar details, but the principles of the present invention are not limited to these or any other implementation details.

[0027] Referring to FIG. 2, the scheme may include a client 202, a server 204, and one or more network targets 206-1, …, 206-10. The client 202, the server 204, and the one or more network targets 206-1, …, 206-10 may be connected via one or more first network paths, second network paths, and / or third network paths as shown, e.g., in FIG. 1.

[0028] Referring to FIG. 2, the client 202 may include a client interface 252, and the client interface 252 may encapsulate one or more implementation features of a data transfer solution to hide the one or more implementation features from an application 250 running on the client 202. In some embodiments, the application 250 may allocate a client buffer 251 (which may be referred to as an application buffer) for RDMA data transfer from one or more of the network targets 206-1, …, 206-10. In this exemplary embodiment, the network targets 206-1, …, 206-10 may be implemented using the NVMe-oF protocol and may thus be referred to as NVMe targets or NVMe-oF targets. In some embodiments, the client interface 252 may be implemented, for example, using hardware, software, or a combination thereof, including combinational logic, sequential logic, one or more timers, counters, registers, and / or a state machine that executes instructions stored in any type of memory, one or more complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), embedded processors, central processing units (CPUs) (such as complex instruction set computers (CISC) processors (such as x86 processors) and / or reduced instruction set computers (RISC) processors (such as ARM processors)), graphics processing units (GPUs), neural processing units (NPUs), tensor processing units (TPUs), and / or similar components or any combination thereof. In embodiments implemented at least partially using software, the client interface 252 may be implemented at least partially using one or more wrapper data, libraries (such as helper libraries), application programming interfaces (APIs), component drivers, function calls, system calls, and / or similar techniques, or a combination thereof.

[0029] For illustrative purposes, in the exemplary embodiment shown in FIG. 2, the application 250 may allocate a maximum size for 8:2 erasure coding of an adaptable object to the application buffer 251. For example, the total data payload size of an object that may be divided into eight data blocks 253-1, …, 253-8, and two parity blocks 253-9 and 253-10 may be added to the eight data blocks 253-1, …, 253-8. However, any other allocation of data blocks and / or parity blocks may be used.

[0030] In the exemplary embodiment shown in FIG. 2, the server 204 may be implemented as an object storage server with storage disaggregation, which may distribute data blocks and / or parity blocks of an object over one or more of the network targets 206-1, …, 206-10.

[0031] The exemplary embodiment shown in FIG. 2 illustrates an object get operation in which the application 250 may request an object from the server 204. The get operation may start at operation 251-1, where the application 250 may provide the starting address and data length of the application buffer 251 to which the data blocks 253-1, …, 253-8 of the requested object may be transferred. At operation 251-2, the client interface 252 may send a request 214 that transfers the starting address and data length of the application buffer 251 to the server 204. Any technique may be used to send the request 214, such techniques being, for example, any type of protocol (such as HTTP, TCP, and / or a similar protocol), any type of object storage protocol, an object storage service, a remote procedure call for object storage, an API service for an object, and / or a similar technique (e.g., via HTTP, TCP, and / or a similar protocol).

[0032] At operation 251-3, the server 204 may translate the starting address and data length of the application buffer 251 into ten different starting addresses and data lengths 257-1, …, 257-10 of the ten blocks (e.g., eight data blocks and two parity blocks) within the application buffer 251. At operations 252-4-1, …, 252-4-8, the server 204 may send commands 216-1, …, 216-8 to the network targets 206-1, …, 206-8, respectively. In some embodiments, the commands 216-1, …, 216-8 may include the corresponding starting addresses and data lengths 257-1, …, 257-8 within the application buffer 251 to which the corresponding data blocks 253-1, …, 253-8 may be transferred. In some embodiments, the server 204 may send the one or more commands 216-1, …, 216-8 using, for example, NVMe, NVMe-oF, or any other protocol.

[0033] In response to commands 216-1, …, 216-8, network targets 206-1, …, 206-8 can retrieve corresponding data blocks 253-1, …, 253-8 from corresponding storage resources (e.g., an object (e.g., key-value (KV) pair) storage distributed over the storage resources and managed by server 204). At operations 253-5-1, …, 253-5-8, network targets 206-1, …, 206-8 can transmit the corresponding data blocks 253-1, …, 253-8 to corresponding portions of application buffer 251 indicated by start addresses and data lengths 257-1, …, 257-8, for example, using RDMA or any other memory access protocol.

[0034] In some embodiments, one or more of network targets 206-1, …, 206-8 can send one or more completions to server 204 based on successful completion of one or more data transmissions at operations 253-5-1, …, 253-5-8.

[0035] In some embodiments, depending on the success or failure of one or more of the data transmissions at operations 253-5-1, …, 253-5-8, server 204 can send one or more of commands 216-9 and 216-10 to corresponding network targets 206-9 and 206-10, respectively, to instruct targets 206-9 and 206-10 to retrieve one or more of parity blocks 253-9 and 253-10, respectively. For example, if one or both of the data transmissions at operations 253-5-1, …, 253-5-8 fail, server 204 can send one or more of commands 216-9 and 216-10 to retrieve one or more parity blocks and transmit them to application buffer 251 so that client interface 252 can reconstruct the one or two lost data blocks. In some embodiments, server 204 can send one or more of commands 216-9 and 216-10 using, for example, NVMe, NVMe-oF, or any other protocol.

[0036] In some embodiments, commands 216-9 and 216-10 can include corresponding start addresses and data lengths 257-9 and 257-10 within application buffer 251 to which corresponding parity blocks 253-9 and 253-10 can be transmitted. Then, one or more of targets 206-9 and 206-10 can retrieve one or more of corresponding parity blocks 253-9 and 253-10 from corresponding storage resources and transmit the one or more parity blocks 253-9 and 253-10 to the corresponding portions of application buffer 251, for example, using RDMA or any other memory access protocol.

[0037] In some embodiments, one or more data transfers may potentially be implemented, in part or in whole, using zero-copy transfer. In some embodiments, implementing zero-copy transfer may involve, for example, transferring data between the memories of a target and a client using a memory access protocol (e.g., RDMA). For example, in some embodiments, one or more data transfers may be implemented using zero-copy transfer by directly transferring data to the memory of a receiving component (e.g., memory 120 shown in FIG. 1 and / or buffer 251 shown in FIG. 2). As another example, in some embodiments, one or more data transfers may be implemented by at least partially using zero-copy transfer to a first memory and then transferring to a second memory. For example, the embodiment shown in FIG. 2 may be modified to include a transfer buffer (e.g., an RDMA buffer) associated with the client interface 252. One or more of the data blocks 253 may be transferred to such a transfer buffer using zero-copy transfer and then transferred to the buffer 251.

[0038] FIG. 3A shows a first part of a sequence diagram of an embodiment of a data transfer scheme for remote data transfer to a memory using a memory access protocol according to an exemplary embodiment of the present disclosure. FIG. 3B shows a second part of a sequence diagram of an embodiment of a data transfer scheme for remote data transfer to a memory using a memory access protocol according to an exemplary embodiment of the present disclosure. FIGS. 3A and 3B together form a single diagram that may be referred to as FIG. 3.

[0039] The embodiment shown in FIG. 3 may be implemented using the embodiments shown in FIGS. 1 and 2 or any other embodiments described herein, or may be used to implement the embodiments shown in FIGS. 1 and 2 or any other embodiments described herein. The embodiment shown in FIG. 3 may include one or more components similar to those shown in FIGS. 1 and 2 or any other embodiments described herein, and components having reference numerals ending with the same number may have similar structures and / or functions. For illustrative purposes, the embodiment shown in FIG. 3 may be described in the context of some specific implementation details, such as a server implemented as an object (e.g., a KV) storage server. However, the principles of the present invention are not limited to these or any other implementation details. In FIG. 3, information, signals, messages, software invocations and / or returns, and / or similar elements may propagate horizontally generally as indicated by the arrows, and time may generally proceed downward. The operating periods of the components are shown as solid lines above their respective vertical timelines.

[0040] Referring to FIG. 3, the data transmission system may include a client 302, a server 304, and one or more network targets 306-1, …, 306-N (collectively referred to as 306) that can provide access to one or more resources controlled by the server 304. A client interface 352 may run on the client 302. In some embodiments, the client 302 (including the client interface 352) may communicate with the server 304, for example, via one or more first network paths and / or a network stack that may implement one or more protocols (such as HTTP, TCP, and / or similar protocols), any type of object storage protocol, an object storage service, a remote procedure call for object storage, an API service for objects, and / or similar technologies (e.g., via HTTP, TCP, and / or similar protocols). In some embodiments, the server 304 may communicate with the one or more network targets 306-1, …, 306-N via one or more second network paths that may implement a memory access protocol (such as RDMA). In some embodiments, the client 302 (including the client interface 352) may communicate with the one or more network targets 306-1, …, 306-N via one or more third network paths that may implement a memory access protocol (such as RDMA).

[0041] For illustrative purposes, some of the embodiments described herein may use exemplary names for function invocations (e.g., init(), GetObject(), and / or similar names), but these names are merely examples and any names may be used. When the startup function Init() of the invocation client interface 352 is applied, the startup procedure 359 for the memory access protocol (e.g., RDMA) may begin. The client interface 352 may invoke a read configuration invocation at operation 361-1, which may cause the client 302 to send a get configuration request to the server 304 at operation 361-2. The server 304 may respond to the client 302 at operation 361-3 with a message that sends the requested network configuration information, erasure coding information, and / or similar information of the one or more network targets 306-1, …, 306-N, and the message may be transferred to the client interface 352 at operation 361-4. In some embodiments, the network configuration information may include, for example, the following information: the type, number, location, capabilities, and / or similar elements of the devices connected to the network, the location of one or more hubs, switches, routers, gateways, and / or similar components connected to the network, one or more identifiers (e.g., network identifiers), network addresses (e.g., Internet Protocol (IP) addresses), handles, authentication information, port information, connection identifiers (IDs), encryption keys, security keys, digital certificates, and / or any other information that may help the client, server, network targets, and / or other components determine how to transmit data via the network.

[0042] Using the network configuration information, the client interface 352 may initiate a memory access protocol (e.g., RDMA) connection with one or more of the network targets 306-1, …, 306-N by sending startup messages at operations 361-5-1, …, 361-5-N. At operations 361-6-1, …, 361-6-N, one or more of the network targets 306-1, …, 306-N may respond with one or more success messages that include one or more connection reference numbers, handles, and / or other elements.

[0043] In some embodiments, when the client interface 352 can invoke a create bucket call at operation 361-7 that may cause the client 302 to send a create bucket request to the server 304 at operation 361-8, the startup procedure 359 can continue. The server 304 can respond to the client 302 at operation 361-9 with a message indicating the successful creation of the bucket, and the message can be forwarded to the client interface 352 at operation 361-10. The startup procedure 359 can end with the client interface 352 returning the completion status of the startup function Init() to the application. In some embodiments, creating a bucket can involve the client determining a bucket name, region, storage management options, ownership, access permissions, and / or similar elements for the bucket. In some embodiments, creating a bucket can involve the server implementing a bucket name, region, storage management options, ownership, access permissions, and / or similar elements for the bucket as determined by the client, for example.

[0044] When the application invokes the GetObject() call of the client interface 352, the data transfer procedure 363 (in this example, the object get procedure) can begin. At operation 365-0, the client interface 352 can begin the GetObject() call by performing one or more administrative functions. For example, the client interface 352 can reuse the application buffer, collect one or more starting addresses and / or data lengths of the application buffer, allocate one or more buffers for parity data, collect one or more starting addresses and / or data lengths of the parity buffers, and / or perform similar operations. In this exemplary embodiment, the key of the KV pair can be used to store address and / or data length information, protocol connection information, and / or similar information. Thus, at operation 365-0, the client interface 352 can append address, data length, protocol connection information, and / or similar information before the key.

[0045] When the client interface 352 invokes a getObject invocation at operation 365-1 that may cause the client 302 to send a fetch request to the server 304 at operation 365-2, the actual data transfer process may begin. At operations 365-3-1, …, 365-3-N, the server 304 may respond to the fetch request by sending one or more commands (e.g., Command_NT1, …, Command_NTN) to one or more of the network targets 306-1, …, 306-N. In some embodiments, one or more of the commands may include information (e.g., RDMA information such as data length, starting address, client identifier, and / or similar information) that enables the one or more network targets 306-1, …, 306-N to perform data transfer operations using a memory access protocol (e.g., RDMA). At operation 365-4 (as shown for network target 306-1), the one or more network targets 306-1, …, 306-N may begin to respond to the command by performing one or more internal tasks, such as allocating a local buffer, reading one or more blocks of the requested data from one or more local storage elements, obtaining a connection handle (e.g., a connection reference number) from the command, and / or similar tasks.

[0046] At operations 365-5-1, …, 365-5-N, the one or more network targets 306-1, …, 306-N may perform one or more transmissions of the requested data as a data payload to the client 302 using a memory access protocol (e.g., RDMA), such as using information such as data length, starting address, and / or similar elements. At operations 365-6-1, …, 365-6-N, the one or more network targets 306-1, …, 306-N may send one or more completions corresponding to the one or more data payload transmissions to the server 304.

[0047] At operation 365-7, the server 304 may send a fetch completion message to the client 302, and the client 302 may send a return to the client interface 352 at operation 365-8. In some embodiments, at operation 365-9, the client interface 352 may return getObject along with the data transmitted from the one or more network targets 306-1, …, 306-N or a pointer to the data to the application. However, in some embodiments, for example, since the one or more network targets 306-1, …, 306-N may have directly transmitted the requested data to the application buffer using a memory access protocol (e.g., RDMA), the getObject return may be ignored. The data transfer procedure 363 may end with the client interface 352 returning the completion status of the GetObject() function to the application.

[0048] FIG. 4 illustrates an exemplary embodiment of a method of a client side flow of a data transfer scheme for remote data transfer to memory using a memory access protocol according to an exemplary embodiment of the present disclosure. The embodiment shown in FIG. 4 may be implemented using any one of the clients disclosed herein (e.g., the clients 102, 202, and / or 302 shown in FIGS. 1, 2, and / or 3), or may be used to implement any one of the clients disclosed herein. For illustrative purposes, the embodiment shown in FIG. 4 may be described in the context of some specific implementation details, such as a server implemented as an object (e.g., KV) storage server, a memory access protocol implemented using RDMA, and / or similar details, but the principles of the present invention are not limited to these or any other implementation details.

[0049] Referring to FIG. 4, the method may start when invoking the GetObject() function at operation 480-1. At operation 480-2, the method may determine whether a user buffer has been passed to the function. If a user buffer has been passed, then at operation 480-3, the method may capture the starting address and / or data length of the user buffer. Otherwise, at operation 480-4, the method may allocate a buffer (e.g., having a maximum buffer size for any data and / or parity block) and capture the starting address and / or data length of the newly allocated buffer. The method may continue from operation 480-3 or operation 480-4 to operation 480-5, in which the client may retrieve network topology (e.g., network configuration) information obtained using the init() function (e.g., as shown in FIG. 3). In some embodiments, the network topology and / or configuration information may include, for example, the following information: the type, number, location, capabilities, and / or similar elements of devices connected to the network, the location of one or more hubs, switches, routers, gateways, and / or similar components connected to the network, one or more identifiers (e.g., network identifiers), network addresses (e.g., IP addresses), handles, authentication information, port information, connection IDs, encryption keys, security keys, digital certificates, and / or any other information that may help the client, server, network destination, and / or other components determine how to transmit data over the network. The method may also determine the number of parity blocks and / or bits to be used.

[0050] At operation 480-6, the method may check the number of parity blocks and / or bits. If the number is zero, the method may continue to operation 480-8. However, if at operation 480-6, the number of parity blocks and / or bits is determined to be non-zero, it may indicate that an error correction (e.g., erasure coding) technique has not been selected. Thus, the method may continue to operation 480-7, in which the client may allocate a buffer having a size based on, for example, the erasure coding (EC) block size and / or the number of parity blocks. The method may also capture the starting address and / or data length of the parity buffer and continue to operation 480-8.

[0051] At operation 480-8, the method can collect a memory access protocol (e.g., RDMA) handle established during an initialization phase (e.g., using an init() function as shown in FIG. 3). At operation 480-9, the method can provide information that enables one or more network targets to perform data transfer to a client using the memory access protocol. Such information can include, for example, information for identifying the data to be transferred. Such information can include, for example, access information for determining how to perform memory access protocol operations. For example, in some embodiments, information that enables one or more network targets to perform data transfer to a client using the memory access protocol can be provided by including one or more of data length, data start address, parity length, parity start address, handles of one or more network targets, and / or similar elements in one or more fields (e.g., unused fields) in a request message of an object storage protocol (e.g., in one or more fields separated by commas and / or colons). As another example, in some embodiments, information that enables one or more network targets to perform data transfer to a client using the memory access protocol can be provided by generating an encoded object key by, for example, encoding one or more of the following information in the object key: data length, data start address, parity length, parity start address, one or more handles of one or more network targets (e.g., network targets 306-1, …, 306-N shown in FIG. 3), and / or the actual (e.g., original) content of the object key. In some embodiments, providing information in fields of a message and / or encoding the information in an encoded key can involve using one or more tags, delimiters, and / or similar elements that may not be usable as a key (e.g., one or more special characters). These examples are provided for illustrative purposes, and any other technique can be used to provide information for enabling one or more network targets to perform data transfer to a client using the memory access protocol.

[0052] At operation 480-10, the method may initiate data transfer between one or more network targets and a client by, for example, invoking a get object call (e.g., the GetObject() call 363 shown in FIG. 3), using a memory access protocol. In some embodiments, the data transferred between one or more network targets and a client using a memory access protocol may include an actual (e.g., raw) object key, and the data length, data start address, parity length, parity start address, handle of one or more network targets, and / or one or more of similar elements encoded in the encoded object key may be used by the client (e.g., client 102, client 202, and / or client 302 shown in FIGS. 1, 2, and / or 3), the server (e.g., server 104, server 204, and / or server 304 shown in FIGS. 1, 2, and / or 3), and / or one or more network targets (e.g., target 106, target 206, and / or target 306-1, …, 306-N shown in FIGS. 1, 2, and / or 3) to implement the transfer of data from one or more network targets to a client using a memory access protocol. In some embodiments, the get object call and / or return may be sent via a network (e.g., one or more first network paths) and / or a network stack that may implement one or more protocols (e.g., HTTP, TCP, and / or similar protocols), any type of object storage protocol, object storage service, remote procedure call for object storage, API service for objects, and / or similar technologies (e.g., via HTTP, TCP, and / or similar protocols).

[0053] At operation 480-11, the method may determine whether the get object call at operation 480-10 was successful. For example, if all of the owners in the data transfers at operations 365-5-1, …, 365-5-N in FIG. 3 were successfully completed, i.e., if all of the owners in the data blocks were successfully transferred directly to the client buffer, then the get object call may be considered successful. If the get object call is successful, the method may proceed to operation 480-12, in which the method may obtain the actual length and checksum (e.g., md5sum) that may have been returned in a buffer (e.g., a dummy buffer). At operation 480-13, the method may determine whether the actual length and checksum match what was transferred via the get object call at operation 480-10. If the actual length and checksum match, the method may end in a success state at operation 480-14, otherwise, the method may end in a failure state at operation 480-15.

[0054] However, if at operation 480-11, the method determines that the fetch object invocation at operation 480-10 was unsuccessful (e.g., not all owners in the data blocks of the requested object were successfully transmitted), then the method may proceed to operation 480-16, where the method may determine whether the fetch object invocation at operation 480-10 has transmitted degraded data. In some embodiments, if enough data blocks have been transmitted such that the lost blocks can be reconstructed from the parity blocks (e.g., if the number of lost blocks is less than or equal to the number of parity blocks), then the transmitted data may be considered degraded. If the transmitted data cannot be considered degraded (e.g., the object invocation at operation 480-10 did not transmit enough data blocks), then the method may end in a failure state at operation 480-17.

[0055] However, if at operation 480-16, the method determines that the fetch object invocation at operation 480-10 transmitted degraded data, then the method may proceed to operation 480-18, where the method may obtain the one or more parity blocks that can be used by it to reconstruct the lost one or more data blocks. Then, the method may proceed to operation 480-13 as described above.

[0056] FIG. 5A shows a first part of an exemplary embodiment of a method of a server side flow of a data transmission scheme for remote data transmission to a memory using a memory access protocol according to an exemplary embodiment of the present disclosure. FIG. 5B shows a second part of an exemplary embodiment of a method of a server side flow of a data transmission scheme for remote data transmission to a memory using a memory access protocol according to an exemplary embodiment of the present disclosure. FIGS. 5A and 5B may be collectively referred to as FIG. 5. The embodiment shown in FIG. 5 may be implemented using any one of the servers disclosed herein (e.g., servers 104, 204, and / or 304 shown in FIGS. 1, 2, and / or 3), or may be used to implement any one of the servers disclosed herein. The embodiment shown in FIG. 5 may implement one or more server side operations, such as interacting with one or more of the client side operations shown in FIG. 4. For illustrative purposes, the embodiment shown in FIG. 5 may be described in the context of some specific implementation details, such as a server implemented as an object (e.g., KV) storage server, a memory access protocol implemented using RDMA, and / or similar details, but the principles of the present invention are not limited to these or any other implementation details.

[0057] Referring to FIG. 5, when the server can receive an object request (e.g., the request shown in operation 365-2 in FIG. 3), the method can start at operation 582-1. At operation 582-2, the method can receive information that enables one or more network targets to perform data transfer to a client using a memory access protocol. For example, this information can be received in one or more fields (e.g., unused fields) in a request message of an object storage protocol (e.g., in one or more fields separated by commas and / or colons). As another example, in some embodiments, the information can be received by decoding one or more of the following information from an object key: data length, data start address, parity length, parity start address, one or more handles of one or more network targets (e.g., network targets 306-1, …, 306-N shown in FIG. 3), and / or the actual (e.g., original) content of the object key, which may have been encoded in the encoded object key (e.g., as described with reference to operation 480-9 in FIG. 4).

[0058] At operation 582-3, the method can determine whether it is desired to directly transfer data between one or more targets and the client (e.g., transfer to / from the memory at the client using a memory access protocol (e.g., RDMA)). In some embodiments, the method can determine the state of a flag, for example, by determining that the data length sent with the object request is greater than zero and / or a similar conclusion, to determine that it is desired to directly transfer data between one or more targets and the client. If it is not desired to directly transfer data between one or more targets and the client (e.g., if the data length sent with the object request is zero), the method can continue to operation 582-4, in which the method can complete the request by transferring data between one or more targets and the client via the server. For example, the server can send one or more commands to one or more network targets, the one or more network targets can return the requested data to the server (e.g., using a memory access protocol (e.g., RDMA)), and the server can then send the requested data to the client (e.g., via a network (e.g., one or more first network paths) and / or a network stack, which can implement one or more protocols (e.g., HTTP, TCP, and / or similar protocols), any type of object storage protocol, object storage service, remote procedure call for object storage, API service for an object, and / or similar technologies (e.g., via HTTP, TCP, and / or similar protocols)).

[0059] However, if at operation 582-3, the method determines that data is to be transferred directly between one or more targets and the client (e.g., transferring to / from the memory at the client using a memory access protocol such as RDMA), then the method may proceed to operation 582-5, where the method may issue a metadata read. At operation 582-6, the method may determine whether the metadata read was successful. If the metadata read was not successful, then the method may end in a failure state at operation 582-7. However, if the metadata read was successful, then the method may proceed to operation 582-8, where the method may obtain information that can be used to retrieve the requested data object from one or more storage resources accessible using one or more network targets. For one or more network targets, this information may include one or more of the following: error correction (e.g., erasure coding) information, data length, checksum (e.g., md5 checksum), data address, one or more parity storage elements, and / or the like.

[0060] At operation 582-9, the method may use the data obtained at operation 582-8 to map (e.g., deterministically) one or more data address ranges and / or parity address ranges to one or more network targets. At operation 582-10, the method may generate one or more error correction (e.g., erasure coding) key names. At operation 582-11, the method may call the API of one or more of the network targets using, for example, the erasure coding key name, the memory access protocol (e.g., RDMA) handle, the address range, and / or the like.

[0061] At operation 582-12, the method may initiate the transfer of the requested data (e.g., object) from one or more network targets to the memory buffer of the client. This may include, for example, the server sending one or more commands to one or more network targets (e.g., operations 365-3-1, …, 365-3-N as shown in FIG. 3), the one or more network targets retrieving one or more of the requested data blocks and / or transferring them to the client (e.g., operations 365-5-1, …, 365-5-N as shown in FIG. 3), and / or the one or more network targets sending one or more completions to the server (e.g., operations 365-6-1, …, 365-6-N as shown in FIG. 3).

[0062] At operation 582-13, the method may determine whether the data transfer at operation 582-12 was successful. For example, if the owners in the data payload transfers at operations 365-5-1, …, 365-5-N in FIG. 3 are all successfully completed, that is, if the owners in the data blocks are all successfully transferred directly to the client buffer, then the data transfer may be considered successful. If the data transfer is successful, the method may proceed to operation 582-14, in which the server may fill the return buffer with the actual length and checksum (e.g., md5sum) for the client to verify the checksum of the transferred data. The method may then end at operation 582-15 with a success status.

[0063] However, if at operation 582-13 the server determines that the data transfer at operation 582-12 was not successful (e.g., not all owners in the data blocks of the requested object were successfully transferred), the method may proceed to operation 582-16, in which the method may determine whether enough data blocks were transferred at operation 582-12 to enable reconstruction of the lost blocks from the parity blocks (e.g., if the number of lost blocks is less than or equal to the number of parity blocks). If not enough data blocks were successfully transferred to enable reconstruction of the lost blocks from the parity blocks (e.g., if the number of lost blocks is greater than the number of parity blocks), the method may end at operation 582-17 with a failure status.

[0064] However, if at operation 582-16 the method determines that operation 582-12 transferred enough data blocks to enable reconstruction of the lost blocks from the parity blocks (e.g., if the number of lost data blocks is less than or equal to the number of parity blocks), the transferred data may be considered degraded. Accordingly, the method may proceed to operation 582-18, in which the server may initiate the transfer of one or more parity blocks from one or more network targets to the client's memory buffer. This may include, for example: the server sending one or more commands to one or more network targets (e.g., similar to operations 365-3-1, …, 365-3-N as shown in FIG. 3), the one or more network targets retrieving one or more of the requested parity blocks and / or transferring them to the client (e.g., similar to operations 365-5-1, …, 365-5-N as shown in FIG. 3), and / or the one or more network targets sending one or more completions to the server (e.g., similar to operations 365-6-1, …, 365-6-N as shown in FIG. 3).

[0065] At operation 582-19, the method may determine whether the transmission of the parity block is successful (e.g., determine whether the number of transmitted parity blocks is sufficient to reconstruct the one or more lost data blocks). If the parity block is not successful, the method may end in a failure state at operation 582-20. However, if at operation 582-19, the transmission of the parity block is successful, the method may proceed to operation 582-21, in which the server may fill the return buffer with the actual length and checksum (e.g., md5sum) for the client to verify the checksum of the transmitted data. The method may then end in a degraded state at operation 582-22.

[0066] For illustrative purposes, the embodiments described with respect to FIGS. 2, 3, 4, and 5 may illustrate examples of the get operation. However, the put operation and / or other operations may also be implemented in a similar manner (e.g., directly transmitting the data payload from the client memory (e.g., using RDMA) to one or more target memories at one or more network targets, and / or the one or more network targets sending one or more completions to the server).

[0067] FIG. 6 shows an exemplary embodiment of a client according to an exemplary embodiment of the present disclosure. The client 602 shown in FIG. 6 may be implemented using any one of the clients disclosed herein (e.g., the client 102, client 202, and / or client 302 shown in FIGS. 1, 2, and / or 3), or may be used to implement any one of the clients disclosed herein. For illustrative purposes, the embodiment shown in FIG. 6 may be described in the context of some specific implementation details, but the principles of the present invention are not limited to those or any other implementation details.

[0068] Referring to FIG. 6, the client 602 may include a memory 646, a client interface 652, one or more NICs 671, and one or more computing units 677. The memory 646 and / or the one or more computing units 677 may be used by one or more applications 650 running on the client 602. Thus, in some embodiments, for the one or more applications 650, the client 602 may act as a computing server. The client interface 652 may be implemented, for example, using one or more envelope databases (which may also be referred to as helper libraries), which may hide one or more of the data transfer operations from the one or more applications 650. In some embodiments, the client interface 652 may use a memory access protocol (e.g., RDMA) to perform remote data transfer to a memory (e.g., the memory 646).

[0069] FIG. 7 shows an exemplary embodiment of a server cluster according to an exemplary embodiment of the present disclosure. The server cluster 784 may include two servers 704-1 and 704-2, and any one of the two servers 704-1 and 704-2 may be implemented by any one of the servers disclosed herein (e.g., the server 104, the server 204, and / or the server 304 shown in FIGS. 1, 2, and / or 3), or may be used to implement any one of the servers disclosed herein. For illustrative purposes, the embodiment shown in FIG. 7 may be described in the context of some specific implementation details, but the principles of the present invention are not limited to these or any other implementation details.

[0070] Referring to FIG. 7, the server cluster 784 may also include three dual-port NICs 771, 773, and 775, one or more peer-to-peer network connections 764 and / or 766, a target interface 756, a target interface subsystem 758, and / or a resource pool 760. The resource pool 760 may include, for example, one or more resources 762 (e.g., storage resources, computing resources, and / or similar resources).

[0071] In some embodiments, port 1 and port 2 of the first NIC 771 may respectively provide the first server 704-1 with access to one or more front-end networks and one or more back-end networks. Similarly, port 1 and port 2 of the second NIC 773 may respectively provide the second server 704-2 with access to one or more front-end networks and one or more back-end networks. In some embodiments, the one or more front-end networks may implement a network stack that can communicate with one or more clients (e.g., using HTTP, TCP, and / or similar protocols, any type of object storage protocol, object storage service, remote procedure call for object storage, API service for objects, and / or similar technologies (e.g., via HTTP, TCP, and / or similar protocols)), and the one or more back-end networks may use a memory access protocol (e.g., RDMA) to implement remote data transfer to memory.

[0072] In some embodiments, the one or more backend networks may enable the servers 704-1 and / or 704-2 to access the resource pool 760, for example, using the target interface 756 and / or the target interface subsystem 758, which may be implemented using NVMe, for example. In some embodiments, the one or more peer-to-peer network connections 764 and / or 766 may enable the servers 704-1 and / or 704-2 to access resource pools at other server clusters and enable other server clusters to access the resource pool 760.

[0073] In some embodiments, a third NIC 775 may provide additional network connectivity to enable, for example, the target interface 756 to perform remote data transfer to memory with one or more clients using a memory access protocol (e.g., RDMA), as described above with reference to FIGS. 1, 2, and / or 3.

[0074] FIG. 8 illustrates an exemplary embodiment of a system for performing remote data transfer to memory using a memory access protocol in accordance with an exemplary embodiment of the present disclosure. The embodiment shown in FIG. 8 may include one or more clients 802 and one or more server clusters 884. In some embodiments, the one or more clients 802 may be implemented using one or more of the clients 602 shown in FIG. 6. In some embodiments, the one or more server clusters 884 may be implemented using one or more of the server clusters 784 shown in FIG. 7. For illustrative purposes, the embodiment shown in FIG. 8 may be described in the context of some specific implementation details, such as resources implemented using a key-value store using solid state drives (SSDs) and / or network targets implemented using NVMe-oF, but the principles of the present invention are not limited to these or any other implementation details.

[0075] The embodiment shown in FIG. 8 may include a first network configuration 891 and a second network configuration 893. The first network configuration 891 may provide a first front-end network between one or more of the servers in the server cluster 884 and one or more of the clients 802. The first network configuration 891 may be implemented, for example, using one or more network stacks, which may communicate using one or more relatively high-cost and / or latency protocols (such as HTTP, TCP, and / or similar protocols), any type of object storage protocol, object storage service, remote procedure call for object storage, API service for objects, and / or similar technologies (e.g., via HTTP, TCP, and / or similar protocols). In contrast, the second network configuration 893 may be implemented using a memory access protocol (such as RDMA) that may have relatively low cost, latency, and / or similar factors to perform remote data transfer to memory. In some embodiments, the second network configuration 893 may enable a network target (e.g., an NVMe-oF target in one or more of the server clusters 884) to directly transfer data from a KV store (e.g., a KV storage) to the memory of one or more clients 802 in response to a command from a server.

[0076] In embodiments in which one or more of the resources are implemented as one or more storage components, the storage components may include any type of non-volatile storage media such as, for example, solid-state media (e.g., SSD), magnetic media (e.g., hard disk drive (HDD)), optical media, and / or the like, or any combination thereof. For example, in some embodiments, the storage components may be implemented as SSDs based on NAND (not-AND) flash memory, persistent memory (e.g., cross-gridded nonvolatile memory), memory with bulk resistance change, phase change memory (PCM), and / or the like, or any combination thereof. Any such storage component may be implemented in any form factor using any connector configuration (e.g., serial advanced technology attachment (SATA), small computer system interface (SCSI), serial attached SCSI (SAS), U.2, M.2, and / or the like), the form factor being, for example, 3.5 inches, 2.5 inches, 1.8 inches, M.2, Enterprise and Data Center SSD Form Factor (EDSFF), NF1, and / or the like. Any such storage component may be implemented in whole or in part using a server cabinet, server rack, data room, data center, edge data center, mobile edge data center, and / or any combination thereof and / or used in combination therewith.

[0077] Any of the functions described herein (including any of the host functions, client functions (e.g., client interface), server functions, target functions (e.g., target interface and / or subsystem), and / or the like) may be implemented using hardware, software, or a combination thereof, including combinational logic, sequential logic, one or more timers, counters, registers, and / or state machines executing instructions stored in any type of memory, one or more CPLDs, FPGAs, ASICs, embedded processors, CPUs (e.g., CISC processors (e.g., x86 processors) and / or RISC processors (e.g., ARM processors)), GPUs, NPUs, TPUs, and / or the like, or any combination thereof.

[0078] FIG. 9 shows an embodiment of a method for remote data transmission to a memory using a memory access protocol according to an exemplary embodiment of the present disclosure. The method may start at operation 902. At operation 904, the method may receive, at a target, a command, information for identifying data, and access information from a server to perform data transmission using the memory access protocol. For example, in some embodiments, the information for identifying data may be implemented using an object key, and the access information may include RDMA information. At operation 906, the method may perform data transmission between the target and a client based on the command, based on the access information, using the memory access protocol. For example, in some embodiments, the target may transmit data to the client using RDMA. The method may end at operation 908.

[0079] The owners in the embodiment shown in FIG. 9 and other embodiments described herein are exemplary operations and / or exemplary components. In some embodiments, some operations and / or components may be omitted and / or other operations and / or components may be included. Additionally, in some embodiments, the temporal order and / or spatial order of the operations and / or components may be changed. Although some components and / or operations may be shown as separate components, in some embodiments, some components and / or operations shown separately may be integrated into a single component and / or operation, and / or some components and / or operations shown as a single component and / or operation may be implemented using multiple components and / or operations.

[0080] The embodiments disclosed above have been described in the context of various implementation details, but the principles of this disclosure are not limited to these or any other specific details. For example, some functions have been described as being implemented by certain components, but in other embodiments, the functions may be distributed among different systems and components located at different positions and having various user interfaces. Certain embodiments have been described as having specific processes, operations, etc., but these terms also encompass embodiments in which a specific process, operation, etc. can be implemented using multiple processes, operations, etc. or embodiments in which multiple processes, operations, etc. can be integrated into a single process, step, etc. A reference to a component or assembly may refer only to a part of the component or assembly. For example, a reference to an integrated circuit may refer to the entire integrated circuit or only a part thereof, and a reference to a block may refer to the entire block or one or more sub-blocks. Unless the context is obvious, the use of terms such as "first" and "second" in this disclosure and the claims may only be for the purpose of distinguishing the things modified by these terms and may not indicate any spatial or temporal order. In some embodiments, a reference to something may refer to at least a part of the thing, for example, "based on" may mean "at least partially based on", "access" may mean "at least partially access" and / or the like. A reference to a first component may not imply the existence of a second component.

[0081] According to the inventive principles disclosed in this patent, various details and embodiments described above can be combined to produce additional embodiments. Since the inventive principles disclosed in this patent can be arranged and refined in terms of details without departing from the inventive concept, such changes and refinements are considered to fall within the scope of the following claims.

[0082] 102, 202, 302, 602, 802: Client 104, 204, 304: Server 106, 206-9, 206-10, 306-1~306-N: Network Target / Target 108: First Network Path 110: Second Network Path / Network Path 112: Third Network Path 114, 214: Request 116, 216-1, 261-2~216-10, Command_NT1~Command_NTN: Command 118: Data Payload 120: Client Memory / Memory 122: Target Memory 206-1, 206-2 to 206-8: Network targets 250, 650: Applications 251: Client buffer / Application buffer / Buffer 251-1, 251-2, 251-3, 252-4-1, 252-4-2 to 252-4-8, 361-1, 361-2, 361-3, 361-4, 361-7, 361-8, 361-9, 361-10, 361-5-1 to 361-5-N, 361-6-1 to 361-6-N, 365-0, 365-1, 365-2, 365-3-1 to 365-3-N, 365-4, 365-5-1 to 365-5-N, 365-6-1 to 365-6-N, 365-7, 365-8, 365-9, 480-1, 480-2, 480-3, 480-4, 480-5, 480-6, 480-7, 480-8, 480-9, 480-10, 480-11, 480-12, 480-13, 480-14, 480-15, 480-16, 480-17, 480-18, 582-1, 582-2, 582-3, 582-4, 582-5, 582-6, 582-7, 582-8, 582-9, 582-10, 582-11, 582-12, 582-13, 582-14, 582-15, 582-16, 582-17, 582-18, 582-19, 582-20, 582-21, 582-22, 902, 904, 906, 908: Operations 252, 352, 652: Client interfaces 253-1, 253-2 to 253-8: Data blocks 253-9, 253-10: Parity blocks 257-1, 257-2 to 257-10: Starting addresses and data lengths 359: Startup procedure 363: Data transfer procedure 646: Memory 671: Network Interface Card (NIC) 677: Computing unit 704-1: First server / Server 704-2: Second server / Server 756: Target interface 758: Target interface subsystem 760: Resource pooling field 762: Resources 764, 766: Peer-to-peer network connection 771: First NIC / Dual-port NIC 773: Second NIC / Dual-port NIC 775: Third NIC / Dual-port NIC 784, 884: Server cluster 891: First network configuration 893: Second network configuration Init(): Startup function

Claims

1. A method for remotely transferring data to memory, comprising: At the target location, the server receives commands, information for identifying data, and access information to perform data transfer using memory access protocols. The data transfer between the target and the client is performed based on the command, the access information, and the memory access protocol, wherein the access information includes a connection handle for the target.

2. The method as described in request item 1, wherein: The information used to identify the data includes an object key, and the object key and the access information are at least partially encoded in the encoded object key.

3. The method as described in claim 1 further includes sending a completion message from the target to the server based on the data transmission.

4. The method as described in request 3 further includes sending a success indication from the server to the client based on the completion.

5. The method as described in claim 1, wherein the target is a first target, the command is a first command, the information for identifying the data is first information for identifying the first data, the access information is first access information, and the data transmission is first data transmission and includes direct transmission of payload between the first target and the client, the method further comprising: The target receives a second command, second information for identifying second data, and second access information from the server to perform second data transmission using the memory access protocol; obtains the second data based on the second information for identifying the second data; and performs the second data transmission between the target and the client based on the second command, the second access information, and the memory access protocol.

6. The method as described in claim 5, further comprising: Based on the first data transmission, a first completion is sent from the target to the server; Based on the second data transmission, a second completion is sent from the target to the server; and based on the first completion and the second completion, a success indication is sent from the server to the client.

7. The method as described in request item 1, wherein the data is configured for error correction.

8. The method of claim 1, wherein the data transmission is a first data transmission and the target is one or more targets, the method further comprising: Determine the status of the first data transmission; And, based on the state, to perform a second data transmission of parity data between the client and one of the one or more targets using the memory access protocol.

9. The method of claim 8 further includes reconstructing the data based on the parity data.

10. The method as described in request item 9, wherein the reconstruction is performed at the client.

11. The method of claim 1, further comprising initializing the connection between the client and the target with respect to the memory access protocol.

12. The method as described in claim 11, further comprising: The server sends configuration information for the target to the client, wherein the connection is based on the configuration information.

13. The method of claim 1, wherein performing the data transfer includes transferring the data from the target to the client.

14. The method of claim 1, wherein performing the data transfer includes transferring the data from the client to the target.

15. A system for remote data transfer to memory, comprising: The target is configured to: receive commands and access information using a first network path; and perform data transmission using a second network path and a memory access protocol based on the commands and the access information, wherein the access information includes a connection handle for the target and the target being memory.

16. The system as described in claim 15 further includes a server configured to: receive a request using a third network path; and send the command to the target using the first network path based on the request.

17. The system as described in claim 16 further includes a client configured to: send the request to the server using the third network path; and perform the data transfer using the second network path and the memory access protocol.

18. An apparatus for remotely transferring data to memory, comprising: Target interface; and one or more network interfaces configured to connect the target interface to a first network path and a second network path, wherein the target interface is configured to: receive commands and access information using the first network path; and perform transmissions based on the commands and the access information, using the second network path, and using a memory access protocol, wherein the access information includes a connection handle for the target interface and the target interface being memory.

19. The apparatus of claim 18, wherein the target interface is configured to initialize the connection for the second network path and for the memory access protocol.

20. The apparatus of claim 18 further includes one or more resources, wherein the target interface is configured to provide access to the one or more resources using the second network path and the memory access protocol.

Images