Enabling fabric-based non-volatile memory high speed (NVMe) (NVMe-oF) traffic over interface using scalable endpoint (SEP) addressing mechanism

Abstract

A method is provided that allows fabric-based non-volatile memory high speed (NVMe) (NVMe-oF) traffic to be implemented over an interface using a scalable endpoint (SEP) addressing mechanism. In a computer network, a communication connection may be established between a host device of a domain object and a computing device of the computer network to allow data transfer. To establish a connection between the domain object and the network, a set of CPU core connections in the network may be determined. A set of transmit and receive contexts may be generated for the domain object, where each set of transmit and receive contexts is aligned with one of the set of CPU core connections. A scalable endpoint (SEP) may be generated for the domain object. The SEP may be used to map the set of transmit and receive contexts to the set of CPU core connections and the domain object, which may be used to transmit data throughout the network.

Description

Background Art

[0001] Non-Volatile Memory Express (NVMe) is an open logical device interface specification for accessing a computer's non-volatile storage media. NVMe is designed to exploit the low latency and internal parallelism of solid-state storage devices. NVMe over Fabrics (NVMe-oF) is an extension of the NVMe network protocol to Ethernet and Fibre Channel, providing faster and more efficient connections between storage and servers and reducing central processing unit (CPU) utilization of application host servers. The NVMe-oF protocol can help bridge the gap between direct-attached storage and storage area networks, enabling organizations to support workloads that require high throughput and low latency. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] The present disclosure is described in detail according to one or more different examples with reference to the following drawings. The drawings are provided for illustration purposes only and depict only typical examples or illustrative examples. These drawings are provided to facilitate the reader's understanding of the disclosed technology and should not be considered as limiting its breadth, scope or applicability. It should be noted that for clarity and ease of illustration, these drawings are not necessarily drawn to scale.

[0003] Figure 1 is an illustration of an example computing system of a network device according to the examples described in this disclosure.

[0004] Figure 2 is an illustration of an example computing system of a network for establishing a communication link between a network device and a client device according to examples described in the present disclosure.

[0005] Figure 3 is an illustration of an example process for implementing a communication connection in a network according to examples described in the present disclosure.

[0006] Figure 4 is a diagram of an example network for establishing a communication link between a domain object and a CPU core using a scalable endpoint (SEP) according to examples described in the present disclosure.

[0007] Figure 5 is an illustration of an example network for performing push operations between a host device and a client CPU core using a SEP according to examples described in the present disclosure.

[0008] Figure 6 is an illustration of an example network for performing pull operations between a host device and a client CPU core using a SEP according to examples described in the present disclosure.

[0009] Figure 7is an illustration of an example computing component according to an example described in the present disclosure, the example computing component including one or more hardware processors and a machine-readable storage medium storing a set of machine-readable / machine-executable instructions that, when executed, cause the one or more hardware processors to perform an illustrative method for establishing a communication link between a domain object and a network using a SEP.

[0010] Figure 8 A block diagram of an example computer system is illustrated upon which various examples of the present disclosure may be implemented.

[0011] The drawings are not intended to be exhaustive or to limit the invention to the precise forms disclosed. It should be understood that the invention can be practiced with modification and alteration, and that the disclosed technology is limited only by the claims and their equivalents. DETAILED DESCRIPTION

[0012] Traditional systems of the NVMe-oF specification need to create multiple queues to establish connection links between multiple devices (such as NVMe devices). Creating queues for connection links requires a large amount of memory. Scalable Endpoints (SEP) refers to a mechanism that solves the problem of large amounts of memory required for data storage to facilitate connection links between devices. Although the concept of SEP can solve such memory problems and provide performance improvements, current systems do not utilize SEP to facilitate connection links between devices. Traditional systems may require a large amount of memory for data storage to facilitate connection links between multiple devices in a network.

[0013] For example, a conventional system may create a set of queues for establishing a data transfer connection between a device such as a host (e.g., an NVMe device, a storage device) and a client (e.g., a network) of multiple devices comprised of a CPU core. The queues created may facilitate a data transfer connection with the host, thereby allowing data to be transferred to each of the multiple NVMe devices. This conventional system architecture is not optimal because it requires a large amount of memory for each set of queues created. Moreover, when the network includes thousands of nodes running thousands of CPU cores and multiple clients are connected to the network, a large amount of memory is used to store data to facilitate all connections between the CPU cores of the network and the host devices (e.g., remotely addressed data for each set of queues).

[0014] Examples of the present disclosure may provide systems and methods configured to establish an improved data transmission connection between a host and other devices in a network by utilizing SEP. Using SEP can reduce the amount of memory required for data storage. For example, SEP can use a reduced and constant memory footprint associated with connection resources, regardless of the number of other devices (e.g., CPU cores, NVMe devices) in the network. Some examples of the present disclosure may generate SEPs to establish a connection between a single host device and the CPU cores of other devices in the network. In some examples, the system and method may create a queue for each connection between the host device and the CPU core of one of the other devices in the network, and use SEP to reduce the amount of memory required. The amount of memory may be small (compared to other memory usage) and relatively constant for data storage. The memory scheme may cover one or more queues for connecting to a CPU core in the network. In this way, SEP may allow a single host device to have a data transmission connection with each CPU core of other devices in the network without using a large amount of memory and without negatively affecting the processing performance of the data transmission performed.

[0015] In some examples, the SEP addressing mechanism can implement NVMe-oF traffic on the interface. SEP can be used to establish a communication connection between one or more host devices of a domain object and one or more CPU cores of a device in a network to allow the transmission of one or more messages and data. A domain object can represent a domain that can include one or more devices (such as host devices) that exist in the same domain. A network can include one or more devices, such as client computing devices, and each device in the network can include a CPU core that runs operations of the corresponding device. SEP can be generated using address vectors for storing remote addresses of devices in the network, including, for example, remote addresses of CPU cores of devices in the network, remote addresses of domain objects, and remote addresses of one or more host devices of domain objects. Address vectors can reduce the memory requirements required to store remotely addressed data.

[0016] In order to establish a communication connection between the host device of the domain object and the CPU core of a device in the network, the SEP may align the domain object connection of the domain object with the number / value / group (hereinafter "N" used interchangeably with the number, value, or group) of the CPU core connections of the device in the network. In some examples, each CPU core of a device in the network may have a corresponding CPU core connection with the network. Each of the CPU core connections may be used as a communication connection between a corresponding CPU core of a device and another device connected to the network. The network may have N devices and N CPU cores and CPU core connections. Each of the CPU core connections may include a non-volatile memory express (NVMe) queue pair, thereby creating a set of NVMe queue pairs. NVMe queue pairs may be used to transmit messages and data between a corresponding CPU core of a device and other devices connected to the network using a corresponding CPU core connection.

[0017] A domain object may have a domain object connection that may allow messages and data to be transmitted between each host device of the domain object and other devices in the network. The domain object connection may include N sets of send and receive contexts to match the domain object to the N CPU cores of the device in the network. The SEP may align the domain object connection of the domain object with the N CPU core connections of the device in the network by linking the N sets of send and receive contexts of the domain object connection to the N sets of NVMe queue pairs of the CPU core connection. Aligning a set of send and receive contexts with the CPU core connection may associate and map the communication connections to each other.

[0018] Each of the N sets of send and receive contexts of the domain object connection can be mapped to a different NVMe queue pair of the CPU core connection to establish a 1-to-1 mapping connection between the domain object and each of the CPU cores in the network. In this way, one or more messages and data transmitted between the domain object and the CPU cores of the devices of the network can use a specific set of send and receive contexts in the domain object connection that is associated with the connection of the CPU core connection of the corresponding CPU core of the network. For example, the first host device of the domain object may want to communicate with the first CPU core and transfer data between the two. The SEP can use the CPU core connection of the first CPU core and the set of send and receive contexts of the domain object connection associated with the first CPU core to transfer data between the first host device of the domain object and the first CPU core.

[0019] N completion queues may be generated for a domain object. Each of the N completion queues may be used to store events and operations (hereinafter "operations") that have been completed between the domain object and the network. For example, operations may include sending messages or data, receiving messages or data, reading messages or data, and writing messages or data between the domain object and the network. The N completion queues may record each operation that has been completed between the domain object and the network. This may include recording each operation that has been completed between each client device of the domain object and the network.

[0020] Each of the N completion queues of the domain object can be associated with a different set of send and receive contexts in the N sets of send and receive contexts of the domain object connection. Each of the N completion queues can further be associated with a corresponding CPU core connection in the CPU core connection, which is aligned with the corresponding set of send and receive contexts of the domain object connection. Each of the N completion queues can therefore be associated with a different CPU core of the network. In this way, each of the N completion queues will track and record operations performed and completed between the domain object and the corresponding CPU core of the corresponding device in the network. This can allow the recording and separation of records of operations that occur between the domain object and each CPU core of each device in the network.

[0021] An event queue may also be generated for a domain object. The event queue may be used to track data transmissions flowing through the domain object. Data transmissions flowing through the domain object may include any data transmissions received from or sent to any of one or more host devices in the domain object. In this manner, the event queue may be used to track all data transmissions occurring between the domain object and the network.

[0022] The SEP may be defined by characteristics that may include a network address and a designated port of a domain object. The network address and designated port of a domain object may indicate that a SEP is designated for a domain object. As mentioned above, a SEP may establish a communication connection between one or more host devices in a domain object and each of the CPU cores of a device in a network. SEP may increase the amount of memory required to establish a communication connection between a domain object and all CPU cores in a network. Using SEP may establish all connections between a host device in a domain object and all CPU cores in a network using only a small and constant amount of memory. In this way, only a small and constant amount of memory is required to establish a data transmission connection between a domain object and a CPU core, regardless of the number of CPU cores of a device in a network. SEP may have the additional benefit of using a single base address of an address vector for multiple sets of send and receive contexts. By using a single base address (also referred to as a single transfer-level address), the memory requirements required to store remotely addressed data may be reduced.

[0023] SEP can allow the creation of multiple independent send and receive contexts. In some examples, SEP can be similar to the concept of "multi-queue" in networking, and can improve the performance of multi-threaded and parallel applications by allowing threads to access independent send and receive context queues. Therefore, in the case where an application has many processing connections, by assigning each connection its own send and receive context queue, the application can avoid serializing (i.e., locking) access to hardware queues. SEP has the additional benefit of allowing the creation of multiple send and receive contexts, the addresses of which can be calculated to use a single base address and have a single transfer-level address (i.e., address vector), making SEP more beneficial than standard endpoints. In other words, using SEP with address vectors can solve the large memory problem of the connection link of the current system by reducing the memory requirements required to store all remotely addressed data.

[0024] Technical improvements can be achieved throughout the disclosure. For example, using SEP can reduce the amount of memory used to establish a data transmission connection between a single host device (e.g., NVMe device, storage device) and a network composed of CPU cores of devices (e.g., client devices).

[0025] In some examples, the described systems and methods can also reduce the overhead processing associated with memory registration in remote direct memory access (RDMA) operations. For example, in a typical NVMe-oF RDMA operation involving an initiator and a target, the NVMe-oF specification can perform a memory registration operation in the target to pass the address and access key to the initiator. The initiator can perform the allocation of the receive buffer, register on the initiator RDMA NIC, and then issue a send request using the registered virtual address and permission key of the target. The target RDMA NIC can then look up the permission key and perform a read or write operation on the memory address. Because this model registers the target-side memory buffer before data is transferred, there is overhead processing for configuring hardware and removing components. The described system can avoid expensive memory registration operations associated with traditional RDMA-based transmissions, thereby improving performance.

[0026] In some examples, the SEP solution for NVMe-oF can also enable the advantages listed above in the context of NVMe-oF. In traditional systems, NVMe queues are aligned with CPU cores so that input / output (I / O) submissions can take advantage of core affinity. Here, the described systems and methods can pair NVMe queues with the transmit (TX) / receive (RX) contexts and corresponding completion queues (CQs) of client devices. This correlation can create a 1-to-1 mapping between the client device and each CPU core of the device in the network.

[0027] Figure 1 1 is an illustration of an example computing system 100 that includes one or more computing components, which may include any of a server 110, a router 120, a switch 122, a network controller 124, an access point 126, and a DHCP server 128. In some examples, router 120 may be associated with firewall 121. Server 110 may further include or be associated with a database or cache 112 (hereinafter "database") that stores attributes of specific client devices, storage devices, and access control lists or policies associated with specific computing devices (such as computing devices 130-139, which are connected to the network via access point 126). In some examples, any or all of computing devices 130-139 may include plug-and-play devices. Although Figure 1 Only ten computing devices are illustrated in FIG. 1 , but a greater or lesser number of computing devices may be implemented. For example, any number of computing devices may be connected to the network via access point 126.

[0028] The database 112 may be integrated or embedded in the server 110, or may be spatially separated from the server 110. The access control list may be stored as a file and / or may be indexed. In some examples, the access control list or policy may include specific access levels and / or access privileges to be assigned to each computing device depending on the group or classification to which the computing device belongs. For example, the access privileges may indicate a subset (e.g., a portion or all) of data resources, such as a specific data server, database, platform, object, file directory, storage area or file that each computing device is authorized to access, a specific protocol (e.g., Hypertext Transfer Protocol (HTTP) or File Transfer Protocol (FTP)) that each computing device may use to access data resources, a sending speed or rate to be provided to each of the computing devices, one or more vendor-specific attributes (VSAs), and / or a specific VLAN to be assigned to each computing device. In some examples, the VSA may include bandwidth for incoming and / or outgoing traffic and download and / or upload speeds. The access control lists or policies may be stored in the database 112 of the server 110 rather than on other computing components such as the router 120, so that the server 110 may centrally update the access control lists or policies and propagate any updates to other computing components in the network.

[0029] Each of the computing components may include one or more hardware processors and logic that implements instructions to perform the functions of the computing components. For example, the server 110 may include or be associated with one or more hardware processors and logic 114 that implements instructions or protocols to perform the functions of the server 110. The logic 114 may execute instructions to perform one or more steps as follows: generate a SEP to establish a connection between a host device (e.g., a storage device) of a domain object and a computing device (e.g., a client device) of a network; and perform operations to read and write data between the host device and the computing device.

[0030] Figure 2 2 is a diagram of an example network 200 through which communication connections are implemented according to various examples of the present disclosure. In some examples, network 200 may include or include one or more computing components, which may include server 202, network device 220, network controller 222, access point 224, and any of devices 210, 230, 232, 234, and 236. Figure 2 The information exchange between the components can be illustrated. The server 202 can be, for example, Figure 1 Server 110.

[0031] The network 200 may include a plurality of network devices, including a network device 220, a network controller 222, and an access point 224. The devices 210, 230, 232, 234, and 236 may be any of computing devices such as computers, storage devices, mobile phones, tablet devices, and the like. The devices 210, 230, 232, 234, and 236 may be implemented as Figure 1 Any of the computing devices 130 to 139. The network device 220 may be implemented as Figure 1 The router 120, switch 122, or network controller 124 of the present invention. The network device 220 may be configured to connect various computing components in the network, such as the device 210, device 230, device 232, device 234, device 236, network controller 222, access point 224, and server 202 in the network 200. The server 202 may further include or be associated with a database or cache 204 (hereinafter "database 204") that stores attributes of specific client devices, storage devices, and access control lists or policies associated with the devices 210, 230, 232, 234, and 236 that are connected to the network (such as the network 200) via an access point (such as the access point 224).

[0032] In some examples, devices 210, 230, 232, 234, and 236 can access the Internet wirelessly via Wi-Fi (e.g., IEEE 802.11), Bluetooth (e.g., IEEE 802,15,1), or a cellular connection (e.g., a 5th generation cellular network of long-term evolution, etc.) to wirelessly access server 202 via network device 220 and / or access point 210. Server 202 can implement software and / or hardware, such as a network server, an application server, a communication server, a database server, etc. Server 202 can access the Internet via Wi-Fi, Bluetooth, a telephone line, or a LAN / WLAN network interface. In other examples, network device 220 can be an intranet (e.g., a private network), and devices 210, 230, 232, 234, and 236 can access the intranet wirelessly via network device 220 to access data files, storage files, or other enterprise data. In some cases, network device 220 may be a network link (e.g., Wi-Fi, Ethernet port, router, switch, etc.) that allows multiple computing components to communicate with each other. Network controller 222 and access point 224 may be configured to allow computing components in the network (such as device 210, device 230, device 232, device 234, device 236, and server 202) to connect through network device 220. In this example, network device 220 may establish device-to-device communication between device 210 and each of devices 230, 232, 234, and 236.

[0033] In some examples, a communication connection may be established between device 210 and each of the other devices 230, 232, 234, and 236 in network 200. Each of devices 210, 230, 232, 234, and 236 may have a communication link with network 200, such as communication links 240, 250, 252, 254, and 256. Server 202 may be configured to act as a link to establish device-to-device communication between device 210 and each of devices 230, 232, 234, and 236 using communication links 240, 250, 252, 254, and 256. Server 202 and devices 210, 230, 232, 234, and 236 may be connected to each other in network 200 via network device 220, network controller 222, and / or access point 224. Network device 220, network controller 222, and access point 224 may be a few of a plurality of network devices in a network, including other network devices, controllers, and access points. Network 200 may allow each computing device in the network to communicate with each other. Device 210 may be connected to any of a plurality of computing devices in network 200. Devices 210, 230, 232, 234, and 236 may be a few of a plurality of devices that may be connected to access point 224 in network 200.

[0034] Figure 3 3 is an illustration of an example process 300 for implementing a communication connection in a network according to various examples of the present disclosure. A server 310, a host device 312, a SEP 314, and a client device 316 are illustrated.

[0035] At block 320, host device 312 may send a request message to server 310 of the network. The request message may include a request to establish a communication connection between host device 312 and one or more client devices in the network, such as client device 316. In some examples, SEP 314 of the domain object may not be generated prior to receiving the request. In some examples, the communication connection between host device 312 and client device 316 may be established within a time threshold of receiving the request message.

[0036] At block 330, the server 310 may receive a message from the host device 312. The message from the host device 312 may include a data packet (e.g., unicast and multicast), a report (e.g., a buffer status report (BSR), a quality of service (QoS) NULL data frame, a transmission control protocol (TCP) acknowledgment, etc.). The message received by the server 310 from the host device 312 may include a request to establish a communication connection between the host device 312 and one or more client devices (such as the client device 316) in the network.

[0037] At block 340, the server 310 may read the request message received from the host device 312 and extract information from the request message. After the server 310 receives the request message from the host device 312 requesting to establish a communication connection with other devices in the network (such as the client device 316), the server 310 may read the request message to obtain identity information of the host device 312. The identity information may include, but is not limited to, a source device address, a destination device address, a source device port, a destination device port, and / or attributes such as a source device type, a destination device type, a frame length or packet size sent, a number of frames or packets sent, a change in the size of a frame or packet, a change in the length of a frame or packet, a frequency of packets, a frequency of frames, and / or an overall size of data within a frame or packet. In some examples, the request message may include an advertisement or a query for a service such as video streaming, video conferencing, or Internet Protocol Television (IPTV).

[0038] Server 310 can record the identity information and store it in a database. For example, once server 310 has read the request message at the first time and obtained the identity information from the request message, server 310 can store the information at the second time. Server 310 can further analyze the identity information of the request message and the stored identity information of the message previously sent in the database. Server 310 monitors the total amount of messages sent from each device in the network by, for example, allowing the identity information to be stored in the database and using the identity information stored in the server 310 analysis database and the identity information of the request message. The message monitored can include messages going to / from host device 312 and client device 316.

[0039] At block 350, server 310 may determine a communication link for a client device in the network, including client device 316. The network may include one or more client devices, such as client device 316. Each client device in the network may include a CPU core that runs operations of the corresponding client device. Each CPU core of a client device in the network may have a corresponding CPU core connection with the network. Each of the CPU core connections may be used as a communication connection between a corresponding CPU core of a client device and another device connected to the network, such as host device 312.

[0040] The network may have a certain number (N) of client devices and N CPU cores and CPU core connections. Each of the CPU core connections may include a non-volatile memory express (NVMe) queue pair. The NVMe queue pair may be used to transmit messages and data between the corresponding CPU core of the client device and other devices connected to the network using the corresponding CPU core connection. Using one or more hardware processors, databases, and logic, the server 310 may determine the number (N) of client devices (such as client device 316) currently associated with the network. By determining the number (N) of client devices, the server 310 may then determine the number (N) of CPU cores, CPU core connections, and NVMe queue pairs.

[0041] At block 360, server 310 may generate a communication link for host device 312. A set of communication links, such as send and receive contexts, may be generated for host device 312. The number of communication links in the set for host device 312 may be the same as the number (N) of client devices currently associated with the network.

[0042] At block 370, server 310 may generate a SEP for host device 312. To establish a SEP (such as SEP 314) for host device 312, an address vector may be generated for host device 312. The address vector may store remote addresses of client devices (such as client device 316) in the network to allow host device 312 to communicate with each of the client devices in the network. The address vector may be used to generate SEP 314 for host device 312 for use in establishing a communication connection between host device 312 and each of the client devices in the network (including client device 316). SEP 314 may be implemented as a network device in the network, such as a client device 316. Figure 2 The network device 220 and Figure 1 The SEP 314 may use the address vector to establish a connectionless communication structure between devices (such as between the host device 312 and the client device 316). The SEP 314 may be implemented as Figure 2 Any of devices 220, 222 or 224.

[0043] At block 380, SEP 314 may be used to establish a communication connection between host device 312 and client device 316. Server 310 may use SEP 314 to map a set of communication links for host device 312 with each of the communication links of a client device in the network, such as client device 316. Each of the set of communication links for host device 312 may be aligned and linked to a different communication link of a client device in the network to establish a 1-to-1 mapping of communication connections between host device 312 and each of the client devices, such as client device 316.

[0044] At block 390, once a communication connection is established between the host device 312 and each of the client devices, such as client device 316, SEP 314 may allow messages to be sent and received between the host device 312 and any of the client devices, such as client device 316. SEP 314 may be used to transfer data between the host device 312 and each of the client devices in the network, including client device 316. SEP 314 may also be used to track the performance of operations, such as send operations, receive operations, read operations, and write operations, that occur between the host device 312 and each of the client devices in the network. Many variations are possible.

[0045] At block 392, the host device 312 receives the data and sends the data to the client device 316. At block 394, the client device 316 receives the data and sends the data to the host device 312. For example, the host device 312 may send a message including, for example, a command to perform a read operation or a write operation. A read operation may include, for example, a request to cause the host device 312 to read data from a client device 316 in the network. A write operation may include, for example, a request to write data from the host device 312 to the client device 316 in the network.

[0046] Figure 4 406. As described herein, SEP 460 may be generated and used to establish a data transfer connection between a domain object 410 and a plurality of CPU cores 420 of a device 406 in a network. In some examples, one of CPU cores 420 of network 400 may request to establish a connection with a domain object, such as domain object 410. Domain object 410 may represent a domain that may include one or more devices, such as host device 402, that exist within the same domain. A message may be sent from network 400 to domain object 410. The message may include, for example, a request to establish a communication connection between domain object 410 and network 400. The message may be sent from a CPU core of network 400 that wants to establish a communication connection with host device 402 in domain object 410. The message may be sent from each of the CPU cores of network 400 that wants to establish a communication connection with one or more host devices in domain object 410. Many variations are possible.

[0047] The network 400 may include one or more computing devices, such as client computing devices 406. Each client computing device 406 in the network 400 may include a CPU core, such as CPU core 420. The network 400 may include a certain number (N) of CPU cores 420 based on the number (N) of client computing devices 406 in the network 400. Each CPU core 420 may have a corresponding connection with the network 400. The connection of all number (N) of CPU core connections of the N CPU cores 420 may be represented by CPU core connections 430. Each of the connections of the CPU core connections 430 may be used as a communication connection between the corresponding CPU core 420 of the client computing device 406 and another device connected to the network 400. Each of the CPU core connections 430 may include an NVMe queue pair. The NVMe queue pair may be used to transmit messages and data between the corresponding CPU core 420 of the client computing device 406 and other devices connected to the network 400.

[0048] A communication connection, such as domain object connection 440, may be generated between network 400 and domain object 410. Domain object connection 440 may allow messages and data to be transmitted between domain object 410 and network 400. Domain object connection 440 may allow messages and data to be transmitted between each host device 402 of domain object 410 and network 400. Domain object connection 440 may include multiple sets of send and receive contexts for domain objects. The number of sets of send and receive contexts may be the same as the number (N) of CPU cores 420 of devices in network 400. Each set of the N sets of send and receive contexts in domain object connection 440 may be aligned with one of the N CPU core connections 430. Aligning a set of send and receive contexts with a CPU core connection may associate and map the communication connections with each other.

[0049] A certain number (N) of completion queues 450 may be generated for a domain object 410. Each of the N completion queues 450 may be used to keep track of events or operations (hereinafter "operations") that have been completed between the domain object 410 and the network 400. For example, operations may include sending messages or data, receiving messages or data, reading messages or data, and writing messages or data between the domain object 410 and the network 400. The N completion queues 450 may record each operation that has been completed between the domain object 410 and the network 400. This may include recording each operation that has been completed between each host device 402 of the domain object 410 and the network 400.

[0050] Each of the N completion queues 450 of the domain object 410 may be associated with a different set of the N sets of send and receive contexts of the domain object connection 440. Each of the N completion queues 450 may be further associated with a respective one of the CPU core connections 430 that is aligned with the respective set of send and receive contexts of the domain object connection 440. Each of the N completion queues 450 may thus be associated with a different CPU core 420 of the network 400. In this manner, each of the N completion queues 450 will track and record operations performed and completed between the domain object 410 and the respective CPU core 420 of the respective device in the network 400. This may allow for the recording and separation of records of operations occurring between the domain object 410 and each CPU core 420 of each device 406 in the network 400.

[0051] A scalable endpoint (SEP), such as SEP 460, may be generated for domain object 410. SEP 460 may link domain object connection 440 of domain object 410 to CPU core connection 430 of all CPU cores 420 of devices 406 in network 400. SEP 460 may increase the amount of memory required to establish communication connections between domain object 410 and all CPU cores 420 in network 400. SEP 460 may be very similar to the concept of "multi-queue" in networking, and may improve the performance of multi-threaded and parallel applications by allowing threads to access independent send and receive context queues. SEP 460 may have the added benefit of using a single address for multiple sets of send and receive contexts. By using a single base address (also referred to as a single transfer-level address), the memory requirements required to store remotely addressed data may be reduced.

[0052] A single base address of SEP 460 may be generated for N sets of send and receive contexts for domain object connection 440. The single base address of SEP 460 may be an address vector. The address vector of SEP 460 may be used to store remote addresses of a connectionless structure. The address vector may store remote addresses of devices in network 400, including, for example, remote addresses of N CPU cores 420 of device 406 in network 400, remote addresses of domain object 410, and remote addresses of one or more host devices 402 of domain object 410. The address vector may reduce the memory requirements required to store remotely addressed data. The address vector of SEP 460 may be used to generate SEP 460. SEP 460 may also include a network address and a designated port of domain object 410, indicating that SEP 460 is designated for domain object 410. SEP 460 can establish a communication connection between each of one or more host devices 402 in domain object 410 and each of CPU cores 420 of network 400, wherein only a small and constant amount of memory is used for all queues of all connections to all CPU cores in the network. In this way, only a small and constant amount of memory is required to establish a data transfer connection between domain objects and CPU cores, regardless of the number of CPU cores of devices in the network.

[0053] SEP 460 may be used to establish a complete communication connection between domain object 410 and each of N CPU cores 420 of device 406 in network 400. SEP 460 may be used to establish a complete communication connection between each of one or more host devices 402 in domain object 410 and each of N CPU cores 420 of device 406 in network 400. SEP 460 may establish a connectionless communication structure between domain object 410 and each of N CPU cores 420 in network 400.

[0054] To establish a complete communication connection, SEP 460 may be used to map N sets of send and receive contexts of domain object connection 440 to NVMe queue pairs of CPU core connection 430. Each of the N sets of send and receive contexts of domain object connection 440 may be mapped to a different NVMe queue pair of CPU core connection 430 to establish a 1-to-1 mapping connection between domain object 410 and each of CPU cores 420 in network 400. In this manner, one or more messages and data transmitted between domain object 410 and CPU core 420 of device 406 of network 400 may use a specific set of send and receive contexts in domain object connection 440 that is associated with a connection of CPU core connection 430 of a corresponding CPU core 420 of network 400. For example, host device 402 of domain object 410 may want to communicate with CPU core 420(1) and transfer data between the two. SEP 460 may use the connections of CPU core connection 430 associated with CPU core 420(1) and the associated send and receive contexts of domain object connection 440 to transfer data between host device 402 of domain object 410 and CPU core 420(1). Completion queues of N completion queues 450 associated with CPU core 420(1) may be used to track and record operations performed and completed between each of one or more host devices 402 of domain object 410 and a corresponding CPU core 420(1) of network 400.

[0055] The SEP 460 may allow one or more messages and data to be transmitted between a host device 402 of a domain object 410 and a CPU core 420 of a device 406 in a network 400. Each of the communication connections established using the SEP 460 may represent a 1-to-1 mapping between a domain object 410 and one of the CPU cores 420 in the network 400. Each host device 402 of the domain object 410 may use the same communication connection to send and receive one or more messages and data to and from a corresponding CPU core 420 in the network 400 associated with the communication connection. The SEP 460 may use only a small and constant amount of memory for all queues of all connections between the domain object 410 and all of the CPU cores 420 in the network 400. A single host device 402 of the domain object 410 may have a communication connection with each of the CPU cores 420 in the network 400 without using a large amount of memory and without negatively impacting the processing performance of the communications (e.g., data transfers) being performed.

[0056] Once a communication connection is established between the domain object 410 and each of the CPU cores 420 of the devices 406 in the network 400 using the SEP 460, messages and data may be transferred (e.g., sent, received, transmitted, etc.) between the host device 402 of the domain object 410 and the CPU core 420 of the network 400. An event queue 415 for the domain object 410 may also be generated. The event queue 415 may be used to track data transfers flowing through the domain object 410. Data transfers flowing through the domain object 410 may include any data transfers flowing from or to any of the one or more host devices 402 in the domain object 410. In this manner, the event queue 415 may be used to track all data transfers occurring with respect to the domain object 410 to the network 400.

[0057] It should be understood that unless otherwise stated, there may be additional, fewer, or alternative steps performed in similar or alternative sequences or in parallel within the scope of the various examples discussed herein. The network 400 may be implemented, for example, as Figure 1 Server 110, Figure 2 Server 202, Figure 3 Server 310, Figure 5 Network 500, Figure 6 Network 600, Figure 7 The computing unit 700 and Figure 8 The computing component 800.

[0058] Figure 5 is a diagram of an example network 500 for performing push operations between a host device (such as a storage device) and a client CPU core in a network using a SEP according to an example described in the present disclosure. When a communication connection is established between a domain object (such as domain object 510) and a CPU core of a device in the network (such as CPU core 520 of network 500) using a SEP (such as SEP 560), one or more operations (such as push operations) may be performed. Network 500, domain object 510, CPU core 520, and SEP 560 may be implemented as, for example, Figure 4 A network 400 , a domain object 410 , a CPU core 420 , and a SEP 460 .

[0059] The push operation may include sending a message, such as message 530, from a client CPU core, such as client CPU core 525, of CPU core 520 in network 500 to a host device, such as host device 515, of domain object 510 using SEP 560. Message 530 may include a read operation, which may include a command to read data from host device 515. Message 530 may be sent from client CPU core 525 to host device 515 using SEP 560. To send message 530 from client CPU core 525 to host device 515, SEP 560 may use send and receive contexts and a CPU core connection associated with client CPU core 525. The CPU core connection of client CPU core 525 may have an NVMe queue pair for transferring data between SEP 560 and client CPU core 525.

[0060] The message 530 may include a unique key tag. The unique key tag may include information such as where the message 530 is sent from, where the message 530 is to be transmitted, data to be collected, data to be written, operations to be performed, etc. The host device 515 may receive the message 530 and may perform a write operation on the data requested from the host device 515. The data of the host device 515 may be written into a client-tagged buffer for the client CPU core to read the data of the host device 515. A first completion event may be generated after the write operation of the data of the host device 515 is completed. The first completion event may indicate that the host device 515 received the message 530 and provided the requested data. The first completion event may be recorded in a completion queue 550 associated with the client CPU core 525.

[0061] The client-tagged buffers with data of the host device 515 may be sent from the host device 515 to the client CPU core 525 using the SEP 560. The client-tagged buffers may be stored in a message, such as message 540. To send the message 540 from the host device 515 to the client CPU core 515, the SEP 560 may use the send and receive contexts and the CPU core connection associated with the client CPU core 525. The CPU core connection of the client CPU core 525 may have an NVMe queue pair for transferring data between the SEP 560 and the client CPU core 525. Upon receiving the message 540, the client CPU core 525 may read the message 540 to extract the data in the client-tagged buffer stored in the message 540.

[0062] A second completion event may be generated after the client CPU core 525 receives a message 540 of a client-tagged buffer including data for the host device 515. The second completion event may indicate that the client CPU core 525 received the requested data for the host device 515. The second completion event may indicate that the client CPU core 525 read the requested data for the host device 515. The second completion event may be recorded in a completion queue 550 associated with the client CPU core 525. Any number of completion events may be generated and stored in the completion queue 550 to indicate the completion of one or more operations between one or more host devices (such as the host device 515) of the domain object 510 and the client CPU core 525 of the network 500.

[0063] It should be understood that unless otherwise stated, there may be additional, fewer, or alternative steps performed in similar or alternative sequences or in parallel within the scope of the various examples discussed herein. Network 500 may be implemented, for example, as Figure 1 Server 110, Figure 2 Server 202, Figure 3 Server 310, Figure 4 Network 400, Figure 6 Network 600, Figure 7 The computing unit 700 and Figure 8 The computing component 800.

[0064] Figure 6 is a diagram of an example network 600 for performing a pull operation between a host device and a client CPU core using a SEP according to an example described in the present disclosure. When a communication connection is established between a domain object (such as domain object 610) and a CPU core of a device in the network (such as CPU core 620 of network 600) using a SEP (such as SEP 660), one or more operations (such as a pull operation) may be performed. Network 600, domain object 610, CPU core 620, and SEP 660 may be implemented as, for example, Figure 3 network 300, domain object 310, CPU core 320 and SEP 360 and Figure 5 A network 500 , a domain object 510 , a CPU core 520 , and a SEP 560 .

[0065] The pull operation may include sending a message, such as message 630, from a client CPU core, such as client CPU core 625, of CPU core 620 in network 600 to a host device, such as host device 615, of domain object 610 using SEP 660. Message 630 may include a write operation, which may include a command to write data from client CPU core 625 to host device 615. Message 630 may be sent from client CPU core 625 to host device 615 using SEP 660. To send message 630 from client CPU core 625 to host device 615, SEP 660 may use send and receive contexts and a CPU core connection associated with client CPU core 625. The CPU core connection of client CPU core 625 may have an NVMe queue pair for transferring data between SEP 660 and client CPU core 625.

[0066] The message 630 may include a unique key tag. The unique key tag may include information such as where the message 630 is sent from, where the message 630 is to be transmitted, data to be collected, data to be written, operations to be performed, etc. The message 630 may include a client-tagged buffer that may include data requested to be written to the host device 615. A first completion event may be generated upon completion of sending the message 630 from the client CPU core 625 to the host device 615 using the SEP 660. The first completion event may indicate that the host device 615 received the message 630. The first completion event may indicate that the host device 615 read the message 630. The first completion event may be recorded in a completion queue 650 associated with the client CPU core 625.

[0067] The host device 615 may receive the message 630 and may perform a read operation on the message 630. The host device 615 may read the message 630 to extract the data in the client-tagged buffer stored in the message 630. A second completion event may be generated after the host device 615 receives the message 630 of the client-tagged buffer of data. The second completion event may indicate that the host device 615 received the data requested to be written. The second completion event may indicate that the host device 615 read the message 630 and obtained the data in the client-tagged buffer. The second completion event may be recorded in a completion queue 650 associated with the client CPU core 625.

[0068] A write operation may be performed on the data extracted from the client-tagged buffer in the message 630 to the host device 615. The extracted data may be written to the host device 615 to store the extracted data in a database of the host device 615. A third completion event may be generated after the write operation of the data to the host device 615 is completed. The third completion event may indicate that the host device 615 stores the data from the client-tagged buffer of the message 630 sent from the client CPU core 625. The third completion event may be recorded in a completion queue 650 associated with the client CPU core 625.

[0069] Any number of completion events may be generated and stored in the completion queue 650 to indicate the completion of one or more operations between one or more devices (such as the host device 615) of the domain object 610 and the client CPU core 625 of the network 600. The first, second, and third completion events may be combined into fewer completion events, where each completion event indicates the completion of multiple operations between the domain object 610 and the target CPU core 625. The first, second, and third completion events may be separated into more completion events, where each completion event indicates the completion of a single operation between the domain object 610 and the client CPU core 625. Many variations are possible.

[0070] It should be understood that unless otherwise stated, there may be additional, fewer, or alternative steps performed in similar or alternative sequences or in parallel within the scope of the various examples discussed herein. The network 600 may be implemented, for example, as Figure 1 Server 110, Figure 2 Server 202, Figure 3 Server 310, Figure 4 Network 400, Figure 5 Network 500, Figure 7 The computing unit 700 and Figure 8 The computing component 800.

[0071] Figure 7 An example computing component 700 is illustrated, which includes one or more hardware processors 702 and a machine-readable storage medium 704 storing a set of machine-readable / machine-executable instructions, which when executed cause the hardware processor(s) 702 to perform an illustrative method of generating a SEP to establish a data transmission connection between a domain object consisting of multiple devices (such as a host device) and all CPU cores of devices in a network (such as a client device). It should be understood that unless otherwise stated, there may be additional, fewer, or alternative steps performed in a similar or alternative order or in parallel within the scope of the various examples discussed herein. The computing component 700 may be implemented, for example, as Figure 1Server 110, Figure 2 Server 202, Figure 3 Server 310 and Figure 8 The computing component 800. Figure 7 Some aspects previously described are summarized and further elaborated.

[0072] In some examples, hardware processor(s) 702 may execute machine-readable / machine-executable instructions stored in machine-readable storage medium 704 to receive a request from a network to establish a connection with a domain object. A domain object may represent a domain that may include multiple devices (such as host devices) that exist within the same domain. A message may be sent from the network to the domain object. The message may include, for example, a request to establish a communication connection between the domain object and the network. The message may be sent from a computing device in the network that wants to establish a communication connection to a host device in the domain object. A message may be sent from each computing device in the network that wants to establish a communication connection with one or more host devices in the domain object to each of the host devices in the domain object. Many variations are possible.

[0073] The network may include multiple computing devices, such as client devices. Each computing device in the network may have a CPU core, and each CPU core may have a CPU core connection to the network. The CPU core connection may be used as a communication connection between a corresponding CPU core of a computing device and another device connected to the network. Establishing a communication connection between the network and the domain object may establish a communication connection between each host device of the domain object and each CPU core of each computing device in the network. Establishing a communication connection between the network and the domain object may allow messages and data to be transmitted between each host device of the domain object and each CPU core of each computing device in the network. Many variations are possible.

[0074] At instruction 708, the hardware processor(s) 702 may execute machine-readable / machine-executable instructions stored in the machine-readable storage medium 704 to determine the number (N) of CPU cores in the network. After receiving a message requesting to establish a connection between a domain object and a network, the number (N) of CPU cores in the network may be determined. The network may include multiple computing devices, such as client devices. Each computing device in the network may have a CPU core. The number (N) of computing devices in the network may be established. The number (N) of computing devices may be the same as the number (N) of CPU cores. Each CPU core may have a CPU core connection to the network. The CPU core connection of a device may be used as a communication connection between a corresponding device and another device connected to the network via a communication connection. Each CPU core connection may include an NVMe queue pair. The NVMe queue pair may be used to transfer data between a corresponding CPU core of the device and another device connected to the network.

[0075] At instruction 710, the hardware processor(s) 702 may execute machine-readable / machine-executable instructions stored in the machine-readable storage medium 704 to generate N sets of send and receive contexts for the domain object. After determining the number (N) of computing devices, CPU cores, and CPU core connections in the network, the N sets of send and receive contexts may be generated. The N sets of send and receive contexts may be communication connections from the domain object to the network. The N sets of send and receive contexts may be the same number of communication connections as the N CPU core connections. Each of the N sets of send and receive contexts may be aligned with one of the N CPU core connections. Aligning a set of send and receive contexts with a CPU core connection may associate the communication connections with each other.

[0076] In some examples, the hardware processor(s) 702 may execute machine-readable / machine-executable instructions stored in the machine-readable storage medium 704 to generate N completion queues for a domain object. The completion queues may be used to keep track of completions of events or operations that occur with respect to a domain object to a network. Events or operations may include, for example, sending messages or data, receiving messages or data, reading messages or data, and writing messages or data. N completion queues may be generated for a domain object. Each completion queue of a domain object may be associated with a different group of N groups of send and receive contexts. Each completion queue of a domain object may be further associated with a CPU core connection aligned with a corresponding group of send and receive contexts. In this manner, each completion queue will track events or operations performed between a domain object and a corresponding CPU core of a corresponding device in the network. This may allow for recording and separation of records of events or operations that occur between a host device of a domain object and each CPU core of each device in the network.

[0077] In some examples, hardware processor(s) 702 may execute machine-readable / machine-executable instructions stored in machine-readable storage medium 704 to generate an address vector of a domain object for storing remote addresses of a connectionless structure. The address vector may store remote addresses of devices in a network, including, for example, remote addresses of all CPU cores of devices in the network, remote addresses of domain objects, and remote addresses of all host devices of the domain object. The address vector may reduce memory requirements required to store remotely addressed data. The address vector of a domain object may be generated for use in generating a SEP.

[0078] At instruction 712, (multiple) hardware processors 702 may execute machine-readable / machine-executable instructions stored in machine-readable storage medium 704 to generate a SEP for a domain object using an address vector. The address vector of a domain object may be used to generate a SEP. SEP may be very similar to the concept of "multiple queues" in networking, and may improve the performance of multi-threaded and parallel applications by allowing threads to access independent send and receive context queues. SEP may have the additional benefit of allowing the creation of multiple send and receive contexts, the addresses of which may be calculated to use a single base address. The single base address of the SEP may be an address vector. SEP may increase the amount of memory required to establish a data transfer connection between multiple host devices in a domain object and a network consisting of multiple CPU cores of a computing device. SEP may also reduce the overhead processing requirements associated with memory registration in remote direct memory access (RDMA) operations. SEP may use address vectors to establish a connectionless communication structure between a host device of a domain object and a CPU core of a device in a network. SEP may also include a network address and a designated port of a domain object.

[0079] At instruction 714, the (multiple) hardware processors 702 may execute machine-readable / machine-executable instructions stored in the machine-readable storage medium 704 to establish a connection between the network and the domain object using the SEP. The SEP may be used to establish a communication connection between all CPU cores of devices in the network and all host devices of the domain object. The communication connection established by using the SEP may be a connectionless communication structure. The communication connection established by using the SEP may be created by mapping N groups of send and receive contexts of the domain object to NVMe queue pairs of N CPU core connections. In this way, messages or data transmitted between the CPU cores of a device of the network and the host device of the domain object may use a specific group of send and receive context communication connections that are associated with the CPU core connections of the corresponding CPU cores of the corresponding devices of the network.

[0080] At instruction 716, (multiple) hardware processors 702 may execute machine-readable / machine-executable instructions stored in machine-readable storage media 704 to transmit data over a network using SEP using a mapping between a transmit and receive context to an NVMe queue pair and a domain object. For example, a communication connection may be established by performing a mapping. Establishing a communication connection using SEP may allow messages and data to be transmitted between a host device of a domain object and a CPU core of a device in a network. Each of the communication connections established using SEP may represent a 1-to-1 mapping between a domain object and a CPU core of a device in a network. Each host device of a domain object may use the same communication connection to send and receive messages and data to a corresponding CPU core of a device in a network associated with the communication connection. In this way, using SEP may use a small and constant amount of memory for all queues of all connections between a domain object and all CPU cores of a device in a network. A single client device of a domain object may have a communication connection with each CPU core of each device in a network without using a large amount of memory and without negatively affecting the processing performance of the communication (such as data transmission) performed.

[0081] Once a communication connection is established between a domain object and all CPU cores of devices in the network using SEP, messages and data can be sent and received between the host device of the domain object and the CPU core of the device in the network. In some examples, an event queue can be generated for the domain object. The event queue can be used to track data transmissions flowing through the domain object. Data transmissions flowing through the domain object can include any data transmission flowing from or to any of the host devices in the domain object. In this way, the event queue can be used to track all data transmissions that occur with respect to the domain object to the network.

[0082] When establishing a communication connection between a domain object and a CPU core of a device in a network using SEP, one or more operations, such as a push operation and a pull operation, may be performed. The push operation may include sending a message from a client CPU core of a client device in the network to a host device of the domain object using SEP. The message may include a command to read data from the host device. The message may be sent from the client CPU core to the host device using SEP. The message may include a unique key tag. The unique key tag may include information such as where the message is sent from, where the message is to be transmitted, data to be collected, etc. The host device of the domain object may receive the message and may perform a write operation on the data requested from the host device of the domain object. The data of the host device may be written into a client-tagged buffer for the client CPU core to read the data of the host device. A first completion event may be generated after completing the write operation of the data of the host device. The first completion event may indicate that the host device has received the message and provided the requested data. The first completion event may be recorded in N completion queues associated with the client CPU core. A client-tagged buffer with data of the host device may be sent from the host device to the client CPU core using SEP. A second completion event may be generated after the client CPU core receives the client-tagged buffer of data of the host device. The second completion event may indicate that the client CPU core received the requested data from the host device. The second completion event may indicate that the client CPU core read the requested data from the host device. The second completion event may be recorded in N completion queues associated with the client CPU core.

[0083] The pull operation may include sending a message from a client CPU core in the network to a host device of a domain object using SEP. The message may include a command to write data to the host device of the domain object. The message may be sent from the client CPU core to the host device using SEP. The message may include a unique key tag. The unique key tag may include information such as where the message is sent from, where the message is to be transmitted, data to be collected, etc. The message may include a client-tagged buffer that may include data requested to be written to the host device. A first completion event may be generated upon completing the sending of the message from the client CPU core to the host device using SEP. The first completion event may indicate that the host device received the message. The first completion event may indicate that the host device read the message. The first completion event may be recorded in N completion queues associated with the client CPU core. The host device may receive the message and may perform a read operation on the client-tagged buffer in the message. Performing a read operation on the client-tagged buffer may extract the data requested to be written to the host device. After completing the read operation, a write operation may be performed. The write operation may include writing data from the client-tagged buffer of the client CPU core to the host device. A second completion event may be generated after the host device receives the client-tagged buffer of the data. The second completion event may indicate that the host device receives the data requested to be written. The second completion event may indicate that the host device reads the data. A third completion event may be generated after completing the write operation of the data to the host device. The third completion event may indicate that the host device stores the data from the buffer marked by the client. The second completion event and the third completion event may be recorded in N completion queues associated with the client CPU core. The first, second, and third completion events may be combined into fewer completion events, wherein each completion event indicates the completion of multiple operations between the domain object and the client CPU core. The first, second, and third completion events may be separated into more completion events, wherein each completion event indicates the completion of a single operation between the domain object and the client CPU core. Many variations are possible.

[0084] Using SEP can increase the amount of memory required to establish data transfer connections between host devices in the same domain object and all CPU cores in the network. Because SEP has a single transfer-level address (i.e., address vector), the memory requirements required to store remotely addressed data can be reduced compared to traditional systems. SEP can establish a data transfer connection between each of multiple host devices in a domain object and all CPU cores of devices in the network, where only a small and constant amount of memory is used for all queues for all connections to all CPU cores in the network. In this way, a small and constant amount of memory can be used to establish a data transfer connection between a domain object and a CPU core, regardless of the number of CPU cores of the device in the network. By also using SEP to align the NVMe queue pairs of a CPU core with the corresponding send / receive context and completion queues of a domain object, SEP can create a 1-to-1 mapping between a domain object and each of the CPU cores of a device in the network.

[0085] The foregoing description provides specific details for a thorough understanding and enabling description of various examples of the present technology. Terms used are intended to be interpreted in their broadest reasonable manner, even when used in conjunction with detailed descriptions of certain examples of the present disclosure.

[0086] It should be understood that the foregoing disclosure is not limited to specific compositions or process steps, and may vary accordingly. Unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" as used herein are intended to include the plural forms as well. In addition, to the extent that the terms "including", "includes", "having", "has", "with", "such as", or variations thereof are used in the specification and / or claims, these terms are not limiting and are intended to be inclusive in a manner similar to the term "comprising". Unless otherwise specified, examples in the specification that recite "comprising" various components are also contemplated as "consisting of" or "consisting essentially of" the components.

[0087] Figure 8 A block diagram of an example computer system 800 is shown in which various examples of the present disclosure may be implemented. The computer system 800 may be implemented as Figure 1 Server 110, Figure 2 Server 202, Figure 3 Server 310 and Figure 7The computing component 700 of the present invention. The computer system 800 may include a bus 802 or other communication mechanism for transmitting information, and one or more hardware processors 804 coupled to the bus 802 for processing information. The (multiple) hardware processors 804 may be, for example, one or more general-purpose microprocessors. The computer system 800 may be an example of a SEP, a client-target communication, or a similar device. The computer system 800 may use the bus 802 and the (multiple) hardware processors 804 to perform operations or transmit instructions. For example, the processor 804 may transmit a request to access instructions stored in the memory 806 or ROM 808 via the bus 802 to receive a message from a host device, read a message from a host device, generate a message for a client device, or send a message for a client device. In another example, the processor 804 may transmit a request to access instructions stored in the memory 806 or ROM 808 via the bus 802 to generate a SEP, determine a communication link for a client device in a network, generate a communication link for a host device in a network, or use a SEP to establish a communication connection between a host device and a client device in a network. Various operations may be transmitted via bus 802 without departing from the essence of the present disclosure.

[0088] The computer system 800 may also include a main memory 806 coupled to the bus 802 for storing information and instructions to be executed by the (multiple) hardware processors 804, such as a random access memory (RAM), a cache, and / or other dynamic storage devices. The main memory 806 may also be used to store temporary variables or other intermediate information during the execution of instructions by the (multiple) hardware processors 804. When stored in a storage medium accessible to the (multiple) hardware processors 804, such instructions make the computer system 800 a special-purpose machine that can be customized to perform the operations specified in the instructions. These instructions may include, for example, generating a SEP, determining a communication link for a client device in a network, generating a communication link for a host device in a network, or using a SEP to establish a communication connection between a host device and a client device in a network.

[0089] The computer system 800 may further include a read-only memory (ROM) 808 or other static storage device coupled to the bus 802 for storing static information and instructions for the hardware processor(s) 804. A storage device 810, such as a magnetic disk, optical disk, or USB thumb drive (flash drive), may be provided and coupled to the bus 802 for storing information and instructions. The ROM 808 and storage device 810 may store information such as identity information of a host device, a client device, an address vector, a SEP, etc. The ROM 808 and storage device 810 may store information and instructions for performing operations. These operations may include, for example, generating a SEP, determining a communication link for a client device in a network, generating a communication link for a host device in a network, or establishing a communication connection between a host device and a client device in a network using a SEP.

[0090] The computer system 800 may further include at least one network interface 812 coupled to the bus 802 for connecting the computer system 800 to at least one network, such as a network interface controller module (NIC), a network adapter, etc., or a combination thereof. The network interface 812 may provide a bidirectional data communication coupling with one or more network links, which are connected to one or more local networks. For example, the network interface 812 may be an integrated services digital network (ISDN) card, a cable modem, a satellite modem, or a modem for providing a data communication connection with a telephone line of a corresponding type. As another example, the network interface 812 may be a local area network (LAN) card for providing a data communication connection with a compatible LAN (or a WAN component for communicating with a WAN). A wireless link may also be implemented. In any such implementation, the network interface 812 may send and receive an electrical signal, an electromagnetic signal, or an optical signal carrying a digital data stream representing various types of information.

[0091] A network link can generally provide data communications with other data devices through one or more networks. For example, a network link can provide a connection to a host computer or to data devices operated by an Internet Service Provider (ISP) through a local network. The ISP, in turn, can provide data communication services through the global packet data communication network now commonly referred to as the "Internet". Both the local network and the Internet use electrical, electromagnetic, or optical signals that can carry digital data streams. Signals through various networks, as well as signals on network links and signals through network interface 812 (which can carry digital data to and from computer system 800) are example forms of transmission media.

[0092] Computer system 800 can send messages and receive data (including program code) through (multiple) networks, network links, and network interface 812. In the Internet example, a server can transmit the requested code of an application program through the Internet, an ISP, a local network, and network interface 812. The received code can be executed by processor 804 as it is received, and / or stored in storage device 810 or other non-volatile storage for later execution. Network interface 812 can be used to send and receive data communications between a host device and one or more client devices using SEP. Various communications are illustrated throughout the disclosure, for example, Figure 5 Data communication for push operations and Figure 6 Data communication for pull operations.

[0093] Generally, as used herein, the words "component", "module", "engine", "system", "database", etc. may refer to logic implemented in hardware or firmware, or to a set of software instructions written in a programming language such as Java, C or C++, which may have entry points and exit points. Software components or modules may be compiled and linked into an executable program, installed in a dynamic link library, or may be written in an interpretive programming language such as BASIC, Perl or Python. It should be understood that software components can be called from other components or from themselves, and / or can be called in response to detected events or interrupts. Software components configured for execution on a computing device (such as computer system 800) may be provided on a computer-readable medium such as an optical disk, a digital video disk, a flash drive, a disk, or any other tangible medium, or may be provided as a digital download (and may be stored originally in a compressed format or an installable format that needs to be installed, decompressed or decrypted before execution). Such software code may be stored in part or in whole on a memory device of an executing computing device for execution by the computing device. Software instructions may be embedded in firmware such as an EPROM. It will be further understood that hardware components may include connected logic units such as gates and flip-flops, and / or may include programmable units such as programmable gate arrays or processors.

[0094] The computer system 800 can implement the techniques described herein using custom hard-wired logic, one or more ASICs or FPGAs, firmware, and / or program logic that are combined with the computer system 800 to make the computer system 800 a special-purpose machine or to program it as a special-purpose machine. According to one or more examples, the computer system 800 performs the techniques described herein in response to the hardware processor(s) 804 executing one or more sequences of one or more instructions contained in the main memory 806. Such instructions can be read into the main memory 806 from another storage medium (such as the storage device 810). The execution of the sequences of instructions contained in the main memory 806 can cause the hardware processor(s) 804 to perform the process steps described herein. In alternative examples, hard-wired circuits can be used in place of or in combination with software instructions.

[0095] As used herein, the term "non-transitory medium" and similar terms refer to any medium that stores data and / or instructions that cause a machine to operate in a particular manner. Such non-transitory media may include non-volatile media and / or volatile media. Non-volatile media may include, for example, optical or magnetic disks, such as storage device 810. Volatile media may include dynamic memory, such as main memory 806. Common forms of non-transitory media include, for example, floppy disks, floppy disks, hard disks, solid-state drives, tapes, or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a pattern of holes, RAM, PROMs and EPROMs, flash EPROMs, NVRAMs, any other memory chips or boxes, and networked versions of these media.

[0096] Non-transient media are distinct from transmission media but can be used in conjunction with transmission media. Transmission media can participate in the transmission of information between non-transient media. For example, transmission media can include coaxial cables, copper wire, and optical fiber, including the wires that comprise bus 802. Transmission media can also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0097] These and other various forms of computer-readable media may involve carrying one or more sequences of one or more instructions to a processing device for execution. Such instructions embodied on the medium are generally referred to as "instructions" or "codes". Instructions may be grouped in the form of computer programs or other groupings. When executed, such instructions may enable a processing device to perform features or functions of the present application as discussed herein.

[0098] As used herein, the term "or" may be interpreted in an inclusive or exclusive sense. In addition, descriptions of resources, operations, or structures in the singular should not be understood to exclude the plural. Unless otherwise expressly stated, or otherwise understood within the context as used, conditional language (such as, among others, "can," "could," "might," or "may") is generally intended to convey that certain examples include and other examples do not include certain features, elements, and / or steps.

[0099] Unless otherwise expressly stated, the terms and phrases used in this document and their variations should be interpreted as open-ended rather than restrictive. Adjectives such as "conventional", "traditional", "normal", "standard", "known" and terms of similar meaning should not be interpreted as limiting the items described to items available for a given time period or before a given time, but should be understood to cover traditional, conventional, normal or standard technologies that may be available or known at any time now or in the future. In some cases, the presence of broad words and phrases (such as "one or more", "at least", "but not limited to" or other similar phrases) should not be understood to mean that narrower situations are intended or required where such broad phrases may not be present.

Claims

1. A method comprising: determining a set of non-volatile memory express (NVMe) queue pairs to which a CPU core of a central processing unit (CPU) core in a network is connected; Generating a set of transmit and receive contexts aligned with the set of NVMe queue pairs connected to the CPU core of the CPU core for the domain object; generating a scalable endpoint (SEP) for the domain object; Mapping the set of send and receive contexts of the domain object to the set of NVMe queue pairs connected to the CPU core using the SEP; as well as Data is transmitted over the network using a mapping between the set of send and receive contexts to the set of NVMe queue pairs connected to the CPU core and the domain object.

2. The method of claim 1, wherein: A SEP for the domain object is generated using an address vector for the domain object, and wherein the address vector for the domain object stores a remote address without a connection structure.

3. The method of claim 1, further comprising: A completion queue is generated for the domain object, the completion queue storing events or operations that have been completed between the domain object and the network.

4. The method of claim 1, further comprising: Prior to generating a SEP for the domain object and within a time threshold, receiving a request to establish a connection with the domain object.

5. The method of claim 1, further comprising: An event queue is generated for the domain object, the event queue storing data transmissions received by the domain object.

6. The method of claim 1, wherein: The characteristics of the SEP include the network address and designated port of the domain object.

7. The method of claim 1, further comprising: A set of completion queues is generated for the domain object, wherein the set of completion queues is associated with the set of send and receive contexts.

8. The method of claim 1, further comprising: A push operation is performed between a host device of the domain object and a client CPU core using the SEP.

9. The method of claim 8, wherein: The push operation includes: sending a read request message with a unique key tag from the client CPU core to the host device using the SEP; performing, at the host device, a write operation of host device data into a buffer marked by the client based on the read request message; generating a first completion event for a first completion queue associated with the client CPU core after completing the write operation; sending the client marked buffer from the host device to the client CPU core using the SEP; and A second completion event is generated for the first completion queue after the client CPU core receives the client marked buffer.

10. The method of claim 1, further comprising: A pull operation is performed between a host device of the domain object and a client CPU core using the SEP.

11. The method of claim 10, wherein: The pulling operation includes: sending a write request message having a unique key tag and a client-tagged buffer of client data from the client CPU core to the host device using the SEP; generating a first completion event for a first completion queue associated with the client CPU core based on the write request message; performing, at the host device, a read operation of the client data from the client-tagged buffer; performing, at the host device, a write operation of the client data to the host device based on the write request message; and A second completion event is generated for the first completion queue after completing the write operation.

12. A computing system comprising: a first domain object of the plurality of domain objects; a network comprising a plurality of central processing unit (CPU) cores; one or more processors; as well as a memory coupled to the one or more processors to store instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including: determining a set of non-volatile memory express (NVMe) queue pairs to which CPU cores of the network are connected; generating a set of send and receive contexts for the first domain object aligned with the CPU core connection; generating a scalable endpoint (SEP) for the first domain object; Mapping the set of send and receive contexts of the domain object to the set of NVMe queue pairs connected to the CPU core using the SEP; as well as Data is transmitted over the network using a mapping between the set of send and receive contexts to the set of NVMe queue pairs connected to the CPU core and the domain object.

13. The computing system of claim 12, wherein: The instructions further cause the processor to perform operations including: A push operation is performed between a client CPU core and a host device of the first domain object using the SEP.

14. The computing system of claim 13, wherein: The push operation includes: sending a read request message with a unique key tag from the client CPU core to the host device using the SEP; After completing the write operation of the host device data based on the read request message into the client marked buffer, generating a first completion event for a first completion queue associated with the client CPU core; sending the client marked buffer from the host device to the client CPU core using the SEP; and A second completion event is generated for the first completion queue after the client CPU core receives the client marked buffer.

15. The computing system of claim 12, wherein: The instructions further cause the processor to perform operations including: A pull operation is performed between a host device of the first domain object and a client CPU core using the SEP.

16. The computing system of claim 15, wherein: The pulling operation includes: sending a write request message having a unique key tag and a client-tagged buffer of client data from the client CPU core to the host device using the SEP; generating a first completion event for a first completion queue associated with the client CPU core based on the write request message; and After completing the write operation of the client data to the host device based on the write request message, a second completion event is generated for the first completion queue.

17. A non-transitory storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising: determining a set of non-volatile memory express (NVMe) queue pairs to which a CPU core of a central processing unit (CPU) core in a network is connected; Generating a set of transmit and receive contexts aligned with the set of NVMe queue pairs connected to the CPU core of the CPU core for the domain object; generating a scalable endpoint (SEP) for the domain object; Mapping the set of send and receive contexts of the domain object to the set of NVMe queue pairs connected to a set of CPU cores using the SEP; as well as Data is transmitted over the network using a mapping between the set of send and receive contexts to the set of NVMe queue pairs connected to the set of CPU cores and the domain object.

18. The non-transitory storage medium of claim 17, wherein: The instructions further cause the at least one processor to perform operations including: A set of completion queues is generated for the domain object, wherein the set of completion queues is associated with the set of send and receive contexts.

19. The non-transitory storage medium of claim 17, wherein: The instructions further cause the at least one processor to perform operations including: Performing a push operation between a host device of the domain object and a client CPU core using the SEP, wherein the push operation includes: sending a read request message with a unique key tag from the client CPU core to the host device using the SEP; After completing the write operation of the host device data based on the read request message into the client marked buffer, generating a first completion event for a first completion queue associated with the client CPU core; sending the client marked buffer from the host device to the client CPU core using the SEP; and A second completion event is generated for the first completion queue after the client CPU core receives the client marked buffer.

20. The non-transitory storage medium of claim 17, wherein: The instructions further cause the at least one processor to perform operations including: Performing a pull operation between a host device of the domain object and a client CPU core using the SEP, wherein the pull operation includes: sending a write request message with a unique key tag and a client-tagged buffer of client data from the client CPU core to a host device of the domain object using the SEP; generating a first completion event for a first completion queue associated with the client CPU core based on the write request message; and After completing the write operation of the client data to the host device based on the write request message, a second completion event is generated for the first completion queue.

Images