Low latency post transmission

Abstract

The invention relates to low latency post-transmission. A system is described having one or more processing devices for preparing a work queue entry (WQE), transmitting the WQE, or providing a doorbell indication indicating that the WQE is available, and in response to determining that an execution failure has occurred, performing at least one additional step to facilitate completion of an operation associated with the WQE.

Description

Technical Field

[0001] This disclosure generally relates to systems, methods, and apparatus for transmitting data between nodes, and more particularly to improvements in kernel-initiated communication. Background Technology

[0002] In modern high-performance computing (HPC) systems, communication between computing devices is typically facilitated by a network of interconnected nodes. Each computing device (which may include a central processing unit (CPU), graphics processing unit (GPU), and / or other hardware peripherals) can be considered a node in the network. Data is transferred between these nodes through a series of discrete operations, with each node acting as a data relay. This architecture supports parallel processing and data sharing, significantly improving overall system performance and supporting complex computational tasks. Communication between nodes is constrained by various protocols, which may vary depending on the specific requirements of the system and the types of devices involved.

[0003] The concept of Queues (QPs) enables efficient inter-network communication. A QP consists of a work queue, which includes a send queue and a receive queue, acting as the endpoints for data transmission between nodes. The send queue holds instructions for sending outgoing data, while the receive queue receives instructions for incoming data. A QP also requires a completion queue to signal the completion of work requests posted to the work queues. The use of QPs enables network technologies such as InfiniBand to provide high-speed, low-latency communication between nodes. However, the implementation and management of QPs can be complex, requiring detailed handling of data transmission protocols and error management.

[0004] Latency and memory consumption are key factors in the performance and efficiency of these communication networks. Latency refers to the delay experienced during data transmission between nodes, which affects the overall performance of real-time or high-speed applications. On the other hand, memory consumption is related to the amount of memory resources used for data transmission and processing. High memory consumption leads to inefficiency, potentially slowing down other processes and limiting overall system performance. Therefore, optimizing latency and memory consumption has always been a continuous challenge in the development and operation of high-performance computing systems. Various strategies and techniques have been adopted to address these issues, aiming to deliver fast, efficient, and reliable communication between devices.

[0005] The technical limitations of traditional computing system networks in terms of memory consumption and latency can negatively impact real-world applications involving artificial intelligence models, mathematical calculations, and other computationally complex applications. Summary of the Invention

[0006] In some communication protocols (such as the MLX5 post-send protocol), submitting a work queue entry (WQE) involves enqueuing the WQE into a circular buffer and updating the head pointer to submit the work to a peripheral device, such as a network interface card (NIC) or a similar type of input / output (I / O) device. Specifically, the post-send protocol may include: (1) writing the WQE (or multiple WQEs) into the work queue (WQ) buffer; (2) updating the doorbell record (DBR); and ringing the doorbell (DB).

[0007] In GPUDirect Async–Kernel Initiated networking protocol (GDA-KI), the WQ and DBR reside in GPU memory. The DB is typically provided on the NIC. In CPU-centric libraries (such as libibverbs), the WQ and DBR reside in host memory, while the DB resides on the NIC.

[0008] Traditional post-send algorithms, designed for CPU use, are inherently sequential processes. Communication protocols such as GDA-KI, which utilize GPUs instead of CPUs, submit WQEs to the NIC via GPU streaming multiprocessors (SMs). If strictly adhering to traditional WQE submission algorithms, the GPU would need to (1) lock network QPs, which limits concurrency, or (2) create a QP for each thread, potentially consuming hundreds of gigabytes of GPU memory in practical applications. Furthermore, each WQE submission requires issuing a memory barrier, which introduces significant latency to the GPU SM.

[0009] The embodiments disclosed herein can operate in an architecture with a scalable multithreaded SM array. Each SM may include a set of execution units, a set of registers, and a shared memory block. The basic execution unit of a processing unit (e.g., CPU or GPU) may be called a warp. A warp may correspond to a set of threads executed concurrently by the SM (e.g., a warp may have 32 threads). Multiple warps may execute concurrently on a single SM.

[0010] A Computational Thread Array (CTA), also known as a thread block, corresponds to a group of threads that cooperate by sharing data through shared memory and synchronizing their execution. A CTA can be executed by one or more SMs, and multiple CTAs can run in parallel on different SMs. Each CTA can access the shared memory space visible to all threads within the CTA, allowing for efficient communication and data sharing between the threads of the CTA.

[0011] The kernel mesh referred to in this article can be a group of threads started by a single kernel. Threads within a kernel mesh can be grouped into Common Action Tasks (CTAs), allowing threads to share resources and execute synchronously. When a program on the host CPU invokes a kernel mesh, it can enumerate the mesh's CTAs and distribute them to Streaming Service Controllers (SMs) with available execution capabilities. A thread of a CTA can execute concurrently on a single SM, and multiple CTAs can also execute concurrently on a single SM. When a CTA terminates, a new CTA can be started on the vacated SM.

[0012] A CTA can include one or more thread bundles. For example, a CTA can include 128 threads, which can be divided into four thread bundles, each containing 32 threads. Each thread bundle can be scheduled and executed independently, either in parallel or sequentially.

[0013] The embodiments of this disclosure improve communication efficiency by providing a post-send algorithm with lower latency compared to conventional post-send algorithms. The embodiments of this disclosure can reduce latency in several computing environments, including, for example, when using one processing device (e.g., CPU or GPU) to communicate with another processing device (e.g., another CPU or GPU) or a peripheral device (e.g., NIC or solid-state drive (SSD)).

[0014] While embodiments of this disclosure are described in conjunction with an architecture for GPU-NIC communication, it should be understood that the features described herein can be used with other architectures. Specifically, but not limited to, embodiments of this disclosure can be deployed in any computing architecture that issues WQE slot reservation and / or WQE creation instructions / requests via any thread.

[0015] One aspect of this disclosure is reducing the amount of time required to ring the database (DB) after an application writes data. This method involves ringing the DB without first updating the DBR buffer. A completion indicator, such as a completion queue (CQ) or a completion counter, can be used. In some embodiments, the CQ can be polled periodically to determine if an execution failure has occurred. Execution failures described herein may occur due to the target device not receiving a WQE, or receiving a WQE but failing to complete the operation associated with the WQE as expected. Execution failures may also occur when a timeout occurs. Timeouts described herein can be represented as timeout states or timeout indicators. While some embodiments described herein may be described in a manner that determines whether a timeout has occurred or a timeout state exists, it should be understood that the same or similar embodiments may involve determining that any type of execution failure has occurred. That is, the systems and methods described herein should not be considered limited to determining that a timeout has occurred and not to any other type of execution failure.

[0016] QP can be connected to a completion counter instead of a queue, or WQE can indicate a completion queue or counter. Either of these mechanisms can be used to indicate that a job has been successfully completed. Incomplete may mean the job is still pending, or it may mean that a partial job submission failed and a full job submission is required. If an execution failure occurs, the DBR buffer can be updated, and the DB can be re-ringed to resolve the timeout state.

[0017] While a complete work commit protocol is desirable to guarantee work execution, some partial work commit protocols exist that often result in work execution. Work completion or incompleteness indicators can be used to identify situations where work execution has not yet begun (at which point the commit protocol can be definitively determined). An implementation scheme that executes WQE first, then barriers, then DBR, then barriers again, and then DB is an example of a complete commit protocol. An implementation scheme that executes WQE first, then barriers, and then DB is an example of a partial commit protocol.

[0018] The execution of WQE can be combined with registers (such as BlueFlame (BF) registers). The description of BF might involve a low-latency mechanism that can be used to send time-critical WQEs to the device. When using BF, WQEs can be written directly to the device's PCIe BAR so that the device can handle them directly, thus reducing execution latency. BF may be necessary when the device load is light. For high-bandwidth scenarios, regular post-processing (e.g., without BF) may be used. A "complete" protocol using BF might include the following: execute WQE first, then execute the barrier, then execute the DBR, then execute the barrier again, and then execute BF.

[0019] The system and method described in this paper have the advantage of reducing the time and computational cost required to perform post-send. While current DB ringing methods require updating the DBR before ringing the DB, the method described in this paper can achieve this by ringing the DB without updating the DBR. One benefit of not updating the DBR is the avoidance of memory barriers. Because the DBR is not updated, this post-send algorithm can be completed with fewer memory barriers compared to traditional post-send algorithms, thereby reducing the amount of time required to perform the post-send algorithm.

[0020] According to at least some embodiments, the solutions described herein can be deployed where two work submission methods are feasible: (1) a full implementation and / or (2) a partial implementation. In a full implementation, work execution is guaranteed. In a partial implementation, work execution may not be guaranteed, but execution may be non-destructive. In a partial implementation, the work will either execute correctly, not execute at all, or not execute incorrectly. Non-execution cases can be detected (e.g., by a completion indication, a missing completion indication, or some other mechanism). Non-execution cases can be corrected by supplementing the operation with additional work execution in the initial partial implementation. The DB and BF examples described herein may include full and / or partial implementations. Other full or partial implementations may also be used. For example, partial streams of WQE buffers may be supported, such that all locations without a valid WQE are marked with a predetermined code (e.g., valid / legal opcode, invalid opcode, or non-destructive (e.g., NOP) opcode), and / or BF can be written without writing to WQE at all. If an error completion is received due to a bad or non-destructive opcode: the system can be configured to rewrite the WQE starting from the erroneous CQE, then write to the DBR, and finally write to the DB. It should also be understood that for some devices, the DBR may not be necessary. For such devices, the BF full / partial flow may still be applicable and utilized. In some embodiments, all WQEs can be pre-marked with a NOP opcode. This opcode is considered valid and legal. It can instruct its receiver (e.g., NIC) not to perform any action, but simply to generate a CQE. This particular opcode may not be needed in other flows. For example, after ringing a BF, it may be necessary to wait for the corresponding CQE. This CQE provides the opcode that has been executed. If it is NOP, it can be inferred that the BF has been degraded to DB, and the release may need to be re-executed.

[0021] In view of the above, one or more of the following are envisioned:

[0022] One aspect of this disclosure is to provide a system comprising one or more circuits for: preparing a work queue entry (WQE); transmitting the WQE to a target device or providing a doorbell indication to the target device indicating that the WQE is available; and, in response to determining that an execution failure has occurred, performing at least one step to facilitate the completion of an operation associated with the WQE.

[0023] In some embodiments, the WQE is written to memory before the WQE is transmitted or the doorbell indication is provided.

[0024] In some embodiments, the execution failure includes failing to receive a completion indication within a predetermined time period.

[0025] In some embodiments, the execution failure includes receiving a status indicating failure.

[0026] In some embodiments, one or more circuits are further configured to: after determining that the execution failure has occurred, repeatedly transmit the WQE to the target device or provide the doorbell indication to the target device.

[0027] In some embodiments, the at least one step for facilitating the completion of the operation associated with the WQE includes: writing a doorbell record to a memory and repeatedly transmitting the WQE to the target device, or providing the doorbell indication to the target device.

[0028] In some embodiments, the one or more circuits are further configured to write the WQE into memory.

[0029] In some embodiments, providing the doorbell indication to the target device includes writing the doorbell (DB) into the memory of the target device, wherein the DB enables the target device to read the WQE.

[0030] In some embodiments, the at least one step for facilitating the completion of the operation associated with the WQE includes updating the doorbell record (DBR) and implementing a memory barrier.

[0031] In some embodiments, the at least one step for facilitating the completion of the operation associated with the WQE further includes: retransmitting the WQE to the target device, or providing the target device with a second doorbell indication indicating that the WQE is available.

[0032] In some embodiments, transmitting the WQE to the target device includes writing the WQE into the memory of the target device.

[0033] In some embodiments, the one or more circuits are also used to mark the WQE buffer.

[0034] In some embodiments, the WQE is transmitted to the target device without writing the WQE to memory.

[0035] In some embodiments, the WQE is transmitted to the target device after being written to the memory, wherein no memory barrier is performed before the WQE is transmitted to the target device.

[0036] In some embodiments, the at least one step for facilitating the completion of the operation associated with the WQE includes: writing the WQE to memory; implementing a memory barrier; and ringing a doorbell (DB).

[0037] In some embodiments, the at least one step further includes: updating the doorbell record (DBR) and implementing a second memory barrier.

[0038] In some embodiments, updating the DBR includes updating the DBR using the producer index.

[0039] In some embodiments, the one or more circuits are configured to repeatedly perform at least one of preparing the WQE and transmitting the WQE to the target device or providing the doorbell indication before performing the at least one step for facilitating the completion of the operation associated with the WQE.

[0040] In some embodiments, the execution failure includes receiving one or more completion indications, which include entries in the completion queue CQ, asynchronous events, status in memory locations, and interrupts.

[0041] In some embodiments, receiving the completion indication includes polling the CQ.

[0042] In some embodiments, the one or more circuits perform fewer than two memory barriers before polling the CQ.

[0043] In some embodiments, transmitting the WQE to the target device includes: using multiple threads to write the WQE into the memory of the target device.

[0044] In some embodiments, consecutive WQEs are written to a single address in the memory of the target device.

[0045] In some embodiments, consecutive WQEs are written to different addresses in the memory of the target device.

[0046] In some embodiments, the one or more circuits are further configured to synchronize the plurality of threads after the WQE is prepared and before the WQE is transmitted.

[0047] In some embodiments, before preparing the WQE, the one or more circuits determine that the number of in-flight WQEs is less than a threshold.

[0048] Another aspect of this disclosure is to provide a communication device including one or more circuits for: preparing a work queue entry (WQE); transmitting the WQE to a target device or providing a doorbell indication to the target device indicating that the WQE is available; and, in response to determining that an execution failure has occurred, performing at least one step to facilitate the completion of an operation associated with the WQE.

[0049] This document describes additional features and advantages that will be apparent from the following description and figures. Attached Figure Description

[0050] The accompanying drawings, incorporated in and forming part of this specification, illustrate several examples of this disclosure. These drawings, together with the specification, explain the principles of this disclosure. The drawings illustrate only preferred and alternative examples of how this disclosure can be implemented and used, and should not be construed as limiting this disclosure to the examples shown and described. Further features and advantages will become apparent from the more detailed description of various aspects, embodiments, and configurations of this disclosure, as illustrated in the following drawings.

[0051] This disclosure is described in conjunction with the accompanying drawings, which are not necessarily drawn to scale:

[0052] Figure 1 It is a block diagram of the computing architecture used according to one or more embodiments described herein;

[0053] Figure 2 This is a block diagram of a system according to one or more embodiments described herein;

[0054] Figure 3 It is a block diagram of system components and their interactions according to one or more embodiments described herein;

[0055] Figure 4 This is a flowchart of a first method according to one or more embodiments described herein;

[0056] Figure 5 This is a flowchart of a second method according to one or more embodiments described herein; and

[0057] Figure 6 This is a flowchart of a third method according to one or more embodiments described herein. Detailed Implementation

[0058] Before explaining any embodiments of this disclosure in detail, it should be understood that the application of this disclosure is not limited to the details of the construction and component arrangement set forth in the following description or shown in the accompanying drawings. This disclosure may have other embodiments and may be practiced or implemented in various ways. Furthermore, it should be understood that the wording and terminology used herein are for descriptive purposes only and should not be considered limiting. The terms “comprising,” “including,” or “having,” and variations thereof, as used herein, are intended to cover items listed thereafter and their equivalents, as well as additional items. Furthermore, this disclosure may use examples to illustrate one or more aspects thereof. Unless expressly stated otherwise, the use or listing of one or more examples (which may be expressed in terms of “for example,” “e.g.,” “like,” “such as,” or similar language) is not intended to limit the scope of this disclosure.

[0059] Details of one or more aspects of this disclosure will be set forth in the accompanying drawings and the following description. Other features, objects, and advantages of the technology described in this disclosure will be apparent from the description, the drawings, and the claims.

[0060] The phrases “at least one,” “one or more,” and “and / or” are open-ended expressions that can be used operationally as both conjunctions and separators. For example, “at least one of A, B, and C,” “at least one of A, B, or C,” “one or more of A, B, and C,” “one or more of A, B, or C,” and “A, B, and / or C” respectively represent a single A, a single B, a single C, A and B together, A and C together, B and C together, or A, B, and C together. When A, B, and C in the above expressions refer to a single element (e.g., X, Y, and Z) or a class of elements (e.g., X1-Xn, Y1-Ym, and Z1-Zo), the phrase is intended to refer to a single element selected from X, Y, and Z, a combination of elements selected from the same class (e.g., X1 and X2), or a combination of elements selected from two or more classes (e.g., Y1 and Zo).

[0061] The term "a" or "one" refers to one or more of the same entity. Therefore, the terms "a" (or "one"), "one or more," and "at least one" are used interchangeably herein. It should also be noted that the terms "comprising," "including," and "having" are used interchangeably.

[0062] As can be understood from the following description, for computational efficiency reasons, system components can be placed anywhere appropriate within a distributed component network without affecting the operation of the system.

[0063] Furthermore, it should be understood that the various links connecting the elements can be wired links, traced links, or wireless links, or any suitable combination thereof, or any other suitable known or subsequently developed element capable of providing data to and / or transmitting data from the connected elements. The transmission medium used as a link can be, for example, any suitable electrical signal carrier, including coaxial cables, copper wires and optical fibers, electrical traces on printed circuit boards (PCBs), etc.

[0064] The terms “determine,” “calculate,” and “operate,” and their variations, as used herein, are used interchangeably and include any appropriate type of method, process, operation, or technique.

[0065] This document will describe various aspects of the present disclosure with reference to the accompanying drawings, which may be schematic diagrams of idealized configurations.

[0066] Any steps, functions, and operations discussed in this article can be performed continuously and automatically.

[0067] The systems and methods disclosed herein can be described in conjunction with processing devices that communicate with peripheral devices and / or another processing device. However, to avoid unnecessarily obscuring this disclosure, some known structures and devices may be omitted in the description. Such omissions should not be construed as limiting the scope of the claimed disclosure. Specific details are provided to aid in understanding this disclosure. However, it should be understood that this disclosure can be implemented in various ways beyond the specific details set forth herein.

[0068] Furthermore, it should be understood that any number of variations and modifications of this disclosure may be used. For example, a system or method may be provided that utilizes certain features of this disclosure without requiring other features.

[0069] Traditional computer systems employing one or more CPUs and one or more GPUs typically use a star topology, placing the CPU at the center of communication between the computer system components. These components include, for example, GPUs, other CPUs, and one or more peripheral devices, such as network devices, additional GPUs, and NVMe SSDs. In such systems, the CPU acts as the central hub through which data flows, including communication between the GPU and various peripheral devices and other components. This means that for the GPU to access or send data to or from peripheral devices and / or processing devices, it must do so via the CPU and relies on the CPU to manage and facilitate data transfer. Because communication is funnel-like through the CPU, this architecture can introduce bottlenecks. Therefore, the efficiency of data transfer and overall system performance depend on the CPU's ability to process data streams, while the GPU's ability to communicate with peripheral devices is limited by the CPU's performance.

[0070] Direct communication between the GPU and other devices (such as peripherals and / or processing devices) eliminates the need for the CPU to act as a middleman, significantly improving system performance and efficiency across various computing environments. This approach offers numerous advantages, particularly in high-performance computing (HPC), data centers, and artificial intelligence (AI) or machine learning applications where speed and efficient data processing are crucial.

[0071] Bypassing the CPU for communication between the GPU and other devices can significantly reduce latency. As mentioned above, while traditional systems rely on the CPU as an intermediary, direct communication between the GPU and peripheral devices (or processing devices) eliminates CPU involvement, thereby enabling faster data exchange.

[0072] A direct path between the GPU and peripheral devices can utilize the full bandwidth of interconnect technologies such as PCIe without sharing or competing for bandwidth with the CPU or other devices on the CPU interconnect bus. Furthermore, applications such as artificial intelligence (AI) and machine learning (ML) can be improved by rapidly offloading data processed by the GPU to storage devices (such as NVMe SSDs) or sending it over the network to other nodes for further processing. Direct GPU-to-peripheral communication simplifies such workflows, enabling more efficient data processing pipelines without making the CPU a bottleneck.

[0073] Offloading communication tasks from the CPU can further free up CPU resources for other critical tasks. This is especially beneficial in systems where the CPU manages multiple tasks simultaneously. Reducing CPU communication overhead can lead to a comprehensive improvement in system performance and responsiveness.

[0074] Direct communication with GPUs also supports better parallelism in system architecture. GPUs designed for parallel processing can interact with multiple peripheral devices simultaneously, thereby enhancing the system's ability to handle concurrent data streams and parallel processing tasks.

[0075] Direct communication between the GPU and peripheral devices can be achieved through technologies such as GPU Direct Asynchronous Kernel-Initiated (GDA-KI) network communication, which allows the GPU to directly access data from memory devices (e.g., NVMe SSDs), NICs, and other peripheral devices. While some implementations described herein specifically relate to GDA-KI and communication between the GPU and NIC, it should be understood that the systems and methods described herein can be used to implement communication between any processing device (e.g., CPU, GPU, Data Path Accelerator (DPA), etc.) and any peripheral device (e.g., NIC, memory device such as NVMe SSD, etc.) or any processing device (e.g., CPU, GPU, DPA, etc.).

[0076] The use of GPUs as a means of offloading computationally intensive tasks from CPUs, and the use of networks of computing nodes to perform computationally intensive tasks (whether executed by CPU or GPU), are becoming increasingly important for users such as scientific researchers seeking to execute AI models and other computationally intensive processes. The growing demand for high-performance computing across various fields, including scientific simulations, machine learning, and image processing, is driving the need for efficient and cost-effective computing resources. The limitations of network communication performance and the increasing importance of parallelism have prompted researchers and other users to explore alternatives to performing data processing using single computing devices. Therefore, GPUs have emerged as a method for offloading computationally intensive tasks from CPUs, and networks of computing systems are becoming extremely useful for executing complex processing applications.

[0077] In traditional systems, whenever the GPU needs to send data to a node (e.g., via a peripheral device or directly to the processing device), it first notifies the CPU. The CPU prepares a WQE, updates the DBR on the NIC, and rings the DB. This process can be called a post-transmission algorithm. Next, the NIC can access the data from the GPU's memory and send it to the node. For example, while GPUDirect RDMA can be used in some implementations to enable direct communication between the NIC and GPU, the control path still flows through the CPU. Although the systems and methods described herein are generally associated with GPUDirect, it should be understood that in some implementations, other techniques can be used to enable the NIC to access data from the GPU's memory.

[0078] As mentioned above, GDA-KI is a network software technology that moves the control path to the GPU, using the GPU's SM to prepare WQE, write DBR, and ring DB. Therefore, the message rate achievable by the GPU is increased several times (e.g., up to 36 times faster than traditional systems). The embodiments of this disclosure can be useful when the GPU interacts with any I / O device that employs a similar command submission mechanism (such as the NIC mechanism described herein), or when the GPU communicates directly with another GPU or other types of processing devices.

[0079] While GDA-KI offers numerous advantages by enabling the GPU to communicate with the NIC, these advantages can be enhanced by reducing the amount of time required for such communication. A major factor affecting latency is the need to execute multiple memory barriers when performing post-transmission algorithms. The system and method described in this paper reduce the number of memory barriers required when performing post-transmission algorithms.

[0080] The memory barrier described in this article can be a function that ensures the correct ordering and visibility of memory operations across different parts of a system. A memory barrier may be referred to as one or more memory barriers (membar), thread fences, thread fence blocks, or other terms. Memory barriers can be used to ensure that memory operations (including reads and writes) are performed in the desired order. When multiple threads or processes execute concurrently, memory operations can become out of order. This can lead to data inconsistency, where the program's outcome incorrectly depends on the timing of the threads.

[0081] Memory barriers ensure that all global and shared memory writes performed by the currently executing thread are visible to all threads in the grid. Memory barriers also ensure that all memory operations issued before the memory barrier complete before any subsequent memory operations begin. Such memory barriers can be used to guarantee correct data disposal when performing parallel computations or sharing resources in different parts of a pipeline.

[0082] Examples of memory barriers described in this article include "Membar.gl" and "Membar.sys". Membar.gl is a global memory barrier used to ensure the completion and / or visibility of memory operations within the global memory space. When executed by the GPU, this memory barrier ensures the ordering and visibility of write operations between multiple threads within the GPU. Membar.sys is a system-wide memory barrier used to ensure the completion and visibility of memory operations across the entire system, including global memory, shared memory, and other processing and / or peripherals. When executed by the GPU, this memory barrier ensures the ordering and visibility of write operations between multiple threads within the GPU and other processing and / or peripherals.

[0083] Both Membar.gl and Membar.sys are relatively time-consuming compared to other functions involved in the post-send algorithm. However, Membar.sys is even more time-consuming than Membar.gl (for example, Membar.sys may take four times or more the time required to execute Membar.gl). Therefore, the post-send algorithm (which avoids the need for one or more Membar.sys functions and / or one or more Membar.gl functions) allows DB to be ringed in a shorter amount of time compared to traditional methods.

[0084] By reducing the number of memory barriers to be executed when executing the post-transfer algorithm, the systems and methods described herein provide a low-latency mechanism for communication between processing devices (e.g., GPUs, CPUs, or other devices) and peripheral devices (e.g., NICs, SSDs, or other devices) and / or another processing device (e.g., GPUs or other devices).

[0085] Now for reference Figure 1 and Figure 2 The computing architecture 100 and system 200 according to one or more embodiments described herein are shown respectively. Although Figure 1 and Figure 2 The computing architecture 100 and system 200 are shown to include specific components and / or a specific number of components, but it should be understood that the systems and methods described herein can be implemented with more or fewer components and / or components not shown. As an example, Figure 2 A system 200 including a GPU 220 and GPU memory 232 is illustrated. It should be understood that implementing the systems and methods described herein may not require a GPU. Instead, the functions of the GPU 220 and GPU memory 232 can be performed by a CPU, DPA, or other processing device. Furthermore, peripheral devices 224 within the system 200 are illustrated. The systems and methods described herein can be performed without peripheral devices, for example, when a first processing device communicates with a second processing device using the systems and methods described herein without the involvement of peripheral devices.

[0086] like Figure 1 As shown, computing architecture 100 may include one or more peer-to-peer computing systems 108a-d that communicate with each other via network 104. Each of the one or more peer-to-peer computing systems 108a-d can be interconnected and can connect to other peer devices 112 via network 104 to access shared resources, services, and data. Peer-to-peer computing systems 108a-d may be, for example, client devices (e.g., personal computers, laptops, smartphones, IoT devices) and switches or servers, or any type of computing system capable of sending and receiving data via network 104.

[0087] Each peer device 112 may include a network interface, including, for example, a transceiver. Some or all of the peer devices 112 may receive and send packets according to an applicable protocol (e.g., TCP), but other protocols may also be used. In some embodiments, the peer devices 112 may be configured to communicate with the peer computing system 108a using memory transactions and a local system bus (e.g., PCIe, CXL, Nvlink, Nvlink-C2C, etc.). Although the peer devices 112 are shown as connected to the peer computing systems 108a-d, it should be understood that the peer devices 112 may also be configured to receive and transmit packets directly to and / or directly from the network 104.

[0088] In some implementations, one or more peer-to-peer computing systems 108a-d and devices 112 may be switches, proxies, gateways, load balancers, etc. Such systems 108a-d and devices 112 may act as intermediaries between clients and / or servers, relaying or modifying communication between them. In some implementations, one or more peer-to-peer computing systems 108a-d and devices 112 may be Internet of Things (IoT) devices connected to network 104, such as sensors, actuators, and / or embedded systems. Such IoT devices may act as clients, servers, or both, depending on the implementation and the specific IoT application. For example, the first peer-to-peer computing system or device may be a smart thermostat acting as a client, while the second peer-to-peer computing system or device may be a central server for analytics or a smartphone running applications.

[0089] It should be understood that in the field of high-performance computing, a large number of peer-to-peer computing systems 108a-d can utilize QP and WQ buffers for network communication. For example, in server farms, data centers, or clusters used for big data analytics and scientific computing, the CPUs and / or GPUs of peer-to-peer computing systems 108a-d can use QP and WQ buffers to send and receive data between each other, for example, via protocols including InfiniBand or Ethernet.

[0090] like Figure 2 As shown, a system 200 having one or more processing devices (e.g., GPU 220) can correspond to examples of peer-to-peer computing systems 108a-d and / or peer device 112. The advent of general-purpose computing on GPUs has enabled the widespread use of GPU 220 in tasks beyond graphics rendering, particularly in fields such as machine learning, deep learning, and data mining. GPU 220 is capable of handling thousands of threads simultaneously, making it well-suited for processing massively parallel tasks.

[0091] While system 200 can be configured to communicate with other systems 200 via network 104 (as described herein), it should be understood that system 200 can also communicate with other peer computing systems 108a-d and / or peer devices 112, which may or may not use network 104.

[0092] Figure 1 The network 104 shown can rely on various network hardware and protocols to establish communication between peer-to-peer computing systems 108a-d and other peer-to-peer computing systems 108a-d and / or peer devices 112. Such infrastructure may include one or more routers, switches, and / or access points, as well as wired and / or wireless connections. Network 104 may be, for example, a local area network (LAN) connecting peer-to-peer computing systems 108a-d and peer devices 112. The LAN may use Ethernet or Wi-Fi technology to provide communication between the peer-to-peer computing systems.

[0093] In some implementations, network 104 may be, for example, a wide area network (WAN) used to connect peer device 112 to one or more peer computing systems 108a-d. The WAN may include, for example, one or more line, satellite link, or cellular network. The WAN can use various transmission technologies (e.g., leased lines, satellite links, or cellular networks) to provide long-distance communication. For example, Transmission Control Protocol (TCP) communication over the WAN can be used to enable reliable long-distance communication between peer computing systems 108a-d. In some implementations, network 104 may include the Internet, one or more mobile networks (e.g., 4G, 5G, LTE), virtual networks (e.g., VPNs), or combinations thereof.

[0094] Similar to peer-to-peer computing systems 108a-d, system 200 can be a client device or may include client devices, and can encompass a variety of devices, including desktop computers, laptops, smartphones, IoT devices, etc. Such system 200 can execute one or more applications that communicate with other systems 200 to access resources or services. For example, the first system 200 can execute a web browser, while the second system 200 can act as a web server. The first system 200 can communicate with the second system 200 to request and display web content. As another example, the first system 200 can execute a file-sharing application, while the second system 200 can act as a file server. The first system 200 can communicate with the second system 200 to upload or download files. As yet another example, the first system 200 can act as an AI server, which can be used by the second system 200 to offload computationally intensive processes so that they can be executed in parallel by one or more GPUs 220 of the first system 200. Applications running on system 200 can be responsible for initiating communication with other systems 200, making requests for resources or services, and processing data. Network 104 enables system 200 to communicate simultaneously with any number of peer computing systems 108a-d and / or peer devices 112, and to handle any number of concurrent communication contents.

[0095] It should also be understood that, in some embodiments, the systems and methods described herein can be performed without a network 104 connection. For example, one or more peer-to-peer computing systems 108a-d (or system 200) can communicate directly with other peer-to-peer computing systems 108a-d (or system 200) without relying on any particular network 104.

[0096] like Figure 2 As shown, each system 200 may include one or more GPUs 220 and one or more peripheral devices 224. In some embodiments, each system 200 may also include one or more CPUs 204. Each of the CPUs 204, GPUs 220, and peripheral devices 224 may communicate via interface 216.

[0097] The peripheral device 224 described herein may include one or more circuits capable of acting as interfaces between components of system 200 (e.g., CPU 204 and GPU 220). Peripheral device 224 may also act as an interface between components of system 200 and network 104. Peripheral device 224 may include, for example, a NIC; however, the systems and methods described herein can be applied to any type of peripheral device, such as a GPU, CPU, storage device (e.g., NVMe SSD), or other components. With the NIC, peripheral device 224 may enable data transmission and reception, enabling peer-to-peer computing systems 108a-d to communicate with system 200. In other embodiments, peripheral device 224 may perform other functions. In some embodiments, peripheral device 224 may include one or more peripheral component high-speed interconnect (PCIe) cards, network adapters, USB adapters, and / or may be integrated into a PCB (e.g., a motherboard). Peripheral device 224 may also include one or more processors 234. The processor 234 of peripheral device 224 can be, for example, a streaming multiprocessor (SM), a RISC processor, a CISC processor, a DSP, a baseband processor, an ASIC, an FPGA, an RFIC, other processors (including those discussed herein), or any suitable combination thereof. Peripheral device 224 can support any number of network protocols, such as Ethernet, Wi-Fi, Fibre Channel, etc.

[0098] One or more CPUs 204 of system 200 may each include one or more circuits capable of executing instructions and performing calculations. The CPUs 204 may be able to interpret and process data received by system 200 via peripheral devices 224. Each CPU 204 of system 200 may include one or more arithmetic logic units (ALUs) capable of performing arithmetic and / or logical operations, such as addition, subtraction, and bitwise operations. The CPUs 204 may also, or alternatively, include one or more control units (CUs) capable of managing instruction and data flow within the CPUs 204. The CUs of the CPUs 204 may be configured to fetch instructions from CPU memory 208 or system memory 212, decode instructions, and instruct appropriate components to perform operations according to the instructions.

[0099] The CPU 204 of system 200 may include, for example, a Reduced Instruction Set Computing (RISC) processor, a Complex Instruction Set Computing (CISC) processor, a Digital Signal Processor (DSP) (e.g., a baseband processor), an Application-Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), a Radio Frequency Integrated Circuit (RFIC), other processors (including those described herein), or any suitable combination thereof. Similarly, the GPU 220 described herein may include a processor 228 (e.g., an SM, RISC processor, CISC processor, DSP, baseband processor, ASIC, FPGA, RFIC, other processors (including those described herein), or any suitable combination thereof).

[0100] The processor 228 of the CPU 204 and / or GPU 220 described herein may include multiple processing cores, thereby allowing the CPU 204 (and / or GPU 220) to execute multiple instructions simultaneously, and / or may be able to perform hyperthreading to execute multiple threads simultaneously.

[0101] One or more GPUs 220 in system 200 may each include one or more circuits capable of acting as dedicated processing components for computationally intensive tasks, such as rendering graphics and performing complex mathematical calculations. GPUs 220 may be able to perform general-purpose tasks in parallel with CPU 204.

[0102] As described above, GPU 220 may include one or more SMs, CUs, or processors 228, which can be responsible for executing instructions in parallel. Each SM, CU, or processor 228 of GPU 220 may contain one or more processing cores or ALUs, which can be capable of performing arithmetic and / or logical operations simultaneously.

[0103] One, some, or all of the GPUs 220 in System 200 can perform tasks such as scientific simulations, machine learning, and data analysis. For example, the GPUs 220 in System 200 can be designed to run in a workstation environment, such as for performing scientific simulations, executing and / or training machine learning models, performing data analysis, etc.

[0104] The GPU 220 can execute one or more kernels. Kernels executed by the GPU 220 can perform specific, parallelizable tasks on the GPU 220. These kernels can be written using GPU programming languages ​​or frameworks such as CUDA.

[0105] System 200's interface 216 may include one or more circuits enabling the connection of peripheral devices (e.g., peripheral device 224, one or more GPUs 220, and one or more CPUs 204) to the motherboard of system 200, and one or more devices for system memory 212. Interface 216 may include one or more high-speed channels. Each channel may be, for example, a serial channel and may consist of a pair of signaling lines for transmitting and / or receiving data. Interface 216 may be, for example, a PCIe bus.

[0106] Devices used for system memory 212 may include solid-state drives (SSDs), such as NVMe SSDs. System memory 212 may be able to provide fast and efficient data access and storage. Each of CPU 204, GPU 220, and peripheral device 224 is able to send data to and read data from system memory 212 via interface 216. Each of CPU 204, GPU 220, and peripheral device 224. Exemplary, but not limited to: CPU 204 may access dedicated CPU memory 208, while GPU 220 may have one or more devices dedicated to GPU memory 232.

[0107] The disclosed systems and methods are applicable to systems with and without GPU 220. Embodiments of this disclosure may include GPU 220 performing data packet preparation, scheduling, sending, and / or transmission. In some embodiments, CPU 204 may instruct GPU 220 (or multiple GPUs 220) to perform various tasks. Such platforms may employ GPU-accelerated signal processing, for example, by using GDA-KI to enable GPU 220 to prepare network working descriptors or WQEs and submit these descriptors to peripheral device 224.

[0108] Since the CPU 204 and GPU 220 may have different memory spaces, data that the GPU 220 needs to process can be moved from the CPU 204 to the GPU 220 before computation begins, and the computation result can be moved back to the CPU 204 after computation is complete. System memory 212 can refer to system-level memory accessible to all threads of the processing device (e.g., GPU 220) and the host (e.g., CPU 204). In some implementations, global memory can be allocated and released by the host and can be used to initialize data that the GPU 220 will process.

[0109] Now for reference Figure 3Further details regarding the interaction between the processing unit 300 and the device 304 will be described based on at least some embodiments of this disclosure. The CPU 204 and / or GPU 220 may correspond to examples of the processing unit 300. Other examples of the processing unit 300 include, but are not limited to, a DPA, a DPU, a microprocessor, an array of CPUs 204, an array of GPUs 220, an array of DPUs, an array of microprocessors, etc.

[0110] Peripheral device 224 may correspond to an example of device 304. It should be understood that embodiments of this disclosure can be applied to other queue-based producer-consumer models, such as NVMe submission queues. In this context, device 304 may correspond to different types of devices, such as memory devices. In some implementations, CPU 204 and / or GPU 220 may correspond to an example of processing unit 300. Other examples of processing unit 300 include, but are not limited to, DPA, DPU, microprocessors, sets of CPUs 204, sets of GPUs 220, sets of DPUs, sets of microprocessors, etc.

[0111] Processing unit 300 is shown to include one or more SMs 328. Within the context of processing unit 300, a basic execution unit may be referred to as a warp. As described above, a warp is a collection of threads executed concurrently by SMs 328. One SM 328 can execute multiple warps at a time.

[0112] When a program on processing unit 300 calls the kernel grid, it can enumerate the CTAs of the kernel grid and distribute them to one or more SM 328s with available execution capabilities. A thread of a CTA executes concurrently on one SM 328, and multiple CTAs can execute concurrently on one SM 328. When a CTA terminates, a new CTA can be started on an empty SM 328.

[0113] The application executed by the SM 328 of the processing unit 300 can write data 312 to memory 332. When data 312 needs to be transferred to a destination, the processing unit 300 can use WQ 316, DBR 320, and CQ 324 to transfer data 312 to and / or receive data 312 from device 304. Each of WQ 316, DBR 320, and CQ 324 can contain one or more buffers or logical storage units within memory 332. WQ 316, DBR 320, and CQ 324 can enable the SM 328 of the processing unit 300 to interact directly with device 304. Device 304 can store DB register 308. The processing unit 300 can be able to ring the DB by writing to DB register 308. Using this mechanism, device 304 can be notified that there is one or more WQEs available for processing. In some implementations, processing unit 300 can ring the DB by writing a producer index or serial number associated with one or more WQEs to the DB register 308, and device 304 can use the producer index or serial number to retrieve these WQEs using a read operation.

[0114] Each WQ 316 can store one or more WQEs, which can represent operations to be performed. A WQ 316 can be used as a repository for WQEs. Each WQE in a WQ 316 may contain information about the operation, such as the operation type, the location of the data, and other control information. Therefore, each WQ 316 consumes memory for each WQE it holds, thus affecting the overall memory consumption of the QP.

[0115] The information contained in a WQE may include a control segment (referred to as the ctrl segment in this document) and other information, such as a data segment. The ctrl segment of a WQE may include, for example, an opcode modifier (referred to as OPC_MOD in this document), a WQE index (referred to as wqe_index or producer index in this document), an opcode (referred to as OPCODE in this document), a queue number or send queue identifier (referred to as qp_or_sq in this document), a data segment length (DS), and / or other information.

[0116] CQ 324 can be configured to track the completion status of WQEs. When the operation associated with a WQE is completed, a completion event can be generated and placed in CQ 324. A QP can be associated with one or more CQ 324s. Therefore, for each QP, memory may be required to store the completion events in the CQ 324. In some embodiments, one or more QPs can be associated with a single CQ 324. Alternative implementations can replace the CQ with a counter, and / or each WQE can be associated with a different CQ 324 / counter.

[0117] DBR 320 can be used as a mechanism to issue a notification when a new WQE is placed into WQ 316, or to request a completion notification from CQ 324. Each QP can be associated with a corresponding DBR.

[0118] The memory and / or storage device of system 200 may store instructions, such as software, programs, applications, or other executable code, for causing at least any one of the CPU 204, GPU 220, and peripheral device 224, individually or in combination, to perform one or more of the methods described herein. In some embodiments, these instructions may reside wholly or partially in [the storage device]. Figures 1 to 3 In at least one memory / storage device or any suitable combination thereof shown.

[0119] In some embodiments, Figures 1 to 3 Alternatively, electronic devices, networks, systems, chips, circuits, or components, or portions thereof, or implementations thereof, shown in other figures herein may be configured to perform one or more processes, techniques, or methods, or portions thereof, as described herein. Such processes may be as described below. Figure 4 , Figure 5 and Figure 6 The flowchart is shown.

[0120] During operation, the processing unit 300 can... Figure 4 , Figure 5 and Figure 6 Methods 400, 500, and 600 illustrate a series of operations that interact with device 304 (or another processing unit). While the examples provided herein involve the use of GDA-KI, it should be understood that the same or similar methods can also be used for CPU-based post-transmission, thereby enabling asynchronous post-transmission by the CPU.

[0121] Traditional post-transmission algorithms involve preparing the WQE and sending it to the NIC for execution. For proper operation, traditional post-transmission algorithms must ensure that the data and control information are correctly formatted, ordered, and visible to the NI before the NI performs the specified operation, such as data transfer or computation tasks.

[0122] Traditional post-processing algorithms are designed for sequential processors such as CPUs. These algorithms ensure that the NIC can accurately interpret and execute the task described by the WQE.

[0123] In traditional post-transfer algorithms, the processor prepares the WQE by writing the necessary task information to a designated area in memory. Once the WQE is ready, the processor updates the DBR in memory. This update notifies the NIC that the new WQE is ready for processing. The DBR mechanism signals the NIC about the existence of a new WQE. The DBR typically contains a pointer to the WQE or an index indicating its position in the queue. After updating the DBR, the processor rings the DB by performing a write operation to the write register, instructing the NIC to acquire and execute the WQE (or multiple WQEs).

[0124] In traditional last-transmission algorithms, it is crucial that the writes to WQE, DBR, and DB be performed strictly in sequence. This is essential for maintaining data consistency and ensuring that the NIC correctly interprets and processes the WQE. To enforce this sequence, traditional last-transmission algorithms require memory barriers to be implemented between each of the following: WQE preparation, DBR update, and DB ringing.

[0125] As mentioned above, each memory barrier is a time-consuming task. Because traditional post-transmission algorithms require multiple memory barriers, they are a time-consuming process. As described herein, post-transmission algorithms can be implemented in a way that reduces the number of required memory barriers and / or uses less time-consuming memory barriers.

[0126] In some implementations of the systems and methods described herein, a WQE can be prepared and used to ring a DB without updating the DBR. Therefore, at least one memory barrier and DBR update required by conventional post-transmission algorithms can be removed from the steps required to ring the DB. Using such a system or method can reduce the time and latency required to execute the post-transmission algorithm.

[0127] Method 400 begins at 404, where data associated with the WQE is written. In some cases, the data associated with the WQE may be contained (or inlined) within the WQE itself. In this case, the step of writing data at 404 can be omitted. The data written at 404 may be written to the dedicated memory of the processing device executing method 400. For a GPU, such data may be called vidmem, which may be dedicated memory within the GPU. The data written at 404 may be data created by an application executed by the processing device executing method 400. The purpose of creating such data is to transmit it to a peripheral device or other processing device using a post-transmission algorithm.

[0128] At 408, a WQE associated with the written data can be prepared. The processing device performing method 400 can prepare the WQE by writing a WQE data structure to a specified memory location (e.g., vidmem). The WQE can contain any information required by the peripheral device (or other processing device) for a task to be performed. In some embodiments, a memory barrier can be performed after the WQE preparation to ensure that the WQE preparation is complete and visible before proceeding to the next step. Such a memory barrier can be membar.gl as described above.

[0129] After the WQE is prepared at 408, and in some implementations that enforce memory barriers, the DB can be ringed using the WQE at 412. Ringing the DB can be performed by writing to the DB register or other memory location. Ringing the DB effectively notifies peripheral devices or other processing devices that the WQE is ready and available to be read from memory. Writing to the DB register or other memory location can act as a trigger for the NIC or other device to read the WQE and begin processing the data associated with that WQE. Thus, the processing device performing method 400 transfers the WQE to its destination before updating the DBR using the producer index. In some implementations, a memory release function (e.g., st.release.sys) can be used to ring the DB. In some implementations, ringing the DB may involve transferring the WQE to the target device (e.g., a peripheral device or processing device) or providing a doorbell indication to the target device indicating that the WQE is available. In some implementations, the WQE can be written to the memory of the processing device or target device performing method 400 before transferring the WQE or providing a doorbell indication. Ringing the DB enables the target device to read the WQE from the processing device executing method 400 or the memory of the target device.

[0130] As described herein, transmitting WQE to a target device may involve writing the WQE directly to the target device's memory, or ringing the DB by writing the DB to the target memory as described above. In some embodiments, the WQE or a copy thereof may be maintained in the memory of the processing device, while in other embodiments, a copy of the WQE may not be maintained.

[0131] Prior to polling the CQ at point 416 described below, steps 404, 408, and 412 above can be performed iteratively multiple times, each iteration preparing and sending a different WQE. The number of iterations may depend on the system or application and can be any of one or more iterations. These steps may be located in the critical path before polling the CQ. Because these steps can be performed in a loop without updating the DBR, the goal of reducing latency can be achieved compared to traditional post-send algorithms.

[0132] After ringing the DB at 412, the processing device executing method 400 can poll the CQ at 416 to determine the progress of the NIC (or other peripheral device or processing device) and confirm whether a WQE is being processed. To poll the CQ, the thread can check if a Completion Queue Event (CQE) is available. Polling can occur within a specific time frame. For example, if no CQE is available within a threshold time frame, the thread can determine that an execution failure has occurred, such as a timeout.

[0133] After polling the CQ at 416, the processing device can determine whether an execution failure has occurred. If no execution failure has occurred, method 400 can continue preparing the new WQE by returning to the write data at 404 as described above. By continuing without a timeout state, method 400 avoids updating the DBR and any memory barriers associated with updating the DBR. On the other hand, if an execution failure has occurred, method 400 may involve performing several steps to facilitate the completion of the operation associated with the WQE. These steps may include, for example, updating the DBR at 408 using the producer index of the latest WQE used for ringing the DB, and re-ringing the DB at 412 using the latest WQE after updating the DBR at 424. These steps may also, or alternatively, include, for example, writing the WQE to memory, implementing memory barriers, and re-ringing the DB.

[0134] In some implementations, updating the DBR may involve writing the DBR to memory and then retransmitting the WQE to the target device (i.e., a peripheral or processing device), or providing a doorbell indication to the target device. In some implementations, the memory barrier can be performed before and / or after updating the DBR.

[0135] The timeout described herein may refer to the following scenario: a WQE has been transmitted to the target device, or a doorbell indication has been provided to the target device indicating that the WQE is available, but the WQE has not been consumed after a predetermined amount of time. As described above, while some embodiments of this disclosure describe how to determine whether a timeout has occurred, the same or similar systems and methods can also be applied to other types of execution failures. The execution failures described herein may occur because the target device has not received the WQE, or has received the WQE but has not completed the operation associated with the WQE as expected. The systems and methods described herein should not be considered limited to determining that a timeout has occurred, and not to any other type of execution failure.

[0136] While the systems and methods described herein are for determining whether a timeout has occurred, it should be understood that the same or similar systems and methods may also involve determining whether an execution failure has occurred. Execution failures described herein may involve events where the target device fails to perform an operation that should have been prompted by the WQE. For example, the target device may fail to consume the WQE, or consume the WQE but fail to successfully perform the operation associated with the WQE.

[0137] As described herein, determining whether a timeout has occurred or whether a timeout state exists may include determining whether a timeout condition exists. A timeout condition may indicate that an execution failure has occurred, a timeout has occurred, or other circumstances have occurred that may require one or more steps to facilitate the completion of operations associated with the WQE. As an example, an execution failure or timeout may involve failing to receive a completion indication within a predetermined amount of time. In such an example, determining whether a timeout condition exists may involve determining that a completion indication has not been received within the predetermined amount of time. In some implementations, a completion indication may be in the form of an entry in the CQ (or other queue), an asynchronous event, an interrupt, or an entry not in the queue (e.g., an entry in a memory location with a counter and / or status indication).

[0138] Another example is that execution failure or timeout might involve a state where receiving an instruction has failed. In such an example, determining that a timeout condition exists might involve a state where receiving an instruction has already failed. This type of state could include an execution failure state.

[0139] In some implementations, execution failure or timeout may involve receiving a completion indication containing entries in the CQ, for example, by polling the CQ. Whether due to a timeout or execution failure, when a timeout condition exists, it may be necessary to resend the WQE, generate a new WQE, or perform one or more other steps to facilitate the completion of the operation associated with the WQE. In some implementations, the steps of preparing the WQE at 408 and ringing the DB at 412 may be repeated multiple times before performing one or more steps to facilitate the completion of the operation associated with the WQE.

[0140] In some implementations, a memory barrier (e.g., Membar.gl) may be required after updating the DBR and before re-ringing the DB. However, as should be understood from the description provided herein, method 400 enables a processing device performing method 400 to execute fewer than two memory barriers before polling the CQ at 416. By updating the DBR only in the event of a timeout, the time and computational overhead required to update the DBR and any associated memory barriers can be reduced or completely avoided, since in practice, timeouts are expected to occur rarely or never.

[0141] In some implementations, a copy of each WQE may be maintained in the memory of the device performing method 400 for a period of time. In other implementations, the WQE buffer may be tagged and the WQEs written to the target device, instead of maintaining a copy of each WQE in memory or as a supplement to maintaining a copy of each WQE in memory.

[0142] The system and method described in this paper can be used in conjunction with DB post-send algorithms and BlueFlame (BF) and other network protocols. From a network perspective, BF aims to achieve low latency. However, conventional post-send algorithms used for BF have the same drawbacks as the conventional DB post-send algorithms described above. Method 400 can be executed by, for example, a CUDA thread running on a GPU. Method 400 assumes that the size of the WQE being prepared and used for ringing the DB is one WQE basic block (WQEBB), or 64 bytes. However, it should be understood that the same or similar methods can be performed using WQEs of any size.

[0143] When ringing BF, the entire WQE must be written to the BF register. The write to WQE must have similar write combination behavior, which means that (if the WQE size is one WQEBB), the GPU (or other processing device) must generate a 64-byte PCIe write transaction to the NIC (or another device).

[0144] To ring BF using the GPU SM, a warp must be used. If the WQE size is one WQEBB (64 bytes), eight threads from the warp can be used, each simultaneously issuing a store operation (8 bytes) to the BF register in a write-combined manner to write the entire 64 bytes of the WQE. The GPU can merge these store operations and generate a single 64B PCIe write transaction. However, generating eight 8-byte PCIe write transactions may cause the transaction to degrade from the BF protocol to DB. The `store.mmio` instruction cannot be used because it prevents merging.

[0145] By convention, when using the BF register in GDA-KI, the BF register is split into two addresses. In each iteration, an alternating odd-even algorithm is used to use the different address from the two addresses as the BF register. Similar to the DB-after-transfer algorithm described above, each BF in an iteration must arrive at the NIC before the BF of the next iteration. However, due to the alternation between the two addresses, the order of identical addresses on the GPU cannot be guaranteed. Therefore, membar.sys is typically required at the end of each iteration.

[0146] BF typically uses the traditional BF post-transfer algorithm, which involves updating the DBR in memory in each iteration before writing the entire WQE to the BF register. Memory barriers must be executed immediately before and after updating the DBR. Furthermore, a system memory barrier (membar.sys) needs to be executed at the end of each iteration.

[0147] One advantage of using BF (Browser-Focused) compared to traditional DB (Database-After-Send) algorithms is that the NIC (or other device) does not need to read the corresponding WQE from the processing device memory where the WQE is prepared. For this reason, latency is reduced by one PCIe round trip. However, the overhead of a membar.sys is similar to that of a single PCIe round trip. Furthermore, BF may silently degrade to DB when the NIC is busy. This degradation occurs in high message rate scenarios. Therefore, the cost of membar.sys is incurred without any benefit. For this reason, the cost of BF outweighs its benefits when implemented using traditional systems and methods.

[0148] As described in this article, systems or methods using BF can provide low-latency use cases for BF, such as in GDA-KI, without incurring the costs associated with regular BF use. The systems and methods described in this article are particularly useful when sending a single message to another party and expecting that message to arrive at that party as quickly as possible. Figure 5 As shown, this method 500 can be executed by a processing device. Although Figure 5 The implementation described is for the BF post-transmission algorithm, but the same or similar implementation can also be applied to existing CPU-based systems and other systems that do not use BF.

[0149] Figure 5 The flowchart shown can be used in a variety of different ways. Various optional elements are described below when describing the characteristics of method 500. These optional elements can be used in any combination to execute method 500 in any number of ways.

[0150] At the start of method 500, a pointer to the BF register can be set at 504. The BF register may be a segment of memory within the processing device executing method 500. This segment of memory can be identified by an address (or a BF address). In some implementations, a single BF address can be used to avoid the address switching required by the conventional BF post-transmission algorithm as described above, while in other implementations, switching can occur between two or more BF addresses per iteration. In implementations using two or more BF addresses, the BF register can be effectively split into two or more registers, each of which is a contiguous segment of memory.

[0151] In some implementations, a determination can be made regarding whether the same BF address can be used. Such a determination can be made at least in part based on the size of the WQE to be prepared. For example, if the size of the WQE is equal to a WQEBB, then the same BF address can be used for each iteration. Using the same BF address for each iteration eliminates the need for the relatively expensive system-wide memory barriers (e.g., membar.sys) required by conventional BF-after-transmission algorithms, offering an advantage over such algorithms.

[0152] In cases where WQE is at most a single WQEBB, using the same address for each iteration can be useful. In this case, the GPU (or other processing device) guarantees that the same address is ordered, thus avoiding the relatively expensive membar.sys. While this approach may not be universal, it may be sufficient in many use cases, such as NVSHMEMPUT and NVSHMEM GET, which have connections using reliable connection (RC) transports and leverage Remote Direct Memory Access (RDMA) to provide communication between two GPUs.

[0153] In some cases, after setting the pointer to the BF register, the `stg.ordered.weak` command or other ordered store commands can be executed. Such ordered store commands can ensure that operations are performed in a predictable order. Weak ordering commands (such as `stg.ordered.weak`) can cause memory operations to be performed out of sequence, while strong ordering commands (such as `stg.ordered.strong` (described below)) enforce strict memory operation sequence.

[0154] At point 508, data to be transmitted via BF can be written into the memory of the processing device executing method 500. For example, the data can be written to vidmem, or to a dedicated memory device within the processing device executing method 500 or communicating with the processing device executing method 500. If the data is inline in WQE, the data writing can be omitted. In some embodiments, the stg.ordered.weak command as described above can be executed after the data is written to memory.

[0155] At position 512, the WQE can be prepared in memory. Preparing or creating the WQE can be performed by creating a ctrl segment containing specific instructions for the peripheral device or other processing device with which the processing device executing method 500 attempts to communicate. For a GPU executing method 500, the WQE can be prepared in vidmem.

[0156] After WQE is ready, a synchronization function such as __syncWarp can be executed to synchronize the threads in the thread bundle of method 500. The synchronization function ensures that the threads in the thread bundle arrive at the same execution point before continuing execution.

[0157] At 516, a memory barrier can be executed. The memory barrier can be, for example, the membar.gl command. If an ordering command (e.g., stg.ordered.weak) has already been executed before preparing the WQE at 512, the memory barrier may not be necessary, as described above. In this case, the stg.ordered.strong command can be used to write the WQE to the BF register at 520 as described below.

[0158] At position 520, the WQE can be written to the BF register. For a WQE of size one WQEBB, each of the 8 threads in the thread bundle can write a portion of the WQE to the BF register (as indicated by the pointer). For a WQE of size n WQEBB, each of the n×8 threads can write a portion of its WQE to the BF register (as indicated by the pointer).

[0159] Once a WQE is written to the BF register, one of three scenarios may occur in some implementations. In the example of GPU-NIC communication, the NIC can accept the entire BF, in which case the NIC can immediately process the WQE written to the BF. In another scenario, the NIC may downgrade the BF to a DB, in which case the NIC may need to read the corresponding WQE from GPU memory. By performing a memory barrier at 516, the visibility of the WQE should be guaranteed. In a third scenario, the NIC may discard the BF instead of downgrading it. To avoid this, in some implementations, method 500 may ring the DB to the BF pointer address after writing the entire WQE to the BF register, and then perform a memory barrier via membar.sys, which should effectively notify the NIC to use the DB again to process the WQE.

[0160] After writing WQE to the BF register, another synchronization mechanism (such as __syncwarp) can be executed to ensure thread synchronization and complete the operation of writing WQE to the BF register before proceeding.

[0161] At this point, at least in some implementations, one or more threads can be freed up to perform useful computations. Outside the critical communication path, the above combination can be used... Figure 4 The described method polls the CQ periodically (after each WQE, after every n WQEs, etc.).

[0162] In some implementations, only one WQE is allowed in transit at any given time, and membar.sys can be avoided at the end of each iteration. Instead, the communication section can be exited, and the thread can be freed to perform useful computations. In some implementations, the processing device can be configured to determine whether the number of WQEs in transit is less than a threshold before preparing the WQEs. This threshold can be one WQE, or it can be any other number of WQEs.

[0163] Before proceeding to the next iteration, CQ can be polled. Unless the computation time is extremely short, CQE should have already reached CQ. Therefore, polling CQ can be expected to return immediately. Next, it can be confirmed that there are no BF writes in transit from the GPU to the NIC, since WQE has been processed, and it is safe to proceed to the next iteration, as the BF write of iteration i cannot arrive after the BF write of iteration i+1.

[0164] Figure 6 A method 600 is shown in which the DB is ringed to the BF pointer without performing a memory barrier within the communication path. Method 600 enables the BF to be used to perform a post-transmission algorithm without incurring the drawbacks of the conventional BF post-transmission algorithm described above.

[0165] Method 600 involves three phases: setup phase 604, communication phase 608, and polling CQ phase 612.

[0166] Method 600 may begin with 616, in which a processing device such as a CPU or GPU executes an application that attempts to communicate with peripheral devices or other processing devices. In some implementations, each of the three stages may be executed by a separate processing device, while in other implementations, a single processing device may execute two or three of the stages.

[0167] The setup phase 604 can be performed by the CPU or other processing device. At 620, the processing device can create queue pairs (QPs) of depth D, where D is a power of 2, and create one or more WQEs. The created WQEs may contain NOP opcodes with wqe_idx values ​​ranging from 0 to D-1. The WQEs can fill all available WQ slots.

[0168] Communication phase 608 can be executed by a CUDA warp. If the initiator has data associated with the WQE, that data should be inlined into the WQE. This avoids memory barriers (e.g., membar.gl) because the data is not written to memory. The functionality of communication phase 608 can be executed in a for loop for each WQE created at 620. For the first WQE, communication phase 608 may include setting a pointer to the BF register at 624. Setting the pointer to the BF register can be as described above, and in some implementations, the BF register can iterate between two or more different BF register addresses.

[0169] At 628, the first WQE can be prepared by setting the wqe_idx associated with the WQE in a register or local memory. The WQE is not written to the WQ buffer. Once the WQE is ready, a synchronization mechanism can be performed using, for example, __syncWarp.

[0170] At position 632, multiple threads can be used to write the entire WQE to the BF register, with each thread writing a separate eight bytes of the WQE to the BF register. This functionality is similar to that described above. Figure 5 The operation of writing WQE to BF register 520 in method 500.

[0171] After communication phase 608 is completed, one or more threads that executed communication phase 608 can be released to perform other operations, while the same or another CUDA thread bundle executes polling CQ phase 612.

[0172] During the polling phase 612, CQ can be polled at position 636. Polling CQ can be combined with the above-mentioned... Figure 4 The method described in 400 polls CQ at position 416 in the same or similar manner. By polling CQ at position 636, it is possible to determine whether a timeout occurred at position 640. Determining whether a timeout occurred at position 640 can be related to the above-mentioned... Figure 4 The features used in method 400 to determine whether a timeout has occurred at 420 are the same or similar.

[0173] If a timeout occurs, polling the CQ phase 612 may include: at 644, updating the DBR and ringing the DB to the pointer set in the BF register at 624. On the other hand, if no timeout occurs (or after updating the DBR and ringing the DB), method 600 may end at 648.

[0174] The systems and methods described herein can provide a post-send algorithm that is particularly useful in low message rate use cases where low latency is a critical factor. Such systems and methods can be implemented as a feature along with other post-send algorithms that provide high message rates. In some implementations, the QP can be divided into two groups: 1) high message rate and 2) low latency. If the user or application prioritizes high message rate, a high message rate post-send algorithm can be used. If the user or application prioritizes low latency (which may come at the expense of message rate), a low latency post-send algorithm described herein can be used.

[0175] In some implementations, the low-level communication API can be configured to accept a flag to distinguish between the two use cases. A flag can be provided for the QP that can be used to determine whether a high message rate algorithm or a low latency algorithm should be used to process the QP.

[0176] Such low-level APIs can be exposed to applications, or in some implementations, heuristics can be created at higher layers to determine when to use a high message rate and when to use a low latency algorithm. Such heuristics can consider factors such as the busyness of the NIC (or other peripherals), the number of WQEs in transit at any given time (which can be tracked, for example, by libraries such as NVSHMEM), and / or other factors to make an informed decision about whether to use a low-latency or high-message rate algorithm. For example, in some implementations, a low-latency group is only used when the number of WQEs in transit is less than a threshold.

[0177] The aforementioned aspects of the systems and network devices include any one or more features substantially disclosed herein combined with any one or more other features substantially disclosed herein.

[0178] The aspects of the aforementioned systems and network devices include any one aspect / feature / embodiment combined with any one or more other aspects / features / embodiments.

[0179] The aforementioned aspects of the systems and network devices include any use of any one or more aspects or features disclosed herein.

[0180] It should be understood that any feature described herein may be combined with any other feature described herein for claim protection, regardless of whether such features are derived from the same described embodiment.

[0181] The foregoing discussion of this disclosure is for illustrative and descriptive purposes only. It is not intended to limit this disclosure to the form disclosed herein. For example, in the foregoing detailed description, various features of this disclosure are combined in one or more embodiments, configurations, or aspects to simplify the disclosure. Features of embodiments, configurations, or aspects of this disclosure may be combined in alternative embodiments, configurations, or aspects other than those described above. This approach to disclosure should not be construed as reflecting an intention that the claimed disclosure requires more features than those expressly listed in each claim. Rather, as reflected in the following claims, the inventive aspect lies in fewer than all features of a single foregoing disclosed embodiment, configuration, or aspect. Therefore, the following claims are hereby incorporated into this detailed description, each claim being an independent preferred embodiment of this disclosure.

[0182] Furthermore, while the description of this disclosure includes descriptions of one or more embodiments, configurations, or aspects, as well as certain variations and modifications, other variations, combinations, and modifications are also within the scope of this disclosure, for example, those variations, combinations, and modifications that may be within the skill and knowledge of someone skilled in the art after understanding this disclosure. This invention is intended to be entitled to include alternative embodiments, configurations, or aspects within the permissible scope, including structures, functions, scopes, or steps that are substitutes for, interchangeable with, and / or equivalent to the structures, functions, scopes, or steps claimed in the claims, regardless of whether such substitutes, interchangeable with, and / or equivalent structures, functions, scopes, or steps are disclosed herein, and is not intended to disclose any patentable subject matter.

Claims

1. A system comprising one or more circuits, said one or more circuits being used for: Prepare work queue entries (WQE); Transmit the WQE to the target device or provide a doorbell indication to the target device, indicating that the WQE is available; and In response to determining that an execution failure has occurred, at least one step is performed to facilitate the completion of the operation associated with the WQE.

2. The system according to claim 1, wherein, The WQE is written to memory before it is transmitted or the doorbell indication is provided.

3. The system according to claim 1, wherein, The execution failure includes failing to receive a completion instruction within the predetermined time frame.

4. The system according to claim 1, wherein, The execution failure includes receiving a status indicating failure.

5. The system according to claim 1, wherein, The one or more circuits are further configured to: after determining that the execution failure has occurred, repeatedly transmit the WQE to the target device or provide the doorbell indication to the target device.

6. The system according to claim 1, wherein, The at least one step used to facilitate the completion of the operation associated with the WQE includes: writing a doorbell record to a memory and repeatedly transmitting the WQE to the target device, or providing the doorbell indication to the target device.

7. The system according to claim 1, wherein, The one or more circuits are also used to write the WQE into memory.

8. The system according to claim 7, wherein, Providing the doorbell indication to the target device includes writing the doorbell DB to the memory of the target device, wherein the DB enables the target device to read the WQE.

9. The system according to claim 1, wherein, The at least one step used to facilitate the completion of the operation associated with the WQE includes: updating the doorbell record DBR and performing a memory barrier.

10. The system according to claim 9, wherein, The at least one step used to facilitate the completion of the operation associated with the WQE further includes: retransmitting the WQE to the target device, or providing the target device with a second doorbell indication indicating that the WQE is available.

11. The system according to claim 1, wherein, Transmitting the WQE to the target device includes writing the WQE into the memory of the target device.

12. The system according to claim 1, wherein, The one or more circuits are also used to mark the WQE buffer.

13. The system according to claim 12, wherein, The WQE is transmitted to the target device without being written to the memory.

14. The system according to claim 12, wherein, After the WQE is written to the memory, the WQE is transmitted to the target device, wherein no memory barrier is performed before the WQE is transmitted to the target device.

15. The system according to claim 1, wherein, The at least one step used to facilitate the completion of the operation associated with the WQE includes: Write the WQE into memory; Execute memory barriers; and Ring the doorbell DB.

16. The system according to claim 15, wherein, The at least one step further includes: updating the doorbell record DBR and implementing a second memory barrier.

17. The system according to claim 16, wherein, Updating the DBR includes updating the DBR using the producer index.

18. The system according to claim 1, wherein, The one or more circuits are configured to repeatedly perform at least one of preparing the WQE and transmitting the WQE to the target device or providing the doorbell indication before performing the at least one step for facilitating the completion of the operation associated with the WQE.

19. The system according to claim 1, wherein, The execution failure includes receiving one or more completion indications, which include entries in the completion queue CQ, status in a memory location, asynchronous events, and interrupts.

20. The system according to claim 19, wherein, Receiving the completion indication includes polling the CQ.

21. The system according to claim 20, wherein, The one or more circuits execute fewer than two memory barriers before polling the CQ.

22. The system according to claim 1, wherein, Transmitting the WQE to the target device includes: using multiple threads to write the WQE into the memory of the target device.

23. The system according to claim 22, wherein, Write consecutive WQEs to a single address in the memory of the target device.

24. The system according to claim 22, wherein, Write consecutive WQEs to different addresses in the memory of the target device.

25. The system according to claim 22, wherein, The one or more circuits are also used to synchronize the multiple threads after the WQE is prepared and before the WQE is transmitted.

26. The system according to claim 1, wherein, Before preparing the WQE, the one or more circuits determine that the number of WQEs in transit is less than a threshold.

27. A communication device comprising one or more circuits, said one or more circuits being used to: Prepare work queue entries (WQE); Transmit the WQE to the target device or provide a doorbell indication to the target device, indicating that the WQE is available; and In response to determining that an execution failure has occurred, at least one step is performed to facilitate the completion of the operation associated with the WQE.

Images