Cross-node communication system and method, electronic equipment and storage medium

Abstract

The invention relates to the technical field of computers, in particular to a cross-node communication system and method, electronic equipment and a storage medium, and the system comprises a host which is used for coordinating training tasks of a plurality of computing nodes; the computing nodes are mutually connected through a network, and each computing node comprises a streaming multiprocessor and special communication hardware; wherein an equipment process runs on the single computing node; and the equipment process is used for calling the special communication hardware to execute a cross-node data communication operation so as to replace the streaming multiprocessor to execute an inter-node communication task. According to the embodiment of the invention, the resource contention between the computing task and the communication task can be effectively reduced, and the utilization efficiency of computing resources is improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a cross-node communication system, method, apparatus and electronic device. Background Technology

[0002] As the scale of artificial intelligence models continues to expand, single devices can no longer meet the storage and computing requirements for training, making distributed training the mainstream solution. In a distributed training system, multiple computing nodes are typically interconnected via a network to collaboratively complete training tasks. Each computing node generally includes processing units (such as streaming multiprocessors) for performing numerical computations and communication resources for data exchange between nodes.

[0003] Typically, in the aforementioned systems, data communication between nodes is handled or coordinated by processing units (such as streaming multiprocessors) that perform computational tasks. This means that the communication process consumes processing resources that would otherwise be used for numerical computation. When processing units perform computational and communication tasks simultaneously, they compete for limited hardware resources. This resource contention can lead to delays or interruptions in computational tasks, thereby affecting the overall efficiency of the training task. Summary of the Invention

[0004] In view of this, this disclosure proposes a cross-node communication technology solution.

[0005] According to one aspect of this disclosure, a cross-node communication system is provided, comprising:

[0006] The host is used to coordinate training tasks across multiple computing nodes.

[0007] Multiple computing nodes are interconnected via a network, and a single computing node includes streaming multiprocessors and dedicated communication hardware;

[0008] In this embodiment, a device process runs on a single computing node; the device process is used to call the dedicated communication hardware to perform cross-node data communication operations, in order to replace the streaming multiprocessor in performing inter-node communication tasks.

[0009] In one possible implementation, the host runs a host process, which is used for:

[0010] Establish a memory access mapping for the multiple computing nodes so that any computing node can directly access the memory of other computing nodes;

[0011] After the video memory access mapping is established, each device process is instructed to create a communication buffer in the video memory of the local computing node;

[0012] Collect the addresses of the communication buffers created by each of the device processes, and distribute all the collected addresses to each of the device processes so that each device process obtains the communication buffer addresses of all peer computing nodes.

[0013] In one possible implementation, the device process is used to:

[0014] Based on the communication buffer addresses of all peer computing nodes received, create multiple communication streams for this computing node corresponding to the number of peer computing nodes;

[0015] Assign a first synchronization flag and a second synchronization flag to this computing node. The first synchronization flag is used for synchronization during the cross-node computing phase, and the second synchronization flag is used for synchronization during the intra-node communication phase.

[0016] In one possible implementation, the device process is used to:

[0017] Each of the communication streams is bound to an independent queue of the dedicated communication hardware, so that multiple communication streams can perform data operations in parallel through the dedicated communication hardware.

[0018] In one possible implementation, the device process is used to:

[0019] After executing the local computing task, the computing results are written to the communication buffer of this computing node;

[0020] Through each of the aforementioned communication streams, an atomic increment operation is performed on the first synchronization flag of the corresponding peer computing node to notify the peer computing node that the node has completed the computation.

[0021] In one possible implementation, the device process is used to:

[0022] When the value of the first synchronization flag reaches the first preset threshold, the dedicated communication hardware is invoked to read data from the communication buffer of the peer computing node to the communication buffer of this computing node based on the communication buffer address of the peer computing node.

[0023] Wherein, the first preset threshold is the number of peer computing nodes, and the value of the first synchronization flag is obtained by each peer computing node by performing an atomic increment operation on it.

[0024] In one possible implementation, the device process is used to:

[0025] During the process of reading data through the dedicated communication hardware, if the current communication mode is the reduction distributed mode, the dedicated communication hardware is instructed to perform an atomic accumulation operation by calling the atomic operation interface provided by the software stack, and the read data is accumulated with the data at the corresponding position in the communication buffer of this computing node.

[0026] When the current communication mode is full collection mode, the asynchronous copy interface provided by the software stack is called to instruct the dedicated communication hardware to perform a data copy operation, copying the read data to the corresponding position in the communication buffer of this computing node.

[0027] In one possible implementation, the device process is used to:

[0028] After the dedicated communication hardware completes the data reading and processing operation on a peer computing node through each communication stream, it performs an atomic increment operation on the second synchronization flag of the local computing node.

[0029] After the value of the second synchronization flag reaches the second preset threshold, the communication results in the communication buffer of this computing node are copied to the preset output storage location, where the second preset threshold is the number of peer computing nodes.

[0030] In one possible implementation, the device process is also configured with a unified communication interface for use by upper-layer algorithms to perform communication operations; the device process is used for:

[0031] The system receives communication requests from upper-layer algorithms through a unified communication interface and parses the configuration parameters in the communication requests.

[0032] Based on the communication algorithm type in the parsed configuration parameters, the system selects to call the atomic operation interface of the dedicated communication hardware to perform an atomic accumulation operation, or to call the asynchronous copy interface to perform a data copy operation.

[0033] Based on the other configuration parameters parsed out, control the data segmentation, storage location, and data type conversion during the communication process.

[0034] In one possible implementation, each of the computing nodes is connected to a fully interconnected network, enabling any one of the computing nodes to directly read and write the communication buffers of other computing nodes via the dedicated communication hardware.

[0035] According to another aspect of this disclosure, a cross-node communication method is provided, applied to a cross-node communication system, the system including a host and multiple computing nodes interconnected via a network, each computing node including a streaming multiprocessor and dedicated communication hardware; each computing node running a device process; the method includes:

[0036] The device process invokes the dedicated communication hardware to perform cross-node data communication operations, replacing the streaming multiprocessor in performing inter-node communication tasks.

[0037] According to another aspect of this disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described method.

[0038] According to another aspect of this disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described method.

[0039] According to another aspect of this disclosure, a computer program product is provided, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above-described method.

[0040] In this embodiment, the cross-node communication system includes a host and multiple computing nodes. The host coordinates the training tasks of the multiple computing nodes, which are interconnected via a network. Each computing node includes a streaming multiprocessor and dedicated communication hardware. A device process runs on each computing node, and this device process invokes the dedicated communication hardware to perform cross-node data communication operations, replacing the streaming multiprocessor in performing inter-node communication tasks. Therefore, during distributed training, when computing nodes need to perform cross-node data communication, the device process invokes the dedicated communication hardware to perform the communication operation, freeing the streaming multiprocessor from inter-node communication tasks. This avoids the communication operations consuming the streaming multiprocessor's computing resources and reduces resource contention between computing and communication tasks. As a result, the streaming multiprocessor can focus more on executing local computing tasks, improving the utilization efficiency of computing resources and thus contributing to the overall training performance and operating efficiency of the distributed training system.

[0041] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0042] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0043] Figure 1 A block diagram of a cross-node communication system according to an embodiment of the present disclosure is shown.

[0044] Figure 2This is a block diagram illustrating an apparatus for cross-node communication according to an exemplary embodiment. Detailed Implementation

[0045] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0046] As used herein, the terms “comprising,” “including,” “having,” or variations thereof are open-ended and include one or more of the stated features, integrals, elements, steps, components, or functions, but do not exclude the presence or addition of one or more other features, integrals, elements, steps, components, functions, or groups thereof.

[0047] When an element is referred to as “connected,” “coupled,” “responding,” or a variation thereof relative to another element, it may be directly connected, coupled, or responding to another element, or there may be an intermediate element present.

[0048] Although the terms first, second, third, etc., may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another. Therefore, without departing from the teachings of the inventive concept, a first element / operation in some embodiments may be referred to as a second element / operation in other embodiments.

[0049] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0050] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0051] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant regions.

[0052] In distributed training systems, multiple computing nodes typically work together to handle large-scale model training tasks. Each computing node is usually equipped with a core processing unit (such as a streaming multiprocessor) for performing numerical computations and a communication interface for data exchange between nodes.

[0053] In related technologies, cross-node data communication tasks are often initiated or processed by the same processing unit (i.e., a streaming multiprocessor) that executes local computing tasks. This means that during inter-node data synchronization or transmission, communication operations require the computing resources and execution cycles of the streaming multiprocessor. When the streaming multiprocessor processes computing and communication tasks simultaneously, they may compete for limited hardware resources, causing delays or interruptions in the computing process, which may affect the overall throughput efficiency of the training task.

[0054] Based on this, the present disclosure provides a cross-node communication technology solution to at least partially improve the above situation.

[0055] Figure 1 A block diagram of a cross-node communication system according to an embodiment of the present disclosure is shown. Figure 1 As shown, the system includes:

[0056] Host 11 is used to coordinate the training tasks of multiple computing nodes 12;

[0057] Multiple computing nodes 12 are interconnected via a network, and each computing node includes a streaming multiprocessor 121 and dedicated communication hardware 122;

[0058] In this embodiment, a device process 123 runs on a single computing node; the device process 123 is used to call the dedicated communication hardware 122 to perform cross-node data communication operations, in order to replace the streaming multiprocessor 121 in performing inter-node communication tasks.

[0059] A host can be a logical entity or physical device that performs management, scheduling, or coordination functions in a distributed training architecture. It can execute operations through hosts running on top of hosts. In some implementations, a host can be a standalone server, a control process running on a computing node, or a software-defined coordination service. The host coordinates the training tasks of multiple computing nodes by distributing task instructions to each node, synchronizing training states, or managing resource allocation, enabling these nodes to collaboratively achieve a unified model training objective.

[0060] Computation nodes are the specific computational units participating in distributed training. These nodes are interconnected via a network to exchange data and synchronize information. Each computation node, as a complete computational entity, may contain processing units that perform core mathematical operations (such as matrix multiplication and convolution), as well as communication components for sending and receiving data with the network or other nodes. For example, a computation node may be an accelerator card containing a graphics processing unit (GPU) or a virtual machine instance in the cloud.

[0061] In a compute node, a streaming multiprocessor (SM) is a processing unit used to perform massively parallel computing tasks. A streaming multiprocessor can contain multiple computing cores and can efficiently handle tensor operations common in deep learning training. Dedicated communication hardware, on the other hand, is physical circuitry or a coprocessor integrated into the compute node specifically designed to handle data communication between nodes.

[0062] This dedicated communication hardware operates independently of the streaming multiprocessor and has a dedicated data processing path and interface for performing communication-related operations such as data packing, sending via the network interface, receiving from the network, and data unpacking.

[0063] This dedicated communication hardware supports a variety of operations directly addressing communication needs. For example, data copying, i.e., moving data blocks between different storage areas. Exemplarily, this operation can be initiated by calling a specific asynchronous memory copy interface, enabling direct data transfer between different storage locations within a compute node, or cross-node transfer between the video memory of different compute nodes, without requiring any instructions from the streaming multiprocessor. This dedicated communication hardware also supports atomic operations, particularly atomic accumulation for half-precision floating-point data types. For example, the dedicated communication hardware can perform atomic addition operations on BF16 or FP16 formatted data, completing the read, add, and write operations in a single, indivisible step, suitable for scenarios requiring the merging of multiple data sources.

[0064] Working in conjunction with this dedicated communication hardware is a software stack that provides a hardware-oriented programming interface. Atomic operation interfaces allow software (such as device processes) to directly instruct the dedicated communication hardware to perform atomic operations. By calling such interfaces, software can initiate an atomic accumulation request, which is received and executed by the dedicated communication hardware, thus avoiding the overhead of traditionally requiring a complete kernel function to run on a streaming multiprocessor.

[0065] In addition, the software stack provides synchronization primitives, specifically implemented through atomic operation interfaces and combinations of synchronization interfaces combined with data copy and stream wait value interfaces. The atomic operation interface instructs dedicated communication hardware to perform atomic accumulation operations. The synchronization interface combined with data copy can be used to atomically update a flag value located in video memory while the dedicated communication hardware completes a data operation. The stream wait value interface allows an execution flow to pause its subsequent tasks until the value of a certain video memory address it is monitoring becomes the expected condition. Through this combination, conditional waiting between different execution flows can be implemented to coordinate the timing dependencies between computational and communication tasks, ensuring that subsequent operations only begin after the necessary preceding operations have been completed.

[0066] On a single compute node, a device process runs. This device process can be a software-implemented instance of control logic that manages the compute and communication resources on that node. In response to training task requirements, when cross-node data exchange (such as gradient synchronization or parameter broadcasting) is needed, the device process invokes dedicated communication hardware on the local compute node to perform the specific cross-node data communication operations.

[0067] By invoking dedicated communication hardware through device processes, inter-node communication tasks that originally required the participation or control of streaming multiprocessors can be delegated to the dedicated communication hardware. In other words, when data needs to be sent to other computing nodes or received from other computing nodes during training, the device process will schedule these communication requests to the dedicated communication hardware for execution.

[0068] During this process, the streaming multiprocessor can continue to execute its local core computational tasks such as model forward propagation and backward gradient calculation, without having to allocate computing resources and execution cycles to handle communication details such as data packet preparation, sending, or receiving acknowledgments. In this way, the two different types of tasks, computation and communication, can be separated at the hardware level and processed in parallel by different dedicated components.

[0069] In this embodiment, the cross-node communication system includes a host and multiple computing nodes. The host coordinates the training tasks of the multiple computing nodes, which are interconnected via a network. Each computing node includes a streaming multiprocessor and dedicated communication hardware. A device process runs on each computing node, and this device process invokes the dedicated communication hardware to perform cross-node data communication operations, replacing the streaming multiprocessor in performing inter-node communication tasks. Therefore, during distributed training, when computing nodes need to perform cross-node data communication, the device process invokes the dedicated communication hardware to perform the communication operation, freeing the streaming multiprocessor from inter-node communication tasks. This avoids the communication operations consuming the streaming multiprocessor's computing resources and reduces resource contention between computing and communication tasks. As a result, the streaming multiprocessor can focus more on executing local computing tasks, improving the utilization efficiency of computing resources and thus contributing to the overall training performance and operating efficiency of the distributed training system.

[0070] In one possible implementation, the host runs a host process, which is used for:

[0071] Establish a memory access mapping for the multiple computing nodes so that any computing node can directly access the memory of other computing nodes;

[0072] After the video memory access mapping is established, each device process is instructed to create a communication buffer in the video memory of the local computing node;

[0073] Collect the addresses of the communication buffers created by each of the device processes, and distribute all the collected addresses to each of the device processes so that each device process obtains the communication buffer addresses of all peer computing nodes.

[0074] A host process can run on the host machine. A host process is a software instance or service that runs on the host machine and is used to perform specific resource coordination and management operations. Before communication begins, the host process first establishes a memory access mapping for all computing nodes participating in the computation.

[0075] Video memory mapping is a hardware and driver-supported feature that establishes direct access channels between the video memory address spaces of multiple compute nodes. By establishing this mapping, a processing unit or communication hardware on one compute node can directly read or write to a specific region of video memory on another compute node without going through the host memory.

[0076] For example, this mapping can be established by calling a specific inter-process communication interface, enabling all computing nodes participating in distributed training to access each other's GPU memory space.

[0077] After the mapping is established, the host process issues instructions to the device processes running on each compute node, instructing each device process to create a communication buffer in the local video memory of its respective compute node. After creating its own communication buffer, each device process reports the starting address information of the buffer to the host process.

[0078] A communication buffer is a pre-allocated storage area in the video memory of a computing node, specifically used to temporarily store communication data to be sent or received. For example, during gradient synchronization or parameter aggregation phases, intermediate results generated by computation can be written to the local node's communication buffer for the peer node to read, or data can be read from the peer node's communication buffer for local aggregation.

[0079] The host process collects the communication buffer addresses from all device processes, forming a global address information set. The host process then distributes this set to each device process. In this way, each device process not only knows the location of its own communication buffer but also obtains the address information of the communication buffers of all other peer computing nodes.

[0080] In this context, a peer computing node refers to another computing node that exchanges data with the current computing node during a specific communication operation. In distributed training, a computing node may need to communicate with one, several, or even all other computing nodes, all of which are its current peer computing nodes.

[0081] In this embodiment, under the coordination of the host process, the host process establishes a memory access mapping for multiple computing nodes, enabling any computing node to directly access the memory of other computing nodes. After the memory access mapping is established, each device process is instructed to create a communication buffer in the memory of its own computing node. The addresses of the communication buffers created by each device process are collected, and all collected addresses are distributed to each device process, so that each device process obtains the communication buffer addresses of all peer computing nodes. Thus, through the centralized coordination and information distribution of the host, each computing node has completed the resource preparation and address awareness required for communication before starting actual data communication. This avoids the need for each node to separately perform complex address negotiation and discovery during communication, simplifying the communication initialization process. Based on the complete address information obtained, each device process can accurately drive dedicated communication hardware to directly read and write the memory buffers of peer nodes, thereby creating conditions for subsequent efficient and direct cross-node data communication operations and helping to improve the efficiency of the communication phase in distributed training.

[0082] In one possible implementation, the device process is used to:

[0083] Based on the communication buffer addresses of all peer computing nodes received, create multiple communication streams for this computing node corresponding to the number of peer computing nodes;

[0084] Assign a first synchronization flag and a second synchronization flag to this computing node. The first synchronization flag is used for synchronization during the cross-node computing phase, and the second synchronization flag is used for synchronization during the intra-node communication phase.

[0085] After obtaining the communication buffer addresses of all peer computing nodes, the device process can prepare further communication resources based on this address information. The device process will create multiple communication streams for its own computing node based on the received address information. By creating multiple independent communication streams, the device process can schedule communication tasks destined for different peer computing nodes into different streams.

[0086] In this context, communication flows are execution queues on computing nodes used to schedule and order a series of related operations (such as memory copying). Tasks in these flows can be executed in parallel or in an overlapping manner, which helps improve the utilization of hardware resources and communication throughput.

[0087] Specifically, the number of communication streams can correspond to the number of peer computing nodes. For example, in a training task with 8 computing nodes, if one of the computing nodes has 7 peer computing nodes, then the device process on that computing node can create 7 communication streams, each of which can be used to manage communication tasks with a specific peer computing node.

[0088] A synchronization flag is a specific variable stored in video memory whose value can be atomically read and modified by multiple execution units (such as different streaming multiprocessors or different communication streams). Synchronization flags are used to establish a coordination mechanism between multiple concurrently executing tasks.

[0089] The device process can assign two synchronization flags to this computing node: a first synchronization flag and a second synchronization flag. The first synchronization flag is mainly used for synchronization of computing phases across nodes. In one example, when each computing node completes a portion of its local computing task (such as a matrix multiplication block), it can notify other nodes by updating the first synchronization flag. All nodes then check the value of this flag to determine whether they can jointly enter the next phase involving data exchange, thereby coordinating any potential differences in computing speed between different computing nodes.

[0090] The second synchronization flag is used for communication phase synchronization within a node. For example, when a compute node uses multiple communication streams it has created to send data read requests to all peer compute nodes in parallel, each communication stream updates the second synchronization flag upon completing its corresponding data read and processing operations. The device process monitors whether the value of the second synchronization flag reaches the expected level (e.g., equal to the number of peer nodes) to determine whether all parallel communication subtasks of its node have been completed, thereby determining whether it is safe to proceed with subsequent operations, such as copying the aggregation result from the communication buffer to the final output location.

[0091] In this embodiment, the device process creates multiple communication streams corresponding to the number of peer computing nodes for its own computing node based on the received communication buffer addresses of all peer computing nodes. It also assigns a first synchronization flag and a second synchronization flag to the computing node. The first synchronization flag is used for synchronization during cross-node computing phases, and the second synchronization flag is used for synchronization during intra-node communication phases. Thus, by creating independent communication streams for each peer node, the device process can organize concurrent communication tasks, fully utilize the parallel processing capabilities of dedicated communication hardware, and improve the efficiency of cross-node data exchange. Simultaneously, by setting and utilizing the first and second synchronization flags, precise synchronization control is achieved in both the global computing progress and local communication completion dimensions. This synchronization mechanism effectively coordinates the inconsistency in execution speed caused by task load or hardware differences, ensuring that data is accessed and processed at the correct time, avoiding computational errors caused by data contention or incomplete reading, thereby guaranteeing the correctness and stability of distributed training tasks in high-concurrency communication modes.

[0092] In one possible implementation, each of the device processes is further configured to:

[0093] Each of the communication streams is bound to an independent queue of the dedicated communication hardware, so that multiple communication streams can perform data operations in parallel through the dedicated communication hardware.

[0094] Device processes can perform more granular hardware resource management for the execution scheduling of communication tasks. Here, an independent queue of dedicated communication hardware refers to an independent physical or logical channel within the hardware used for receiving, sorting, and executing operation commands. Each queue can independently process the sequence of instructions dispatched to it without interference. Binding the communication flow at the software level to the independent queue at the hardware level ensures that all communication task instructions in the communication flow are sent to the designated hardware queue for queuing and execution.

[0095] In this implementation, after each device process creates multiple communication streams corresponding to the peer computing node, it can further perform binding operations. The device process assigns and associates each communication stream to an independent queue of dedicated communication hardware. For example, if the dedicated communication hardware provides 8 independent command queues, and the device process creates 7 communication streams, it can bind the 7 streams to 7 different hardware queues respectively.

[0096] Through this one-to-one binding relationship, when a device process initiates multiple asynchronous communication operations (such as simultaneously reading data from multiple peer nodes) through different communication flows, these operation requests are distributed to different independent queues in the dedicated communication hardware. The hardware can retrieve instructions from these queues in parallel and execute them, enabling data transmission operations for different peer nodes to be truly concurrent at the hardware level.

[0097] In this embodiment of the disclosure, each device process also binds its respective communication stream to an independent queue of the dedicated communication hardware, enabling multiple communication streams to perform data operations in parallel through the dedicated communication hardware. Thus, by establishing a binding from logical communication streams to physical hardware queues, the device process achieves isolation and dedicated allocation of communication tasks on hardware execution resources. This binding mechanism avoids instruction contention or transmission blocking problems that may occur when multiple concurrent communication streams are in a single hardware queue, allowing the dedicated communication hardware to fully utilize its internal multi-queue parallel processing capabilities.

[0098] In one possible implementation, the device process is used to:

[0099] After executing the local computing task, the computing results are written to the communication buffer of this computing node;

[0100] Through each of the aforementioned communication streams, an atomic increment operation is performed on the first synchronization flag of the corresponding peer computing node to notify the peer computing node that the node has completed the computation.

[0101] Local computation tasks are the core computational work performed by the streaming multiprocessors of the compute nodes. For example, in distributed training scenarios, this typically includes forward propagation of a neural network model and gradient calculation during backpropagation. The results of these tasks, such as updated gradient tensors or intermediate activation values ​​that need to be aggregated, are the objects that subsequent cross-node communication operations need to process.

[0102] After completing its local computing task, the device process writes the resulting data to a pre-created communication buffer on the local computing node. This communication buffer acts as a data relay, moving the computing results from the dedicated storage area of ​​the computing core to an address space directly accessible to the communication hardware, thus preparing for subsequent cross-node data transmission.

[0103] Subsequently, the device process utilizes the multiple communication streams created earlier to perform notification operations. For each communication stream, the device process initiates an atomic increment operation on the first synchronization flag of the peer computing node corresponding to that communication stream.

[0104] An atomic increment operation is an atomic operation whose indivisibility is guaranteed by hardware. It increments an integer value stored at a specified memory address by 1. This operation guarantees the correctness of the result when multiple threads or processes concurrently access the same memory location; that is, each increment takes effect and is not lost due to concurrent access. In cross-node synchronization scenarios, this operation can be performed using dedicated communication hardware to update the synchronization flag located in the video memory of the peer computing node.

[0105] For example, if the current computing node has 3 peer nodes and 3 communication streams are created accordingly, the device process will increment the first synchronization flag of each of the 3 peer nodes by one through these 3 streams.

[0106] The essence of performing an atomic increment operation on the first synchronization flag is that the current computing node sends a signal to each peer node, informing them that "this node has completed its task for the current computing phase, and the relevant result data has been stored in this node's communication buffer." The device process on each peer node can continuously monitor the value of its own first synchronization flag. When it detects an increase in the flag's value, it knows that another peer node is ready. By accumulating the number of received increments, each node can perceive the computation completion status of all relevant nodes.

[0107] In this embodiment, after executing a local computation task, the device process writes the computation result into the communication buffer of its local computing node. Through each communication stream, it performs an atomic increment operation on the first synchronization flag of the corresponding peer computing node to notify that the local computing node has completed the computation. Thus, by centrally storing the computation results in the communication buffer, a unified source is created for direct data reading by dedicated communication hardware, avoiding complex interactions between the communication hardware and the computing unit when accessing data. By concurrently sending atomic increment signals to all peer nodes using communication streams, the device process implements an efficient and lightweight distributed completion status broadcast mechanism. Compared to centralized polling or complex message passing protocols, this allows each node to quickly and accurately perceive the global computation progress, thereby providing precise triggering conditions for subsequent communication phases (such as aggregation operations) that depend on the readiness of data from all nodes. This reduces idle waiting time between nodes and improves the overall efficiency of the training pipeline.

[0108] In one possible implementation, the device process is used to:

[0109] When the value of the first synchronization flag reaches the first preset threshold, the dedicated communication hardware is invoked to read data from the communication buffer of the peer computing node to the communication buffer of this computing node based on the communication buffer address of the peer computing node.

[0110] Wherein, the first preset threshold is the number of peer computing nodes, and the value of the first synchronization flag is obtained by each peer computing node by performing an atomic increment operation on it.

[0111] The first preset threshold here is a pre-defined numerical condition used to determine whether a certain synchronization stage has met the prerequisites for continuing subsequent operations. In this implementation, the first preset threshold can be the number of all peer computing nodes.

[0112] The device process can continuously monitor the value changes of the first synchronization flag. The value of this flag is accumulated by all peer computing nodes performing atomic increment operations on it. When the device process detects that the value of the first synchronization flag reaches a first preset threshold, which is equal to the total number of peer computing nodes, it determines that all peer computing nodes that need to exchange data have completed their local computing tasks (such as matrix multiplication Gemm calculations) and stored the results in their respective communication buffers. The entire system has reached a consistent state for the current computing phase and can safely enter the data exchange phase.

[0113] At this point, the device process begins to invoke the dedicated communication hardware of this computing node to perform data transfer. Based on the communication buffer addresses of all peer computing nodes, the device process initiates an independent asynchronous read request for each peer node, reading data from a specified position in the communication buffer of each peer node into the corresponding position in its own communication buffer.

[0114] Because these read operations are performed asynchronously by dedicated communication hardware, the device process does not need to block and wait for each read to complete after initiating this series of requests. The communication hardware will use its internal resources to process these cross-node data transfer tasks in parallel.

[0115] In this embodiment, when the value of the first synchronization flag reaches a first preset threshold, the device process invokes the dedicated communication hardware to read data from the communication buffer of the peer computing node to its own communication buffer based on the address of the peer computing node's communication buffer. The first preset threshold is the number of peer computing nodes, and the value of the first synchronization flag is obtained by each peer computing node through an atomic increment operation. Therefore, by explicitly defining the first preset threshold as the number of peer nodes and using the first synchronization flag reaching this threshold as a unified condition for triggering communication, the data reading operation is ensured to start only when all necessary data is ready. This fundamentally avoids the problem of reading invalid or intermediate data due to incomplete computation by some nodes, ensuring the correctness of the communication results. Simultaneously, using dedicated communication hardware for reading offloads large-scale data transfer work from the computing core, allowing for deep overlap between computation and communication. This significantly improves the overall throughput efficiency of the cross-node data aggregation stage and reduces the training iteration cycle time.

[0116] In one possible implementation, the device process is used to:

[0117] During the process of reading data through the dedicated communication hardware, if the current communication mode is the reduction distributed mode, the dedicated communication hardware is instructed to perform an atomic accumulation operation by calling the atomic operation interface provided by the software stack, and the read data is accumulated with the data at the corresponding position in the communication buffer of this computing node.

[0118] When the current communication mode is full collection mode, the asynchronous copy interface provided by the software stack is called to instruct the dedicated communication hardware to perform a data copy operation, copying the read data to the corresponding position in the communication buffer of this computing node.

[0119] When managing dedicated communication hardware to perform asynchronous data reading, the device process can adopt corresponding data processing strategies based on different global communication needs. The communication mode defines the flow and processing method of data between nodes in a cross-node data exchange. For example, the reduction-distribution mode typically involves summarizing (reducing) a portion of the data from each node and then distributing the different parts of the complete result to each node; while the full collection mode refers to each node collecting data from all other nodes.

[0120] The software stack provides hardware-oriented programming interfaces to encapsulate the complexity of low-level operations. Atomic operation interfaces allow callers to instruct the hardware to perform an indivisible read-modify-write operation, such as adding a value to an existing value at a target address. Asynchronous copy interfaces allow callers to instruct the hardware to perform a data transfer task and return immediately after the task is submitted, without waiting for the transfer to complete.

[0121] In some application scenarios, the data to be output is not stored contiguously in memory; its elements are logically organized by rows, columns, or higher dimensions, while existing at fixed address intervals in physical memory. To efficiently and accurately handle the movement of such data with non-contiguous memory layouts, the software stack further provides an asynchronous interface that supports two-dimensional data copying. This interface can precisely control the copying process based on the independent row length and row span parameters of the source and target data, thereby correctly and accurately moving the result data, which may have a specific storage structure in the communication buffer, to the final output location, while maintaining its original logical data layout.

[0122] When the device process drives the dedicated communication hardware to asynchronously read data from the peer node, it determines how to process the read data based on the current communication mode configured in the training task. If the current communication mode is a reduction-distribution mode, it indicates that the purpose of this communication is not only to collect data, but also to merge and calculate (e.g., sum) data from multiple nodes. In this case, the device process instructs the dedicated communication hardware to perform an atomic accumulation operation by calling the atomic operation interface provided by the software stack.

[0123] Specifically, while transmitting the data block read from the peer node to the local node's communication buffer, the hardware adds the data block to the existing value at the corresponding position in the buffer and writes the result back to the original position. This implements a distributed accumulation operation directly performed by the communication hardware.

[0124] Conversely, if the current communication mode is full collection mode, the purpose of communication is mainly to collect data, without the need for real-time computation. In this case, the device process instructs the dedicated communication hardware to perform standard data copy operations by calling the asynchronous copy interface provided by the software stack. The hardware simply places the data blocks read from the peer node into the pre-planned corresponding positions in the local node's communication buffer, overwriting the original content at those positions or filling empty areas.

[0125] In this embodiment, when the device process asynchronously reads data through the dedicated communication hardware, if the current communication mode is a reduction-distributed mode, it instructs the dedicated communication hardware to perform an atomic accumulation operation by calling the atomic operation interface provided by the software stack, accumulating the read data with the data at the corresponding position in the communication buffer of the local computing node. If the current communication mode is a full collection mode, it instructs the dedicated communication hardware to perform a data copy operation by calling the asynchronous copy interface provided by the software stack, copying the read data to the corresponding position in the communication buffer of the local computing node. Thus, by flexibly switching the software interface called within the same hardware communication process according to the requirements of higher-level algorithms, the dedicated communication hardware can adapt to various mainstream distributed communication primitives. This approach pushes the data computation logic (such as accumulation) required for specific communication modes down to the communication hardware level, avoiding the additional overhead of reading data back to the computing unit for processing and then writing it back, further strengthening the separation of computation and communication.

[0126] In one possible implementation, the device process is also configured with a unified communication interface for use by upper-layer algorithms to perform communication operations; the device process is used for:

[0127] The system receives communication requests from upper-layer algorithms through a unified communication interface and parses the configuration parameters in the communication requests.

[0128] Based on the communication algorithm type in the parsed configuration parameters, the system selects to call the atomic operation interface of the dedicated communication hardware to perform an atomic accumulation operation, or to call the asynchronous copy interface to perform a data copy operation.

[0129] Based on the other configuration parameters parsed out, control the data segmentation, storage location, and data type conversion during the communication process.

[0130] In this implementation, the device process simplifies the invocation of complex communication processes by different upper-layer algorithms by providing a standardized service entry point. The unified communication interface is a set of function or method definitions exposed by the device process to the upper-layer algorithms. This interface encapsulates all communication details at the underlying level, including dedicated communication hardware, synchronization mechanisms, and buffer management, presenting them as a concise function call.

[0131] In practical applications, there is often more than one upper-level algorithm. For example, in distributed training, there are pipeline algorithms or bulk algorithms that manage the overlap of computation and communication strategies. These algorithms first process the input data according to their own logic. Pipeline algorithms often divide large computational tasks into multiple sequentially executed chunks and plan the computation and communication order of each chunk; while bulk algorithms may process the entire computational task as a whole.

[0132] After completing their own data processing and scheduling logic, different upper-layer algorithms will generate a structured communication request. This request encapsulates all the descriptive information, i.e., configuration parameters, required to perform this cross-node data exchange. Subsequently, the upper-layer algorithm sends this communication request to the unified communication interface configured by the device process.

[0133] The device process receives communication requests sent by the upper-layer algorithm through the unified communication interface and parses the configuration parameters carried in the requests. The device process then executes subsequent communication tasks based on these parameters, which may include at least one of the following:

[0134] a) Algorithm type, used to indicate whether the caller is using a Pipeline algorithm or a Bulk algorithm;

[0135] b) Data segmentation identifier, used to indicate whether the input data has been segmented into multiple blocks;

[0136] c) Communication algorithm type, used to indicate whether a reduce-scatter operation or an all-gather operation needs to be performed;

[0137] d) Storage mode, used to indicate whether the communication results are stored locally in the communication buffer or need to be stored off-site to another specified output location;

[0138] e) Data type, used to indicate whether the data to be communicated is BF16, FP16 or other floating-point format.

[0139] Based on the parsed communication algorithm type, the device process internally branches and selects to call the corresponding low-level interface provided by the dedicated communication hardware. If the communication algorithm type indicates that this communication requires reduction-scatter computation (such as Reduce-Scatter), the device process will choose to call the atomic operation interface provided by the software stack, which will ultimately instruct the dedicated communication hardware to perform an atomic accumulation operation. If the algorithm type indicates that this communication is mainly for data collection (such as All-Gather), the device process will choose to call the asynchronous copy interface, instructing the hardware to perform simple data transfer.

[0140] Simultaneously, the device process comprehensively utilizes all other parsed configuration parameters to precisely control the specific execution details of the communication process. For example, based on the data segmentation identifier and algorithm type, the device process determines the number of loops and the size and offset of each data block. Depending on the storage mode, the device process decides whether an additional data copy needs to be initiated after communication ends, transferring the results from the communication buffer to the final location. Based on the data type, the device process configures hardware for automatic data format conversion or verification when necessary to meet the stringent data precision requirements of the computing unit.

[0141] In this embodiment, the device process is configured with a unified communication interface for upper-layer algorithms to call and perform communication operations. The unified communication interface receives communication requests containing configuration parameters from upper-layer algorithms and parses these parameters, which include algorithm type, data segmentation identifier, communication algorithm type, storage mode, and data type. Based on the parsed communication algorithm type, the process selects to call the atomic operation interface of the dedicated communication hardware to perform an atomic accumulation operation, or to call the asynchronous copy interface to perform a data copy operation. Based on other parsed configuration parameters, the process controls data segmentation, storage location, and data type conversion during communication. Thus, by providing a unified communication interface, the device process establishes an efficient and flexible abstraction layer between the complex underlying hardware communication capabilities and the diverse upper-layer algorithm strategies (such as Pipeline and Bulk). This interface design allows upper-layer algorithms to focus on their core parallel strategies and data planning, declaring their complete communication intent through only a set of structured parameters. The device process is responsible for translating all parameter intentions into specific, optimized sequences of hardware instructions. This significantly reduces the complexity of adapting novel computational communication overlap algorithms to dedicated hardware, and improves the modularity and scalability of the software stack. Fine-grained parameter control ensures that the performance potential of the dedicated communication hardware can be fully and accurately utilized to adapt to the specific requirements of different algorithms for data partitioning, storage layout, and computational accuracy, thereby achieving a balance between performance, flexibility, and ease of use at the system level.

[0142] In one possible implementation, the device process is used to:

[0143] After the dedicated communication hardware completes the data reading and processing operation on a peer computing node through each communication stream, it performs an atomic increment operation on the second synchronization flag of the local computing node.

[0144] After the value of the second synchronization flag reaches the second preset threshold, the communication results in the communication buffer of this computing node are copied to the preset output storage location, where the second preset threshold is the number of peer computing nodes.

[0145] When managing concurrent cross-node data reading and processing, device processes require a mechanism to reliably confirm that all parallel tasks have been completed in order to safely proceed with subsequent operations. Here, "completion" refers not only to the transmission of data from the peer node to the local node, but also to real-time processing (such as accumulation) that has been completed during transmission according to the communication mode (such as protocol distribution).

[0146] Whenever the dedicated communication hardware completes the data reading from a specific peer computing node and any on-the-fly processing (such as atomic accumulation or simple copy) required by the communication mode through a communication stream, the communication stream can perform an atomic increment operation on the second synchronization flag of this computing node.

[0147] The second synchronization flag acts as a progress counter within this node, incrementing its value with each concurrent communication subtask completed. After initiating all concurrent read requests, the device process continuously checks the current value of the second synchronization flag. It waits for the current value of the second synchronization flag to reach a second preset threshold; this process can be implemented through the stream wait value interface.

[0148] The second preset threshold is set to the number of peer computing nodes, i.e., the total number of all parallel communication tasks that must succeed. When the device process detects that the value of the second synchronization flag has reached this preset threshold, it determines that the data of all peer nodes has been successfully read and has been merged or aggregated as expected in the local communication buffer.

[0149] At this point, the device process can initiate the final data transfer step, moving the complete communication results (e.g., the final gradient block after reduction and dispersion, or the complete parameter set after full collection) that are ready in the communication buffer of this computing node to the preset output storage location through a single memory copy operation. This output location can be tensor memory for subsequent computation kernel functions to use directly, or another storage area specified by the framework.

[0150] In this embodiment, after the dedicated communication hardware completes data reading and processing operations on a peer computing node through each communication stream, the device process performs an atomic increment operation on the second synchronization flag of its own computing node. Once the value of the second synchronization flag reaches a second preset threshold, the communication results in the communication buffer of this computing node are copied to a preset output storage location, where the second preset threshold is the number of peer computing nodes. Thus, by using the second synchronization flag as a completion counter for all parallel communication subtasks within the node, and setting the threshold to the number of peer nodes, the device process can clearly know when all concurrent hardware operations have finished, thereby ensuring that the data in the communication buffer is a complete and consistent final result. The result copying operation performed on this basis is safe, avoiding the risk of erroneous transfer before the data is fully ready. This maintains high throughput while ensuring the correctness and reliability of the data processing flow.

[0151] In one possible implementation, each of the computing nodes is connected to a fully interconnected network, enabling any one of the computing nodes to directly read and write the communication buffers of other computing nodes via the dedicated communication hardware.

[0152] The interconnection between computing nodes can adopt a fully interconnected topology. A fully interconnected topology is a network connection structure in which each computing node establishes a direct physical or logical communication link with all other computing nodes. That is, there is a dedicated connection path between any two nodes, and data can be transmitted without being relayed by other nodes or forwarded by network switching equipment.

[0153] For example, on a training platform consisting of multiple GPU accelerator cards, full interconnection can be achieved through dedicated high-speed interconnect technologies (such as certain custom interconnect links). Specifically, in an 8-card server, each GPU card can have 7 independent physical channels, directly connected to the other 7 GPU cards, forming a pairwise interconnection pattern. This connection provides dedicated bandwidth resources for data exchange between each pair of nodes.

[0154] Based on this fully interconnected topology, when a device process on a compute node needs to perform cross-node data communication, it can drive its local dedicated communication hardware to initiate direct access. Because the network provides point-to-point direct paths, read / write requests to remote communication buffers generated by the dedicated communication hardware can be delivered to the target compute node via the corresponding direct link, and manipulate specified buffer areas in its video memory. For example, in a Reduce-Scatter operation, a compute node can simultaneously read data blocks from the communication buffers of all other peer nodes in parallel via multiple direct links and perform local accumulation.

[0155] In this embodiment of the disclosure, each computing node is connected to a fully interconnected network, enabling any computing node to directly read and write the communication buffers of other computing nodes via the dedicated communication hardware. Thus, the fully interconnected topology provides the shortest and most direct physical path for inter-node communication, avoiding the overhead of queuing, store-and-forward, and bandwidth contention that data may experience when transmitted in a multi-hop network.

[0156] After all computation and communication tasks are submitted, the device process inserts synchronization events on all active computation and communication streams. Subsequently, the device process waits for all these events to complete on the default stream, which is a predefined main command queue in the compute node used for final coordination and synchronization after all parallel computation and communication streams have finished, ensuring that all tasks are complete before proceeding to the next step. Finally, the device process releases the temporary resources allocated during this communication process sequentially, including the communication stream, event objects, and synchronization flag variables in video memory, restoring the system to a state ready for the next task.

[0157] The following describes a specific embodiment of this disclosure, which provides a specific execution flow for overlapping computation and communication in distributed training, including a complete process from initialization to resource cleanup. Before the training task begins, the host process coordinates all participating computing nodes to establish a memory access mapping, and then instructs the device process on each computing node to create a communication buffer in the local memory. The host process collects the base addresses of all buffers and distributes them to each device process, so that each device process obtains the communication buffer addresses of all peer nodes. Based on these addresses, the device process creates multiple communication streams for its local node, equal to the number of peer nodes, for example, creating 7 communication streams in an 8-node system, and allocates a first synchronization flag (“flag_inter”) for cross-node synchronization and a second synchronization flag (“flag_intra”) for intra-node synchronization in the memory of each node.

[0158] When an upper-layer algorithm (such as the Pipeline or Bulk algorithm) initiates a communication request through a unified communication interface, the execution enters the parameter checking and preparation phase. The device process first parses the configuration parameters in the request, checks the specified data type (e.g., whether it is BF16 or FP16), and configures the atomic operation data format of the dedicated communication hardware according to this type. Subsequently, based on the total data volume, segmentation information, and communication algorithm type in the parameters, the device process calculates the size of the data block transmitted from each peer node, the offset address in the buffer, and the specific position of the synchronization flag.

[0159] Next, the computation and communication overlap execution phase begins. Taking the Reduce-Scatter operation under the Pipeline algorithm as an example, the streaming multiprocessor performs matrix multiplication calculations in the pre-segmented order, and each calculation result is written to the corresponding position in the local node's communication buffer. After all calculations are completed, the device process performs an atomic increment operation on the "flag_inter" of each other peer node by using all communication streams in a loop, to notify the peer that the local node's calculation is complete. At the same time, the device process waits on each communication stream for the local node's "flag_inter" value to reach the threshold of the number of peer nodes, in order to achieve synchronization of the global computation phase. After synchronization is complete, the device process enters the communication loop: for each peer node, the device process selects the corresponding communication stream, and according to the calculated offset address, calls the atomic operation interface to instruct the dedicated communication hardware to perform asynchronous reading and complete atomic accumulation. Each such operation is executed in the corresponding hardware independent queue. Whenever a communication stream completes its data reading and accumulation operation, the device process drives the stream to perform an atomic increment operation on the local node's "flag_intra". The device process continuously monitors the value of "flag_intra". When this value reaches the threshold for the number of peer nodes, it indicates that all concurrent communication subtasks have been completed, and the final result is now in the communication buffer. Based on the storage mode specified in the request, if it is off-site storage, the device process calls the asynchronous copy interface to copy the result from the communication buffer to the specified output location.

[0160] Finally, during the stream synchronization and resource release phase, the device process inserts synchronization events on all active compute and communication streams and waits for these events to complete on the default stream to ensure that all GPU tasks have finished executing. Subsequently, the device process releases the temporary resources allocated during this communication process in sequence, including the communication stream, event objects, and synchronization flag variables in video memory.

[0161] In this embodiment, through centralized coordination and address distribution of the host process, fine-grained management of multi-stream and synchronization flags by the device process, and support for concurrent reading and atomic operations by dedicated communication hardware, a complete separation and deep overlap of computing and communication in terms of hardware resources is achieved. Detailed parameter checking and multi-level synchronization mechanisms ensure the correctness of high-concurrency operations, while complete resource lifecycle management ensures the robustness and predictability of the system, thus providing reliable and efficient underlying communication support for diverse parallel training algorithms at higher levels.

[0162] It should be noted that although the Reduce-Scatter operation under the Pipeline algorithm was used as an example to illustrate the distributed training system, those skilled in the art will understand that this disclosure is not limited to this. In fact, users can flexibly configure the upper-layer algorithm according to their personal preferences and / or actual application scenarios, as long as dedicated communication hardware can be called to perform cross-node data communication operations.

[0163] According to another aspect of this disclosure, a cross-node communication method is provided, applied to a cross-node communication system, the system including a host and multiple computing nodes interconnected via a network, each computing node including a streaming multiprocessor and dedicated communication hardware; each computing node running a device process; the method includes:

[0164] The device process invokes the dedicated communication hardware to perform cross-node data communication operations, replacing the streaming multiprocessor in performing inter-node communication tasks.

[0165] In one possible implementation, the host runs a host process, and the method further includes:

[0166] The host process establishes a video memory access mapping for the multiple computing nodes, so that any computing node can directly access the video memory of other computing nodes; after the video memory access mapping is established, it instructs each device process to create a communication buffer in the video memory of its own computing node; it collects the addresses of the communication buffers created by each device process and distributes all the collected addresses to each device process, so that each device process obtains the communication buffer addresses of all peer computing nodes.

[0167] In one possible implementation, the method further includes: the device process creating multiple communication streams for the local computing node corresponding to the number of peer computing nodes based on the communication buffer addresses of all peer computing nodes received; and allocating a first synchronization flag and a second synchronization flag to the local computing node, wherein the first synchronization flag is used for cross-node computing phase synchronization and the second synchronization flag is used for intra-node communication phase synchronization.

[0168] In one possible implementation, the step of invoking the dedicated communication hardware to perform cross-node data communication operations includes: the device process binding each of the communication streams to an independent queue of the dedicated communication hardware, so that multiple communication streams can perform data operations in parallel through the dedicated communication hardware.

[0169] In one possible implementation, the step of invoking the dedicated communication hardware to perform cross-node data communication operations includes: after the device process executes a local computing task, it writes the computing result into the communication buffer of the local computing node; and through each of the communication streams, it performs an atomic increment operation on the first synchronization flag of the corresponding peer computing node to notify the peer computing node that the local node has completed the computing.

[0170] In one possible implementation, the step of invoking the dedicated communication hardware to perform cross-node data communication operations includes: when the value of the first synchronization flag reaches a first preset threshold, the device process invokes the dedicated communication hardware to read data from the communication buffer of the peer computing node to the communication buffer of the local computing node based on the communication buffer address of the peer computing node.

[0171] Wherein, the first preset threshold is the number of peer computing nodes, and the value of the first synchronization flag is obtained by each peer computing node by performing an atomic increment operation on it.

[0172] In one possible implementation, the invocation of the dedicated communication hardware to perform cross-node data communication operations includes:

[0173] When the device process reads data through the dedicated communication hardware, if the current communication mode is reduction-distributed mode, it instructs the dedicated communication hardware to perform an atomic accumulation operation by calling the atomic operation interface provided by the software stack, accumulating the read data with the data at the corresponding position in the communication buffer of this computing node; if the current communication mode is full collection mode, it instructs the dedicated communication hardware to perform a data copy operation by calling the asynchronous copy interface provided by the software stack, copying the read data to the corresponding position in the communication buffer of this computing node.

[0174] In one possible implementation, the invocation of the dedicated communication hardware to perform cross-node data communication operations includes:

[0175] After the dedicated communication hardware completes the data reading and processing operation on a peer computing node through each communication stream, the device process performs an atomic increment operation on the second synchronization flag of the computing node; after the value of the second synchronization flag reaches a second preset threshold, the communication result in the communication buffer of the computing node is copied to a preset output storage location, where the second preset threshold is the number of peer computing nodes.

[0176] In one possible implementation, the device process is further configured with a unified communication interface for use by upper-layer algorithms to perform communication operations; the invocation of the dedicated communication hardware to perform cross-node data communication operations includes:

[0177] The device process receives communication requests from the upper-layer algorithm through a unified communication interface and parses the configuration parameters in the communication request. Based on the communication algorithm type in the parsed configuration parameters, it selects to call the atomic operation interface of the dedicated communication hardware to perform an atomic accumulation operation, or to call the asynchronous copy interface to perform a data copy operation. Based on other parsed configuration parameters, it controls the data segmentation, storage location, and data type conversion during the communication process.

[0178] In one possible implementation, each of the computing nodes is connected to a fully interconnected network, enabling any one of the computing nodes to directly read and write the communication buffers of other computing nodes via the dedicated communication hardware.

[0179] In some embodiments, the steps or processes of the methods provided in this disclosure can be used to implement the system described in the above system embodiments. The specific implementation can be referred to the description of the above system embodiments, and for the sake of brevity, it will not be repeated here.

[0180] This disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0181] This disclosure also provides a non-volatile computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described method.

[0182] This disclosure also provides a computer program product, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above method.

[0183] Figure 2 This is a block diagram illustrating an apparatus for cross-node communication according to an exemplary embodiment. For example, apparatus 1900 may be provided as a server or terminal device to implement the host or computing node of this disclosure. (Refer to...) Figure 2 The apparatus 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0184] Device 1900 may also include a power supply component 1926 configured to perform power management of device 1900, a wired or wireless network interface 1950 configured to connect device 1900 to a network, and an input / output interface 1958 (I / O interface). Device 1900 can operate on an operating system, such as Windows Server, stored in memory 1932. TM macOS X TM Unix TM Linux TM FreeBSD TM Or similar.

[0185] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of the device 1900 to perform the above-described method.

[0186] Computer-readable storage media can be tangible devices capable of holding and storing programs / instructions used by instruction execution devices. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0187] The computer program (or computer-readable program instructions) described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage medium in the respective computing / processing device.

[0188] The computer program (or computer program instructions) used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions to implement various aspects of this disclosure.

[0189] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0190] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0191] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0192] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0193] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A cross-node communication system, characterized in that, include: The host is used to coordinate training tasks across multiple computing nodes. Multiple computing nodes are interconnected via a network, and a single computing node includes streaming multiprocessors and dedicated communication hardware; In this embodiment, a device process runs on a single computing node; the device process is used to call the dedicated communication hardware to perform cross-node data communication operations, in order to replace the streaming multiprocessor in performing inter-node communication tasks.

2. The system according to claim 1, characterized in that, The host machine runs a host process, which is used for: Establish a memory access mapping for the multiple computing nodes so that any computing node can directly access the memory of other computing nodes; After the video memory access mapping is established, each device process is instructed to create a communication buffer in the video memory of the local computing node; Collect the addresses of the communication buffers created by each of the device processes, and distribute all the collected addresses to each of the device processes so that each device process obtains the communication buffer addresses of all peer computing nodes.

3. The system according to claim 1, characterized in that, The device process is used for: Based on the communication buffer addresses of all peer computing nodes received, create multiple communication streams for this computing node corresponding to the number of peer computing nodes; Assign a first synchronization flag and a second synchronization flag to this computing node. The first synchronization flag is used for synchronization during the cross-node computing phase, and the second synchronization flag is used for synchronization during the intra-node communication phase.

4. The system according to claim 3, characterized in that, The device process is used for: Each of the communication streams is bound to an independent queue of the dedicated communication hardware, so that multiple communication streams can perform data operations in parallel through the dedicated communication hardware.

5. The system according to claim 3, characterized in that, The device process is used for: After executing the local computing task, the computing results are written to the communication buffer of this computing node; Through each of the aforementioned communication streams, an atomic increment operation is performed on the first synchronization flag of the corresponding peer computing node to notify the peer computing node that the node has completed the computation.

6. The system according to claim 3, characterized in that, The device process is used for: When the value of the first synchronization flag reaches the first preset threshold, the dedicated communication hardware is invoked to read data from the communication buffer of the peer computing node to the communication buffer of this computing node based on the communication buffer address of the peer computing node. Wherein, the first preset threshold is the number of peer computing nodes, and the value of the first synchronization flag is obtained by each peer computing node by performing an atomic increment operation on it.

7. The system according to claim 5, characterized in that, The device process is used for: During the process of reading data through the dedicated communication hardware, if the current communication mode is the reduction distributed mode, the dedicated communication hardware is instructed to perform an atomic accumulation operation by calling the atomic operation interface provided by the software stack, and the read data is accumulated with the data at the corresponding position in the communication buffer of this computing node. When the current communication mode is full collection mode, the asynchronous copy interface provided by the software stack is called to instruct the dedicated communication hardware to perform a data copy operation, copying the read data to the corresponding position in the communication buffer of this computing node.

8. The system according to claim 6, characterized in that, The device process is used for: After the dedicated communication hardware completes the data reading and processing operation on a peer computing node through each communication stream, it performs an atomic increment operation on the second synchronization flag of the local computing node. After the value of the second synchronization flag reaches the second preset threshold, the communication results in the communication buffer of this computing node are copied to the preset output storage location, where the second preset threshold is the number of peer computing nodes.

9. The system according to claim 1, characterized in that, The device process is also configured with a unified communication interface for upper-layer algorithms to call and perform communication operations; the device process is used for: The system receives communication requests from upper-layer algorithms through a unified communication interface and parses the configuration parameters in the communication requests. Based on the communication algorithm type in the parsed configuration parameters, the system selects to call the atomic operation interface of the dedicated communication hardware to perform an atomic accumulation operation, or to call the asynchronous copy interface to perform a data copy operation. Based on the other configuration parameters parsed out, control the data segmentation, storage location, and data type conversion during the communication process.

10. The system according to claim 1, characterized in that, Each computing node is connected to a fully interconnected network, enabling any computing node to directly read and write the communication buffers of other computing nodes via the dedicated communication hardware.

11. A cross-node communication method, characterized in that, The invention is applied to a cross-node communication system, which includes a host and multiple computing nodes, which are interconnected via a network. Each computing node includes a streaming multiprocessor and dedicated communication hardware. A device process runs on a single computing node; the method includes: The device process invokes the dedicated communication hardware to perform cross-node data communication operations, replacing the streaming multiprocessor in performing inter-node communication tasks.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method of claim 11.

13. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method of claim 11.

14. A computer program product comprising a computer program, or a non-volatile computer-readable storage medium carrying a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method of claim 11.

Images