Time division multiplexing method for grid computing switch resources

Abstract

The application provides a time division multiplexing method for in-network computing switch resources, and relates to the technical field of time division multiplexing. When a global reduction operation AllReduce of one task is about to start: it is judged whether other tasks are using in-network computing resources at present, if yes, the in-network computing resources are not allowed to be used by the current AllReduce; otherwise, it is judged whether the AllReduce of other tasks will arrive during the execution of the current AllReduce, if no, the in-network computing resources are allowed to be used by the current AllReduce; otherwise, the CalcScore function is used to calculate the scores of the current AllReduce and the AllReduce of all other tasks about to arrive: if the score of the current AllReduce is higher than the scores of the AllReduce of other tasks about to arrive, the in-network computing resources are allowed to be used by the current AllReduce; otherwise, the in-network computing resources are not allowed to be used, and the in-network computing resources are given to the AllReduce of other tasks about to arrive.

Description

Technical Field

[0001] This invention relates to the field of time-division multiplexing technology, and more particularly to a time-division multiplexing method for computing switch resources on the network. Background Technology

[0002] In distributed machine learning training, data parallelism refers to dividing the training data into multiple subsets, with each computing node processing one subset and independently calculating the gradient. To synchronize gradient updates across nodes, the AllReduce communication mode is often used. This means that after each node calculates its local gradient, the AllReduce operation aggregates and distributes the gradients from all nodes equally, thereby ensuring the consistency of model parameters.

[0003] In-network computing accelerates parallel training by directly performing data aggregation operations (such as AllReduce) within network devices (e.g., switches). During data transmission, in-network computing leverages network devices to aggregate gradients in real time, reducing data transmission volume and communication latency, thereby significantly improving training efficiency.

[0004] Scalable Hierarchical Aggregation and Reduction Protocol (SHARP) is an on-network computing protocol designed by NVIDIA for InfiniBand data center networks. NVIDIA's Quantum series switches all support this protocol and are widely deployed in data centers and supercomputers. However, due to performance and cache capacity limitations of the switches, this protocol restricts a switch to supporting only one on-network computing task at a time. This prevents the use of on-network computing technology to accelerate distributed training in multi-task, multi-tenant scenarios of Machine Learning as a Service (MLaaS).

[0005] The ATP protocol, proposed by Tsinghua University, is an Ethernet-based on-network computing protocol that supports multi-task, multi-tenant scenarios. It achieves multi-task on-network computing by allocating switch buffers to multiple running tasks. However, due to the limited buffer capacity of switches, the buffer capacity allocated to each task is usually insufficient to allow on-network computing to reach its theoretical maximum line speed, making it a bottleneck for on-network computing.

[0006] NetPack is a system for scheduling machine learning tasks in data center clusters. It schedules tasks onto compute nodes to minimize the probability of conflicts between tasks using on-network computing resources. Despite reducing the probability of conflicts, the system still employs a strategy of allowing multiple tasks to share the limited cache of a switch. When multiple tasks share this cache, the overall training speed will be limited.

[0007] Chinese invention patent CN118509370A discloses a method for generating aggregation trees for online computing, an electronic device, and a computer storage medium. It proposes a scheme to avoid data congestion and reduce network latency by generating the optimal aggregation tree based on the current cluster network state. However, this patent does not explicitly consider the time-based sharing strategy between tasks on shared switches, but only selects the switch with the highest remaining performance in the current state as the root node. Summary of the Invention

[0008] Purpose of the invention: To propose a time-division multiplexing method for on-network computing switch resources, and further propose a system architecture for implementing the method. The aim is to accelerate the on-network aggregation of All-Reduce collection communication operators in distributed training scenarios by using time-division multiplexing of on-network computing resources of switches, thereby maximizing the utilization of on-network computing resources and solving the above-mentioned problems existing in the prior art.

[0009] This invention proposes a time-division multiplexing method for on-network computing switch resources. When a global reduction operation (AllReduce) of one of the tasks is about to begin, the following process is executed:

[0010] S1. Determine if any other tasks are currently using online computing resources. If so, do not allow the current AllReduce to use online computing resources; otherwise, proceed to step S2.

[0011] S2. Determine whether any other AllReduce tasks will arrive during the current AllReduce execution. If not, allow the current AllReduce to use the on-network computing resources; otherwise, proceed to step S3.

[0012] S3. Use the CalcScore function to calculate the scores of the current AllReduce and all other upcoming AllReduce tasks:

[0013] If the current AllReduce score is higher than the scores of other upcoming AllReduce tasks, then the current AllReduce is allowed to use on-network computing resources.

[0014] Otherwise, it is not allowed. The priority of using online computing resources is given according to the score, and the online computing resources are used according to the predetermined priority.

[0015] In a further embodiment, when an AllReduce is not allowed to use on-network computing resources, it can be executed using a conventional algorithm instead of waiting for on-network computing resources; the conventional algorithm includes Ring AllReduce or TreeAllReduce.

[0016] In a further embodiment, when an AllReduce request arrives, it is not directly added to the CUDA stream queue, but is instead stored in a custom queue to ensure that only one kernel function is always executing in the CUDA stream queue.

[0017] In a further embodiment, a host callback function is executed whenever a kernel function in the CUDA stream queue completes execution;

[0018] In the callback function, a new AllReduce request is retrieved from the system queue, it is determined whether it can use on-network computing resources for acceleration, and after configuration, it is inserted into the CUDA stream queue.

[0019] In a further embodiment, this method utilizes a shared controller to ensure that compute nodes performing the same task reach a consensus on whether each AllReduce should use on-net compute acceleration.

[0020] The shared controller synchronizes the cluster task execution status to each compute node, enabling each compute node within the same task to calculate the same result based on the same cluster status, thereby controlling whether each AllReduce uses on-network computing acceleration.

[0021] In a further embodiment, a shared controller is used to ensure that compute nodes performing the same task reach a consensus on whether each AllReduce should use on-network compute acceleration, specifically including:

[0022] When the state of any task sharing the same switch changes, the shared controller sends a "ready to update" message to all compute nodes using the switch.

[0023] Upon receiving the message, each compute node attempts to acquire the lock;

[0024] After successfully acquiring the lock, the compute node sends a confirmation message to the shared controller, which includes the number of the current AllReduce.

[0025] The shared controller verifies whether the AllReduce numbers received from all compute nodes for the same task are the same;

[0026] If the numbers are the same, the shared controller sends information to each node to update its state; otherwise, it aborts the attempt and retryes after a period of time.

[0027] Each compute node updates its state based on the received information and releases the lock;

[0028] Each compute node then sends a confirmation message to the shared controller to acknowledge receipt, thus completing the process.

[0029] Furthermore, this invention proposes a time-division multiplexing system for on-net computing switch resources. The system includes a cluster control node, multiple GPU training accelerator cards, and multiple switches with on-net computing capabilities. The multiple GPU training accelerator cards can train multiple tasks in a distributed and parallel manner, and perform multi-card All-Reduce aggregate communication during the training process.

[0030] The cluster control node contains at least one shared controller; the shared controller can execute the time-division multiplexing method of the above-mentioned on-net computing switch resources, thereby calculating in real time whether the current All-Reduce communication uses on-net computing resources.

[0031] Furthermore, the present invention also proposes a computer-readable storage medium storing at least one executable instruction, which, when executed on a shared controller, causes the shared controller to perform the aforementioned time-division multiplexing method for computing switch resources on the network.

[0032] Beneficial effects: Compared with existing solutions (which partition switch cache in the spatial dimension), this application partitions switch cache in the temporal dimension for different tasks, proposing a time-division multiplexing scheme for switch cache. Considering the network communication characteristics of distributed machine learning training tasks, this application can effectively improve the problem of low switch cache utilization, allowing multiple tasks to share a limited switch cache while almost not affecting the execution speed of these tasks. Attached Figure Description

[0033] Figure 1 This is a diagram showing the relationship between the calculation stage and the corresponding aggregation stage of each gradient layer in this embodiment of the invention.

[0034] Figure 2 This is a flowchart of the time-division multiplexing method for computing switch resources on the network in an embodiment of the present invention.

[0035] Figure 3 This is a diagram of the shared controller synchronization protocol in an embodiment of the present invention. Detailed Implementation

[0036] In the following description, numerous specific details are set forth in order to provide a more thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention can be practiced without one or more of these details. In other instances, certain technical features well-known in the art have not been described in order to avoid obscuring the invention.

[0037] The applicant's research found that existing on-network computing implementations are limited by switch performance and cache capacity, thus restricting the number or speed of simultaneously supported on-network computing tasks. The SHARP protocol limits a switch to supporting only one on-network computing task at a time; the ATP protocol and NetPack system divide limited switch cache across multiple tasks, sacrificing the training speed of each task. This strategy based on switch cache allocation results in low switch cache utilization because when a task is not engaged in network communication (such as during the computation phase), the switch cache allocated to it remains idle and cannot be effectively utilized.

[0038] Figure 1 This illustrates the relationship between the gradient computation phase and the corresponding aggregation phase for each layer. In GPUs, communication and computation have their own queues (called CUDA streams). The arrow between the first and second lines indicates the dependency between the computation and communication phases: the corresponding aggregation phase can only begin after the gradient computation for each layer is complete. Comparing the second and third lines shows that using on-premises computation acceleration for all AllReduce tasks effectively reduces task execution time compared to not using it. This proposal suggests that the same level of overall task execution time reduction can be achieved by using on-premises computation acceleration only for a subset of AllReduce tasks, such as... Figure 1 As shown in the second and fourth lines, this is because the task completion time is determined by the end time of the last AllReduce. Accelerating the preceding AllReduces does not actually speed up task execution, as they are affected by computational performance bottlenecks. Based on this finding, this application can reduce the time each task occupies on-network computing resources, achieving the same acceleration effect.

[0039] In its first aspect, this embodiment proposes a time-division multiplexing method for on-net computing switch resources to control how on-net computing resources are allocated to multiple tasks in the time dimension. The core idea of ​​this method is to selectively accelerate AllReduce, which has a significant impact on the final task execution time.

[0040] For a task, such as Figure 1As shown in the fourth line, on-network computation can be used to accelerate the latter half of an AllReduce training step without accelerating the first half. For the case of multiple tasks, we need to assign importance scores to each AllReduce for each task. This proposal defines the importance of an AllReduce as the impact of using on-network computation acceleration versus not accelerating it on the final task execution time.

[0041] The specific execution flow of the time-division multiplexing method for network computing switch resources is as follows: Figure 2 When one of the AllReduce operations for one of the tasks is about to begin:

[0042] S1. Determine if any other tasks are currently using online computing resources. If so, do not allow the current AllReduce to use online computing resources; otherwise, proceed to step S2.

[0043] S2. Determine whether any other AllReduce tasks will arrive during the current AllReduce execution. If not, allow the current AllReduce to use the on-network computing resources; otherwise, proceed to step S3.

[0044] S3. Use the CalcScore function to calculate the scores of the current AllReduce and all other upcoming AllReduce tasks.

[0045] If the current AllReduce score is higher than the scores of other upcoming AllReduce tasks, then the current AllReduce is allowed to use on-network computing resources.

[0046] Otherwise, it is not allowed. The priority of using online computing resources is given according to the score, and the online computing resources are used according to the predetermined priority.

[0047] When an AllReduce operation is not allowed to use on-network computing resources, it will execute using a traditional algorithm instead of waiting for on-network computing resources. Traditional AllReduce algorithms can be Ring AllReduce or Tree AllReduce. Ring AllReduce arranges the nodes into a logical loop, with each node sending its received information to its neighbors. Tree AllReduce arranges the nodes into a logical tree topology, with each node sending its information to its parent node. The root node receives all the information, sums them, and then sends the sum back to each node.

[0048] In step S3, the CalcScore function calculates whether the upcoming AllReduce will use INA, and the execution times d1 and d2 of the current step. Then it divides these values ​​by the duration of INA resources it will occupy (t). now The acceleration effect of INA resources used per unit time on the overall task is calculated as: (d1-d2) / t now .

[0049] The aforementioned time-division multiplexing sharing algorithm places high demands on the state synchronization between multiple computing nodes. Therefore, in the second aspect of this embodiment, a shared controller architecture is proposed. Its main responsibilities are to efficiently: 1) ensure that computing nodes executing the same task reach a consensus on whether each AllReduce should use on-network computing acceleration; 2) prevent multiple tasks from using on-network computing resources simultaneously, thereby preventing conflicts.

[0050] The shared controller disclosed in this embodiment is a software system that centrally controls the entire data center network, running on a dedicated cluster control node (in the data center network, the shared controller of this proposal can run on the same node as the subnet manager and aggregation manager). To avoid the additional overhead caused by each task needing to communicate with the shared controller before each AllReduce, this proposal suggests running the shared algorithm on each compute node, while the shared controller is responsible for synchronizing the cluster task execution status to each compute node. This allows each compute node within the same task to calculate the same result based on the same cluster status, thereby controlling whether each AllReduce uses on-network computation acceleration.

[0051] This embodiment uses an improved two-phase commit (2PC) protocol to ensure data consistency. For example... Figure 3 As shown, specifically:

[0052] (1) When the state of any task sharing the same switch changes, the shared controller sends a "ready to update" message to all computing nodes using the switch;

[0053] (2) Upon receiving the message, each compute node attempts to acquire the lock;

[0054] (3) After successfully acquiring the lock, the compute node sends a confirmation message to the shared controller, which includes the number of the current AllReduce;

[0055] (4) The shared controller verifies whether the AllReduce numbers received from all compute nodes of the same task are the same;

[0056] (5) If the numbers are the same, the shared controller sends information to each node to update the state; otherwise, it aborts the attempt and retryes after a period of time.

[0057] (6) Each computing node updates its state based on the received information and releases the lock;

[0058] (7) Each computing node finally sends a confirmation message to the shared controller to complete the process.

[0059] The above ensures the correctness of the process: 1) During the update process when the system is in an inconsistent state, the mutex locks of all compute nodes within the task are always held; 2) The shared controller checks the consistency of the AllReduce number to avoid situations where some compute nodes have started AllReduce while others have not.

[0060] Furthermore, existing GPU network aggregation communication libraries typically need to determine how to execute AllReduce (such as whether to use on-premises computation, message block size, number of GPU threads, etc.) immediately upon the arrival of an AllReduce request, and then add the configured CUDA kernel function to the CUDA stream queue, after which it cannot be modified. In contrast, the on-premises computation sharing scheme designed in this proposal can only determine whether to use on-premises computation before AllReduce is actually executed, rather than when the AllReduce request arrives.

[0061] To address this, the third aspect of this embodiment presents a novel CUDA kernel function queuing method. When an AllReduce request arrives, the system does not directly add it to the CUDA stream queue, but instead stores it in a custom queue. The system ensures that only one kernel function is always executing in the CUDA stream queue. Whenever the kernel function completes its execution, a host callback function is executed. In the callback function, a new AllReduce request is retrieved from the system queue, a shared algorithm is invoked to determine whether on-network computing acceleration can be used, and after configuration, it is inserted into the CUDA stream queue.

[0062] A fifth aspect of this embodiment also discloses a time-division multiplexing control system for on-network computing resources. The system includes a cluster control node, multiple GPU training accelerator cards, and multiple switches with on-network computing capabilities. The multiple GPU training accelerator cards can train multiple tasks in a distributed parallel manner and perform multi-card All-Reduce aggregate communication during the training process.

[0063] The cluster control node includes a shared controller (second aspect) and a novel CUDA kernel function queuing method (third aspect). The shared controller includes a time-division multiplexing shared on-net computing resource algorithm (first aspect) to calculate in real time whether the current All-Reduce communication uses on-net computing resources.

[0064] In summary, this application proposes time-division multiplexing of limited switch cache resources in the time dimension, and designs an efficient distributed consensus protocol to synchronize the state between computing nodes, as well as a flexible CUDA kernel function startup method to support this sharing strategy.

[0065] Compared to existing on-network computing implementation schemes, the method proposed in this application can effectively improve the utilization of limited on-network computing resources, enabling multiple distributed machine learning tasks to achieve 98.2% of the training speed when sharing the same switch, which is far higher than the baseline of 26.3%.

[0066] The time-division multiplexing method for on-network computing switch resources proposed in this application can guide the allocation of on-network computing resources during the distributed training process of large models, maximizing the capabilities of on-network computing devices and effectively reducing task completion time. The intelligent computing clusters upon which large-scale distributed training relies, including GPU training accelerator cards and intelligent computing switches, are all high-cost hardware devices. Efficient on-network computing technology can significantly reduce training time and communication traffic, while improving the utilization efficiency of computing and network resources, thus greatly saving the training cost of large models.

[0067] The time-division multiplexing method for on-network computing switch resources proposed in this application can improve training speed and accelerate the iteration efficiency of large models for self-use large model training clusters, facilitating commercialization while saving training costs. For intelligent computing public cloud scenarios, the efficient on-network computing acceleration service can shorten the training task time for users and enhance the competitiveness of intelligent computing public cloud products.

[0068] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0069] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0070] In the several embodiments provided in this application, it should be understood that the disclosed system can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0071] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0072] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0073] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0074] As described above, although the invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the invention itself. Various changes in form and detail may be made without departing from the spirit and scope of the invention as defined in the appended claims.

Claims

1. A time division multiplexing method for switching resources in a mesh computing, characterized in that, When a global reduction operation of one of the tasks, AllReduce, is about to start: S1, judge whether there is other task using in-network computing resources at present, if yes, the current AllReduce is not allowed to use in-network computing resources; otherwise, go to step S2; S2, judge whether there will be other task's AllReduce arriving during the execution of the current AllReduce, if no, the current AllReduce is allowed to use in-network computing resources; otherwise, go to step S3; S3, use the CalcScore function to calculate the score of the current AllReduce and all other tasks' AllReduce about to arrive, and score the importance of each AllReduce of each task, the importance is defined as: the impact on the final task execution time of using in-network computing acceleration and not using in-network computing acceleration for the AllReduce is the importance of the AllReduce; If the score of the current AllReduce is higher than that of other tasks' AllReduce about to arrive, the current AllReduce is allowed to use in-network computing resources; Otherwise, it is not allowed, and the priority of using in-network computing resources is given according to the score, and in-network computing resources are used according to the predetermined priority.

2. The time division multiplexing method of cloud computing switch resources according to claim 1, characterized in that: When a certain AllReduce is not allowed to use in-network computing resources, a traditional algorithm can be used for execution, rather than waiting for in-network computing resources all the time; the traditional algorithm includes Ring AllReduce or Tree AllReduce.

3. The time division multiplexing method of cloud computing switch resources as claimed in claim 1, wherein: When an AllReduce request arrives, the AllReduce request is not directly added to the CUDA stream queue, but saved in a custom queue, ensuring that there is only one kernel function executing in the CUDA stream queue at all times.

4. The time division multiplexing method of cloud computing switch resources according to claim 3, wherein, A host callback function is executed every time the kernel function in the CUDA stream queue completes execution; In the callback function, a new AllReduce request is taken out from the system queue, and it is judged whether in-network computing resources can be used for acceleration, and after configuration, it is inserted into the CUDA stream queue.

5. The time division multiplexing method of fog computing switch resources according to any one of claims 1 to 4, characterized in that, A shared controller is used to ensure that the computing nodes executing the same task reach a consensus on whether in-network computing acceleration is used for each AllReduce.

6. The time division multiplexing method of cloud computing switch resources according to claim 5, wherein, The shared controller synchronizes the cluster task execution state to each computing node, so that each computing node in the same task can calculate the same result according to the same cluster state, thereby controlling whether each AllReduce uses in-network computing acceleration.

7. The time division multiplexing method of cloud computing switch resources according to claim 6, wherein, The shared controller is used to ensure that the computing nodes executing the same task reach a consensus on whether in-network computing acceleration is used for each AllReduce, specifically including: When the state of any task sharing the same switch changes, the shared controller sends a "ready to update" message to all computing nodes using the switch; After receiving the message, each computing node tries to acquire a lock; After successfully acquiring the lock, the computing node sends a confirmation message to the shared controller, including the number of the current AllReduce; The shared controller verifies whether the AllReduce numbers received from all compute nodes of the same task are the same; If the numbers are the same, the shared controller sends information for updating the state to each node; otherwise, it aborts this attempt and retries after a period of time; Each compute node updates its state according to the received information and releases the lock; Each compute node finally sends a confirmation message to the shared controller to complete the process.

8. A time-multiplexed system for switching resources in a mesh computer, characterized by, The system comprises a cluster control node, a plurality of GPU training accelerator cards and a plurality of network computing capable switches; The plurality of GPU training accelerator cards can distribute and parallel train a plurality of tasks and perform multi-card All-Reduce collective communication during the training process; The cluster control node comprises at least one shared controller; the shared controller can execute the time division multiplexing method of the network computing switch resource as claimed in any one of claims 1 to 7, thereby calculating in real time whether the current All-Reduce communication uses the network computing resource.

9. A computer-readable storage medium, characterized in that, The storage medium stores at least one executable instruction; when the executable instruction runs on the shared controller, the shared controller executes the time division multiplexing method of the network computing switch resource as claimed in any one of claims 1 to 7.

Images