Time division multiplexing method for computing switch resources in network

Abstract

The invention provides a time division multiplexing method for network computing switch resources, and relates to the technical field of time division multiplexing. When a certain global reduction operation AllReduce of one task is about to start, judging whether other tasks are currently using the on-network computing resources or not, and if so, not allowing the current AllReduce to use the on-network computing resources; otherwise, judging whether AllReduce of other tasks arrives or not during the execution period of the current AllReduce, and if not, allowing the current AllReduce to use the online computing resources; otherwise, calculating scores of the current AllReduce and the AllReduce to which other tasks are about to arrive by utilizing a CalcScore function; if the score of the current AllReduce is higher than the scores of the AllReduce to which other tasks are about to arrive, allowing the current AllReduce to use online computing resources; otherwise, not allowing, and offering the on-network computing resources to AllReduce to which other tasks are about to arrive for use.

Description

Technical Field

[0001] The invention relates to the technical field of time division multiplexing, and in particular to a time division multiplexing method for computing switch resources on a network. Background Art

[0002] In distributed machine learning training, data parallelism means dividing the training data into multiple subsets, with each computing node processing a subset and independently calculating the gradient. In order to synchronize the gradient updates of each node, the AllReduce communication mode is often used, that is, after each node calculates the local gradient, the gradients of all nodes are aggregated and evenly distributed through the AllReduce operation, thereby ensuring the consistency of model parameters.

[0003] In-network computing technology accelerates data parallel training by directly performing data aggregation operations (such as AllReduce) in network devices (such as switches). In the process of data transmission, in-network computing technology uses network devices to aggregate the gradients in real time, reducing the amount of data transmission and communication delay, thereby significantly improving training efficiency.

[0004] Scalable Hierarchical Aggregation and Reduction Protocol (SHARP) is an in-network computing protocol designed by NVIDIA based on InfiniBand data center networks. The company's Quantum series switches all support this protocol, and this series of switches has been widely deployed in major data centers and supercomputers. However, due to the performance and cache capacity limitations of the switch, the protocol limits a switch to only support one in-network computing task at a time, which makes it impossible to use in-network computing technology to accelerate distributed training in the multi-task and multi-tenant scenarios of "Machine Learning as a Service" (MLaaS).

[0005] The ATP protocol is an Ethernet-based on-network computing protocol proposed by Tsinghua University that supports multi-task and multi-tenant scenarios. It implements multi-task on-network computing by dividing the switch cache to multiple running tasks. However, due to the limited switch cache capacity, the cache capacity allocated to each task is usually not enough to allow the on-network computing speed to reach the theoretical maximum line speed, making it a bottleneck for on-network computing.

[0006] NetPack is a system for scheduling machine learning tasks in a data center cluster. The system schedules tasks to computing nodes to minimize the probability of conflicts between tasks in using network computing resources. Although the system reduces the probability of conflicts, it still adopts the strategy of letting multiple tasks share the limited cache of the switch. When multiple tasks share, the overall training speed will be limited.

[0007] The Chinese invention patent with publication number CN118509370A discloses a method for generating an aggregation tree for network computing, an electronic device, and a computer storage medium. It proposes a solution that can avoid data congestion and reduce network latency by generating an aggregation tree with optimal performance according to the current cluster network status. However, the patent does not explicitly consider the sharing strategy between tasks of the shared switch in the time dimension, but only selects the switch with the most remaining performance in the current state as the root node. Summary of the invention

[0008] Purpose of the invention: To propose a method for time-division multiplexing of switch resources for network computing, and further propose a system architecture for implementing the method, aiming to perform network aggregation acceleration on the All-Reduce collective communication operators in distributed training scenarios through time-division multiplexing of switch network computing resources, maximize the utilization of network computing resources, and solve the above-mentioned problems existing in the prior art.

[0009] The present invention proposes a time division multiplexing method for computing switch resources in a network. When a global reduction operation AllReduce of one of the tasks is about to start, the following process is executed: S1. Determine whether there are other tasks currently using the online computing resources. If so, the current AllReduce is not allowed to use the online computing resources; otherwise, proceed to step S2; S2. Determine whether there will be AllReduces of other tasks arriving during the execution of the current AllReduce. If not, allow the current AllReduce to use the computing resources on the network; otherwise, proceed to step S3. S3. Use the CalcScore function to calculate the scores of the current AllReduce and all other tasks that are about to arrive at the AllReduce: If the current AllReduce score is higher than the AllReduce score of other tasks that are about to arrive, the current AllReduce is allowed to use the online computing resources; Otherwise, it is not allowed, and the priority of using the online computing resources is given according to the score, and the online computing resources are used according to the predetermined priority.

[0010] In a further embodiment, when a certain AllReduce is not allowed to use the computing resources on the network, a traditional algorithm may be used for execution instead of waiting for the computing resources on the network; the traditional algorithm includes Ring AllReduce or TreeAllReduce.

[0011] In a further embodiment, when an AllReduce request arrives, the AllReduce request is not directly added to the CUDA stream queue, but is saved in a custom queue, ensuring that only one kernel function is always executed in the CUDA stream queue.

[0012] In a further embodiment, whenever a kernel function in a CUDA stream queue is executed, a host callback function is executed; In the callback function, a new AllReduce request is taken from the system queue to determine whether it can be accelerated using the computing resources on the network, and then inserted into the CUDA stream queue after configuration.

[0013] In a further embodiment, the method utilizes a shared controller to ensure that computing nodes executing the same task reach a consensus on whether each AllReduce uses on-network computing acceleration.

[0014] The shared controller synchronizes the cluster task execution status to each computing node, so that each computing node in the same task can calculate the same result based on the same cluster status, thereby controlling whether each AllReduce uses or does not use network computing acceleration.

[0015] In a further embodiment, a shared controller is used to ensure that computing nodes executing the same task reach a consensus on whether each AllReduce uses on-network computing acceleration, specifically including: When the state of any task sharing the same switch changes, the shared controller sends a “prepare to update” message to all compute nodes using the switch; After receiving the message, each computing node tries to acquire the lock; After successfully acquiring the lock, the computing node sends a message to the shared controller to confirm receipt, including the number of the current AllReduce; The shared controller verifies that the AllReduce number received from all compute nodes of the same task is the same; If the numbers are the same, the shared controller sends information to each node to update the status; otherwise, it aborts the attempt and tries again after a while; Each computing node updates its state based on the information received and releases the lock; Each computing node finally sends a message to the shared controller to confirm receipt, completing the process.

[0016] In addition, the present invention also proposes a time-division multiplexing system for in-network computing switch resources, which includes a cluster control node, multiple GPU training accelerator cards, and multiple switches with in-network computing functions; multiple GPU training accelerator cards can perform distributed parallel training of multiple tasks, and perform multi-card All-Reduce collective communication during the training process; The cluster control node includes at least one shared controller; the shared controller can execute the above-mentioned time division multiplexing method of the on-line computing switch resources, so as to calculate in real time whether the current All-Reduce communication uses the on-line computing resources.

[0017] In addition, the present invention also proposes a computer-readable storage medium, which stores at least one executable instruction. When the executable instruction runs on a shared controller, the shared controller executes the above-mentioned time division multiplexing method of in-network computing switch resources.

[0018] Beneficial effects: Compared with the existing solution (dividing the switch cache in the spatial dimension), this application divides the switch cache into different tasks in the time dimension and proposes a time-division multiplexing solution for the switch cache. Considering the network communication characteristics of distributed machine learning training tasks, this application can effectively improve the problem of low switch cache utilization, allowing multiple tasks to share a limited switch cache while hardly affecting the execution speed of these tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a relationship diagram between the calculation stage of each layer of gradient and the corresponding aggregation stage in the embodiment of the present invention.

[0020] Figure 2 It is a flow chart of a time division multiplexing method for computing switch resources in a network according to an embodiment of the present invention.

[0021] Figure 3 4 is a diagram of a shared controller synchronization protocol in an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In the following description, a large number of specific details are provided to provide a more thorough understanding of the present invention. However, it is apparent to those skilled in the art that the present invention can be implemented without one or more of these details. In other examples, in order to avoid confusion with the present invention, some technical features known in the art are not described.

[0023] The applicant found that existing on-line computing implementation methods are limited by switch performance and cache capacity, which limits the number or speed of on-line computing tasks that can be supported simultaneously. The SHARP protocol limits a switch to only one on-line computing task at a time; the ATP protocol and the NetPack system both divide the limited switch cache into multiple tasks, sacrificing the training speed of each task. This strategy based on switch cache division results in low switch cache utilization, because when a task does not communicate on the network (such as in the calculation stage), the switch cache allocated to it is idle and cannot be effectively used.

[0024] Figure 1 It shows the relationship between the calculation phase of each layer of gradient and the corresponding aggregation phase. In the GPU, communication and calculation have their own queues (called CUDA streams). The arrow between the first and second rows indicates the dependency between the calculation phase and the communication phase: the corresponding aggregation phase can only start after the gradient calculation of each layer is completed. Comparing the second and third rows, it can be seen that using on-line computing acceleration for all AllReduce can effectively reduce the task execution time compared to not using on-line computing acceleration. This proposal points out that it is possible to use on-line computing acceleration for only part of AllReduce to achieve the same degree of overall task execution time reduction, such as Figure 1 This is because the task completion time is determined by the end time of the last AllReduce. Accelerating the previous AllReduce does not really speed up the task execution because they are affected by the computing performance bottleneck. Based on this finding, the present application can reduce the time each task occupies on-line computing resources to achieve the same acceleration effect.

[0025] In the first aspect of this embodiment, a time division multiplexing method for in-network computing switch resources is proposed to control how to allocate in-network computing resources to multiple tasks in the time dimension. The core idea of ​​this method is to selectively accelerate those AllReduces that have a greater impact on the final task execution time.

[0026] For a task, such as Figure 1 As shown in the fourth row, the latter half of the AllReduce in a training step can be accelerated by using on-line computing without accelerating the former half. For multiple tasks, we need to score the importance of each AllReduce of each task. This proposal defines the importance of AllReduce as: the impact of using on-line computing to accelerate the AllReduce and not accelerating it on the final task execution time is the importance of the AllReduce.

[0027] The specific execution process of the time division multiplexing method of the network computing switch resource is shown in Figure 2 , when an AllReduce of one of the tasks is about to start: S1. Determine whether there are other tasks currently using the online computing resources. If so, the current AllReduce is not allowed to use the online computing resources; otherwise, proceed to step S2; S2. Determine whether there will be AllReduces of other tasks arriving during the execution of the current AllReduce. If not, allow the current AllReduce to use the computing resources on the network; otherwise, proceed to step S3. S3. Use the CalcScore function to calculate the scores of the current AllReduce and all other tasks' upcoming AllReduces.

[0028] If the current AllReduce score is higher than the AllReduce score of other tasks that are about to arrive, the current AllReduce is allowed to use the online computing resources; Otherwise, it is not allowed, and the priority of using the online computing resources is given according to the score, and the online computing resources are used according to the predetermined priority.

[0029] When an AllReduce is not allowed to use on-network computing, the AllReduce will be executed using a traditional algorithm instead of waiting for on-network computing resources. Traditional AllReduce algorithms can be Ring AllReduce or TreeAllReduce. Ring AllReduce forms nodes into a logical loop, and each node sends the information it receives to its neighboring nodes. Tree AllReduce forms nodes into a logical tree topology, and each node sends its own information to the parent node. The root node receives all the information and sums it up, then sends it back to each node.

[0030] In step S3, for the CalcScore function, it calculates whether the upcoming AllReduce uses INA, the execution time d of the current step, and 1 and d 2 Then divide it by how long it will take up the INA resources (t now ), we can get the acceleration degree of the task as a whole by occupying INA resources per unit time: (d 1 -d 2 ) / t now .

[0031] The above-mentioned time-division multiplexing sharing algorithm places high demands on the state synchronization between multiple computing nodes. Therefore, in the second aspect of this embodiment, a shared controller architecture is proposed. Its main responsibilities are to efficiently: 1) ensure that the computing nodes executing the same task reach a consensus on whether each AllReduce uses the on-line computing acceleration; 2) avoid multiple tasks using the on-line computing resources at the same time to prevent conflicts.

[0032] The shared controller disclosed in this embodiment is a software system that centrally controls the entire data center network and runs in a dedicated cluster control node (in the data center network, the shared controller of this proposal can run on the same node as the subnet manager and aggregation manager). In order to avoid the additional overhead caused by the need to communicate with the shared controller over the network before each AllReduce of each task, this proposal proposes running the shared algorithm in each computing node, and the shared controller is responsible for synchronizing the cluster task execution status to each computing node, so that each computing node in the same task can calculate the same result based on the same cluster status, thereby controlling whether each AllReduce uses or does not use network computing acceleration.

[0033] This embodiment uses an improved two-phase commit (2PC) protocol to ensure data consistency. Figure 3 As shown, specifically: (1) When the state of any task sharing the same switch changes, the shared controller sends a "prepare update" message to all computing nodes using the switch; (2) After receiving the message, each computing node attempts to acquire the lock; (3) After successfully acquiring the lock, the computing node sends a message to the shared controller to confirm receipt, which includes the number of the current AllReduce; (4) The shared controller verifies that the AllReduce numbers received from all compute nodes of the same task are the same; (5) If the numbers are the same, the shared controller sends information to each node to update the status; otherwise, it aborts the attempt and tries again after a while; (6) Each computing node updates its status based on the received information and releases the lock; (7) Each computing node finally sends a message to the shared controller to confirm receipt, completing the process.

[0034] The above guarantees the correctness of the process: 1) When the system is in an inconsistent state during the update, the mutex locks of all computing nodes in the task are always held; 2) The shared controller checks the consistency of the AllReduce number to avoid the situation where some computing nodes have started AllReduce while others have not.

[0035] In addition, existing GPU network collective communication libraries usually need to determine how to execute AllReduce (such as whether to use on-line computing, the size of message blocks, the number of GPU threads, etc.) immediately when an AllReduce request arrives, and then add the configured CUDA kernel function to the CUDA stream queue, which cannot be modified afterwards. The on-line computing sharing solution designed in this proposal can only determine whether to use on-line computing before AllReduce is actually executed, rather than when the AllReduce request arrives.

[0036] To this end, the third aspect of this embodiment designs a new CUDA kernel function enqueue method. When an AllReduce request arrives, the system does not directly add it to the CUDA stream queue, but saves it in a custom queue. The system ensures that there is always only one kernel function executing in the CUDA stream queue, and each time the kernel function is executed, a host callback function will be executed. In the callback function, a new AllReduce request will be taken out of the system queue, and the sharing algorithm will be called to obtain whether the on-line computing acceleration can be used, and it will be inserted into the CUDA stream queue after configuration.

[0037] In a fifth aspect of this embodiment, a time-division multiplexing control system for online computing resources is also disclosed. The system includes a cluster control node, multiple GPU training accelerator cards, and multiple switches with online computing functions. Multiple GPU training accelerator cards can perform distributed parallel training of multiple tasks and perform multi-card All-Reduce collective communication during the training process.

[0038] The cluster control node includes the shared controller of the second aspect and the novel CUDA kernel function enqueueing method of the third aspect. The shared controller includes the time-division multiplexing shared network computing resource algorithm of the first aspect, and calculates in real time whether the current All-Reduce communication uses the network computing resources.

[0039] In summary, the present application proposes time-division multiplexing of limited switch cache resources in the time dimension, and designs an efficient distributed consistency protocol to synchronize the states between computing nodes, as well as a flexible CUDA kernel function startup method to support the sharing strategy.

[0040] Compared with existing on-line computing implementation solutions, the method proposed in this application can effectively improve the utilization rate of limited on-line computing resources, allowing multiple distributed machine learning tasks to achieve 98.2% of the training speed when they share the same switch, which is much higher than the baseline of 26.3%.

[0041] The time-division multiplexing method of on-line computing switch resources proposed in this application can guide the allocation of on-line computing resources during the distributed training of large models, maximize the capabilities of on-line computing devices, and effectively reduce the time to complete tasks. The intelligent computing cluster that large-model distributed training relies on, the hardware equipment includes GPU training accelerator cards and intelligent computing switches, etc., all of which are extremely expensive hardware equipment. Efficient on-line computing technology can significantly reduce training time and communication traffic, while improving the utilization efficiency of computing resources and network resources, and greatly saving the training cost of large models.

[0042] The time-division multiplexing method of in-network computing switch resources proposed in this application can improve the training speed, accelerate the iteration efficiency of large models, and commercialize the process for self-use large model training clusters, while saving training costs. For intelligent computing public cloud scenarios, efficient in-network computing acceleration services can shorten the training task duration of users and improve the competitiveness of intelligent computing public cloud products.

[0043] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0044] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0045] In the several embodiments provided in the present application, it should be understood that the disclosed system can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0046] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0047] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0048] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0049] As described above, although the present invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the present invention itself. Various changes in form and details may be made without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims

1. A time division multiplexing method for computing switch resources in a network, characterized in that: When a global reduction operation AllReduce of one of the tasks is about to start: S1. Determine whether there are other tasks currently using the online computing resources. If so, the current AllReduce is not allowed to use the online computing resources; otherwise, proceed to step S2; S2. Determine whether there will be AllReduces of other tasks arriving during the execution of the current AllReduce. If not, allow the current AllReduce to use the computing resources on the network; otherwise, proceed to step S3. S3. Use the CalcScore function to calculate the scores of the current AllReduce and all other tasks that are about to arrive at the AllReduce: If the current AllReduce score is higher than the AllReduce score of other tasks that are about to arrive, the current AllReduce is allowed to use the online computing resources; Otherwise, it is not allowed, and the priority of using the online computing resources is given according to the score, and the online computing resources are used according to the predetermined priority.

2. The time division multiplexing method for computing switch resources on the network according to claim 1, characterized in that: When a certain AllReduce is not allowed to use the computing resources on the network, a traditional algorithm can be used for execution instead of waiting for the computing resources on the network; the traditional algorithm includes Ring AllReduce or Tree AllReduce.

3. The time division multiplexing method for computing switch resources on the network according to claim 1, characterized in that: When an AllReduce request arrives, it is not directly added to the CUDA stream queue, but saved in a custom queue to ensure that there is always only one kernel function executing in the CUDA stream queue.

4. The time division multiplexing method for computing switch resources on the network according to claim 3, characterized in that: Whenever a kernel function in a CUDA stream queue is executed, a host callback function is executed; In the callback function, a new AllReduce request is taken from the system queue to determine whether it can be accelerated using the computing resources on the network, and then inserted into the CUDA stream queue after configuration.

5. The time division multiplexing method for computing switch resources on the network according to any one of claims 1 to 4, characterized in that: A shared controller is used to ensure that computing nodes executing the same task reach a consensus on whether each AllReduce uses on-network computing acceleration.

6. The time division multiplexing method for computing switch resources on the network according to claim 5, characterized in that: The shared controller synchronizes the cluster task execution status to each computing node, so that each computing node in the same task can calculate the same result according to the same cluster status, thereby controlling each AllReduce to use or not use network computing acceleration.

7. The time division multiplexing method for computing switch resources on the network according to claim 6, characterized in that: The use of a shared controller to ensure that computing nodes executing the same task reach a consensus on whether each AllReduce uses on-line computing acceleration specifically includes: When the status of any task sharing the same switch changes, the shared controller sends a "ready to update" message to all compute nodes using the switch; After receiving the message, each computing node tries to acquire the lock; After successfully acquiring the lock, the computing node sends a message to the shared controller to confirm receipt, including the number of the current AllReduce; The shared controller verifies that the AllReduce number received from all compute nodes of the same task is the same; If the numbers are the same, the shared controller sends information to each node to update the status; otherwise, it aborts the attempt and tries again after a while; Each computing node updates its state based on the information received and releases the lock; Each computing node finally sends a message to the shared controller to confirm receipt, completing the process.

8. A time division multiplexing system for computing switch resources on a network, characterized in that: It includes a cluster control node, multiple GPU training accelerator cards and multiple switches with online computing capabilities; Multiple GPU training accelerator cards can be used to train multiple tasks in a distributed and parallel manner, and perform multi-card All-Reduce collective communication during the training process; The cluster control node includes at least one shared controller; the shared controller can execute the time-division multiplexing method of the on-line computing switch resources as described in any one of claims 1 to 7, so as to calculate in real time whether the current All-Reduce communication uses the on-line computing resources.

9. A computer-readable storage medium, characterized in that: The storage medium stores at least one executable instruction. When the executable instruction is executed on the shared controller, the shared controller executes the time division multiplexing method for in-network computing switch resources according to any one of claims 1 to 7.

Images