Automatic online tuning method, system, device and storage medium of NCCL library

Abstract

The present invention discloses an automatic online tuning method, system, device and storage medium of an NCCL library, which are corresponding schemes. In the relevant scheme, under the premise of ensuring that the collective communication tasks of the NCCL library can be correctly executed, online tuning can be performed according to the execution result of each configuration, so as to continuously optimize the configuration, improve the performance and reduce the time. The method is suitable for complex distributed communication scenarios without introducing too much tuning overhead. In addition, the relevant scheme can quickly find a configuration with better performance in a huge search space through an effective tuning process. Finally, the evaluation of multiple clusters and models shows that the present invention can quickly converge on DNN model training and can ensure higher performance benefits than NCCL.

Description

Technical Field

[0001] The present invention relates to the field of high performance computing of computers, and in particular to an automatic online tuning method, system, device and storage medium of an NCCL library. Background Art

[0002] As the size of Deep Neural Networks (DNN) models and datasets continues to grow, it has become standard practice to use multiple GPU (graphics processing unit) nodes for distributed parallel training. Commonly used parallel strategies include data parallelism, tensor parallelism, and pipeline parallelism. Although data parallelism is effective within the memory limitations of a single GPU, when the model exceeds the limit, more complex communication modes must be adopted, which often leads to communication becoming the main bottleneck for training. In order to optimize communication performance, researchers have proposed a variety of solutions, such as scheduling, data compression, sparsity exploration, and topology-based algorithm design. Most methods assume that the underlying communication library has been optimized. Experiments have shown that communication tuning is crucial to improving bandwidth performance.

[0003] Distributed deep neural network training introduces a specific communication mode, called collective communications, which is used to aggregate or propagate data across multiple GPUs. The NVIDIA Collective Communication Library (NCCL) is designed to optimize multi-GPU and multi-node communications between NVIDIA GPUs and network systems, providing high-performance primitives such as ReduceScatter, AllGather, and AllReduce to achieve low-latency and high-bandwidth communication. In DNN training, the NCCL library is often used to handle collective communications, and the appropriate communication primitives are selected based on the number of model parameters, parallel training methods, and the number of GPUs. For example, the VGG-19 model (a 19-layer convolutional neural network model) uses 32-way data parallelism and uses AllReduce to exchange six model gradients in each iteration. The Yi-1.5-34B model (a large language model developed by the Zero One team) performs AllGather and ReduceScatter operations in one training iteration, transferring 56MB of activation data.

[0004] When the training framework calls the underlying NCCL communication primitive, the NCCL runtime will create a corresponding collective communication task and assign it a configuration of<algorithm, protocol, transport, Nchannel,Nthread, chunk_size> Among them, algorithm determines how data is distributed, combined or aggregated on the GPU; protocol determines how data is managed in GPU memory, transport determines how data is transmitted between GPUs, and the other configurations determine the computing resources allocated to collective communication tasks. Experiments show that changes in these configurations have an important impact on the performance of collective communication, so it is necessary to tune these parameters; on the other hand, in large model training scenarios, there is a problem of mutual interference between computing and communication, which makes the configuration found by offline tuning not optimal, so it is necessary to automatically tune these parameters online to dynamically optimize communication performance.

[0005] In view of this, the present invention is proposed. Summary of the invention

[0006] The purpose of the present invention is to provide an automatic online tuning method, system, device and storage medium for an NCCL library, which can perform online tuning according to the execution result of each configuration, continuously optimize the configuration, improve the performance, and reduce the time, while ensuring that the collective communication tasks of the NCCL library can be correctly executed. The method is suitable for complex distributed communication scenarios and can achieve rapid convergence during the DNN model training process.

[0007] The objective of the present invention is achieved through the following technical solutions:

[0008] An automatic online tuning method for an NCCL library, comprising:

[0009] When receiving the collective communication task of the NCCL library, it queries its own configuration table to obtain the best configuration and executes it, completes the corresponding collective communication task of the NCCL library, and records the execution time; wherein, the NCCL library is the NVIDIA collective communication library;

[0010] updating the history table in combination with the optimal configuration and its execution time, and starting the tuning process in combination with the history table to obtain a new optimal configuration; wherein, in the tuning process, the parameters in the configuration are divided into parameters related to implementation and parameters related to computing resource allocation, and the optimal combination of the parameters related to implementation and the parameters related to computing resource allocation is selected with performance as an indicator, and is used as the new optimal configuration;

[0011] The new optimal configuration is updated into the configuration table.

[0012] An automatic online tuning system for an NCCL library, comprising: a leader and a worker; wherein the leader comprises: an optimizer, a coordinator and a first executor; the worker comprises a second executor; the leader and the worker both store a configuration table;

[0013] The first executor and the second executor are used to query their own configuration tables to obtain the best configuration and execute when receiving the collective communication task of the NCCL library, so as to complete the corresponding collective communication task of the NCCL library. At the same time, after the first executor executes, the execution time is also recorded; wherein the NCCL library is the NVIDIA collective communication library;

[0014] The optimizer is used to update the history table in combination with the optimal configuration and the execution time recorded by the first executor, and start the tuning process in combination with the history table to obtain a new optimal configuration; wherein, during the tuning process, the parameters in the configuration are divided into parameters related to implementation and parameters related to computing resource allocation, and the optimal combination of the parameters related to implementation and the parameters related to computing resource allocation is selected with performance as an indicator as the new optimal configuration;

[0015] The coordinator is used to update the new optimal configuration into the configuration tables of the leader and the workers.

[0016] A processing device, comprising: one or more processors; a memory for storing one or more programs;

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0018] A readable storage medium stores a computer program, which implements the above method when the computer program is executed by a processor.

[0019] It can be seen from the technical solution provided by the present invention that, firstly, the performance of collective communication tasks can be improved by tuning the configuration parameters without introducing too much tuning overhead; secondly, an effective tuning solution can be used to quickly find a configuration with better performance in a huge search space; finally, the evaluation of multiple clusters and models shows that the present invention can converge quickly in DNN model training and can guarantee higher performance benefits than NCCL. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0021] Figure 1 A flowchart of an automatic online tuning method for an NCCL library provided by an embodiment of the present invention;

[0022] Figure 2 A schematic diagram of the overall architecture of an automatic online tuning method for an NCCL library provided in an embodiment of the present invention;

[0023] Figure 3 A flowchart from upper layer calling communication primitives to executing configuration, as well as online tuning and configuration update provided by an embodiment of the present invention;

[0024] Figure 4 A flowchart of a subspace-based tuning method provided by an embodiment of the present invention;

[0025] Figure 5 A schematic diagram of an embodiment of the present invention applied to DNN training;

[0026] Figure 6 A schematic diagram of a processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

[0028] First, the terms that may be used in this article are explained as follows:

[0029] The terms "include", "comprises", "contains", "has" or other descriptions with similar semantics should be interpreted as non-exclusive inclusion. For example, including certain technical feature elements (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or products, etc.) should be interpreted as including not only certain technical feature elements explicitly listed, but also other technical feature elements known in the art that are not explicitly listed.

[0030] The term "consisting of..." means excluding any technical feature elements not explicitly listed. If this term is used in a claim, it will make the claim closed, so that it does not contain technical feature elements other than the technical feature elements explicitly listed, except for the conventional impurities related to them. If this term only appears in a clause of a claim, it only limits the elements explicitly listed in the clause, and the elements recorded in other clauses are not excluded from the overall claim.

[0031] The following is a detailed description of an automatic online tuning method, system, device and storage medium for an NCCL library provided by the present invention. The contents not described in detail in the embodiments of the present invention belong to the prior art known to professional and technical personnel in the field. If no specific conditions are specified in the embodiments of the present invention, the conventional conditions in the field or the conditions recommended by the manufacturer are followed. The instruments used in the embodiments of the present invention, if the manufacturer is not specified, are all conventional products that can be purchased commercially.

[0032] Embodiment 1

[0033] The embodiment of the present invention provides an automatic online tuning method for NCCL library, such as Figure 1 As shown, it mainly includes the following steps:

[0034] Step 1: When receiving a collective communication task from the NCCL library, query its own configuration table to obtain the best configuration for calling the NCCL library communication primitive and execute it.

[0035] In the embodiment of the present invention, on the one hand, the corresponding NCCL library collective communication task is executed through the optimal configuration, and on the other hand, the execution time needs to be recorded for subsequent tuning process.

[0036] Step 2: Update the history table in combination with the optimal configuration and its execution time, and start the tuning process in combination with the history table to obtain a new optimal configuration.

[0037] In an embodiment of the present invention, during the tuning process, the parameters in the configuration are divided into parameters related to implementation and parameters related to computing resource allocation. Taking performance as an indicator, the best combination of parameters related to implementation and parameters related to computing resource allocation is selected and used as the new best configuration.

[0038] Step 3: Update the new optimal configuration into the configuration table.

[0039] In the above scheme provided by the embodiment of the present invention, when the upper layer continuously calls the communication primitive, the configuration is dynamically adjusted online according to the execution time of each configuration, and under the condition that each communication can be correctly executed, the configuration with better performance can be continuously tried according to the tuning algorithm; at the same time, the tuning process is performed asynchronously, ensuring the minimization of the tuning overhead each time; the tuning process does not modify the underlying logic of NCCL too much, and is coupled with the code of NCCL itself. The tuning can infer the next optimal configuration according to the online execution time of each communication primitive, and can overcome the influence of computing communication interference on the actual communication execution time in model training, so it can be widely used in scenarios such as model training.

[0040] In order to more clearly demonstrate the technical solution and technical effects provided by the present invention, the method provided by the embodiment of the present invention is described in detail with reference to specific embodiments below.

[0041] 1. General introduction of the plan.

[0042] The solution provided by the embodiment of the present invention mainly includes two roles, namely the leader and the worker, both of which are GPUs, and both of which store configuration tables and update synchronously. Usually, one GPU is designated as the leader, and all other GPUs are workers; the worker is mainly responsible for executing the collective communication task (referred to as the collective communication task) of the NCCL library, and gives priority to the use of the tuned configuration to ensure the best execution effect. The leader leads the tuning process, determines the best configuration and updates it to all worker nodes. The leader mainly includes: the first executor, which is used to execute the collective communication task and feedback performance data for continuous optimization; the optimizer: responsible for collecting performance data, determining and executing configuration tuning, so as to dynamically improve system efficiency; the coordinator: synchronously updates the configuration of the entire communication group to ensure that all nodes use the latest optimized configuration; here, the workers and leaders are regarded as nodes, and all nodes include all workers and leaders.

[0043] Figure 2 The overall structure of the present invention is provided, and each part is introduced in detail below.

[0044] Leader: In the present invention, the execution of collective communication tasks is led by a GPU designated as a leader. Figure 2 Task1 and task2 represent two collective communication tasks. The leader is responsible for running the tuning algorithm, determining the best configuration, and updating it to other GPUs (i.e., workers). This is fundamentally different from the design of NCCL, in which each GPU is an equivalent peer, and each peer independently generates a default configuration based on a deterministic cost model. In the example of the present invention, the leader plays a core control and optimization role.

[0045] The optimizer is an important component responsible for performance optimization in the example of the present invention. It mainly collects performance data (i.e., the execution time corresponding to the configuration) provided by the executor and decides whether tuning is needed. When it is considered that tuning will help improve performance, the optimizer starts the tuning process. It can search for the best configuration independently without the coordination of other nodes. The function of the optimizer enables the example of the present invention to flexibly respond to performance changes during execution and adjust the configuration in real time, thereby improving the efficiency of the overall system.

[0046] The Coordinator is notified and started by the Optimizer to update the configuration of all nodes in the entire collective communication group. Specifically, the Coordinator updates the new tuning configuration generated by the leader to the configuration table of all nodes in the communication group through broadcast operations. This means that when a new configuration is found to be valid, the Coordinator ensures that the configuration is applied synchronously throughout the communication group, so that all subsequent identical collective communication tasks can use the latest tuning configuration to achieve an overall optimization effect.

[0047] The executor is mainly located on the worker and the leader. To distinguish, it is called the first executor when it is located on the leader, and the second executor when it is located on the worker. The second executor is responsible for executing the actual collective communication task. When receiving the collective communication task, the second executor will select the corresponding configuration according to the configuration table to execute it. The first executor will also select the corresponding configuration according to the configuration table to execute the collective communication task. It will also record the execution time of the task under the corresponding configuration and feed it back to the optimizer as an important basis for the optimization process. This enables the entire system to continuously accumulate runtime data, thereby gradually optimizing the configuration.

[0048] Worker: The worker in the present invention is similar to a peer in NCCL. Each worker has a configuration table that stores the default configuration of the task (generated by the cost model of NCCL) and the tuned configuration generated by the leader. When the worker receives a collective communication task, the second executor checks the configuration table and gives priority to the tuned configuration. If there is no tuned configuration, the default configuration is used to execute the task. In this way, the role design of the worker is relatively simple, and its main function is to execute the task and apply the configuration provided by the leader. Figure 2 The default and tuned configurations are omitted.

[0049] In the above solution, the optimizer collects the performance data of the executor and changes the running configuration in time to achieve the purpose of real-time feedback and tuning. At the same time, the configuration table is maintained between nodes and coordinated by the coordinator to ensure that all nodes use consistent configurations. At the same time, the update of the optimizer and the execution of the executor are asynchronous, which minimizes the overhead caused by tuning.

[0050] In the embodiment of the present invention, for a communication primitive called by the upper layer, the execution function will directly obtain the next configuration from the configuration table maintained by the current process, and assign the configuration to the communication primitive for execution. At the same time, the asynchronous thread will call the optimizer, and the optimizer will update the history table according to the execution time of the communication primitive corresponding to each configuration, and determine the next appropriate configuration based on the information in the history table by the tuning algorithm and send it to the coordinator. The coordinator will broadcast the configuration information and write it into the configuration table corresponding to each node.

[0051] The above scheme mainly involves the following four parts:

[0052] (1) Real-time feedback and tuning: Based on the actual time feedback of each collective communication task execution, the optimizer automatically generates a new configuration to improve communication performance and reduce execution time;

[0053] (2) Configuration table and coordination mechanism: Maintain the configuration table. The coordinator is responsible for broadcasting update information to ensure that all nodes use the latest and consistent configuration.

[0054] (3) Efficient tuning mechanism: The asynchronous optimizer executes and obtains configurations from the configuration table, dynamically adjusting the configuration of communication primitives to optimize task execution efficiency;

[0055] (4) Low coupling design: The method maintains low coupling with the underlying logic of the NCCL library, allowing effective tuning without modifying the original code execution logic, and is suitable for various distributed computing scenarios.

[0056] In the embodiment of the present invention, during the tuning process, the parameters in the configuration are divided into parameters related to implementation and parameters related to computing resource allocation, and the group coordinate descent method is executed based on the classification to continuously find points with better performance.

[0057] To illustrate the effect of the present invention, it is applied to dynamically optimize communication performance during DNN model training, including: through online tuning in each round of communication, the method of the embodiment of the present invention can usually converge to a near-optimal configuration within the first 1-3 iterations, showing the advantage of quickly achieving efficient configuration in a new environment, a capability that cannot be achieved by traditional offline tuning.

[0058] 2. Detailed introduction of the plan.

[0059] 1. The overall process of the plan.

[0060] like Figure 3 As shown, a schematic diagram of the NCCL automatic tuning method provided by an embodiment of the present invention, the rounded rectangles in the figure represent different GPU processes in collective communication, there are two roles, namely leader and worker, where all roles have a configuration table (Config Table) to provide the configuration required for the next execution of the same communication primitive, the configuration table is updated by the coordinator, and the executors within all roles will call the execution function to complete the execution of the communication primitive; in addition, the leader node will have a history table (History Table) to record the time required for each configuration, and an optimizer (Optimizer) to update the history table, execute the tuning algorithm, and deduce the configuration for the next execution based on the previous results. The coordinator (Coordinator) will broadcast the configuration obtained by the optimizer to all roles. The parameters in the configuration include<algorithm, protocol, transport, Nchannel, Nthread, chunk_size> , Figure 3 The abbreviations are: A, P, T, NC, NT, C.

[0061] Figure 3 A specific example is also provided in the text, which shows the workflow of the embodiment of the present invention in detail to illustrate how the system can effectively perform and optimize the collective communication task. On the leader, the initial configuration table only contains an initial configuration config1 (<Ring, LL, SHM, 8, 32, 2MB> ), which is used to process the collective communication task task2. In the subsequent process, configuration config2 will be gradually generated during the tuning process. In this process, steps a1 and a2 show the process of obtaining and executing the current optimal configuration, and steps b1 to b5 detail the specific process of online tuning and configuration update.

[0062] (1) Obtain and execute the current best configuration.

[0063] Assume that the leader and worker nodes simultaneously receive a new collective communication task, whose communication primitives, message size, and communication group members are exactly the same as the previously recorded collective communication task task2. For example, the task type (type) is AllGather operation, the message size (size) is 40MB, and the size of the communication group (Group) is 2. In order to execute this task, the leader and worker first need to query their own configuration table to obtain the corresponding target configuration (step a1). Since the configuration table is completely replicated between the leader and the worker, they can both retrieve the same configuration config1 and use it to submit the collective communication task task2 to the NCCL runtime for execution (step a2). Because the tuning method is tightly coupled with the NCCL runtime, NCCL can easily accept the configurations passed from the upper layer and perform corresponding execution operations based on these configurations, thereby ensuring that the task can be performed in an optimized manner.

[0064] Those skilled in the art can understand that NCCL Runtime is a technical term in the art, and can be understood as an NCCL library function being executed.

[0065] (2) Online tuning.

[0066] After the task is executed, the executor on the leader node returns the actual execution time of the task (14 milliseconds) to the optimizer (step b1). After receiving the execution time, the optimizer updates the data in the history table (step b2) so that the historical records can reflect the latest task execution. Next, the optimizer retrieves the historical execution data related to task2 and starts a new round of tuning process (step b3). Based on the historical data, the optimizer observes that when the NT parameter is increased from 16 to 32, the communication time decreases from 16 milliseconds to 14 milliseconds (not shown in the table). Therefore, the optimizer decides to further increase the NT parameter and set it to 64, thereby generating a new configuration config2. The optimizer then notifies the collaborative coordinator of this update (step b4) so ​​that the new configuration can be applied throughout the system.

[0067] (3) Configuration update.

[0068] The coordinator updates the new configuration config2 to the configuration tables of the leader and the worker (step b5) through a broadcast operation with synchronous semantics to replace the original configuration config1. This synchronous update mechanism ensures that the configuration tables of the leader and the worker remain consistent, so that the subsequent collective communication task task2 can use the latest optimized configuration. In this way, the subsequent execution of task2 will use the more optimized config2 to continue to reduce communication time and improve the overall performance of the system. The online tuning process will continue until the optimal configuration is found, so that the execution efficiency of the collective communication task reaches the optimal level.

[0069] It should be noted that in the above example, the provided task2, config1, and config2 are mainly used to identify the collective communication task and two configuration information. In actual applications, the content of the collective communication task, the specific parameter content of the configuration, and their identification can be adjusted according to actual conditions, and the present invention does not limit it.

[0070] The above solution provided by the embodiment of the present invention enables the system to continuously optimize the configuration according to the real-time execution results by dynamically adjusting the task configuration, gradually improving the communication performance and reducing the execution time. This method is particularly suitable for distributed computing scenarios with complex communication requirements. Under different load conditions, various parameters can be flexibly adjusted to ensure that the entire system always runs in an efficient state.

[0071] 2. Online tuning implementation plan.

[0072] In the embodiment of the present invention, the parameters in the configuration include<algorithm, protocol, transport,Nchannel, Nthread, chunk_size> . Among them, algorithm, protocol and transport represent the algorithm, protocol and transport respectively. They are implementation-related parameters; they jointly determine the topological structure of the algorithm execution logic, the method of data transmission, and the software calls required to complete the primitive semantics. These parameters are crucial, but the search space is relatively small. For example, there are only two options for the algorithm (Ring and Tree), three options for the protocol (LL, LL128 and Simple), and only two options for the transport (SHM and P2P). The symbols involved in the above parameters are all common forms in this field. Among them, Ring means connecting GPUs in a ring manner to perform communication; Tree means connecting GPUs in a tree manner to perform communication; LL and LL128 are two low-latency protocols of different granularity; Simple is a simple protocol that can achieve high bandwidth; SHM means that the communication between GPUs will pass through the shared memory on the CPU side; P2P means that the communication between GPUs will be achieved through GPU direct memory access.

[0073] Nchannel, Nthread, and chunk_size represent the number of communication blocks (the more the more obvious the competition, the lower the GPU resource utilization), the number of threads, and the size of the communication block, respectively. They are parameters related to the allocation of computing resources. As we all know, the collective communication tasks in the model training process require network data transmission and GPU resource operations (such as accumulation and averaging). These three parameters together determine how to parallelize the utilization of network bandwidth and GPU computing power. The search space for these parameters is relatively large, with options such as chunk size up to 8,192 and Nchannel up to 128. It is further observed that the analysis of the impact of these parameters on performance must be based on the previous selection of the first three implementation parameters. In other words, the selection of the first three parameters provides the basis for the modeling analysis of the next three parameters. Although the first three parameters can be fully searched, the combination of the next three parameters is almost impossible to fully search. Therefore, it is necessary to establish a performance model for these resource allocation related parameters.

[0074] Based on the classification of the above parameters, the embodiment of the present invention proposes a subspace-based tuning method, which uses different combinations of implementation-related parameters as different subspaces.<algorithm, protocol, transport> Triples are used as subspaces, and configurations with the same triples belong to the same subspace. In each subspace, a unified performance model is used to determine the best combination of parameters related to computing resource allocation; the best combinations corresponding to all subspaces are compared, and the global best combination is selected as the new best configuration. This method helps avoid the explosion of resource parameter search space and eliminates the reliance on implementation-related parameter heuristic rules.

[0075] The subspace-based tuning method uses the form of grouped coordinate descent. When the upper layer calls the optimizer to update the history table, the tuning algorithm will be executed to select the next appropriate configuration. The specific process is as follows: Figure 4 As shown, at the beginning, it will determine whether all configurations of all current subspaces have been traversed. If they have been traversed, the best configuration will be returned directly, otherwise it will return to step ①.

[0076] Step 1: Traverse each subspace. For efficiency, if the gap between the current subspace performance and the best global performance value exceeds the threshold, filter out the current subspace; otherwise, go to step 2.

[0077] Step ②: Use the current subspace to randomly generate a configuration p, and initialize the current tuning dimension and direction, as well as the learning rate. The dimension is used to control the parameters in the current tuning configuration (that is, one of Nchannel, Nthread, chunk_size), the direction is used to control the increase or decrease of the parameters in the current tuning configuration, and the learning rate is used to control the span of adjusting the current dimension parameters when selecting a new configuration in the next step; wherein, the configuration p includes the current subspace and randomly generated parameters related to computing resource allocation.

[0078] Step 3: Call NCCL runtime to execute configuration p, obtain the execution time corresponding to configuration p, and compare it with the execution time of the best configuration of the current subspace. If this is the first tuning, the best configuration of the current subspace is the execution time of the default configuration. If this is not the first tuning, the best configuration is the configuration with the minimum execution time of the current subspace that has been recorded.

[0079] Step ④: If it is greater than the execution time of the optimal configuration of the current subspace, use configuration p to update the optimal configuration of the current subspace, and adjust the current learning rate according to the difference in bandwidth before and after the update, generate the next configuration, and step ③; where the bandwidth is calculated by the execution time.

[0080] Those skilled in the art will understand that bandwidth is a technical term in the art, which refers to the speed (efficiency) of executing a communication primitive.

[0081] Step 5: If it is not greater than the execution time of the optimal configuration of the current subspace, adjust the current tuning dimension or direction, reset the learning rate, generate the next configuration, and return to step 3.

[0082] Repeat until the set stop condition is reached to obtain the optimal configuration of the current subspace. Specifically: if the execution time after adjusting all dimensions and directions cannot exceed the optimal configuration of the current subspace, then randomly select a new direction and dimension and return to step ③, and stop after the set number of random selections to obtain the optimal configuration of the current subspace.

[0083] The above subspace-based tuning method first divides the configuration parameters into implementation-related parameters and performance-related parameters. First, dividing the parameters can ensure the priority processing of implementation-related parameters, which is in line with the underlying operating logic of the NCCL library. Secondly, the coordinate descent method is implemented for tuning resource-related parameters, which can quickly find the optimal solution when the search space is large and conform to the performance laws of resource-related configurations.

[0084] 3. Application examples of the solution.

[0085] like Figure 5 As shown, an example of applying the above scheme in DNN model training is provided. Figure 5 In the figure, the box represents the iteration round of model training, the length of the box represents the length of model training time, the partner node represents that there is no functional difference between GPUs under NCCL execution, while there is a difference between leaders and workers in the example of the present invention, k represents that there is tuning in the communication of the first k rounds of training and the optimal performance solution has been found in the first k rounds of training, N represents the total number of training rounds, and the end-to-end benefit represents the saved training time. The example of the present invention can perform online tuning in each round of communication, and apply the best optimized configuration to the next execution of the communication primitive. For a normal Phi-2B model (a small language model with 2.7 billion parameters developed by Microsoft Research), there are about 2000-3000 AllGather and ReduceScatter in one iteration, and the embodiment of the present invention can tune the nearly optimal configuration after thousands of AllGather or ReduceScatter executions, so in the actual execution process, performance convergence can often be achieved in the first 1-3 rounds of iterations. The solution reasonably provided by the implementation of the present invention can quickly find a sufficiently efficient configuration in only a few iterations or a few minutes through online tuning, showing its advantage of fast tuning in a new environment, which is difficult to achieve with offline tuning strategies.

[0086] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by means of software plus necessary general hardware platforms. Based on such understanding, the technical solutions of the above embodiments can be embodied in the form of software products, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0087] Embodiment 2

[0088] The present invention also provides an automatic online tuning system for NCCL library, which is mainly used to implement the method provided in the above embodiment. The overall architecture of the system can be found in the above Figure 2 , the system mainly includes: a leader and a worker; wherein the leader includes: an optimizer, a coordinator and a first executor; the worker includes a second executor; the leader and the worker both store a configuration table;

[0089] The first executor and the second executor are used to query their own configuration tables to obtain the best configuration and execute when receiving the collective communication task of the NCCL library, so as to complete the corresponding collective communication task of the NCCL library. At the same time, after the first executor executes, the execution time is also recorded; wherein the NCCL library is the NVIDIA collective communication library;

[0090] The optimizer is used to update the history table in combination with the optimal configuration and the execution time recorded by the first executor, and start the tuning process in combination with the history table to obtain a new optimal configuration; wherein, during the tuning process, the parameters in the configuration are divided into parameters related to implementation and parameters related to computing resource allocation, and the optimal combination of the parameters related to implementation and the parameters related to computing resource allocation is selected with performance as an indicator as the new optimal configuration;

[0091] The coordinator is used to update the new optimal configuration into the configuration tables of the leader and the workers.

[0092] In the embodiment of the present invention, the configuration content is divided into parameters related to implementation and parameters related to computing resource allocation, including: the parameters in the configuration include<algorithm, protocol, transport, Nchannel,Nthread, chunk_size> ; Among them, algorithm, protocol and transport represent the algorithm, protocol and transport respectively, which are parameters related to implementation; Nchannel, Nthread and chunk_size represent the number of communication blocks, the number of threads and the size of communication blocks respectively, which are parameters related to computing resource allocation.

[0093] In the embodiment of the present invention, the method uses performance as an indicator to select the best combination of parameters related to implementation and parameters related to computing resource allocation as a new best configuration; including: taking different combinations of parameters related to implementation as different subspaces, and in each subspace, using a unified performance model to determine the best combination of parameters related to computing resource allocation; comparing the best combinations corresponding to all subspaces, and selecting the global best combination as the new best configuration.

[0094] In the embodiment of the present invention, the use of a unified performance model to determine the best combination of parameters related to computing resource allocation in each subspace includes:

[0095] Step 1: Traverse each subspace. If the difference between the current subspace performance and the global best performance value exceeds the threshold, filter out the current subspace; otherwise, go to step 2;

[0096] Step 2: Use the current subspace to randomly generate a configuration p, and initialize the current tuning dimension and direction, as well as the learning rate. The dimension is used to control the parameters in the current tuning configuration, the direction is used to control the increase or decrease of the parameters in the current tuning configuration, and the learning rate is used to control the span of adjusting the current dimension parameters when selecting a new configuration in the next step; wherein the configuration p includes the current subspace and randomly generated parameters related to computing resource allocation;

[0097] Step 3: Call NCCL runtime to execute configuration p, obtain the execution time corresponding to configuration p, and compare it with the execution time of the best configuration of the current subspace. If this is the first tuning, the best configuration of the current subspace is the execution time of the default configuration. If this is not the first tuning, the best configuration is the configuration with the minimum execution time of the current subspace that has been recorded.

[0098] Step 4: If it is greater than the execution time of the optimal configuration of the current subspace, use configuration p to update the optimal configuration of the current subspace, and adjust the current learning rate according to the difference in bandwidth before and after the update, generate the next configuration, and return to step 3; where the bandwidth is calculated by the execution time;

[0099] Step 5: If it is not greater than the execution time of the optimal configuration of the current subspace, adjust the current tuning dimension or direction, reset the learning rate, generate the next configuration, and return to step 3;

[0100] Repeat until the set stop condition is reached and the optimal configuration of the current subspace is obtained.

[0101] Technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0102] Embodiment 3

[0103] The present invention also provides a processing device, such as Figure 6 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the methods provided in the aforementioned embodiments.

[0104] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0105] In the embodiment of the present invention, the specific types of the memory, input device and output device are not limited; for example:

[0106] The input device may be a touch screen, an image acquisition device, a physical button or a mouse, etc.;

[0107] The output device may be a display terminal;

[0108] The memory may be a random access memory (RAM) or a non-volatile memory such as a disk storage.

[0109] Embodiment 4

[0110] The present invention also provides a readable storage medium storing a computer program, which implements the method provided in the above embodiment when the computer program is executed by a processor.

[0111] In the embodiment of the present invention, the readable storage medium as a computer-readable storage medium can be set in the aforementioned processing device, for example, as a memory in the processing device. In addition, the readable storage medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk, etc., which can store program codes.

[0112] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. An automatic online tuning method for NCCL library, characterized in that: include: When receiving the collective communication task of the NCCL library, it queries its own configuration table to obtain the best configuration and executes it, completes the corresponding collective communication task of the NCCL library, and records the execution time; wherein, the NCCL library is the NVIDIA collective communication library; updating the history table in combination with the optimal configuration and its execution time, and starting the tuning process in combination with the history table to obtain a new optimal configuration; wherein, in the tuning process, the parameters in the configuration are divided into parameters related to implementation and parameters related to computing resource allocation, and the optimal combination of the parameters related to implementation and the parameters related to computing resource allocation is selected with performance as an indicator, and is used as the new optimal configuration; Updating the new optimal configuration into the configuration table; The method of using performance as an indicator to select the best combination of parameters related to implementation and parameters related to computing resource allocation as a new best configuration includes: using different combinations of parameters related to implementation as different subspaces, and in each subspace, using a unified performance model to determine the best combination of parameters related to computing resource allocation; comparing the best combinations corresponding to all subspaces, and selecting the global best combination as the new best configuration; Wherein, in each subspace, using a unified performance model to determine an optimal combination of parameters related to computing resource allocation includes: Step 1: Traverse each subspace. If the difference between the current subspace performance and the global best performance value exceeds the threshold, filter out the current subspace; otherwise, go to step 2; Step 2: Use the current subspace to randomly generate a configuration p, and initialize the current tuning dimension and direction, as well as the learning rate. The dimension is used to control the parameters in the current tuning configuration, the direction is used to control the increase or decrease of the parameters in the current tuning configuration, and the learning rate is used to control the span of adjusting the current dimension parameters when selecting a new configuration in the next step; wherein the configuration p includes the current subspace and randomly generated parameters related to computing resource allocation; Step 3: Call NCCL runtime to execute configuration p, obtain the execution time corresponding to configuration p, and compare it with the execution time of the best configuration of the current subspace. If this is the first tuning, the best configuration of the current subspace is the execution time of the default configuration. If this is not the first tuning, the best configuration is the configuration with the minimum execution time of the current subspace that has been recorded. Step 4: If it is greater than the execution time of the optimal configuration of the current subspace, use configuration p to update the optimal configuration of the current subspace, and adjust the current learning rate according to the difference in bandwidth before and after the update, generate the next configuration, and return to step 3; where the bandwidth is calculated by the execution time; Step 5: If it is not greater than the execution time of the optimal configuration of the current subspace, adjust the current tuning dimension or direction, reset the learning rate, generate the next configuration, and return to step 3; Repeat until the set stop condition is reached and the optimal configuration of the current subspace is obtained.

2. The automatic online tuning method of the NCCL library according to claim 1, characterized in that: The division of the parameters in the configuration into parameters related to implementation and parameters related to computing resource allocation includes: The parameters in the configuration include<algorithm, protocol, transport, Nchannel, Nthread,chunk_size> ; Among them, algorithm, protocol and transport represent the algorithm, protocol and transport respectively, which are parameters related to implementation; Nchannel, Nthread and chunk_size represent the number of communication blocks, the number of threads and the size of communication blocks respectively, which are parameters related to computing resource allocation.

3. An automatic online tuning system for NCCL library, characterized in that: include: A leader and a worker; wherein the leader includes: an optimizer, a coordinator and a first executor; the worker includes a second executor; the leader and the worker both store a configuration table; The first executor and the second executor are used to query their own configuration tables to obtain the best configuration and execute when receiving the collective communication task of the NCCL library, and complete the corresponding collective communication task of the NCCL library. At the same time, after the first executor executes, the execution time is also recorded; wherein the NCCL library is the NVIDIA collective communication library; The optimizer is used to update the history table in combination with the optimal configuration and the execution time recorded by the first executor, and start the tuning process in combination with the history table to obtain a new optimal configuration; wherein, during the tuning process, the parameters in the configuration are divided into parameters related to implementation and parameters related to computing resource allocation, and the optimal combination of the parameters related to implementation and the parameters related to computing resource allocation is selected with performance as an indicator as the new optimal configuration; The coordinator is used to update the new optimal configuration into the configuration tables of the leader and the workers; The method of using performance as an indicator to select the best combination of parameters related to implementation and parameters related to computing resource allocation as a new best configuration includes: using different combinations of parameters related to implementation as different subspaces, and in each subspace, using a unified performance model to determine the best combination of parameters related to computing resource allocation; comparing the best combinations corresponding to all subspaces, and selecting the global best combination as the new best configuration; Wherein, in each subspace, using a unified performance model to determine an optimal combination of parameters related to computing resource allocation includes: Step 1: Traverse each subspace. If the difference between the current subspace performance and the global best performance value exceeds the threshold, filter out the current subspace; otherwise, go to step 2; Step 2: Use the current subspace to randomly generate a configuration p, and initialize the current tuning dimension and direction, as well as the learning rate. The dimension is used to control the parameters in the current tuning configuration, the direction is used to control the increase or decrease of the parameters in the current tuning configuration, and the learning rate is used to control the span of adjusting the current dimension parameters when selecting a new configuration in the next step; wherein the configuration p includes the current subspace and randomly generated parameters related to computing resource allocation; Step 3: Call NCCL runtime to execute configuration p, obtain the execution time corresponding to configuration p, and compare it with the execution time of the best configuration of the current subspace. If this is the first tuning, the best configuration of the current subspace is the execution time of the default configuration. If this is not the first tuning, the best configuration is the configuration with the minimum execution time of the current subspace that has been recorded. Step 4: If it is greater than the execution time of the optimal configuration of the current subspace, use configuration p to update the optimal configuration of the current subspace, and adjust the current learning rate according to the difference in bandwidth before and after the update, generate the next configuration, and return to step 3; where the bandwidth is calculated by the execution time; Step 5: If it is not greater than the execution time of the optimal configuration of the current subspace, adjust the current tuning dimension or direction, reset the learning rate, generate the next configuration, and return to step 3; Repeat until the set stop condition is reached and the optimal configuration of the current subspace is obtained.

4. The automatic online tuning system of the NCCL library according to claim 3, characterized in that: The division of the parameters in the configuration into parameters related to implementation and parameters related to computing resource allocation includes: The parameters in the configuration include<algorithm, protocol, transport, Nchannel, Nthread,chunk_size> ; Among them, algorithm, protocol and transport represent the algorithm, protocol and transport respectively, which are parameters related to implementation; Nchannel, Nthread and chunk_size represent the number of communication blocks, the number of threads and the size of communication blocks respectively, which are parameters related to computing resource allocation.

5. A processing device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 2.

6. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 2 is implemented.

Images