Ensemble communication library parameter automatic tuning method, storage medium, equipment and computer program product

Abstract

The invention discloses an aggregate communication library parameter automatic tuning method, a storage medium, equipment and a computer program product, and the method comprises the steps: in a network communication process, an aggregate communication library generates a large number of aggregate communication library parameter combinations, and a high-dimensional aggregate communication library parameter space is obtained; mapping the high-dimensional set communication library parameter space to a low-dimensional space, and obtaining set communication library parameter features of the low-dimensional space; storing the set communication library parameter characteristics of the low-dimensional space into a corresponding bucket memory according to a set step length, and reflecting the set communication library parameter characteristics of the low-dimensional space into a high-dimensional set communication library parameter space in the bucket memory; in the process of optimizing the parameters of the set communication library by using the Bayesian algorithm, the parameters of the set communication library in different bucket memories are sequentially selected for iterative training, and abnormal parameters of the set communication library are cut off through an online pruning strategy, so that optimization of the parameters of the set communication library is realized. According to the invention, the network communication efficiency and expansibility are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of collective communication library parameter optimization, and in particular to a collective communication library parameter automatic tuning method, storage medium, device and computer program product. Background Art

[0002] The Collective Communication Library (xCCL) provides a set of communication primitives, such as AllReduce, AllToall, and AllGather, that enable efficient communication in multi-GPU and multi-node machine learning workloads.

[0003] In order to further improve the communication quality, some research works focus on optimizing the communication topology, such as improving communication through tree radix. However, this method usually only targets a specific network topology and may not be able to adapt to diverse workload sizes and heterogeneous network environments. At present, there are also hard-coded methods or dynamic strategies to solve the topological heterogeneity problem. However, these methods may be effective in specific scenarios, but their versatility and scalability are limited, and they are difficult to adapt to large-scale and complex network environments. In order to solve this problem, better network balance is achieved through network topology and routing planning to improve the load balancing of network communication, but the flexibility and optimization potential of the collective communication library parameter configuration are usually ignored, thereby limiting communication efficiency and scalability. Summary of the invention

[0004] In view of the problems existing in the prior art, the present invention provides a method, storage medium, device and computer program product for automatically tuning the parameters of a collective communication library, which improves the efficiency and scalability of network communication by directly and automatically optimizing and adjusting the parameters of the collective communication library.

[0005] To achieve the above technical objectives, the present invention adopts the following technical solution: a method for automatically tuning collective communication library parameters, specifically comprising the following steps:

[0006] Step S1, during the network communication process, the collective communication library generates a large number of collective communication library parameter combinations to obtain a high-dimensional collective communication library parameter space;

[0007] Step S2, mapping the high-dimensional collective communication library parameter space to the low-dimensional space, and obtaining the collective communication library parameter features of the low-dimensional space;

[0008] Step S3: storing the parameter features of the set communication library in the low-dimensional space into the corresponding bucket memory according to the set step size, and inversely mapping the parameter features of the set communication library in the low-dimensional space into the high-dimensional set communication library parameter space in the bucket memory;

[0009] Step S4: In the process of optimizing the collective communication library parameters using the Bayesian algorithm, the collective communication library parameters in different bucket memories are selected in turn for iterative training, and abnormal collective communication library parameters are pruned through the online pruning strategy to optimize the collective communication library parameters.

[0010] Furthermore, step S2 includes the following sub-steps:

[0011] Step S2.1, all collective communication library parameters in the high-dimensional collective communication library parameter space form a sample matrix, where each row of the sample matrix represents a group of collective communication library parameters;

[0012] Step S2.2, calculate the covariance matrix of the sample matrix, and calculate the eigenvalues ​​and corresponding eigenvectors of the covariance matrix;

[0013] Step S2.3, select the eigenvector corresponding to the maximum eigenvalue as the transformation matrix, map the sample matrix to the low-dimensional space through the transformation matrix, and obtain the collective communication library parameter characteristics of the low-dimensional space.

[0014] Furthermore, in step S3, the specific process of storing the parameter features of the set communication library in the low-dimensional space into the corresponding bucket memory according to the set step length is:

[0015] i. Arrange the location of the bucket memory according to the set step size and determine the boundary of each bucket memory;

[0016] ii. Arrange the collective communication library parameter features in the low-dimensional space in ascending order, and store the collective communication library parameter features within the boundary range of the bucket memory into the corresponding bucket memory.

[0017] Furthermore, in step S4, the specific process of pruning abnormal collective communication library parameters through the online pruning strategy is: in the process of optimizing the collective communication library parameters using the Bayesian algorithm, error detection is performed on each optimized collective communication library parameter. If it is detected as an error three times in a row, the collective communication library parameter is pruned, and the pruned collective communication library parameter is saved in the pruning set.

[0018] Furthermore, the error detection refers to the detection of abnormalities in network communication caused by the use of the collective communication library parameters during an offline test of network communication.

[0019] Furthermore, the next time the Bayesian algorithm is used to optimize the collective communication library parameters, the pruning set will be prioritized for screening to avoid optimizing the collective communication library parameters in the pruning set.

[0020] Furthermore, the present invention also provides a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the method for automatically tuning the collective communication library parameters.

[0021] Furthermore, the present invention also provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for automatically tuning the collective communication library parameters is implemented.

[0022] Furthermore, the present invention also provides a computer program product, including a computer program, and when the computer program is executed by a processor, the method for automatically tuning the collective communication library parameters is implemented.

[0023] Compared with the prior art, the present invention has the following beneficial effects: the method for automatically tuning the parameters of the collective communication library of the present invention can reduce the search space while adjusting the complete combination of collective communication library parameters through low-dimensional space mapping and bucket memory storage, thereby improving the iterative optimization efficiency of the Bayesian algorithm and ensuring that the optimal solution of the collective communication library parameters is found under a limited number of iterations; in addition, in the process of optimizing the collective communication library parameters using the Bayesian algorithm, the present invention uses an online pruning strategy, which can dynamically identify and exclude those parameter configurations that may cause errors during the tuning process, thereby significantly improving the efficiency and reliability of automatic tuning of collective communication library parameters in complex network environments. Therefore, the automatic tuning method for collective communication library parameters of the present invention can greatly improve the efficiency and scalability of network communication. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 The present invention is a flowchart of the method for automatically tuning parameters of a collection communication library. DETAILED DESCRIPTION

[0025] The technical solution of the present invention is further explained below in conjunction with the accompanying drawings.

[0026] like Figure 1 The flowchart of the method for automatically tuning the parameters of the collective communication library of the present invention is as follows:

[0027] Step S1: During the network communication process, the collective communication library generates a large number of collective communication library parameter combinations to obtain a high-dimensional collective communication library parameter space.

[0028] Step S2, each dimension in the high-dimensional collective communication library parameter space corresponds to a collective communication library parameter, and each point in the high-dimensional collective communication library parameter space represents a set of collective communication library parameters. Although the Bayesian algorithm can model high-dimensional space, it will increase the number of iterations of collective communication library parameter optimization. At the same time, due to the repeated iterations caused by the existence of some similar collective communication library parameters, in order to reduce the number of iterations, the high-dimensional collective communication library parameter space is mapped to a low-dimensional space, and the collective communication library parameter characteristics of the low-dimensional space are obtained, so that when the Bayesian algorithm is used for optimization later, the search space can be narrowed while adjusting the complete collective communication library parameter combination, thereby improving the iterative optimization efficiency of the Bayesian algorithm and ensuring that the optimal solution for the collective communication library parameters is found under a limited number of iterations. Specifically including the following sub-steps:

[0029] Step S2.1, all collective communication library parameters in the high-dimensional collective communication library parameter space form a sample matrix, where each row of the sample matrix represents a group of collective communication library parameters;

[0030] Step S2.2, calculate the covariance matrix of the sample matrix, and calculate the eigenvalues ​​and corresponding eigenvectors of the covariance matrix;

[0031] Step S2.3, select the eigenvector corresponding to the maximum eigenvalue as the transformation matrix, map the sample matrix to the low-dimensional space through the transformation matrix, and obtain the collective communication library parameter characteristics of the low-dimensional space.

[0032] Step S3, store the collective communication library parameter features of the low-dimensional space into the corresponding bucket memory according to the set step size. Specifically, arrange the position of the bucket memory according to the set step size and determine the boundary of each bucket memory, arrange the collective communication library parameter features of the low-dimensional space in ascending order, and store the collective communication library parameter features within the boundary range of the bucket memory into the corresponding bucket memory. Similar collective communication library parameters can be stored in the same bucket memory, thereby reducing the number of optimization iterations of the Bayesian algorithm, and reversely map the collective communication library parameter features of the low-dimensional space to the high-dimensional collective communication library parameter space in the bucket memory.

[0033] Step S4: In the process of optimizing the collective communication library parameters using the Bayesian algorithm, the collective communication library parameters in different bucket memories are selected in turn for iterative training, and abnormal collective communication library parameters are pruned through an online pruning strategy. The online pruning strategy is used to solve the problem of communication failures that may be caused during the automatic optimization of the collective communication library parameters, thereby optimizing the collective communication library parameters and improving the tuning efficiency. The optimal collective communication library parameters optimized by the method of the present invention are used for network communication, which can greatly improve the network communication efficiency.

[0034] The specific process of pruning abnormal collective communication library parameters through the online pruning strategy in the present invention is: in the process of optimizing the collective communication library parameters using the Bayesian algorithm, error detection is performed on each optimized collective communication library parameter. If it is detected as an error three times in a row, the collective communication library parameter is pruned, and the pruned collective communication library parameter is saved in the pruning set. In the next process of optimizing the collective communication library parameters using the Bayesian algorithm, the pruning set will be preferentially screened to avoid optimizing the collective communication library parameters in the pruning set, thereby reducing the occurrence of the same error, wherein error detection refers to the detection of abnormal network communication caused by the use of the collective communication library parameters during the offline test of network communication. Through this online pruning strategy, the present invention can dynamically identify and exclude those parameter configurations that may cause errors during the tuning process, thereby significantly improving the efficiency and reliability of automatic tuning of collective communication library parameters in complex network environments, and is particularly suitable for application scenarios with large parameter spaces, complex network environments, and strict requirements on tuning time.

[0035] The method for automatically tuning the parameters of the collective communication library of the present invention is applied to the NVIDIA Collective Communication Library (NCCL), which is a widely used GPU communication basic component that provides implementations of AllReduce algorithms such as SHARP, ring, and tree. The method for automatically tuning the parameters of the collective communication library of the present invention can achieve parameter tuning in NCCL, and no modification is required to the code base in NCCL. In addition, the method of the present invention is also applicable to the automatic tuning of other collective communication library parameters. The method for automatically tuning the parameters of the collective communication library of the present invention provides ultra-large bandwidth, ultra-low latency, and ultra-high reliability collective communication library parameters for data-intensive and computationally intensive tasks, which can improve training speed and save costs.

[0036] In a technical solution of the present invention, a computer-readable storage medium is also provided, storing a computer program, wherein the computer program enables a computer to execute the method for automatically tuning the collective communication library parameters.

[0037] In a technical solution of the present invention, an electronic device is also provided, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for automatically tuning the collective communication library parameters is implemented.

[0038] In a technical solution of the present invention, a computer program product is also provided, including a computer program, characterized in that when the computer program is executed by a processor, the method for automatically tuning the collective communication library parameters is implemented.

[0039] In the embodiments disclosed in the present application, the computer storage medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. The computer storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the above. More specific examples of computer storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0040] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0041] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A method for automatically tuning collective communication library parameters, characterized in that: The specific steps include: Step S1, during the network communication process, the collective communication library generates a large number of collective communication library parameter combinations to obtain a high-dimensional collective communication library parameter space; Step S2, mapping the high-dimensional collective communication library parameter space to the low-dimensional space, and obtaining the collective communication library parameter features of the low-dimensional space; Step S3: storing the parameter features of the set communication library in the low-dimensional space into the corresponding bucket memory according to the set step size, and inversely mapping the parameter features of the set communication library in the low-dimensional space into the high-dimensional set communication library parameter space in the bucket memory; Step S4: In the process of optimizing the collective communication library parameters using the Bayesian algorithm, the collective communication library parameters in different bucket memories are selected in turn for iterative training, and abnormal collective communication library parameters are pruned through the online pruning strategy to optimize the collective communication library parameters.

2. A method for automatically tuning collective communication library parameters according to claim 1, characterized in that: Step S2 includes the following sub-steps: Step S2.1, all collective communication library parameters in the high-dimensional collective communication library parameter space form a sample matrix, where each row of the sample matrix represents a group of collective communication library parameters; Step S2.2, calculate the covariance matrix of the sample matrix, and calculate the eigenvalues ​​and corresponding eigenvectors of the covariance matrix; Step S2.3, select the eigenvector corresponding to the maximum eigenvalue as the transformation matrix, map the sample matrix to the low-dimensional space through the transformation matrix, and obtain the collective communication library parameter characteristics of the low-dimensional space.

3. A method for automatically tuning collective communication library parameters according to claim 2, characterized in that: The specific process of storing the parameter features of the set communication library in the low-dimensional space into the corresponding bucket memory according to the set step length in step S3 is: i. Arrange the location of the bucket memory according to the set step size and determine the boundary of each bucket memory; ii. Arrange the collective communication library parameter features in the low-dimensional space in ascending order, and store the collective communication library parameter features within the boundary range of the bucket memory into the corresponding bucket memory.

4. A method for automatically tuning collective communication library parameters according to claim 3, characterized in that: The specific process of pruning abnormal collective communication library parameters through the online pruning strategy in step S4 is: in the process of optimizing the collective communication library parameters using the Bayesian algorithm, error detection is performed on each optimized collective communication library parameter. If it is detected as an error three times in a row, the collective communication library parameter is pruned and the pruned collective communication library parameter is saved in the pruning set.

5. A method for automatically tuning collective communication library parameters according to claim 4, characterized in that: The error detection refers to the detection of abnormalities in network communication caused by the use of the collective communication library parameters during the offline test of network communication.

6. A method for automatically tuning collective communication library parameters according to claim 4, characterized in that: The next time the Bayesian algorithm is used to optimize the collective communication library parameters, the pruning set will be prioritized for screening to avoid optimizing the collective communication library parameters in the pruning set.

7. A computer-readable storage medium storing a computer program, characterized in that: The computer program enables a computer to execute the method for automatically tuning collective communication library parameters as described in any one of claims 1 to 6.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for automatically tuning the parameters of a collective communication library as described in any one of claims 1 to 6 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for automatically tuning the collective communication library parameters described in any one of claims 1 to 6 is implemented.

Images