A method, a storage medium, a device and a computer program product for automatically tuning a collective communication library parameter

Abstract

The application discloses a kind of collection communication library parameter automatic tuning method, storage medium, equipment and computer program product, comprising: in network communication process, collection communication library generates a large number of collection communication library parameter combinations, obtains high-dimensional collection communication library parameter space;High-dimensional collection communication library parameter space is mapped to low-dimensional space, and the collection communication library parameter characteristics of low-dimensional space are obtained;The collection communication library parameter characteristics of low-dimensional space are stored in the corresponding bucket memory according to the set step, and the collection communication library parameter characteristics of low-dimensional space are reflected in the bucket memory Mapping to high-dimensional collection communication library parameter space;In the process of optimizing collection communication library parameter using Bayesian algorithm, the collection communication library parameters in different bucket memories are sequentially selected for iterative training, and the abnormal collection communication library parameters are pruned through online pruning strategy, to realize the optimization of collection communication library parameter.The application improves network communication efficiency and expansibility.

Description

Technical Field

[0001] This invention relates to the field of collective communication library parameter optimization technology, specifically to a method, storage medium, device, and computer program product for automatic tuning of collective communication library parameters. Background Technology

[0002] The Collective Communication Library (xCCL) provides a set of communication primitives, such as AllReduce, AllToall, and AllGather, which enable efficient communication in multi-GPU and multi-node machine learning workloads.

[0003] To further improve communication quality, some research focuses on optimizing communication topologies, such as using tree radix to improve communication. However, this method is usually only applicable to specific network topologies and may not be suitable for diverse workload sizes and heterogeneous network environments. Currently, there are also methods that address topology heterogeneity through hard-coding or dynamic strategies. However, these methods may be effective in specific scenarios, but their versatility and scalability are limited, making it difficult to adapt to large-scale and complex network environments. To solve this problem, better network balancing is achieved through network topology and routing planning to improve network communication load balancing. However, this often ignores the flexibility and optimization potential of aggregated communication library parameter configuration, thus limiting communication efficiency and scalability. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention provides a method, storage medium, device, and computer program product for automatic optimization of aggregate communication library parameters. By directly and automatically optimizing and adjusting the aggregate communication library parameters, network communication efficiency and scalability are improved.

[0005] To achieve the above technical objectives, the present invention adopts the following technical solution: an automatic tuning method for aggregate communication library parameters, specifically including the following steps:

[0006] Step S1: During network communication, the collective communication library generates a large number of collective communication library parameter combinations, resulting in a high-dimensional collective communication library parameter space;

[0007] Step S2: Map the high-dimensional set communication library parameter space to a low-dimensional space and obtain the set communication library parameter features in the low-dimensional space;

[0008] Step S3: Store the low-dimensional space set communication library parameter features into the corresponding bucket memory according to the set step size, and in the bucket memory, back-map the low-dimensional space set communication library parameter features to the high-dimensional set communication library parameter space.

[0009] Step S4: In the process of optimizing the set communication library parameters using the Bayesian algorithm, set communication library parameters in different memory buckets are selected for iterative training. Abnormal set communication library parameters are pruned through an online pruning strategy to optimize the set communication library parameters.

[0010] Further, step S2 includes the following sub-steps:

[0011] Step S2.1: Construct a sample matrix from all the set communication library parameters in the high-dimensional set communication library parameter space, where each row of the sample matrix represents a set of set communication library parameters;

[0012] Step S2.2: Calculate the covariance matrix of the sample matrix, and calculate the eigenvalues ​​and corresponding eigenvectors of the covariance matrix;

[0013] Step S2.3: Select the eigenvector corresponding to the largest eigenvalue as the transformation matrix, and map the sample matrix to a low-dimensional space through the transformation matrix to obtain the set communication library parameter features in the low-dimensional space.

[0014] Furthermore, the specific process of storing the low-dimensional space set communication library parameter features into the corresponding bucket memory according to the set step size in step S3 is as follows:

[0015] i. Arrange the positions of the bucket memory according to the set step size and determine the boundaries of each bucket memory;

[0016] ii. Arrange the collection communication library parameter features in the low-dimensional space in ascending order, and store the collection communication library parameter features located within the boundary range of the bucket memory into the corresponding bucket memory.

[0017] Furthermore, the specific process of removing abnormal set communication library parameters through online pruning strategy in step S4 is as follows: during the optimization of set communication library parameters using the Bayesian algorithm, error detection is performed on each optimized set communication library parameter. If an error is detected three times in a row, the set communication library parameter is pruned, and the pruned set communication library parameter is saved to the pruning set.

[0018] Furthermore, the error detection refers to the detection of network communication anomalies caused by the use of the collected communication library parameters during offline network communication testing.

[0019] Furthermore, in the next process of using the Bayesian algorithm to optimize the parameters of the set communication library, the pruning set will be used as the primary filter to avoid optimizing the parameters of the set communication library.

[0020] Furthermore, the present invention also provides a computer-readable storage medium storing a computer program that causes a computer to execute the aforementioned automatic tuning method for collection communication library parameters.

[0021] Furthermore, the present invention also provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the aforementioned automatic tuning method for the parameters of the collection communication library.

[0022] Furthermore, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned automatic tuning method for collection communication library parameters.

[0023] Compared with existing technologies, the present invention has the following advantages: The automatic tuning method for set communication library parameters of the present invention, through low-dimensional space mapping and bucket memory storage, can reduce the search space while adjusting the complete set communication library parameter combination, thereby improving the iterative optimization efficiency of the Bayesian algorithm and ensuring that the optimal solution for the set communication library parameters is found within a limited number of iterations. Furthermore, the present invention employs an online pruning strategy during the optimization of set communication library parameters using the Bayesian algorithm, which can dynamically identify and eliminate parameter configurations that may lead to errors during the tuning process, thus significantly improving the efficiency and reliability of automatic tuning of set communication library parameters in complex network environments. Therefore, the automatic tuning method for set communication library parameters of the present invention can greatly improve network communication efficiency and scalability. Attached Figure Description

[0024] Figure 1 This is a flowchart of the automatic tuning method for the collection communication library parameters of the present invention. Detailed Implementation

[0025] The technical solution of the present invention will be further explained and described below with reference to the accompanying drawings.

[0026] like Figure 1 This is a flowchart of the automatic optimization method for aggregate communication library parameters according to the present invention. The automatic optimization method for aggregate communication library parameters specifically includes the following steps:

[0027] Step S1: During network communication, the collective communication library generates a large number of collective communication library parameter combinations, resulting in a high-dimensional collective communication library parameter space.

[0028] Step S2: In the high-dimensional set communication library parameter space, each dimension corresponds to a set communication library parameter, and each point in the high-dimensional set communication library parameter space represents a set of set communication library parameters. Although the Bayesian algorithm can model a high-dimensional space, it increases the number of iterations for optimizing the set communication library parameters. Furthermore, due to repeated iterations caused by some similar set communication library parameters, to reduce the number of iterations, the high-dimensional set communication library parameter space is mapped to a low-dimensional space, and the features of the set communication library parameters in the low-dimensional space are obtained. This allows the Bayesian algorithm to reduce the search space while adjusting the complete set communication library parameter combination during subsequent optimization, thereby improving the iterative optimization efficiency of the Bayesian algorithm and ensuring that the optimal solution for the set communication library parameters is found within a limited number of iterations. Specifically, this includes the following sub-steps:

[0029] Step S2.1: Construct a sample matrix from all the set communication library parameters in the high-dimensional set communication library parameter space, where each row of the sample matrix represents a set of set communication library parameters;

[0030] Step S2.2: Calculate the covariance matrix of the sample matrix, and calculate the eigenvalues ​​and corresponding eigenvectors of the covariance matrix;

[0031] Step S2.3: Select the eigenvector corresponding to the largest eigenvalue as the transformation matrix, and map the sample matrix to a low-dimensional space through the transformation matrix to obtain the set communication library parameter features in the low-dimensional space.

[0032] Step S3: Store the low-dimensional space set communication library parameter features into the corresponding bucket memory according to the set step size. Specifically, arrange the positions of the bucket memory according to the set step size and determine the boundary of each bucket memory. Arrange the low-dimensional space set communication library parameter features in ascending order. Store the set communication library parameter features within the boundary range of the bucket memory into the corresponding bucket memory. Similar set communication library parameters can be stored in the same bucket memory, thereby reducing the number of optimization iterations of the Bayesian algorithm. In the bucket memory, the low-dimensional space set communication library parameter features are back-mapped to the high-dimensional set communication library parameter space.

[0033] Step S4: In the process of optimizing the set communication library parameters using the Bayesian algorithm, set communication library parameters in different memory buckets are selected sequentially for iterative training. Abnormal set communication library parameters are pruned using an online pruning strategy. The online pruning strategy solves the problem of communication failure that may occur during the automatic optimization of set communication library parameters, thereby optimizing the set communication library parameters and improving the efficiency of tuning. The optimal set communication library parameters obtained by the method of this invention can be used for network communication, which can greatly improve the efficiency of network communication.

[0034] The specific process of pruning abnormal aggregate communication library parameters using an online pruning strategy in this invention is as follows: During the optimization of aggregate communication library parameters using the Bayesian algorithm, error detection is performed on each optimized aggregate communication library parameter. If an error is detected three times consecutively, the aggregate communication library parameter is pruned and saved to a pruning set. In the next optimization process using the Bayesian algorithm, the pruning set will be prioritized for filtering, avoiding optimization of aggregate communication library parameters in the pruning set and reducing the occurrence of the same error. Here, error detection refers to the detection of network communication anomalies caused by the use of the aggregate communication library parameters during offline network communication testing. Through this online pruning strategy, this invention can dynamically identify and eliminate parameter configurations that may lead to errors during the tuning process, thereby significantly improving the efficiency and reliability of automatic tuning of aggregate communication library parameters in complex network environments. It is particularly suitable for application scenarios with large parameter spaces, complex network environments, and strict requirements for tuning time.

[0035] This invention's automatic parameter tuning method for the NVIDIA Collective Communication Library (NCCL) is applied to the NCCL, a widely used GPU communication infrastructure that provides implementations of AllReduce algorithms based on SHARP, rings, and trees. This invention's automatic parameter tuning method enables parameter tuning within the NCCL without requiring any modifications to the NCCL codebase. Furthermore, this method is also applicable to the automatic tuning of parameters in other NCCL libraries. This invention's automatic parameter tuning method for NCCL libraries provides ultra-high bandwidth, ultra-low latency, and ultra-high reliability NCCL parameters for data-intensive and computationally intensive tasks, thereby improving training speed and saving costs.

[0036] In one technical solution of the present invention, a computer-readable storage medium is also provided, storing a computer program that causes a computer to execute the automatic tuning method for the set communication library parameters.

[0037] In one technical solution of the present invention, an electronic device is also provided, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the automatic tuning method for the parameters of the collection communication library.

[0038] In one technical solution of the present invention, a computer program product is also provided, comprising a computer program, characterized in that, when the computer program is executed by a processor, it implements the automatic tuning method for the parameters of the collection communication library.

[0039] In the embodiments disclosed in this application, a computer storage medium may be a tangible medium that may contain or store programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of computer storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0040] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0041] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A method for automatically optimizing parameters of a collection of communication libraries, characterized in that, Specifically, the steps include the following: Step S1: During network communication, the collective communication library generates a large number of collective communication library parameter combinations, resulting in a high-dimensional collective communication library parameter space; Step S2: Map the high-dimensional set communication library parameter space to a low-dimensional space and obtain the set communication library parameter features in the low-dimensional space; Step S3: Store the low-dimensional space set communication library parameter features into the corresponding bucket memory according to the set step size, and in the bucket memory, back-map the low-dimensional space set communication library parameter features to the high-dimensional set communication library parameter space. Step S4: In the process of optimizing the set communication library parameters using the Bayesian algorithm, set communication library parameters in different memory buckets are selected for iterative training. Abnormal set communication library parameters are pruned through an online pruning strategy to optimize the set communication library parameters.

2. The method for automatic tuning of collection communication library parameters according to claim 1, characterized in that, Step S2 includes the following sub-steps: Step S2.1: Construct a sample matrix from all the set communication library parameters in the high-dimensional set communication library parameter space, where each row of the sample matrix represents a set of set communication library parameters; Step S2.2: Calculate the covariance matrix of the sample matrix, and calculate the eigenvalues ​​and corresponding eigenvectors of the covariance matrix; Step S2.3: Select the eigenvector corresponding to the largest eigenvalue as the transformation matrix, and map the sample matrix to a low-dimensional space through the transformation matrix to obtain the set communication library parameter features in the low-dimensional space.

3. The method for automatic tuning of collection communication library parameters according to claim 2, characterized in that, The specific process in step S3 of storing the low-dimensional space set communication library parameter features into the corresponding bucket memory according to the set step size is as follows: i. Arrange the positions of the bucket memory according to the set step size and determine the boundaries of each bucket memory; ii. Arrange the collection communication library parameter features in the low-dimensional space in ascending order, and store the collection communication library parameter features located within the boundary range of the bucket memory into the corresponding bucket memory.

4. The method for automatic tuning of collection communication library parameters according to claim 3, characterized in that, The specific process of pruning abnormal set communication library parameters in step S4 using an online pruning strategy is as follows: During the optimization of set communication library parameters using the Bayesian algorithm, error detection is performed on each optimized set communication library parameter. If an error is detected three times in a row, the set communication library parameter is pruned, and the pruned set communication library parameter is saved to the pruning set.

5. The method for automatic tuning of collection communication library parameters according to claim 4, characterized in that, The error detection refers to the detection of network communication anomalies caused by the use of the set communication library parameters during offline network communication testing.

6. The method for automatic tuning of parameters of a collection communication library according to claim 4, characterized in that, In the next process of using the Bayesian algorithm to optimize the parameters of the set communication library, we will prioritize filtering through the pruning set to avoid optimizing the parameters of the set communication library.

7. A computer-readable storage medium storing a computer program, characterized in that, The computer program causes the computer to execute the automatic tuning method for the collection communication library parameters as described in any one of claims 1-6.

8. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the automatic tuning method for the set communication library parameters as described in any one of claims 1-6.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the automatic tuning method for the collection communication library parameters as described in any one of claims 1-6.

Images