An NCCL communication optimization method and device based on performance change curve analysis

Abstract

The application provides an NCCL communication optimization method and device based on performance change curve analysis, and belongs to the technical field of parameter communication optimization. The method provided by the application comprises the following steps: collecting communication performance data of each node in a cluster under different message sizes; generating performance change curves of each node according to the communication performance data, determining a platform period or an inflection point at which the bus bandwidth decreases with the increase of the message size, and determining an abnormal interval segment according to the platform period or the inflection point; determining a to-be-optimized message size data interval corresponding to the abnormal interval segment, generating an optimization configuration file containing optimized protocol parameters for the to-be-optimized message size data interval, generating a shared library file according to the optimization configuration file, and deploying the shared library file on a node with performance problems in the cluster to perform communication optimization. The NCCL communication optimization method and device based on performance change curve analysis provided by the application are used to solve the problem of difficult repair of performance degradation after tuning.

Description

Technical Field

[0001] This application relates to the field of parameter communication optimization technology, and in particular to an NCCL communication optimization method and apparatus based on performance change curve analysis. Background Technology

[0002] In distributed training, GPU cluster nodes need to frequently synchronize and communicate data. Communication efficiency directly determines the execution speed of training tasks and the utilization of cluster resources. NCCL (NVIDIA Collective Communications Library), as an optimization library for GPU cluster collective communication launched by NVIDIA, has become a key component for improving distributed training performance by optimizing the underlying communication path and parallel strategies. Therefore, the quality of NCCL communication performance has a significant impact on the overall efficiency of distributed training.

[0003] Currently, NCCL parameter tuning mainly relies on manual methods. Operations or R&D personnel need to manually log in to each node of the cluster, modify the BIOS configuration, system parameters, network parameters and NCCL environment variables, manually perform communication performance tests, and gradually adjust the parameters based on the test results. In some scenarios, simple scripts are also used to assist in batch configuration. However, the overall tuning process still requires a lot of manual intervention and judgment. After the tuning is completed, it is usually necessary to manually confirm the configuration recovery status or directly keep the tuning parameters and run.

[0004] However, the aforementioned existing technologies still have the following drawbacks: First, manual tuning relies on a large amount of human intervention, making the tuning process cumbersome and inefficient. Furthermore, the difficulty in synchronizing configurations across multiple nodes makes it hard to guarantee the consistency of parameter configurations across all nodes in the cluster. Second, existing automated tuning solutions mostly focus on searching and optimizing parameter configurations, lacking a systematic verification and in-depth repair mechanism for the overall communication performance of the cluster after tuning. Third, NCCL contains protocol variants and communication algorithms. While its default cost model and dynamic scheduler can make reasonable decisions in most scenarios, in specific hardware environments or message size ranges, inappropriate algorithm or protocol selection may still occur, resulting in some performance bottlenecks hidden in specific data ranges not being effectively identified and resolved. Summary of the Invention

[0005] In view of this, this application provides an NCCL communication optimization method and apparatus based on performance change curve analysis to solve the problem of difficult-to-repair performance degradation after optimization.

[0006] Specifically, this application is implemented through the following technical solution:

[0007] The first aspect of this application provides an NCCL communication optimization method based on performance change curve analysis, the method comprising:

[0008] After the training cluster completes the parameter configuration, run the cluster-level communication test to collect communication performance data of each node in the cluster under different message sizes;

[0009] Based on the communication performance data, generate performance change curves for each node, determine the plateau period or inflection point where bus bandwidth decreases as message size increases, and determine abnormal intervals based on the plateau period or inflection point.

[0010] The abnormal interval is determined, and an optimization configuration file containing optimized protocol parameters is generated for the abnormal interval. A shared library file is generated based on the optimization configuration file, and the shared library file is deployed to the nodes in the cluster with performance problems to optimize communication.

[0011] A second aspect of this application provides an NCCL communication optimization device based on performance change curve analysis, the device comprising an acquisition module, a processing module, and an optimization module;

[0012] The acquisition module is used to run cluster-level communication tests after the training cluster has completed parameter configuration, and to collect communication performance data of each node in the cluster under different message sizes.

[0013] The processing module is used to generate performance change curves for each node based on the communication performance data, determine the plateau period or inflection point where the bus bandwidth decreases as the message size increases, and determine the abnormal interval based on the plateau period or inflection point.

[0014] The optimization module is used to determine the message size data range to be optimized corresponding to the abnormal interval, generate an optimization configuration file containing optimized protocol parameters for the message size data range to be optimized, generate a shared library file according to the optimization configuration file, and deploy the shared library file to the nodes in the cluster with performance problems for communication optimization.

[0015] The NCCL communication optimization method and apparatus based on performance change curve analysis provided in this application, on the basis of parameter tuning, accurately locates communication algorithm / protocol bottlenecks hidden in specific message intervals through performance change curve morphology analysis, and dynamically repairs them through shared libraries, realizing a complete tuning closed loop from parameter-level optimization to algorithm and protocol-level deep optimization. Specifically, by obtaining real performance data of each node under different message sizes through cluster-level communication testing, the capability has been leaped from single-point configuration verification to full-cluster performance scanning. By performing morphological analysis on the performance change curve, the slope and trend of the curve are used as the basis for automatically identifying plateaus and inflection points where bus bandwidth stagnates or declines with the increase of message size. This transforms the traditionally difficult-to-locate hidden performance degradation problems that rely on manual experience into a quantifiable and automatically detectable curve feature identification process, accurately locating the message size data range to be optimized. On this basis, a configuration file containing optimized protocol parameters and communication algorithms is generated for this specific range and compiled into a shared library file for dynamic deployment to the problem node. This allows the optimization measures to be injected into the NCCL runtime environment in a dynamic loading manner without modifying the source code or recompiling the entire communication library. On the basis of the established unified parameter baseline for the entire cluster, the algorithm protocol-level performance bottlenecks in specific message ranges are further eliminated. Finally, a complete technical closed loop from parameter configuration, performance verification, anomaly location to targeted repair is formed, which significantly improves the overall communication efficiency and stability of the distributed training cluster under real business message distribution. Attached Figure Description

[0016] Figure 1 A flowchart of an embodiment of the NCCL communication optimization method based on performance change curve analysis provided in this application;

[0017] Figure 2 A flowchart of Embodiment 2 of the NCCL communication optimization method based on performance change curve analysis provided in this application;

[0018] Figure 3 This is a schematic diagram of the third embodiment of the NCCL communication optimization device based on performance change curve analysis provided in this application. Detailed Implementation

[0019] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0020] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used herein are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0021] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0022] The following specific embodiments are given to illustrate the technical solution of this application in detail.

[0023] Figure 1 This is a flowchart of an embodiment of the NCCL communication optimization method based on performance variation curve analysis provided in this application. Please refer to... Figure 1 The method provided in this embodiment may include:

[0024] S101. After the parameter configuration of the training cluster is completed, run the cluster-level communication test and collect the communication performance data of each node in the cluster under different message sizes.

[0025] Specifically, in a cluster state where parameters have been optimized, select a node as the test initiation node; run a cluster-level communication test command to initiate a collective communication operation covering all nodes in the cluster; and collect communication performance index data for each node in the cluster under different message sizes.

[0026] Furthermore, after parameter tuning at each level, all nodes in the cluster have been configured with their respective optimal parameter combinations. Based on this unified configuration, a full evaluation of the end-to-end communication performance of the entire cluster is conducted through cluster-level communication testing. The test command is initiated by one node in the cluster, calling the NCCL collective communication library to execute collective communication operations covering all participating nodes. The communication modes cover commonly used collective communication primitives in distributed training, such as AllReduce, AllGather, and Broadcast. During the test, the message size gradually increases from a minimum to a maximum, covering the data volume range of typical communication scenarios in distributed training, such as parameter synchronization and gradient aggregation. The communication bandwidth and transmission latency of each node under each message size are recorded in real time and summarized into a complete performance data set for each node in the cluster. Simultaneously, the NCCL performance monitoring tool can be used to collect more granular performance decomposition indicators from each node, including the time distribution of different communication stages and link utilization. This performance data provides comprehensive data support for subsequent analysis of cluster communication bottlenecks and the identification of performance-prone nodes.

[0027] S102. Generate performance change curves for each node based on the communication performance data, determine the plateau period or inflection point where the bus bandwidth decreases as the message size increases, and determine the abnormal interval based on the plateau period or the inflection point.

[0028] Specifically, the system remotely acquires performance data from each node; organizes and formats the performance data; plots a performance change curve with message size as the x-axis and bus bandwidth as the y-axis; performs segmented slope detection on the curve; identifies plateaus where bus bandwidth growth stagnates as message size increases; identifies inflection points where bus bandwidth decreases as message size increases; and marks the message size range corresponding to the plateau or inflection point as an abnormal interval.

[0029] Furthermore, in an ideal linearly expanding communication scenario, the bus bandwidth should increase synchronously with the message size until it reaches the upper limit of the physical bandwidth of the link. However, in actual clusters, due to factors such as the NCCL's internal communication algorithm switching strategy, protocol adaptation capabilities, or topology-aware mechanisms, the performance change curve may experience a plateau in growth within a certain message size range, or even a reverse bandwidth decline, i.e., an inflection point. Such abnormal characteristics indicate that the NCCL has selected a non-optimal communication algorithm or protocol parameters within that data range. To identify these characteristics, S-Curve analysis or similar statistical curve fitting analysis is performed on the collected performance data of each node to extract the trend curve of bus bandwidth changing with message size. The slope of the curve is detected. When the bandwidth growth rate of multiple consecutive message size data points is lower than a preset threshold, the interval is determined to be a plateau period. When the absolute value of bandwidth decreases beyond a preset threshold, the point is determined to be an inflection point. Based on the message size range corresponding to the plateau period or inflection point, an abnormal interval is clearly defined. For the abnormal interval, the optimization tools provided by NVIDIA are called to optimize and adjust the communication algorithm selection strategy and protocol parameters within the interval according to the actual cluster topology and hardware configuration. An optimization configuration file containing the optimized algorithm and protocol configuration is generated. The optimization configuration file describes the optimization information such as the type of communication algorithm and protocol parameters that should be used within a specific message size interval.

[0030] Furthermore, the steps to determine the plateau or inflection point where bus bandwidth decreases with increasing message size include:

[0031] (1) Perform S-curve data analysis on the communication performance data, and plot the performance change curve with message size as the horizontal axis and bus bandwidth as the vertical axis;

[0032] Specifically, the acquired performance data is organized and formatted; the S-curve data analysis method is used, with message size as the independent variable and bus bandwidth as the dependent variable, to plot the performance change curve of each node or the cluster as a whole.

[0033] Furthermore, after completing the cluster-level communication test, the performance data of each node is distributed and stored in the local logs or memory buffers of each node. Through remote data acquisition channels, such as SSH remote query commands or calling the monitoring agent interface deployed on the node, the bus bandwidth and transmission latency data of each node under different message sizes are obtained one by one. The obtained raw data is parsed, cleaned, and formatted, then sorted and organized in ascending order of message size to form a structured performance data table. The S-curve data analysis method is used to fit and visualize the data. The S-curve is a growth curve with typical three-segment characteristics, which can effectively depict the three stages of bus bandwidth growth with message size: slow initial growth, rapid growth, and approaching saturation. With message size as the horizontal axis and bus bandwidth as the vertical axis, each data point is plotted on a coordinate system, and a smooth performance change curve is drawn through curve fitting. The performance change curve intuitively shows the dynamic change law of cluster communication bandwidth with message size, providing a visual analytical basis for subsequent identification of abnormal features in the curve.

[0034] (2) Detect whether there is a segment in the performance change curve where the bus bandwidth increases stagnant or decreases as the message size increases;

[0035] Specifically, along the horizontal axis of the performance change curve, the slope or bandwidth change rate of the curve is calculated segment by segment; the slope or change rate of each segment is compared with the preset normal growth threshold; if there are multiple consecutive data points with slopes lower than the normal growth threshold or negative values, it is determined that the interval segment has abnormal characteristics of growth stagnation or decline.

[0036] Furthermore, in a normal linearly expanding communication scenario, the bus bandwidth performance curve shows a monotonically increasing trend with the increase of message size, until it reaches the upper limit of the physical bandwidth of the link and then tends to flatten and saturate. This flattening and saturation stage is a normal physical limitation phenomenon rather than an anomaly. However, if growth stagnates within the intermediate message size range before reaching the physical bandwidth limit, or if the absolute bandwidth value decreases instead of increasing, it indicates that the NCCL's communication algorithm selection or protocol parameter configuration within this range is unreasonable. To detect such anomalies, the performance change curve is segmented and scanned along the horizontal axis. The ratio of the bandwidth increment to the message size increment between adjacent data points, i.e., the interval slope, is calculated. When the absolute value of the slope for a consecutive preset number of data points is lower than a preset stagnation threshold, it indicates that the bandwidth has almost no growth within this range, and is determined to be a stagnation interval. When the slope of multiple consecutive data points is negative and the bandwidth decrease exceeds a preset decrease threshold, it indicates that the bandwidth has significantly declined, and is determined to be a decrease interval. During the detection process, saturation areas close to the physical bandwidth limit of the link must be excluded to avoid misjudging normal physical bandwidth limits as abnormal stagnation. The detection result is output as whether there is an abnormal interval that meets the conditions. If not, it indicates that the current cluster communication performance is good and no further optimization is required.

[0037] (3) If the interval exists, the interval is marked as the plateau period or inflection point.

[0038] Specifically, intervals determined to be stagnant growth are marked as plateau periods; intervals determined to be declining are marked as inflection points; the start and end ranges of message size and the node identifiers involved are recorded for each plateau period or inflection point.

[0039] Furthermore, a plateau period refers to a horizontal segment on the performance change curve where bandwidth stagnates as message size increases. Within this segment, increasing message size does not result in a corresponding bandwidth increase, reflecting that NCCL has failed to effectively utilize available link bandwidth within this message size range. An inflection point refers to a turning point on the performance change curve where bandwidth decreases significantly. Within this segment, bandwidth decreases instead of increasing message size, reflecting that NCCL may have engaged in unreasonable algorithm switching or protocol degradation within this message size range. For each detected abnormal segment, it is classified according to its curve shape characteristics. Those exhibiting stagnation characteristics are marked as plateau periods, and those exhibiting decline characteristics are marked as inflection points. During marking, the start and end boundaries of the message size for the abnormal segment, as well as the nodes where the anomaly occurs, are recorded simultaneously. This marking information constitutes the basis for subsequent problem localization in algorithm and protocol optimization and will be passed to the optimization tool to specify the data range to be optimized and the target nodes for optimization. By clearly classifying abnormal segments into plateau periods or inflection points, it is helpful to adopt differentiated optimization strategies for different anomaly types, improving the targeting and effectiveness of optimization.

[0040] S103. Determine the message size data range to be optimized corresponding to the abnormal interval, generate an optimization configuration file containing optimized protocol parameters for the message size data range to be optimized, generate a shared library file according to the optimization configuration file, and deploy the shared library file to the nodes with performance problems in the cluster for communication optimization.

[0041] Specifically, the process involves: obtaining the start and end values ​​of the message size within the abnormal interval as the data interval to be optimized; determining the protocol type and communication algorithm currently used by NCCL within this data interval; selecting the optimized target protocol type and target communication algorithm based on the anomaly type; generating an optimization configuration file containing the mapping relationship between the message size interval, target protocol type, and target communication algorithm; compiling the optimization configuration file into a shared library file; identifying nodes in the cluster with performance issues; distributing and deploying the shared library file to the problematic nodes; and verifying the effectiveness of the optimized configuration after deployment.

[0042] Furthermore, the generated optimized configuration files are compiled into shared library files; nodes in the cluster with plateau or inflection point performance issues are identified; the shared library files are deployed to the problematic nodes; all optimized configuration files deployed in the cluster are archived and recorded; after deployment, cluster-level communication tests are run again to verify overall performance; and the final optimized test results are recorded and stored.

[0043] Furthermore, the optimized configuration file, being a declarative configuration description, needs to be further transformed into an execution format that the NCCL runtime can directly call. This optimized configuration file is then compiled into a shared library file using a compilation tool. This shared library file contains the optimized communication algorithm implementation and protocol control logic for a specific message size range. After the shared library file is generated, based on the identified performance data of each node, nodes with plateaus or inflection points are located. The shared library file is then distributed and deployed to the NCCL library loading path of these nodes, ensuring that NCCL can automatically load the shared library file and apply the optimized algorithms and protocol strategies in subsequent communication operations. For nodes where no performance issues are detected, there is no need to deploy the shared library file to avoid unnecessary runtime overhead. All generated optimized configuration files and their correspondence with problematic nodes are uniformly recorded and archived, forming a complete optimized configuration archive. After all deployment work is completed, the same cluster-level communication test command is run again to perform regression testing on the communication performance of the entire cluster, collecting communication bandwidth and latency data for each node after optimization. The optimized test results are compared with the unoptimized test results to verify the effectiveness of the optimization measures. All test results are recorded and stored as part of the tuning report.

[0044] Furthermore, the implementation steps for generating an optimization configuration file containing optimized protocol parameters for the data range of message size to be optimized include:

[0045] (1) Obtain the starting message size and ending message size of the data range of the message size to be optimized;

[0046] Specifically, the starting and ending message sizes of the data range to be optimized are obtained. This step can be further broken down into: reading the starting and ending message sizes from the marked records of the abnormal intervals; verifying the validity of the starting and ending message sizes; and determining a unified boundary for the data range to be optimized based on the set of abnormal nodes.

[0047] Furthermore, the start and end values ​​of the message size corresponding to each detected abnormal interval are recorded in a structured data table. This data table is also associated with an anomaly type identifier and an anomaly node identifier. The start and end message size fields are directly extracted from this data table. For the validity verification of the start and end message sizes, it is necessary to check whether the start value is less than the end value, whether both are positive numbers, and whether the start value is within a reasonable message size range. If the verification fails, it indicates that the detection result is abnormal or the data is corrupted. At this time, the abnormal interval is abandoned and a re-detection or error message is triggered. For the unified processing of multiple abnormal node intervals, when the detected abnormal intervals involve multiple nodes, the boundaries of the abnormal intervals of different nodes may have slight differences. For example, the inflection point of node A appears in the message size range of 256KB to 512KB, and the inflection point of node B appears in the range of 256KB to 1MB. In order to ensure that the subsequently generated optimization configuration file can be uniformly effective on all abnormal nodes, the minimum value of the start value and the maximum value of the end value of the abnormal interval of each node should be used as the lower and upper bounds of the unified message size data range to be optimized, respectively, to ensure that the range covers the performance degradation range of all abnormal nodes. If the boundaries of the abnormal intervals of each node differ too much, then the nodes are grouped according to their types, and differentiated optimization configuration files are generated for different node types to finely match the hardware characteristics of each type of node.

[0048] (2) Based on the size of the start message and the size of the end message, determine the target protocol type from the various communication protocols supported by the NCCL communication library, and determine the target communication algorithm from the various collective communication algorithms supported by the NCCL communication library;

[0049] Specifically, determine the current protocol type and current communication algorithm corresponding to the start and end message sizes in the NCCL communication library; select the optimization strategy direction based on the exception type; determine the target protocol type from the various communication protocols supported by NCCL; determine the target communication algorithm from the various collective communication algorithms supported by NCCL; and verify the effectiveness of the combination of the target protocol type and the target communication algorithm.

[0050] Furthermore, to determine the current protocol type and communication algorithm, the communication protocol type and communication algorithm type selected by the default cost model when NCCL performs collective communication within this message size range can be obtained by enabling NCCL's debug log mode or calling NCCL's performance analysis interface. NCCL internally maintains a protocol selection table and an algorithm selection table. The protocol types include three variants: Simple protocol, LL protocol, and LL128 protocol, which correspond to data transmission strategies under different message sizes. The Simple protocol is suitable for large message transmission, the LL protocol is suitable for low-latency transmission of small messages, and the LL128 protocol is an optimized variant of the LL protocol, which further improves the efficiency of small message transmission through pipelined parallelism and data packing technology. The algorithm types include the Ring algorithm, the Tree algorithm, and the NVLink algorithm. The Ring algorithm is suitable for cross-node large message transmission, the Tree algorithm is suitable for small message or intra-node communication, and the NVLink algorithm is dedicated to the NVLink high-speed interconnect channel between GPUs within the same physical node. When choosing an optimization strategy, the target protocol selection should be determined based on the anomaly type. If the anomaly type is a plateau, meaning the bandwidth stagnates as the message size increases, it indicates that the current protocol has reached its performance limit within this range and cannot expand with the increase in message size. In this case, a protocol type with a higher bandwidth limit should be selected. For example, if a plateau occurs when using the LL protocol in the 256KB to 1MB range, the target protocol type should be the LL128 protocol. If the anomaly type is an inflection point, meaning the bandwidth decreases, it indicates that the current protocol may have triggered a degradation or switching behavior within this range. In this case, a protocol type with a higher performance level than the current protocol should be selected to reverse the rapid degradation trend. The determination of the target communication algorithm needs to consider the message size range and node topology corresponding to the current anomaly interval. If the anomaly occurs in a medium message size interval and involves a large number of nodes, the current algorithm should be switched to one with better scalability, such as switching from the Tree algorithm to the Ring algorithm. If the anomaly occurs in a small message interval and involves a small number of nodes, the current algorithm should be switched to one with lower latency, such as switching from the Ring algorithm to the Tree algorithm. If the performance curve shows that communication anomalies occur within NVLink-reachable nodes, NVLink algorithms should be prioritized as alternative targets. Verification of the effectiveness of the combination of target protocol type and target communication algorithm is crucial. Not all combinations of protocols and algorithms are valid or effective. The NCCL protocol-algorithm compatibility matrix needs to be consulted to confirm a valid implementation path between the selected target protocol and target algorithm. If the combination is invalid, a fallback to the suboptimal choice should be made. It is also necessary to verify whether the selected combination is usable in the node's hardware environment. For example, the LL128 protocol requires the GPU architecture to have corresponding instruction set support; if the node's GPU model does not support the LL128 protocol, this protocol type cannot be selected.After determining the target protocol type and target communication algorithm, it is necessary to compare them with the current configuration. If the target combination is consistent with the combination automatically selected by the current NCCL in this range, it indicates that the abnormality in this range is not actually caused by improper protocol algorithm selection. At this time, record the situation and trigger alternative optimization strategies, such as adjusting other parameter dimensions such as buffer size or number of channels.

[0051] (3) Write the target protocol type and the target communication algorithm as optimized protocol parameters into the optimization configuration file.

[0052] Specifically, determine the format and encoding specifications of the optimized configuration file; encode the start message size, end message size, target protocol type, and target communication algorithm into structured configuration data according to the specifications; add metadata information to the configuration file; and persist the encoded configuration data to the file system.

[0053] Furthermore, the configuration file adopts the standard parsing format agreed upon by the NCCL protocol-algorithm overriding plugin. It can be either JSON or YAML format. JSON format is easy for machines to parse and generate, while YAML format is easy for humans to read and debug. Choose one according to the actual deployment scenario. Both formats support nested structures and key-value pair mappings, which can fully express the mapping relationship between ranges and protocol algorithms. For the encoding of structured configuration data, configuration entries are organized according to the following mapping relationship: the starting message size is the lower bound key, the ending message size is the upper bound key, the target protocol type string identifier is the protocol key, and the target communication algorithm string identifier is the algorithm key. The protocol type string identifier can be standardized to the standard names used internally by NCCL, such as "SIMPLE", "LL", "LL128", and the algorithm type string identifier can be standardized to standard names such as "RING", "TREE", "NVLS", etc. A single optimized entry after encoding fully expresses the following semantics: when the size of the communication message to be transmitted by NCCL is within the closed interval between the starting message size and the ending message size, the NCCL communication library is forced to use the specified target protocol type and target communication algorithm to perform collective communication operations, overriding and replacing the algorithm and protocol scheduling decisions of the NCCL default cost model within this interval. For the addition of metadata information, a version number field is added to the configuration file to record the configuration format version to support subsequent expansion, a generation timestamp field is added to record the configuration generation time for easy traceability, and an exception type field is added to record the exception category corresponding to the optimized entry to assist subsequent analysis and debugging. For the persistence of configuration data, the structured data after encoding is written to the configuration file in UTF-8 encoding. The file name adopts a standard naming method that includes node type identifier, hierarchy identifier and generation timestamp for easy management and identification. After writing, the file is checked for integrity. The correctness of its format and the integrity of its content are verified by rereading and parsing the configuration file. After the verification is passed, the optimized configuration file can enter the subsequent stage of compiling and generating shared library files.

[0054] By accurately extracting the starting and ending message sizes from anomaly detection results, the optimization scope is precisely narrowed from fuzzy intervals to quantifiable and programmable numerical boundaries. This provides a precise domain anchor for all subsequent optimization decisions, ensuring that optimization measures only apply to specific message size intervals where performance deteriorates without interfering with normal communication outside these intervals. Based on this, according to the interval boundaries and anomaly types, target protocol types and algorithms are selectively determined from various communication protocols and collective communication algorithms supported by the NCCL communication library. Detected plateaus or inflection points are directly mapped to explicit optimization decision outputs, making protocol and algorithm selection no longer a general optimal choice but a tailored matching decision based on the characteristics of the current anomaly interval. This solves the problem of inaccurate scheduling in the NCCL default cost model within this interval. The interval boundaries and selected target protocols and algorithms are written into the optimization configuration file in a structured format. The decision results are solidified into a storable, transmissible, and compilable standardized configuration file entity, providing a unified configuration input source for subsequent batch compilation and cluster deployment. This allows the optimization strategy to seamlessly transition from the decision-making stage to the execution stage. It achieves an automated closed loop from performance anomaly detection to the generation of targeted optimization solutions. The entire process requires no manual intervention or subjective judgment. It replaces the inefficient traditional mode of relying on expert experience to debug protocol algorithms one by one with data-driven and automatic decision-making, which significantly improves the efficiency and accuracy of repairing NCCL communication performance bottlenecks in distributed training clusters.

[0055] The method provided in this embodiment, by running cluster-level communication tests after the training cluster has completed parameter configuration, collects real communication performance data of each node under different message sizes, realizing a leap in capability from single-point configuration verification to end-to-end performance scanning of the entire cluster, providing a comprehensive data foundation for subsequent analysis; by performing S-curve fitting and segmented slope detection on the collected performance data, the phenomenon of bus bandwidth growth stagnation or decline hidden in specific message size ranges, which is difficult to detect manually, is transformed into quantifiable and automatically identifiable curve shape features, accurately locating the performance degradation range caused by improper selection of communication algorithms or protocols and the range of nodes involved; based on this, by using various protocol types and algorithms supported by the NCCL communication library... In this type, the target protocol type and target communication algorithm are determined based on the characteristics of the abnormal interval, and an optimization configuration file is generated. The optimization scheme is solidified in a declarative configuration and then compiled into a shared library file for dynamic deployment to the problem node. This allows the optimization logic to be injected into the NCCL runtime environment in a dynamic loading manner without modifying the source code or recompiling the entire communication library. After deployment, the effect is verified by running cluster-level communication tests again. This forms a complete technical closed loop from parameter configuration, performance verification, anomaly location to targeted repair. It effectively solves the technical problem that communication performance degradation within a specific message size range is difficult to identify and repair after parameter tuning, and significantly improves the overall communication efficiency and stability of the distributed training cluster under real business message distribution.

[0056] Furthermore, Figure 2 The flowchart for Embodiment 2 of the NCCL communication optimization method based on performance change curve analysis provided in this application is shown below. Please refer to... Figure 2 The method provided in this embodiment may include:

[0057] S201. Divide each node in the training cluster into layers, obtain the parameters of each layer to form multiple sets of parameters to be tested, each set of parameters to be tested corresponds to a layer, and the layers corresponding to each set of parameters to be tested are different.

[0058] Specifically, a node refers to each GPU computing node in a distributed training cluster, i.e., the physical or virtual server instance that actually executes deep learning model training tasks in the cluster. These nodes are interconnected through a network and collaboratively complete the distributed training of large-scale models. This application aims to improve the overall NCCL collective communication performance of the cluster by systematically and automatically optimizing and setting the underlying configuration parameters that affect the efficiency of inter-GPU communication on each node. Parameters refer to various configuration parameters closely related to the running behavior of the NCCL communication library. These are configuration items spanning three layers: hardware firmware, operating system kernel, and communication software. Specifically, they can include: BIOS firmware settings related to GPU communication in the base layer, such as Resizable BAR and Above 4G Decoding; runtime parameters and network protocol stack parameters that can be adjusted by the operating system kernel in the system layer, such as NUMA binding strategy, memory large page configuration, MTU value, and TCP congestion control algorithm; and environment variable parameters provided by the NCCL communication library itself in the application layer, such as NCCL_IB_DISABLE, NCCL_NET_GDR_LEVEL, and NCCL_SOCKET_IFNAME. The values ​​of these parameters directly affect the efficiency of data transfer paths between GPUs, but they are not defined by deep learning training frameworks or models; instead, they exist independently in the underlying basic software stack of computing nodes.

[0059] Specifically, a three-layer parameter pool is constructed, consisting of a foundation layer, a system layer, and an application layer. Parameters corresponding to nodes are obtained from each layer's parameter pool to form a set of parameters to be tested for each layer.

[0060] Furthermore, the hierarchy includes at least a foundation layer, a system layer, and an application layer. The foundation layer is determined based on the computer's hardware, the system layer is determined based on the computer's operating system, and the application layer is determined based on the computer's applications.

[0061] Furthermore, a three-layer parameter architecture is constructed. The basic layer parameter pool contains low-level hardware configuration items closely related to GPU communication. The system layer parameter pool contains system parameters related to the node's computing performance, as well as system and network parameters related to network performance. The application layer parameter pool contains parameters related to NCCL environment variables. Further, the global parameter pool is matched and compared with the node's hardware specifications to identify and remove invalid parameters that the node's current hardware does not support. For example, the global parameter pool contains parameters for both Ethernet and InfiniBand network cards. If a node is only equipped with an Ethernet card and not an InfiniBand card, the InfiniBand-related parameters are removed from the node's test set. Similarly, if an older motherboard model is detected that does not support Above 4G Decoding, the parameter is removed from the corresponding node's test set. After the above processing, each node has a set of test parameters matching its hardware specifications at the basic, system, and application layers. These three sets are hierarchical, comprehensive, and non-overlapping.

[0062] Furthermore, the foundation layer is determined by the computer's hardware, which refers to the underlying BIOS firmware configuration parameters related to GPU communication, such as Resizable BAR and Above 4G Decoding. These parameters are directly limited by the node's physical hardware model and motherboard capabilities, and are strongly bound to hardware specifications. The system layer is determined by the computer's operating system, referring to the system operating parameters and network protocol stack configuration parameters that the operating system kernel can configure, such as NUMA binding strategies, memory big page configuration, MTU values, and TCP congestion control algorithms. The availability and value range of these parameters depend on the type of operating system and kernel version. The application layer is determined by the computer's application, referring to the environment variable configuration parameters provided by the NCCL collective communication library itself, such as NCCL_SOCKET_IFNAME, NCCL_IB_DISABLE, and NCCL_NET_GDR_LEVEL. These parameters are independent of hardware and operating system, and their behavior is controlled through the application software layer. The three layers, from bottom to top, are hardware support, system scheduling, and application communication. When constructing the parameter pool for each layer, it is necessary to perform validity filtering based on the actual hardware specifications and operating system environment of the node to ensure that the parameter configuration matches the node's capabilities.

[0063] S202. Classify the parameters in the set of parameters to be tested according to the hierarchy to obtain a subset of parameters to be tested in each set of parameters to be tested.

[0064] Specifically, the parameters in the system layer test parameter set are divided into a first system parameter subset that affects the performance of this node and a second system parameter subset that affects the network performance; the parameters in the application layer test parameter set are divided into a first application parameter subset that affects the performance of this node and a second application parameter subset that affects the network performance.

[0065] Furthermore, for the set of system-level parameters to be tested, the parameters are divided into two subsets according to their scope of application: parameters affecting the performance of the local node and parameters affecting network performance. Parameters affecting the performance of the local node refer to configuration items whose scope of application is limited to the internal computing efficiency of a single node, such as NUMA binding strategy and memory big page configuration. Parameters affecting network performance refer to configuration items whose scope of application involves the communication links between nodes, such as MTU value, TCP congestion control algorithm, and network card queue length.

[0066] Furthermore, for the application-layer set of parameters to be tested, the same classification logic is used to divide NCCL environment variables into combinations of parameters affecting local node performance and combinations of parameters affecting network performance. Basic-layer BIOS parameters, due to their underlying global attributes, are not further subdivided and belong to the first system parameter subset. After each subset is generated, it is combined with custom bandwidth and latency weights to form the set of parameters to be tested, providing a classification basis for subsequent differentiated tuning strategies.

[0067] Furthermore, the implementation steps for classifying the parameters in the set of parameters to be tested according to the hierarchy include:

[0068] (1) Determine the impact of each parameter on the nodes and the network;

[0069] Specifically, obtain the technical definition and functional description of each parameter in the set of parameters to be tested; based on the technical definition and functional description, determine whether the object of parameter regulation is the computing or data path resources inside the node or the network data transmission link between nodes; and form a one-to-one correspondence record between each parameter and the result of the influence relationship determination.

[0070] Furthermore, each parameter has a clear technical definition and functional boundaries within the operating system, and its impact can be directly determined based on these definitions. For system-level parameters, NUMA binding establishes an affinity association between a process and the CPU and memory resources of a specific NUMA node. Its control targets the resource allocation strategy within a single node; data flow is entirely completed within the node, without passing through the network interface card (NIC) or external switches. Therefore, its effect directly impacts the node but has no impact on the network. Large page configuration improves address translation efficiency within a single node by changing the page granularity of the memory management unit, similarly affecting only the node. Conversely, the MTU value defines the maximum transmission unit of a single data packet in the network, directly affecting packet fragmentation and encapsulation behavior on the network link. The TCP congestion control algorithm operates on the flow control strategy of the transport layer protocol, regulating end-to-end network transmission behavior. NIC queue length and Ring Buffer size affect the data transmission and reception buffering mechanism of the NIC, directly impacting the caching and forwarding efficiency of network packets. Therefore, these parameters have a direct impact on the network. For application-layer NCCL environment variable parameters, the determination is also based on their technical definitions. For example, NCCL_BUFFSIZE controls the size of the local communication buffer, affecting data transfer between GPUs within a node, thus impacting the node itself. NCCL_IB_TIMEOUT controls InfiniBand network timeout behavior, and NCCL_NET_GDR_LEVEL controls the direct data read level between the GPU and the network card; both affect the network communication link, thus impacting the network. By consulting the technical definition document for each parameter or identifying its functional semantics from the parameter naming rules, the impact relationships of all parameters can be determined without relying on actual measurement data, forming a mapping record between parameter names and impact relationship determination results.

[0071] (2) The parameters that affect the relationship to the node are divided into the first subset of parameters that affect the performance of the local node, and the parameters that affect the relationship to the network are divided into the second subset of parameters that affect the performance of the network.

[0072] Specifically, iterate through all parameters in the set of parameters to be tested and query the result of their influence relationship determination one by one; parameters whose influence relationship is determined to belong to the node are assigned to the first parameter subset; parameters whose influence relationship is determined to belong to the network are assigned to the second parameter subset; for parameters that affect both the node and the network, they can be assigned to both subsets at the same time or assigned to the dominant subset according to the main influence relationship.

[0073] Furthermore, based on the formed parameter-influence relationship mapping record, a classification operation is performed to create an empty second parameter subset and an empty first parameter subset. Each parameter in the set of parameters to be tested is traversed, and its influence relationship determination result is read. If the determination result indicates that the parameter's functional scope only involves resource scheduling or data paths within a single node, the parameter is assigned to the first parameter subset; if the determination result indicates that the parameter's functional scope involves network data transmission links or collective communication behavior between nodes, the parameter is assigned to the second parameter subset. During the partitioning process, if the functional scope of a parameter simultaneously covers both the node's internal and network communication levels—for example, some NCCL environment variables affect both local buffer management and network communication protocol selection—then the parameter can exist simultaneously in both the first and second parameter subsets, and subsequent optimization of both subsets must cover this parameter. For basic layer BIOS configuration parameters, because they operate at the hardware level, their influence typically covers the entire node's PCIe topology and GPU interconnect capabilities, possessing low-level global attributes. Therefore, basic layer parameters are not further partitioned and are directly used as an independent basic layer parameter set in the subsequent optimization process. After the partitioning is completed, each parameter subset is associated with a corresponding subset type identifier. The first parameter subset corresponds to the subsequent use of a single-node testing strategy, and the second parameter subset corresponds to the subsequent use of a multi-node testing strategy.

[0074] Furthermore, after hierarchical division and parameter classification, each node includes five parameter units to be tested, including: a basic layer parameter set, a first subset of system layer parameters, a second subset of system layer parameters, a first subset of application layer parameters, and a second subset of application layer parameters. Each parameter unit is associated with a node type identifier, a hierarchy identifier, a parameter subset type identifier, and a parameter list. The node type identifier records the hardware model classification to which the unit belongs, the hierarchy identifier indicates whether it is a basic layer, system layer, or application layer, the parameter subset type identifier distinguishes whether it affects the performance of this node or the network performance, and the parameter list records the parameters of this unit. All valid parameter items and their selectable value ranges after hardware filtering are included. Through the combination of the above four types of identifiers, any parameter unit has a unique index path in the entire tuning system. For example, it can be accurately located through "node type - level - subset type". The node type identifier determines that the optimal parameter set is only shared in batches among nodes of the same type. The level identifier determines the tuning execution order in the overall process. The subset type identifier determines the subsequent test combination method (single node or multiple nodes). This provides a clear scheduling basis for the automated tuning process to be promoted item by item by category, level and subset.

[0075] S203. Select any untuned node, determine the optimal parameter set based on the performance parameters of each subset of parameters to be tested, configure all untuned nodes of the same type as the untuned node based on the optimal parameter set, and return to the step of selecting any untuned node until all nodes in the training cluster have been tuned.

[0076] Specifically, untuned nodes are randomly selected, and for each subset of parameters to be tested, the parameter set is configured on the untuned nodes. After the configuration of each subset of parameters to be tested is completed, NCCL performance test is run and performance data is collected. According to the preset bandwidth and latency weight evaluation criteria, the performance of all subsets of parameters to be tested is compared, and the subset of parameters to be tested with the best performance is selected as the optimal parameter set.

[0077] Furthermore, on untuned nodes, each generated subset of parameters to be tested is applied sequentially. Taking a two-node combination as an example, the system or network parameters in the subset of parameters to be tested need to be set synchronously on the selected two nodes to ensure that the configurations on both ends are consistent. After each configuration is completed, the NCCL performance testing tool is run remotely. The appropriate test command is selected according to the type of the current parameter subset. Single-node testing uses single-machine commands, and multi-node testing uses multi-machine commands. The performance parameters collected during the test include communication bandwidth and transmission latency under different message sizes. The message size ranges from a few bytes to several GB. Among them, the bandwidth of large messages greater than 4MB and the latency of small messages less than 256KB are the key indicators. Furthermore, a comprehensive score is performed according to a custom weight standard. If the preset is bandwidth priority mode, the bandwidth data of messages greater than 4MB is given a higher weight, with bandwidth maximization as the main optimization goal; if the preset is latency priority mode, the latency data of messages less than 256KB is given a higher weight, with latency minimization as the main optimization goal. The performance scores of all subsets of parameters to be tested are sorted, and the one with the highest score is determined as the optimal parameter set for that level and type of parameter subset.

[0078] Furthermore, the steps to determine the optimal parameter set based on the performance parameters of each subset of parameters to be tested include:

[0079] (1) Randomly select multiple subsets of parameters to be tested, run communication performance tests respectively, and obtain the performance results of each subset of parameters to be tested as the initial observation data;

[0080] Specifically, the initial sampling quantity is determined; within the parameter space, a corresponding number of parameter value combinations are selected using a random sampling method; communication performance tests are sequentially configured and run for each parameter combination; the performance results corresponding to each parameter combination are collected and recorded to construct the initial observation dataset. The parameter space refers to the multidimensional search domain spanned by all parameters within multiple subsets of parameters to be tested and their respective all selectable values.

[0081] Furthermore, the initial sample size is determined based on the dimensionality of the parameter space, typically taking a multiple of the parameter dimension. Sampling methods can employ Latin hypercube sampling or uniform random sampling to generate uniformly distributed initial sampling points within the parameter space. Each sampling point corresponds to a complete set of parameter values, representing an instantiation of a subset of parameters to be tested. Each set of parameters is then sequentially configured onto the test nodes or node combinations, and single-machine or multi-machine communication performance tests are run. Bandwidth and latency data for the corresponding message size are collected and converted into a single performance score based on the preset optimization objective. Each parameter combination and its corresponding performance score constitute an observation record, and all initial observation records are summarized into an initial observation dataset.

[0082] (2) Use multiple sets of parameters to train a Gaussian process to obtain the performance prediction mean and prediction standard deviation in the parameter space;

[0083] Specifically, a Gaussian process surrogate model is established to fit the performance distribution within the parameter space, transforming discrete observation data into a continuous performance prediction surface and quantifying the confidence level of each prediction point. The Gaussian process model is trained using the parameter combination from the initial observation dataset as input and the performance results as output. After training, the Gaussian process model can output the performance prediction mean and prediction standard deviation for any unobserved point within the parameter space.

[0084] Furthermore, the Gaussian process is a nonparametric Bayesian model based on the kernel method. Its core idea is to assume that the performance values ​​between any two points in the parameter space follow a joint Gaussian distribution, and the closer the two points are, the stronger the correlation of their performance values. During training, an appropriate kernel function is selected to measure the similarity between parameter combinations, and the hyperparameters of the kernel function are estimated by maximizing marginal likelihood using initial observation data. After training, the Gaussian process model can provide the mean and standard deviation of performance predictions at any point in the parameter space. The mean performance prediction represents the model's best estimate of the performance value at that point; the standard deviation represents the degree of uncertainty in the model's prediction for that point. A smaller standard deviation near observed points indicates that the model is more confident in the performance of that region; a larger standard deviation in unobserved regions indicates that the model is still uncertain about the performance of that region. These two outputs together constitute the information basis for the calculation of the acquisition function.

[0085] (3) Based on the predicted mean and the predicted standard deviation, calculate the expected improvement value of each subset of parameters to be tested in the parameter space using the acquisition function, and select the subset of parameters to be tested with the largest expected improvement value as the set of parameters to be tested in the next round of testing;

[0086] Specifically, all candidate parameter combinations are sampled or traversed within the parameter space; for each candidate combination, the acquisition function value is calculated based on the predicted mean and predicted standard deviation output by the Gaussian process model; the parameter combination with the largest acquisition function value is selected as the parameter group to be tested in the next round of actual testing.

[0087] Furthermore, the acquisition function is the core mechanism guiding the search direction in Bayesian optimization. Its role is to merge the predicted mean and predicted standard deviation into a comparable score. Taking the expected uplift function as an example, its calculation method is as follows: for any point in the parameter space, calculate the uplift of the predicted mean at that point relative to the current best observation value, and weight the uplift with the predicted standard deviation using the cumulative probability density of a normal distribution to obtain the expected uplift value at that point. When the predicted mean of a point is high, the expected uplift value tends to select that point to utilize known high-performance regions; when the predicted standard deviation of a point is large, the expected uplift value tends to select that point to explore unknown regions that are not yet fully understood. By calculating the expected uplift value of all candidate parameter combinations in the parameter space and selecting the maximum value, an automatic balance between utilization and exploration is achieved. The selected parameter combination becomes the test parameter set for the next round of testing.

[0088] (4) Perform multiple tests on the parameter group to be tested, take the average value of the multiple test results as the performance parameter of the parameter group to be tested, update the Gaussian process based on the performance parameter of the parameter group to be tested, return the step of obtaining the performance prediction mean and prediction standard deviation of the parameter space, stop the iteration when the termination condition is met, and take the subset of the parameter group to be tested corresponding to the parameter group with the best performance parameters as the optimal parameter set.

[0089] Specifically, perform a preset number of repeated communication performance tests on the selected parameter group to be tested; calculate the average value of each repeated test result as the final performance parameter of the parameter group; add new observation data to the observation dataset and retrain the Gaussian process model; determine whether the iteration termination condition is met. If not, return to step (2) to continue iterating. If it is met, terminate the iteration and output the optimal parameter set.

[0090] Furthermore, for the selected parameter group to be tested, the parameter combination is configured on the test node and repeated communication performance tests are performed multiple times. The number of repetitions is usually set to a fixed value, and the average value of the multiple test results is taken to reduce the impact of random fluctuations in a single test. The newly obtained parameter group and its average performance parameters are added to the observation dataset as a new observation record. The Gaussian process model is retrained using the expanded observation dataset, the kernel function hyperparameters are updated, and a more accurate performance prediction mean and prediction standard deviation surface are obtained. After the update is completed, it is determined whether the iteration termination condition is met. The termination condition can be that the best performance improvement observed within a consecutive preset number of iterations is less than a preset threshold, indicating that the optimization has converged to a stable optimal region; the termination condition can also be that the preset maximum number of iterations is reached. If the condition is not met, return to step (2) and enter the next round of recommendation and testing based on the updated model; if the condition is met, stop the iteration, select the parameter combination with the highest performance score from all historical observation records, and determine the corresponding parameter value as the optimal parameter set of the parameter subset of this type at this level. Through the above Bayesian optimization iteration mechanism, the global optimal parameter configuration is approximated with far fewer test times than the full number of combinations.

[0091] Furthermore, identify untuned nodes of the same type as the currently tuned nodes, where the same type means the same hardware model and operating system; batch distribute and configure the optimal parameter set to all untuned nodes of the same type; after verifying that the configuration is effective, mark the node of this type as tuned; determine whether there are still untuned nodes in the cluster. If there are, return to step S203 to continue execution; if not, the entire cluster tuning is completed.

[0092] Furthermore, a categorization index for cluster nodes is established, based on a combination of multiple dimensions including GPU model, CPU model, network interface card (NIC) type, motherboard model, and operating system type and version. Once a representative node of a certain type has determined its optimal parameter set, this optimal parameter set is simultaneously distributed in batches and parallels to all other untuned nodes of that type using automated configuration management tools such as Ansible, SaltStack, or a self-developed agent. For system-level parameters, persistence is achieved by modifying sysctl.conf or the corresponding system configuration file. For network parameters, the changes are made effective via NIC configuration commands or network management tools. For NCCL environment variables, they are written to the node's environment configuration file. After the configuration is distributed, a verification command is executed remotely to check whether the parameter values ​​of each node are consistent with the optimal parameter set, ensuring that the configuration is effective and correct.

[0093] Furthermore, after successful verification, all nodes of that type are marked as tuned in the cluster node classification index. The index is then checked to see if any node types remain untuned, i.e., if any hardware or operating system combinations have not yet been found to be optimal. If so, one untuned node from that type is selected for a new round of hierarchical parameter tuning. The tuning scope covers the subset of parameters in the corresponding hierarchy of that node that have not yet been tuned. As the loop progresses, various parameter subsets at each hierarchy are tuned sequentially, ultimately covering all hierarchical parameters of all types of nodes in the cluster, achieving fully automated, end-to-end tuning of all parameters for all nodes in the training cluster.

[0094] Furthermore, before determining the optimal parameter set, the method provided in this embodiment also includes:

[0095] (1) Determine the range of test nodes based on the type of the subset of parameters to be tested;

[0096] Specifically, the type of the current subset of parameters to be tested is identified; if the subset of parameters to be tested is the first subset of parameters, the test node range is determined to be a single node; if the subset of parameters to be tested is the second subset of parameters, the test node range is determined according to the level, wherein the second subset of parameters at the system level determines the test node range to be two nodes, and the second subset of parameters at the application level determines the test node range to be four nodes.

[0097] Furthermore, obtain the type identifier of the subset of parameters to be tested. The type identifier indicates whether the subset of parameters to be tested belongs to the first subset of parameters affecting the performance of this node or the second subset of parameters affecting network performance. When the type identifier is the first subset of parameters, it means that the scope of the parameters within this subset is limited to the single node. The adjustment of these parameters will not go through the network link. Therefore, running the nccl-test single-machine command on a single node can fully verify its performance impact, and the scope of the test node is determined to be the representative node itself. When the type identifier is the second subset of parameters, it means that the scope of the parameters within this subset involves cross-node network communication links. It must be verified in a real inter-node communication scenario. In this case, the number of nodes is determined according to the level to which the parameters belong. If the level is the system layer, the scope of the test nodes is determined to be two. For a single node, two nodes with identical hardware type and operating system are selected to form a basic test cluster. This is because system-level network parameters such as MTU value and TCP congestion control algorithm affect end-to-end transmission behavior, and two nodes are the smallest and most effective basic unit to reflect changes in link communication. If the layer is the application layer, the test node range is determined to be four nodes. Four nodes with identical hardware type and operating system are selected to form a small-scale cluster. This is because application-level NCCL environment variable parameters such as NCCL_NET_GDR_LEVEL and NCCL_IB_TIMEOUT involve collective communication algorithms and topology-aware behavior. Four nodes can form various communication topologies such as ring and tree, which are closer to real distributed training scenarios and can fully expose the impact of parameter adjustments on collective communication performance.

[0098] (2) Determine the number of parameters participating in the combination based on the aforementioned hierarchy;

[0099] Specifically, the level to which the current optimization object belongs is obtained; based on the preset correspondence between level and number of parameters, the upper limit of the number of target parameters selected from the subset of parameters to be tested to participate in this permutation and combination is determined.

[0100] Furthermore, the number of basic BIOS parameters is relatively limited and the parameters are highly independent. All parameters within this subset can be included in the combination, with the number of parameters equal to the total number of parameters in this subset. The system layer has a large number of parameters, and there are certain dependencies between them. The selection ratio can be set according to the total number of parameters in the subset, or a fixed upper limit can be set. Some core parameters can be selected from the subset and included in this round of combination. The application layer NCCL environment variable parameters are numerous and some parameters have strong coupling effects. Similarly, the number of parameters can be set according to the size of the subset, and parameters with a higher impact on performance can be selected first for combination.

[0101] Furthermore, the number of parameters can be dynamically adjusted based on historical tuning data. For node types undergoing initial tuning, a larger number of parameters can be used to fully explore the parameter space. For node types with existing tuning experience using similar configurations, the number of parameters can be appropriately reduced to accelerate convergence. By controlling the number of parameters hierarchically, the problem of exponential expansion of the subset of parameters to be tested due to the simultaneous combination of all parameters can be effectively avoided, while ensuring coverage of key parameters.

[0102] Furthermore, the method provided in this embodiment also includes:

[0103] (1) Classify multiple nodes in the training cluster according to their hardware models;

[0104] Specifically, the hardware configuration information of each node in the training cluster is collected; the nodes are grouped according to the hardware model as the core classification dimension; and a mapping relationship between node type and the list of nodes to which it belongs is established.

[0105] Furthermore, in large-scale distributed training clusters, nodes typically consist of multiple batches and specifications of hardware. Nodes with different hardware models differ in BIOS configuration items, PCIe topology, GPU models, and interconnection methods. These differences determine the applicable parameter configurations. Therefore, at the beginning of the tuning process, it is necessary to obtain the core hardware identifiers of each node through out-of-band management interfaces or hardware information collection tools on the nodes. This includes key component information such as GPU model, CPU model, motherboard model, and network card model. Using the combination of these hardware identifiers as the classification basis, nodes with identical configurations are grouped into one category. Each category of nodes has the same hardware specifications and component combinations. After classification, a mapping index from type to node list is established, recording all node identifiers and their quantities for each hardware type. This classification index will be repeatedly used in subsequent steps to ensure that tuning operations are only performed within the same type of node, avoiding performance inconsistencies or operational anomalies caused by misconfigured parameters across hardware types.

[0106] (2) Determine the type of untuned node, and configure the optimal parameter set of the untuned node to the same level of all untuned nodes of the same type. The parameter configuration applied to each node of the same type at the same level is exactly the same.

[0107] Specifically, determine the hardware type of the currently tuned but untuned nodes; identify the level targeted by the current tuning; obtain the optimal parameter set of the node at that level after testing and verification; batch distribute and configure the optimal parameter set to the corresponding level parameter positions of all untuned nodes of the same type; and verify the consistency of the configuration of each node after it takes effect.

[0108] Furthermore, once an untuned node completes the testing of a subset of parameters at a certain level and determines the optimal parameter set through the aforementioned steps, that node becomes the tuning benchmark for its represented hardware type. The established node classification index is queried to determine the hardware type to which the node belongs, and a list of all nodes under that type is obtained. The current tuning level identifier is identified, i.e., whether the optimal parameter set is for a subset of parameters at the basic layer, system layer, or application layer. The level identifier corresponds one-to-one with the optimal parameter set, ensuring that the configuration is accurately targeted to the target level when it is distributed. The values ​​of each parameter contained in the optimal parameter set are synchronously written to the corresponding configuration locations of all other nodes of the same type through an automated configuration management tool.

[0109] Furthermore, for base-layer parameters, BIOS configurations are batch-distributed to each node and made effective via out-of-band interfaces; for system-layer parameters, they are batch-deployed using system configuration commands or configuration file distribution tools; for application-layer parameters, they are uniformly set using an environment variable configuration distribution mechanism. After batch configuration, configuration verification is performed on each node to confirm that the actual effective parameter values ​​on each node are completely consistent with the values ​​in the optimal parameter set. Since nodes of the same hardware type apply the exact same parameter configurations at all levels, the communication behavior between similar nodes in the cluster remains consistent, avoiding collective communication performance fluctuations caused by parameter asymmetry between nodes. This step is executed cyclically as different subsets of parameters at different levels are successively optimized, gradually achieving optimal parameter configuration coverage for all types of nodes at all levels.

[0110] By precisely classifying nodes in the training cluster according to hardware models, and after determining the optimal parameter set for representative nodes of a certain type, this parameter set is batch-configured onto the same level of all nodes of the same type. This ensures that the same node type applies completely consistent parameter configurations at the same level, thus achieving an efficient scaling mechanism from single-point testing to cluster coverage. This class-based grouping, one-time tuning, and batch replication mode avoids performance anomalies caused by misconfigured parameters across hardware models and eliminates communication behavior fluctuations caused by inconsistent parameter configurations between nodes of the same type. At the same time, it compresses the workload of repeated node-by-node tuning to be performed by node type, significantly reducing the tuning overhead of large-scale clusters while ensuring configuration consistency and tuning quality.

[0111] Furthermore, the method provided in this embodiment also includes:

[0112] (1) Perform performance verification tests on the nodes that have been configured with the optimal parameter set, filter out abnormal nodes based on the test results, and redetermine the optimal parameter set of the abnormal nodes;

[0113] Specifically, performance verification tests are performed on all nodes of the same type that have been configured with the optimal parameter set, and communication bandwidth and latency data of each node are collected; the test results of each node are compared with the average performance or benchmark performance of the same type of node group; if the performance deviation of a node exceeds the preset deviation threshold, the node is marked as an abnormal node; the complete tuning process is re-executed for the abnormal node, and the optimal parameter set is regenerated for the abnormal node.

[0114] Furthermore, after completing the batch distribution and configuration verification of the optimal parameter set, a performance verification test is run on each configured node. The test can use the same communication benchmark test command as in the tuning phase, collecting bandwidth and latency data within a uniform message size range. After collection, the average performance of the same type of node group under each message size is calculated. The test data of each node is compared with this average one by one, and the deviation is calculated. When the deviation of a node in the main performance indicators exceeds the preset deviation threshold, it indicates that the node has failed to obtain the expected communication performance after applying the unified parameter set, which may be due to individual hardware differences, firmware version differences, or link degradation issues that were not identified in the previous classification. The node is marked as an abnormal node, and the complete tuning process is re-triggered on the node, including re-executing the hierarchical parameter test, generating a subset of parameters to be tested and comparing performance, and generating the optimal parameter set for the abnormal node. In addition, if the cluster as a whole has completed cluster-level algorithm and protocol tuning, this tuning mechanism can also be directly called on abnormal nodes for targeted optimization to ensure that the communication performance of each node in the cluster is guaranteed and to avoid individual nodes becoming the performance bottleneck of the overall distributed training.

[0115] (2) Obtain the initial optimal parameter set of the abnormal node, perform performance testing on the initial optimal parameter set, and determine whether to retain the initial optimal parameter set.

[0116] Specifically, the optimal parameter set when an abnormal node is determined to be abnormal is used as the initial optimal parameter set; for each parameter in the initial optimal parameter set, all other parameters except the parameter are set to default values ​​or default values; multiple random values ​​are selected as test points within the legal value range of the parameter; the corresponding value of the parameter is configured at each test point in sequence and communication performance tests are run; the performance test results under each random value are compared and the performance fluctuation amplitude is calculated; the impact of the parameter on communication performance is determined based on the performance fluctuation amplitude, and then it is decided whether to retain or remove the parameter.

[0117] Furthermore, a control variable method is used to verify the validity of each parameter. For the single parameter being tested, all parameters in the set of parameters to be tested, except for this parameter, are reset to their default values ​​or system factory default values ​​to eliminate the combined effects of other parameters, and performance changes are observed only on the current parameter axis. Within the legal value range of this parameter, several value points are selected using a random sampling method. For example, multiple values ​​are selected evenly or randomly between the minimum and maximum values ​​of continuous numerical parameters, and for enumerated parameters, each option is traversed as a test point. At each test point, the parameter is set to the corresponding value, while keeping the other parameters at their default values. Single-machine or multi-machine communication benchmark tests are run, and the corresponding performance indicators are recorded. After completing all test points, the performance test results of this parameter under different values ​​are compared horizontally, and the performance fluctuation range between each test point is calculated. If the performance fluctuation of this parameter across all test points is less than the preset impact threshold, meaning that communication performance does not change significantly regardless of the parameter's value, then the parameter is deemed to have no substantial impact on communication performance and is considered a redundant parameter that can be removed under the current hardware and software environment; therefore, it is removed from the set of parameters to be tested. If the performance fluctuation caused by this parameter at different values ​​exceeds the preset impact threshold, then the parameter is deemed to have a substantial impact on communication performance and is retained. By iteratively executing the above verification process parameter by parameter, a streamlined parameter set is generated, retaining only the necessary parameters that do have a real impact on communication performance, removing interference from invalid or redundant parameters, making the final configuration of abnormal nodes more refined, stable, and easier for subsequent operation and maintenance management.

[0118] The method provided in this embodiment divides nodes according to type during batch node tuning. Nodes within the same type are further divided according to level and parameter function. This allows for the optimal parameter synchronization of multiple nodes within a single division result. On the one hand, this ensures the consistency of parameters across nodes during distributed cluster training. On the other hand, by dividing the group into three levels—node type, level, and parameter function—it avoids synchronizing all node parameters with any single object at once, further refining the granularity of the synchronization object for each node and improving the performance of the node model. Specifically, a hierarchical classification mechanism incorporates underlying hardware configurations, system kernel parameters, network protocol stack settings, and NCCL application environment variables into a unified tuning framework, ensuring comprehensive parameter coverage and inter-level collaboration. On-demand switching between single-machine and multi-machine testing modes and adaptive determination of test node scale ensures accurate verification of each parameter within a communication environment matching its scope. An expanded approach, grouping by hardware model, testing representative nodes, and batch coverage of similar types, compresses the tuning workload from node-by-node operations to node-by-node execution, significantly reducing the overhead of large-scale cluster tuning, while ensuring strict consistency in parameter configurations for nodes of the same type. A quantitative scoring mechanism using benchmark performance comparison and custom bandwidth latency weights replaces manual experience-based judgment with data-driven approaches, ensuring the objectivity and reproducibility of optimal parameter selection. Initial configuration backup and closed-loop recovery verification ensure that the tuning process does not affect the subsequent stability of the cluster. Finally, cluster-level performance change curve analysis and anomaly detection are performed after layer-by-layer parameter tuning to further locate and repair communication algorithm and protocol bottlenecks under specific message sizes. This solution addresses the core shortcomings of traditional manual tuning, such as low efficiency, poor parameter coordination, insufficient scenario adaptability, and difficulty in ensuring configuration consistency. It can significantly improve the NCCL communication performance of distributed training clusters and reduce operation and maintenance costs.

[0119] The NCCL communication optimization method based on performance change curve analysis provided in this embodiment divides each node of the training cluster into three layers: the basic layer, the system layer, and the application layer. It then categorizes the parameters according to their scope of influence into a first subset affecting local node performance and a second subset affecting network performance. A Bayesian optimization strategy is used to efficiently optimize each parameter subset. After classifying nodes by hardware model, the test results of representative nodes are batch-synchronized to all nodes of the same type. This achieves automated parameter tuning from single-point testing to cluster coverage. While ensuring strict consistency in the configuration of nodes of the same type, the tuning workload is reduced from node-by-node operation to execution by node type, significantly reducing the overhead of large-scale cluster tuning and solving the shortcomings of low efficiency, difficulty in ensuring consistency of multi-node configuration, and poor parameter coordination in manual tuning. In the second phase, based on the unified parameter baseline established in the first phase, cluster-level communication tests covering all nodes were run to collect communication bandwidth and latency data for each node under different message sizes. This achieved a leap in capability from single-point configuration verification to end-to-end performance scanning of the entire cluster. By performing S-curve fitting and segmented slope detection on the performance data, and using changes in the curve slope as a criterion, the system automatically identified plateaus or inflection points where bus bandwidth stagnated or declined with increasing message size. This transformed the traditionally experience-dependent and difficult-to-locate hidden performance degradation problems into a quantifiable and automatically detectable curve feature identification process, accurately pinpointing the message size data range to be optimized and the problematic nodes. Based on this, from NCCL... Among the various communication protocols and collective communication algorithms supported by the communication library, the target protocol type and target communication algorithm are determined based on the abnormal characteristics of plateau periods or inflection points. An optimization configuration file containing the mapping relationship between message size ranges and target protocols / algorithms is generated and compiled into a shared library file for dynamic deployment to problem nodes. This allows optimization measures to be dynamically loaded into the NCCL runtime environment without modifying the source code or recompiling the entire communication library. This effectively solves the technical problem that after parameter tuning, the NCCL default cost model may experience communication performance degradation due to improper algorithm or protocol selection within a specific message size range, which is difficult to identify and repair. After deployment, the effect is verified by running cluster-level communication tests again to ensure that the optimization measures are truly effective. The entire solution replaces the inefficient traditional mode of relying on expert experience to debug parameters and protocols one by one with data-driven and automated decision-making. This significantly improves the overall communication efficiency and stability of the distributed training cluster under real business message distribution and reduces operation and maintenance costs.

[0120] Corresponding to the aforementioned embodiment of the NCCL communication optimization method based on performance change curve analysis, this application also provides an embodiment of an NCCL communication optimization device based on performance change curve analysis.

[0121] Figure 3This is a schematic diagram of the third embodiment of the NCCL communication optimization device based on performance change curve analysis provided in this application. Please refer to... Figure 3 The apparatus provided in this embodiment includes an acquisition module 310, a processing module 320, and an optimization module 330.

[0122] The acquisition module 310 is used to run cluster-level communication tests after the training cluster has completed parameter configuration, and to collect communication performance data of each node in the cluster under different message sizes.

[0123] The processing module 320 is used to generate performance change curves for each node based on the communication performance data, determine the plateau period or inflection point where the bus bandwidth decreases as the message size increases, and determine the abnormal interval based on the plateau period or the inflection point.

[0124] The optimization module 330 is used to determine the message size data range to be optimized corresponding to the abnormal interval if the abnormal interval exists, generate an optimization configuration file containing optimized protocol parameters for the message size data range to be optimized, generate a shared library file according to the optimization configuration file, and deploy the shared library file to the nodes with performance problems in the cluster for communication optimization.

[0125] The apparatus of this embodiment can be used to perform... Figure 1 The steps of the method embodiment shown are similar in principle and process, and will not be repeated here.

[0126] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0127] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0128] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for optimizing NCCL communication based on performance variation curve analysis, characterized in that, The method includes: After the training cluster completes the parameter configuration, run the cluster-level communication test to collect communication performance data of each node in the cluster under different message sizes; Based on the communication performance data, performance change curves for each node are generated to determine the plateau period or inflection point where bus bandwidth decreases as message size increases, and abnormal intervals are determined based on the plateau period or inflection point; wherein, determining the plateau period or inflection point where bus bandwidth decreases as message size increases includes: The communication performance data was analyzed using an S-curve curve, and a performance change curve was plotted with message size on the x-axis and bus bandwidth on the y-axis. Detect whether there are any intervals in the performance change curve where the bus bandwidth growth stagnates or decreases as the message size increases; The intervals determined to be stagnant growth are marked as the plateau period, and the intervals determined to be declining are marked as the inflection points; The abnormal interval is determined, and an optimization configuration file containing optimized protocol parameters is generated for the abnormal interval. A shared library file is generated based on the optimization configuration file, and the shared library file is deployed to the nodes in the cluster with performance problems to optimize communication.

2. The method according to claim 1, characterized in that, Before the training cluster completes parameter configuration, the method further includes: Each node in the training cluster is divided into layers, and the parameters of each layer are obtained to form multiple sets of parameters to be tested. Each set of parameters to be tested corresponds to a layer, and the layers corresponding to each set of parameters to be tested are different. The parameters in the set of parameters to be tested are classified according to the hierarchy to obtain a subset of the parameters to be tested in each set of parameters to be tested. Select any untuned node, determine the optimal parameter set based on the performance parameters of each subset of parameters to be tested, configure all untuned nodes of the same type as the untuned node based on the optimal parameter set, and return to the selected untuned node, until all nodes in the training cluster have been tuned.

3. The method according to claim 2, characterized in that, The classification of parameters in the set of parameters to be tested according to the hierarchy includes: Determine the impact of each parameter on the nodes and the network; The parameters that affect the node are classified as the first subset of parameters affecting the local node's performance, and the parameters that affect the network are classified as the second subset of parameters affecting the network's performance.

4. The method according to claim 2, characterized in that, The hierarchical division of each node in the training cluster includes: The hierarchy includes at least a foundation layer, a system layer, and an application layer. The foundation layer is determined based on the computer's hardware, the system layer is determined based on the computer's operating system, and the application layer is determined based on the computer's applications.

5. The method according to claim 2, characterized in that, The process of determining the optimal parameter set based on the performance parameters of each subset of parameters to be tested includes: Randomly select multiple subsets of parameters to be tested, run communication performance tests on each subset, and obtain the performance results of each subset of parameters to be tested as initial observation data; By training a Gaussian process using multiple sets of parameters, the mean and standard deviation of the performance predictions in the parameter space are obtained. Based on the predicted mean and the predicted standard deviation, the expected improvement value of each subset of parameters to be tested in the parameter space is calculated using the acquisition function. The subset of parameters to be tested with the largest expected improvement value is selected as the set of parameters to be tested in the next round of testing. The test parameter group is tested multiple times, and the average value of the multiple test results is used as the performance parameter of the test parameter group. The Gaussian process is updated based on the performance parameter of the test parameter group. The process returns to the step of obtaining the performance prediction mean and prediction standard deviation of the parameter space. The iteration stops when the termination condition is met, and the subset of test parameters corresponding to the test parameter group with the best performance parameter is taken as the optimal parameter set.

6. The method according to claim 2, characterized in that, The method further includes: The training cluster is classified according to its hardware model. Determine the type of untuned node, and configure the optimal parameter set of the untuned node to the same level of all untuned nodes of the same type. The parameter configuration applied to each node of the same type at the same level is exactly the same.

7. The method according to claim 1, characterized in that, The step of generating an optimization configuration file containing optimized protocol parameters for the data range of the message size to be optimized includes: Obtain the start and end message sizes of the data range to be optimized; Based on the starting message size and the ending message size, the target protocol type is determined from the various communication protocols supported by the NCCL communication library, and the target communication algorithm is determined from the various collective communication algorithms supported by the NCCL communication library. The target protocol type and the target communication algorithm are written into the optimization configuration file as optimized protocol parameters.

8. The method according to claim 2, characterized in that, The method further includes: Perform performance verification tests on the nodes that have been configured with the optimal parameter set, filter out abnormal nodes based on the test results, and redetermine the optimal parameter set for the abnormal nodes; Obtain the initial optimal parameter set of the abnormal node, perform performance testing on the initial optimal parameter set, and determine whether to retain the initial optimal parameter set.

9. An NCCL communication optimization device based on performance change curve analysis, characterized in that, The device includes an acquisition module, a processing module, and an optimization module; The acquisition module is used to run cluster-level communication tests after the training cluster has completed parameter configuration, and to collect communication performance data of each node in the cluster under different message sizes. The processing module is configured to generate performance change curves for each node based on the communication performance data, determine the plateau period or inflection point where bus bandwidth decreases as message size increases, and determine abnormal intervals based on the plateau period or inflection point; wherein, determining the plateau period or inflection point where bus bandwidth decreases as message size increases includes: The communication performance data was analyzed using an S-curve curve, and a performance change curve was plotted with message size on the x-axis and bus bandwidth on the y-axis. Detect whether there are any intervals in the performance change curve where the bus bandwidth growth stagnates or decreases as the message size increases; The intervals determined to be stagnant growth are marked as the plateau period, and the intervals determined to be declining are marked as the inflection points; The optimization module is used to determine the message size data range to be optimized corresponding to the abnormal interval, generate an optimization configuration file containing optimized protocol parameters for the message size data range to be optimized, generate a shared library file according to the optimization configuration file, and deploy the shared library file to the nodes in the cluster with performance problems for communication optimization.

Images