An adaptive distributed training optimization method

Abstract

The application discloses a kind of self-adapting distributed training optimization method, belong to distributed deep learning training technical field, including the following steps: S1, through data acquisition module in distributed training process Iteration performance data is collected in real time;S2, speed variation detection module detects whether system performance appears significant fluctuation;S3, start problem positioning module determines the specific cause of performance degradation;S4, when determining network changes, trigger re-execution performance sampling process, switch training strategy by training strategy adjustment module;S5, when determining computing resource changes, system is automatically switched to benchmark scheduling strategy.The application adopts the above-mentioned self-adapting distributed training optimization method, can real-time sense system performance change, automatically determine the reason of performance degradation and adaptively adjust performance sampling model and scheduling strategy, so that distributed training continuously maintains efficient and stable communication and computing performance in dynamic, heterogeneous and multi-tenant environment.

Description

Technical Field

[0001] This invention relates to the field of distributed deep learning training technology, and in particular to an adaptive distributed training optimization method. Background Technology

[0002] With the rapid development of artificial intelligence, the scale of deep learning models is growing exponentially (e.g., models with hundreds of billions of parameters), and the size of training datasets is also continuously expanding. Against this backdrop, the computing power and storage capacity of a single GPU are no longer sufficient to meet the demands of training timeliness; training ultra-large-scale models with a single GPU often takes months or even years. Therefore, distributed training has become the mainstream solution for large-scale model training.

[0003] To simplify deployment and lower the barrier to entry, current mainstream distributed training frameworks generally adopt a static configuration mode: before training begins, users need to manually set parameters such as communication operators, block size, etc. These parameters remain fixed during training and cannot be dynamically adjusted according to changes in the environment.

[0004] To further improve the efficiency of communication and computation overlap, some research schemes in recent years have proposed a performance analysis-driven distributed training optimization approach. This type of method performs performance sampling operations before training to collect performance metrics such as communication operator latency and computation time, and builds a performance model based on the sampling results, thereby generating a scheduling strategy adapted to the current environment. For example, US-Byte determines tensor fusion and task priorities based on performance sampling results, while LIBRA constructs communication and computation models using performance sampling data to optimize resource allocation.

[0005] However, existing performance analysis-driven methods still have significant shortcomings:

[0006] Poor adaptability: The performance acquisition model is only built before training and cannot reflect changes in communication bandwidth or GPU load during operation;

[0007] Poor compatibility with heterogeneous and fluctuating scenarios: Most methods assume that the computing performance and bandwidth of nodes are stable, but in real clusters, there are often differences in GPU models or resource competition, which leads to model failure.

[0008] Lack of adaptive mechanism: When performance degrades, the system cannot automatically adjust the performance acquisition model or scheduling strategy.

[0009] In summary, existing distributed training solutions (including static configuration and performance acquisition methods) are all inadequate for effectively handling dynamic, heterogeneous, and multi-tenant shared training environments. Summary of the Invention

[0010] The purpose of this invention is to provide an adaptive distributed training optimization method that can sense system performance changes in real time, automatically determine the causes of performance degradation, and adaptively adjust the performance sampling model and scheduling strategy, so that distributed training can maintain efficient and stable communication and computing performance in dynamic, heterogeneous and multi-tenant environments.

[0011] To achieve the above objectives, the present invention provides an adaptive distributed training optimization method, comprising the following steps:

[0012] Step S1: Collect iterative performance data in real time during the distributed training process through the data acquisition module. The iterative performance data includes the time consumed per iteration, GPU computing utilization, communication latency, and bandwidth data.

[0013] Step S2: The speed change detection module detects whether there are significant fluctuations in system performance based on the average time consumption of the sliding window provided by the data acquisition module, combined with a weighted fusion evaluation of iteration speed, GPU computing utilization, and communication latency.

[0014] Step S3: Based on the detection results of step S2, start the problem location module to determine the specific cause of performance degradation. The specific cause includes network changes and computing resource changes. If no computing resource changes or network changes are found, it is determined to be an instantaneous fluctuation and return to step S1.

[0015] Step S4: When a network change is detected, the performance sampling process is re-executed, and the training strategy is switched through the training strategy adjustment module.

[0016] Step S5: When it is determined that there is a change in computing resources, the system automatically switches to the baseline scheduling strategy.

[0017] Preferably, in step S1, the time consumed in a single iteration is recorded in each training iteration. And store it in an initial length of In a sliding window, when the amount of data stored in the sliding window reaches Then, based on the average iteration time within the window, the average iteration speed corresponding to the window is dynamically calculated. :

[0018] ;

[0019] in, The sample batch size for a single iteration; the initial length W of the sliding window ranges from 20 to 100.

[0020] Preferably, in step S2, the sliding window length is adaptively adjusted based on the stability of the first N iterations, where N≥5. When the iteration fluctuation coefficient is greater than 0.1, the sliding window length increases by 30%, and when the iteration fluctuation coefficient is less than 0.05, the sliding window length decreases by 20%.

[0021] After each new window calculation is completed, the velocity change detection module performs the following operations:

[0022] S201. Calculate the average iteration rate of the current window. Average speed compared to the previous window The relative rate of change R:

[0023] ;

[0024] S202, the weighted evaluation value M of computation iteration speed, GPU computing utilization, and communication latency:

[0025] ;

[0026] in, This represents the current GPU computing utilization (values ​​range from 0 to 1). This represents the average time taken for the current window's communication steps. This represents the average time taken for the previous window communication step;

[0027] S203. Calculate the cumulative change based on the weighted evaluation value, using a dynamic threshold. To determine whether significant fluctuations exist, the dynamic threshold is dynamically adjusted based on the standard deviation of the weighted evaluation values ​​from the previous 10 rounds. The calculation formula is as follows:

[0028] ;

[0029] ;

[0030] ;

[0031] in, This represents the cumulative change at the k-th detection. This represents the cumulative change at the (k-1)th detection. This is the weighted evaluation value for the k-th test; The standard deviation of the weighted evaluation values ​​from the first 10 rounds; This is the average of the weighted evaluation values ​​from the previous 10 rounds.

[0032] ;

[0033] If the cumulative change exceeds the dynamic threshold three times consecutively (i.e.) and and If the system environment shows a significant trend of change, then it can be determined that the system environment is changing.

[0034] Preferably, in step S3, the problem localization module determines the cause through the following steps:

[0035] S301. Detect the current number of GPU processes, GPU utilization, and task throughput of each node, and compare them with the initial state data;

[0036] S302. If the rate of change of the number of GPU processes on any node exceeds 20%, or the fluctuation of GPU utilization exceeds 15% for three consecutive periods, and the fluctuation of communication bandwidth is less than 10%, it is determined to be a change in computing resources. At this time, the computing capabilities of each node exhibit heterogeneous characteristics.

[0037] Among them, the rate of change of GPU process count The calculation formula is:

[0038] ;

[0039] GPU utilization fluctuations The calculation formula is:

[0040] ;

[0041] Communication bandwidth fluctuation range The calculation formula is:

[0042] ;

[0043] in, This represents the current number of GPU processes. This represents the initial number of GPU processes. This represents the current GPU utilization rate. This represents the GPU utilization rate in the previous cycle. This represents the current communication bandwidth. This is the initial communication bandwidth;

[0044] S303. If the number of GPU processes and utilization rate of each node do not meet the above conditions, but the communication bandwidth fluctuation exceeds 30% or the communication latency increases to more than twice the initial value, it is determined to be a network change (i.e., a performance change caused by a change in communication bandwidth or an increase in latency).

[0045] S304. If the computing resources and network environment do not meet the above judgment conditions, it is judged as an instantaneous random fluctuation. The system ignores this anomaly and automatically returns to step S1.

[0046] Preferably, in step S4, the adaptive adjustment process of the training strategy includes refitting the communication time model and dynamically switching the scheduling parameters.

[0047] Specifically, after determining network changes, the system triggers a refitting operation of the communication model. This communication model characterizes the relationship between the time consumption of communication tasks and the data scale, guiding the scheduling and fusion strategies for subsequent communication tasks. The model form is shown in the following equation:

[0048] ;

[0049] in, For communication initiation delay; The time overhead of transmitting one byte; Indicates the size of the transmitted data; The communication time estimated by the model.

[0050] Preferably, during the model fitting process in step S4, the system dynamically adjusts the number of samplings based on the historical fitting error. If the previous fitting error is greater than 5%, the number of samplings increases by 50%; if the error is less than 2%, the number of samplings decreases by 30%. The number of samplings ranges from 10 to 100.

[0051] Among them, the fitting error The calculation formula is:

[0052] ;

[0053] in, The communication time predicted by the model; This represents the actual measured communication time.

[0054] Preferably, the system collects multiple sets of actual communication sample data during the performance sampling phase and uses the least squares method to analyze the parameters. and Linear fitting is performed to obtain the communication model under the current network environment;

[0055] The least squares method fits the objective function as follows:

[0056] ;

[0057] in, This refers to the number of samples collected. The sample number; For the first The actual communication time of the group samples; For the first The size of the transmitted data for the group of samples;

[0058] The solution obtained by least squares method and The optimal solution:

[0059] ;

[0060] ;

[0061] in, This is the sum of the actual communication times for all samples. The sum of the data sizes transmitted for all samples; This is the sum of the products of the data size transmitted for all samples and the actual communication time. The sum of squares of the data size transferred for all samples.

[0062] Preferably, after the communication model is updated, the system waits for all tasks in the current communication task queue to complete, and then automatically performs the parameter switching operation through the training strategy adjustment module, specifically:

[0063] S401. Adjust the parameters of the old model in the training strategy adjustment module. , Replace with the newly fitted , ;

[0064] S402. Recalculate tensor block threshold, fusion granularity, and communication priority rules based on the new model:

[0065] Among them, tensor block threshold The calculation formula is:

[0066] ;

[0067] in, For a single communication time slot; 8 represents the number of parallel communication channels; 8 is the unit conversion factor, converting Mbps to MB / s.

[0068] fusion granularity The calculation formula is:

[0069] ;

[0070] Wherein, λ is a preset buffer coefficient, ranging from 2 to 10, indicating that the amount of data after fusion should make the transmission time several times the startup delay, so as to make full use of the bandwidth.

[0071] The communication prioritization rule adopts a dynamic sorting mechanism that prioritizes latency-sensitive tasks, with priority scores. The calculation formula is:

[0072] ;

[0073] in, All are weighting coefficients. Default value , ; It is obtained by segmenting and quantizing tensors based on their topological order in the backpropagation computation graph. Specifically, the system divides all tensors to be transmitted into five priority tiers (1-5) according to the dependencies of backpropagation.

[0074] For the parameters that are needed first in the next forward propagation (i.e., the gradients closest to the input layer), the highest urgency is assigned (5); for the parameters that are needed last (i.e., the gradients closest to the output layer), the lowest urgency is assigned (1).

[0075] This switching process is completed during task idle periods to ensure communication continuity and avoid communication interruptions or gradient synchronization errors during model parameter switching, thereby achieving smooth adaptive optimization of the training system in dynamic network environments. The initial calculation logic for tensor block thresholds, fusion granularity, and communication priority rules is defined synchronously during model fitting in step S4, ensuring the consistency and traceability of parameter adjustments.

[0076] Furthermore, in step S5, when the problem location module determines that the computing resources have changed, the system automatically switches to the baseline scheduling strategy because the original scheduling strategy based on the assumption of "homogeneous node computing capabilities" has become invalid.

[0077] Furthermore, the baseline scheduling strategy refers to the default scheduling scheme adopted when the training system does not perform performance resampling or when the current environmental conditions do not meet the adaptive switching requirements. This strategy does not rely on performance acquisition results (including communication time sampling and computation performance sampling during the forward / backward propagation phases), but instead executes communication and computation task scheduling based on a set of fixed parameter configurations. This strategy features stable implementation and predictable behavior, and can provide reliable performance and consistency guarantees for the distributed training process even when the system is in a state of heterogeneous computing resources or severe environmental fluctuations.

[0078] Baseline scheduling strategies typically include the following characteristics:

[0079] Fixed tensor partitioning rules: Gradient tensor partitioning is performed based on a preset partitioning threshold. This partitioning method remains unchanged during training and is not adjusted with fluctuations in communication bandwidth or computational performance.

[0080] Fixed tensor fusion strategy: adopts static fusion granularity, that is, in the gradient communication stage, multiple small tensors are aggregated at a fixed granularity to reduce communication call overhead, but the fusion granularity does not change with the runtime environment;

[0081] Fixed communication priority rules: Communication tasks are queued in a preset order, and task priorities are not dynamically rearranged based on the communication time model;

[0082] Since the above features are all statically preset and do not rely on data provided by the performance acquisition module, switching to the baseline scheduling strategy can ensure the stability and consistency of the training process when the heterogeneity of node computing capabilities causes the performance analysis-driven method to fail. This effectively avoids communication blockage, synchronization delay or training performance degradation caused by model misjudgment or strategy mismatch.

[0083] Therefore, the present invention employs the above-mentioned adaptive distributed training optimization method, and the technical effects are as follows:

[0084] 1) It can sense changes in system performance in real time and promptly detect problems of declining training efficiency;

[0085] 2) Automatically determine the causes of performance degradation and distinguish between fluctuations in network and computing resources;

[0086] 3) Implement differentiated adaptive strategies based on the judgment results to improve the efficiency of communication-computation overlap.

[0087] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0088] Figure 1 This is a flowchart of an embodiment of an adaptive distributed training optimization method according to the present invention;

[0089] Figure 2 This is a diagram illustrating the operation of an adaptive distributed training optimization method of the present invention in a dynamic distributed training environment. Detailed Implementation

[0090] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0091] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0092] Example 1

[0093] In embodiments of the present invention, unless otherwise stated, the value of the sliding window length during training can be configured according to the actual task scale and system stability requirements.

[0094] Initial length of sliding window The value range is 20 to 100, which is used to balance the smoothness and real-time performance of the detection. It needs to be determined comprehensively based on the model complexity and system response sensitivity. When the window is too short, it is easily affected by occasional jitter, leading to false judgments; when the window is too long, the detection response is delayed. In this embodiment, the initial length of the sliding window... It is 40.

[0095] like Figure 1 As shown, this invention provides an adaptive distributed training optimization method, comprising the following steps:

[0096] Step S1: Collect iterative performance data in real time during distributed training using the data acquisition module; the iterative performance data includes the time consumed per iteration, GPU computing utilization, communication latency, and bandwidth data; record the time consumed per iteration in each training iteration. The data is then stored in a sliding window of length 40. Once the amount of data stored in the sliding window reaches length 40, the average iteration speed corresponding to that window is dynamically calculated based on the average iteration time within that window. :

[0097] ;

[0098] in, This represents the batch size of samples in a single iteration.

[0099] Step S2: The speed change detection module detects whether there are significant fluctuations in system performance based on the average time consumption of the sliding window provided by the data acquisition module in step S1, combined with a weighted fusion evaluation of iteration speed, GPU computing utilization, and communication latency.

[0100] The sliding window length is adaptively adjusted based on the stability of the first 10 iterations. When the iteration fluctuation coefficient is greater than 0.1, the sliding window length increases by 30%, and when the iteration fluctuation coefficient is less than 0.05, the sliding window length decreases by 20%.

[0101] After each new window calculation is completed, the velocity change detection module performs the following operations:

[0102] S201. Calculate the average iteration rate of the current window. Average speed compared to the previous window The relative rate of change R:

[0103] ;

[0104] S202, the weighted evaluation value M of computation iteration speed, GPU computing utilization, and communication latency:

[0105] ;

[0106] in, This represents the current GPU computing utilization (values ​​range from 0 to 1). This represents the average time taken for the current window's communication steps. This represents the average time taken for the previous window communication step;

[0107] S203. Calculate the cumulative change based on the weighted evaluation value, using a dynamic threshold. To determine whether significant fluctuations exist, the dynamic threshold is dynamically adjusted based on the standard deviation of the weighted evaluation values ​​from the previous 10 rounds. The calculation formula is as follows:

[0108] ;

[0109] ;

[0110] ;

[0111] in, This represents the cumulative change at the k-th detection. This represents the cumulative change at the (k-1)th detection. This is the weighted evaluation value for the k-th test; The standard deviation of the weighted evaluation values ​​from the first 10 rounds; This is the average of the weighted evaluation values ​​from the previous 10 rounds.

[0112] ;

[0113] If the cumulative change exceeds the dynamic threshold three times consecutively (i.e.) and and If the system environment shows a significant trend of change, a performance resampling or strategy switching operation will be triggered.

[0114] In a multi-node distributed environment, any node detecting a performance change and meeting the triggering conditions is considered a global performance degradation event. To ensure consistency in judgment results among nodes, this invention utilizes collective communication primitives to synchronize and reduce the detection flags of each node. When the detection result on any node is true, all nodes receive a global signal indicating "re-sampling of performance or policy switching is required," thereby uniformly executing performance resampling or policy adjustment operations at the same time step, ensuring the consistency of judgments and the synchronization of operations in a distributed scenario.

[0115] Step S3: Based on the detection results of step S2, activate the problem localization module to determine the specific causes of performance degradation. The specific causes include network changes and changes in computing resources.

[0116] The problem localization module determines the cause through the following steps:

[0117] S301. Detect the current number of GPU processes, GPU utilization, and task throughput of each node, and compare them with the initial state data;

[0118] S302. If the rate of change of the number of GPU processes on any node exceeds 20%, or the fluctuation of GPU utilization exceeds 15% for three consecutive periods, and the fluctuation of communication bandwidth is less than 10%, it is determined to be a change in computing resources. At this time, the computing capabilities of each node exhibit heterogeneous characteristics.

[0119] Among them, the rate of change of GPU process count The calculation formula is:

[0120] ;

[0121] GPU utilization fluctuations The calculation formula is:

[0122] ;

[0123] Communication bandwidth fluctuation range The calculation formula is:

[0124] ;

[0125] in, This represents the current number of GPU processes. This represents the initial number of GPU processes. This represents the current GPU utilization rate. This represents the GPU utilization rate in the previous cycle. This represents the current communication bandwidth. This is the initial communication bandwidth;

[0126] S303. If the number of GPU processes and utilization rate of each node do not meet the above conditions, but the communication bandwidth fluctuation exceeds 30% or the communication latency increases to more than twice the initial value, it is determined to be a network change (i.e., a performance change caused by a change in communication bandwidth or an increase in latency).

[0127] S304. If the computing resources and network environment do not meet the above judgment conditions, it is judged as an instantaneous random fluctuation. The system ignores this anomaly and automatically returns to step S1.

[0128] Step S4: When the problem module is determined to be a network change, the performance sampling process is re-executed to switch the training strategy.

[0129] The adaptive adjustment process of the training strategy includes refitting the communication time model and dynamically switching the scheduling parameters;

[0130] Specifically, after determining that the network has changed, the system triggers a refitting operation of the communication model. The communication model is used to characterize the relationship between the time consumption of communication tasks and the data scale, so as to guide the scheduling and fusion strategies of subsequent communication tasks. The model form is shown in the following formula:

[0131] ;

[0132] in, For communication initiation delay; The time overhead of transmitting one byte; Indicates the size of the transmitted data; The communication time estimated by the model.

[0133] During the model fitting process in step S4, the system dynamically adjusts the number of samplings based on the historical fitting error. If the previous fitting error is greater than 5%, the number of samplings increases by 50%; if the error is less than 2%, the number of samplings decreases by 30%. The number of samplings ranges from 10 to 100. In this embodiment, the initial number of samplings is 20.

[0134] Among them, the fitting error The calculation formula is:

[0135] ;

[0136] in, The communication time predicted by the model; This represents the actual measured communication time.

[0137] The system collects multiple sets of actual communication sample data (including communication time corresponding to different data volumes) during the performance sampling phase, and uses the least squares method to evaluate the parameters. and Linear fitting is performed to obtain the communication model under the current network environment;

[0138] The least squares method fits the objective function as follows:

[0139] ;

[0140] in, This refers to the number of samples collected. The sample number; For the first The actual communication time of the group samples; For the first The size of the transmitted data for the group of samples;

[0141] The solution obtained by least squares method and The optimal solution:

[0142] ;

[0143] ;

[0144] in, This is the sum of the actual communication times for all samples. The sum of the data sizes transmitted for all samples; This is the sum of the products of the data size transmitted for all samples and the actual communication time. The sum of squares of the data size transferred for all samples.

[0145] After the communication model is updated, the system waits for all tasks in the current communication task queue to complete, and then automatically performs the parameter switching operation through the training strategy adjustment module, specifically:

[0146] S401. Adjust the parameters of the old model in the training strategy adjustment module. , Replace with the newly fitted , ;

[0147] S402. Recalculate tensor block threshold, fusion granularity, and communication priority rules based on the new model:

[0148] Among them, tensor block threshold The calculation formula is:

[0149] ;

[0150] in, For a single communication time slot; 8 represents the number of parallel communication channels; 8 is the unit conversion factor, converting Mbps to MB / s.

[0151] fusion granularity The calculation formula is:

[0152] ;

[0153] Wherein, λ is a preset buffer coefficient, ranging from 2 to 10, indicating that the amount of data after fusion should make the transmission time several times the startup delay, so as to make full use of the bandwidth.

[0154] The communication prioritization rule adopts a dynamic sorting mechanism that prioritizes latency-sensitive tasks, with priority scores. The calculation formula is:

[0155] ;

[0156] in, All are weighting coefficients. Default value , ; It is obtained by segmenting and quantizing tensors based on their topological order in the backpropagation computation graph. Specifically, the system divides all tensors to be transmitted into five priority tiers, from 1 to 5, according to the dependencies of backpropagation.

[0157] For the parameters that are needed first in the next forward propagation (i.e., the gradients closest to the input layer), the highest urgency is assigned (5); for the parameters that are needed last (i.e., the gradients closest to the output layer), the lowest urgency is assigned (1).

[0158] This switching process is completed during task idle periods to ensure communication continuity and avoid communication interruptions or gradient synchronization errors during model parameter switching, thereby achieving smooth adaptive optimization of the training system in dynamic network environments. The initial calculation logic for tensor block thresholds, fusion granularity, and communication priority rules is defined synchronously during model fitting in step S4, ensuring the consistency and traceability of parameter adjustments.

[0159] Step S5: When the problem localization module determines that there is a change in computing resources, since the original performance sampling-driven scheduling strategy based on the assumption of "isomorphic node computing capabilities" has become invalid, the system automatically switches to the baseline scheduling strategy. In this embodiment, it switches to the Horovod original training strategy.

[0160] Example 2

[0161] This embodiment combines Figure 2 The operation process of the method of the present invention in a dynamic distributed training environment is described.

[0162] The first phase is the normal operation phase. In this phase, the system is in a stable distributed training environment, with communication bandwidth and computing resources remaining constant. The iterative data acquisition module continuously collects data on single iteration time, GPU computing utilization, and communication latency, storing it in a sliding window. The weighted evaluation value M calculated by the speed change detection module fluctuates at a low level, with the cumulative change... Always below the dynamic threshold The system determined that no significant fluctuations had occurred and maintained the original training strategy unchanged.

[0163] The second stage is the external interference application stage. During this stage, changes in the external environment occur, such as partial occupancy of communication bandwidth or jitter in the communication link, leading to increased communication latency and decreased training speed. The system continuously monitors changes in iterative performance.

[0164] The weighted evaluation value M spikes: This is due to a decrease in iteration speed and an increase in communication latency. M The value increased significantly.

[0165] Trigger threshold: An increase in the M value drives the cumulative change. Rapid rise, when Exceeding the dynamic threshold adjusted based on the previous 10 standard deviations three times consecutively Subsequently, the speed change detection module determined that the system performance was experiencing abnormal fluctuations.

[0166] The third stage is the performance-aware adaptive stage. After detecting significant performance fluctuations, the system first activates the problem localization module to determine the cause. The module detects that the number of GPU processes and pure computation time on each node are not abnormal, but the communication bandwidth fluctuation exceeds 30% (or the latency increases significantly), thus determining that the performance degradation is caused by network changes.

[0167] The system triggers a performance sampling process, collecting communication time samples under different data volumes, and refits the communication model parameters using the least squares method. Based on the new communication model parameters, the training strategy adjustment module automatically performs parameter switching operations. By applying this new strategy, the system maintains high computation-communication overlap efficiency even under limited bandwidth conditions, resulting in a significant improvement in training speed.

[0168] The fourth stage is the external interference elimination stage. During this stage, the communication bandwidth occupancy in the external environment gradually disappears, and the system's available communication bandwidth returns to normal. As the network environment improves, the time consumed per iteration decreases. The speed change detection module again detects a significant change in the weighted evaluation value M (this time caused by a positive speed increase), determining that a positive change has occurred in the system environment.

[0169] The fifth stage is the environmental re-change and second performance resampling stage. The system triggers the problem localization module again to confirm that the communication bandwidth has returned to its initial level (determined to be a network change). A second performance resampling operation is then triggered to refit the communication model and switch back to a training strategy adapted to the high-bandwidth environment, thereby fully utilizing the recovered network resources.

[0170] Therefore, the present invention employs the aforementioned adaptive distributed training optimization method, which can perceive system performance changes in real time, automatically determine the causes of performance degradation, and adaptively adjust the performance sampling model and scheduling strategy, so that distributed training can maintain efficient and stable communication and computing performance in dynamic, heterogeneous, and multi-tenant environments.

[0171] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. An adaptive distributed training optimization method, characterized in that, Includes the following steps: Step S1: Collect iterative performance data in real time during the distributed training process through the data acquisition module. The iterative performance data includes the time consumed per iteration, GPU computing utilization, communication latency, and bandwidth data. Step S2: The speed change detection module detects whether there are significant fluctuations in system performance based on the average time consumption of the sliding window provided by the data acquisition module, combined with a weighted fusion evaluation of iteration speed, GPU computing utilization, and communication latency. In step S2, the sliding window length is adaptively adjusted based on the stability of the first N iterations, where N≥5. When the iteration fluctuation coefficient is greater than 0.1, the sliding window length increases by 30%, and when the iteration fluctuation coefficient is less than 0.05, the sliding window length decreases by 20%. After each new window calculation is completed, the velocity change detection module performs the following operations: S201. Calculate the average iteration rate of the current window. Average speed compared to the previous window The relative rate of change R: ； S202, the weighted evaluation value M of computation iteration speed, GPU computing utilization, and communication latency: ； in, This represents the current GPU computing utilization rate. This represents the average time taken for the current window's communication steps. This represents the average time taken for the previous window communication step; S203. Calculate the cumulative change based on the weighted evaluation value, using a dynamic threshold. To determine whether significant fluctuations exist, the dynamic threshold is dynamically adjusted based on the standard deviation of the weighted evaluation values ​​from the previous 10 rounds. The calculation formula is as follows: ； ； ； in, This represents the cumulative change at the k-th detection. This represents the cumulative change at the (k-1)th detection. This is the weighted evaluation value for the k-th test; The standard deviation of the weighted evaluation values ​​from the first 10 rounds; This is the average of the weighted evaluation values ​​from the previous 10 rounds. ； If the cumulative change exceeds the dynamic threshold three times consecutively, it is determined that there is a significant trend of change in the system environment. Step S3: Based on the detection results of step S2, start the problem location module to determine the specific cause of performance degradation. The specific cause includes network changes and computing resource changes. If no computing resource changes or network changes are found, it is determined to be an instantaneous fluctuation and return to step S1. Step S4: When a network change is detected, the performance sampling process is re-executed, and the training strategy is switched through the training strategy adjustment module. Step S5: When it is determined that there is a change in computing resources, the system automatically switches to the baseline scheduling strategy.

2. The adaptive distributed training optimization method according to claim 1, characterized in that: In step S1, the time taken for a single iteration is recorded in each training iteration. And store it in an initial length of In a sliding window, when the amount of data stored in the sliding window reaches Then, based on the average iteration time within the window, the average iteration speed corresponding to the window is dynamically calculated. : ； in, The sample batch size for a single iteration; the initial length of the sliding window. The value range is 20 to 100.

3. The adaptive distributed training optimization method according to claim 1, characterized in that: In step S3, the problem localization module determines the cause through the following steps: S301. Detect the current number of GPU processes, GPU utilization, and task throughput of each node, and compare them with the initial state data; S302. If the rate of change of the number of GPU processes on any node exceeds 20%, or the fluctuation of GPU utilization exceeds 15% for three consecutive periods, and the fluctuation of communication bandwidth is less than 10%, it is determined to be a change in computing resources. At this time, the computing capabilities of each node exhibit heterogeneous characteristics. Among them, the rate of change of GPU process count The calculation formula is: ； GPU utilization fluctuations The calculation formula is: ； Communication bandwidth fluctuation range The calculation formula is: ； in, This represents the current number of GPU processes. This represents the initial number of GPU processes. This represents the current GPU utilization rate. This represents the GPU utilization rate in the previous cycle. This represents the current communication bandwidth. This is the initial communication bandwidth; S303. If the number of GPU processes and utilization rate of each node do not meet the conditions in step S302, but the communication bandwidth fluctuation exceeds 30% or the communication latency increases to more than twice the initial value, then it is determined to be a network change. S304. If the computing resources and network environment do not meet the above judgment conditions, it is judged as an instantaneous random fluctuation. The system ignores this anomaly and automatically returns to step S1.

4. The adaptive distributed training optimization method according to claim 1, characterized in that: In step S4, the adaptive adjustment process of the training strategy includes refitting the communication time model and dynamically switching the scheduling parameters. Specifically, after determining network changes, the system triggers a refitting operation of the communication model. This communication model characterizes the relationship between the time consumption of communication tasks and the data scale, guiding the scheduling and fusion strategies for subsequent communication tasks. The model form is shown in the following equation: ； in, For communication initiation delay; The time overhead of transmitting one byte; Indicates the size of the transmitted data; The communication time estimated by the model.

5. The adaptive distributed training optimization method according to claim 4, characterized in that: During the model fitting process in step S4, the system dynamically adjusts the number of samplings based on the historical fitting error. If the previous fitting error is greater than 5%, the number of samplings increases by 50%; if the error is less than 2%, the number of samplings decreases by 30%. The number of samplings ranges from 10 to 100. Among them, the fitting error The calculation formula is: ； in, The communication time predicted by the model; This represents the actual measured communication time.

6. The adaptive distributed training optimization method according to claim 5, characterized in that: The system collects multiple sets of actual communication sample data during the performance sampling phase and uses the least squares method to analyze the parameters. and Linear fitting is performed to obtain the communication model under the current network environment; The least squares method fits the objective function as follows: ； in, This refers to the number of samples collected. The sample number; For the first The actual communication time of the group samples; For the first The size of the transmitted data for the group of samples; The solution obtained by least squares method and The optimal solution: ； ； in, This is the sum of the actual communication times for all samples. The sum of the data sizes transmitted for all samples; This is the sum of the products of the data size transmitted for all samples and the actual communication time. The sum of squares of the data size transferred for all samples.

7. The adaptive distributed training optimization method according to claim 6, characterized in that: After the communication model is updated, the system waits for all tasks in the current communication task queue to complete, and then automatically performs the parameter switching operation through the training strategy adjustment module, specifically: S401. Adjust the parameters of the old model in the training strategy adjustment module. , Replace with the newly fitted , ; S402. Recalculate tensor block threshold, fusion granularity, and communication priority rules based on the new model: Among them, tensor block threshold The calculation formula is: ； in, For a single communication time slot; This represents the number of parallel communication channels. fusion granularity The calculation formula is: ； Wherein, λ is the preset buffer coefficient, which takes a value of 2 to 10; The communication prioritization rule adopts a dynamic sorting mechanism that prioritizes latency-sensitive tasks, with priority scores. The calculation formula is: ； in, All are weighting coefficients. Default value , ; It is obtained by segmenting and quantizing tensors based on their topological order in the backpropagation computation graph. Specifically, the system divides all tensors to be transmitted into five priority tiers from 1 to 5 according to the dependencies of backpropagation. Among them, the parameter that is needed first in the next round of forward propagation is assigned the highest urgency of 5, and the parameter that is needed last is assigned the lowest urgency of 1.

Images