Slow node detection method, apparatus, device, storage medium and program product

Abstract

Embodiments of the present disclosure disclose a slow node detection method, device, equipment, storage medium and program product. The method comprises: obtaining first time length data of a plurality of processing nodes participating in distributed model training; the first time length data represents the time consumption of the corresponding processing node in the current iteration calculation; grouping the first time length data based on the parallel strategy of the distributed model training to obtain a first time length set of a plurality of groups; the parallel strategy represents the correlation of the plurality of processing nodes in the forward and backward propagation process and the training tasks they undertake; based on the probability density function of each of the plurality of groups and a first threshold, the first time length set of each of the plurality of groups is analyzed and processed to obtain a slow node in the plurality of processing nodes. In this way, the slow node in the distributed model training can be intelligently and accurately detected, the efficiency and accuracy of slow node detection are improved, and strong compatibility is achieved.

Description

Technical Field

[0001] This disclosure relates to, but is not limited to, the field of distributed training technology, and in particular to a slow node detection method, apparatus, device, storage medium, and program product. Background Technology

[0002] When training large models using distributed training methods, each processing node is responsible for processing only a portion of the model and data. Iterative updates of the large model are achieved by synchronizing and integrating the model parameters across multiple processing nodes. However, if a processing node slows down during parallel computation, other nodes will be in a waiting state, hindering the entire distributed training process. This slowed-down node is called a straggler. Stragglers have a strong negative impact on distributed training. Currently, however, there is still a lack of real-time and efficient methods for detecting stragglers in distributed training. Summary of the Invention

[0003] In view of this, the present disclosure provides at least one method, apparatus, device, storage medium, and program product for detecting slow nodes.

[0004] The technical solution of this disclosure embodiment is implemented as follows:

[0005] On one hand, embodiments of this disclosure provide a slow node detection method, including:

[0006] Obtain the first duration data for each of the multiple processing nodes participating in the distributed model training; the first duration data represents the computation time of the corresponding processing node in the current iteration;

[0007] The parallel strategy based on distributed model training groups the first duration data into multiple groups of first duration sets; the parallel strategy represents the relationship between multiple processing nodes in the forward and backward propagation process, as well as the training tasks they undertake.

[0008] Based on the probability density function and first threshold of each group, the first duration set of each group is analyzed and processed to obtain the slow nodes among the multiple processing nodes.

[0009] On the other hand, embodiments of this disclosure provide a slow node detection device, including:

[0010] The acquisition module is configured to obtain the first duration data of each of the multiple processing nodes participating in the distributed model training; the first duration data represents the computation time of the corresponding processing node in the current iteration;

[0011] The grouping module is configured to use a parallel strategy based on distributed model training to group the first duration data into multiple groups of first duration sets; the parallel strategy represents the relationship between multiple processing nodes in the forward and backward propagation process, as well as the training tasks they undertake.

[0012] The processing module is configured to analyze and process the first duration set of multiple groups based on the probability density function and the first threshold of each group, and obtain the slow node among multiple processing nodes.

[0013] In another aspect, embodiments of this disclosure provide a computer device, including a memory and a processor. The memory stores a computer program that can run on the processor, and the processor executes the program to implement some or all of the steps in the above-described method.

[0014] In another aspect, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements some or all of the steps in the above-described method.

[0015] In another aspect, embodiments of this disclosure provide a computer program including computer-readable code, which, when executed in a computer device, causes a processor in the computer device to perform some or all of the steps in the above-described method.

[0016] In another aspect, embodiments of this disclosure provide a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, it implements some or all of the steps in the above-described method.

[0017] In this embodiment, the computational score performance of each processing node in distributed model training is evaluated using the first duration data of multiple processing nodes in the current iteration. Grouping the first duration data using a parallel strategy for distributed model training reduces the impact of load imbalance on slow node detection and facilitates use under different parallel strategies. Since the probability density function can characterize the overall distribution of computation time of different groups of Graphics Processing Units (GPUs), analyzing the first duration sets of multiple groups using their respective probability density functions and first thresholds allows for evaluation and detection of processing nodes based on the overall distribution of computation time. Compared to detection based on single characteristics such as the mean, median, or variance of computation time, this method achieves higher accuracy in detecting slow nodes. Therefore, the slow node detection method provided in this embodiment can intelligently and accurately detect slow nodes in distributed model training, improving both efficiency and accuracy. Furthermore, this method can be applied to distributed training scenarios with different numbers of devices, different models, and different training parameters, and can be used under different parallel strategies, demonstrating strong compatibility.

[0018] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this disclosure. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the specification, serve to illustrate the technical solutions of this disclosure.

[0020] Figure 1 A schematic diagram of the implementation process of a slow node detection method provided in this embodiment of the present disclosure. Figure 1 ;

[0021] Figure 2 A schematic diagram of a parallel strategy in a slow node detection method provided in this embodiment of the disclosure;

[0022] Figure 3 A schematic diagram of the implementation process of a slow node detection method provided in this embodiment of the present disclosure. Figure 2 ;

[0023] Figure 4 A schematic diagram of the implementation process of a slow node detection method provided in this embodiment of the present disclosure. Figure 3 ;

[0024] Figure 5 A schematic diagram of the composition structure of a slow node detection device provided in this embodiment of the present disclosure. Figure 1 ;

[0025] Figure 6 A schematic diagram of the implementation process of a slow node detection method provided in this embodiment of the present disclosure. Figure 4 ;

[0026] Figure 7 A schematic diagram of the composition structure of a slow node detection device provided in this embodiment of the present disclosure. Figure 2 ;

[0027] Figure 8 This is a schematic diagram of the hardware entity of a computer device provided in an embodiment of the present disclosure;

[0028] Figure 9 This is a schematic diagram of the hardware entity of a computer program product provided in an embodiment of the present disclosure. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this disclosure clearer, the technical solutions of this disclosure are further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on this disclosure. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0030] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0031] The terms “first / second / third” are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that “first / second / third” may be interchanged in a specific order or sequence where permitted, so that the embodiments of this disclosure described herein can be implemented in an order other than that illustrated or described herein.

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. The terminology used herein is for descriptive purposes only and is not intended to limit the scope of this disclosure.

[0033] Large Language Models (LLMs) are rapidly developing, with the number of model parameters increasing dramatically to over 10 billion. Training requires kilocalories or tens of thousands of kilocalories of computing power. Only by combining distributed training methods can the model be trained in a reasonable time and achieve the expected model performance.

[0034] When training large models using distributed training methods, each GPU is only responsible for processing a portion of the model and a portion of the data. Gradients or parameters are synchronized using ensemble communication to achieve iterative updates of the large model.

[0035] However, some GPUs in the cluster can exhibit poor computational performance during model training due to factors such as heat dissipation, hardware performance limitations, and workload. This directly impacts the efficiency of distributed training. As the cluster size increases, the probability of encountering such poorly performing GPUs also increases, making large-scale distributed training more susceptible to slow node problems.

[0036] The presence of slow nodes can severely hinder the progress of distributed training, leading to a decline in the performance of the entire training process and even causing training failure. Therefore, timely detection of slow nodes during training not only enables real-time monitoring of the cluster's GPU training performance but also helps to address slow node issues in a targeted manner, ensuring that large-scale distributed training proceeds efficiently and stably.

[0037] For training large-scale models (tens of thousands of GPUs), the massive cluster size makes slow node localization exceptionally difficult. Specifically, large-scale model training typically employs multi-dimensional hybrid parallel strategies, leading to workload imbalances among GPU nodes. If the number of GPUs involved in training is large, slow node detection requires collecting and processing a vast amount of data, placing high demands on the efficiency of the detection algorithm. Furthermore, slow node detection methods should consider the needs of real-world scenarios, minimizing the impact on training performance while supporting diverse parameter configurations during large-scale model training or tuning. Therefore, slow node detection algorithms should possess good scalability and generalization capabilities.

[0038] The existing approach uses heatmaps to display the average latency of different devices, and slow nodes are identified by manually observing the darker-colored points in the graph. However, this method requires manual intervention for visualization and lacks quantitative analysis of slow nodes and automated detection processes, resulting in problems such as missed detections, false positives, and low efficiency.

[0039] The existing approach 2 saves the multi-NIC data from each node during actual training and constructs a dataset to train a corresponding nonlinear autoregressive model with exogenous inputs (NARX). This trained NARX model is then used in distributed model training to infer and identify slow nodes. However, this method requires pre-preparing the dataset and training a convergent NARX model before it can be used for slow node detection in subsequent training processes, regardless of the model architecture or training parameter configuration. This results in drawbacks such as long preparation time and insufficient applicability.

[0040] Therefore, this disclosure provides a slow node detection method, which aims to quickly discover slow computation nodes in large-scale model training with tens of thousands of kilobytes of computation, and solve the problems of low detection accuracy, poor real-time performance, and poor scalability in the prior art.

[0041] The slow node detection method provided in this disclosure can be executed by the processor of a computer device. The computer device refers to a device with data processing capabilities, such as a server, laptop, tablet, desktop computer, smart TV, set-top box, or mobile device (e.g., mobile phone, portable video player, personal digital assistant, dedicated messaging device, portable gaming device). Figure 1 As shown, the method includes the following steps 101 to 103:

[0042] Step 101: Obtain the first duration data of each of the multiple processing nodes participating in the distributed model training; the first duration data represents the computation time of the corresponding processing node in the current iteration.

[0043] For large-scale distributed model training tasks, hundreds or thousands of worker nodes need to be scheduled to execute the training task. These multiple worker nodes can be called a cluster. A worker node participating in distributed model training within a cluster corresponds to a machine, including multiple processing nodes, hardware drivers, model training frameworks, training tasks, etc. The processing nodes are used to execute specific training tasks. For example, a processing node can be a GPU, a Central Processing Unit (CPU), or a Tensor Processing Unit (TPU), etc.

[0044] The monitoring node in the cluster, used to monitor training tasks, corresponds to a single machine and may not be equipped with a GPU. It is mainly used to monitor worker nodes. The monitoring node supports using one GPU from a worker node as the smallest monitoring unit, collecting the training status, intermediate data, and training parameters of each GPU, and performing real-time slow node detection and persistent data transfer.

[0045] The first duration data represents the computation time of the corresponding processing node in the current iteration. The first duration data may include the computation time of the processing node during forward and backward propagation in the current iteration. Forward and backward propagation includes both forward and backward propagation. For example, the first duration data may include, but is not limited to: the computation time of forward propagation (or the start and end times of forward propagation), the computation time of backward propagation (or the start and end times of backward propagation), the start time of the current iteration, and the end time of the current iteration.

[0046] In some implementations, the specific way to obtain the first duration data can be: by monitoring the nodes, detecting the computation time of each processing node from start to finish during the forward and backward propagation process, and obtaining the first duration data.

[0047] Step 102: Based on the parallel strategy of the distributed model training, the first duration data is grouped to obtain multiple groups of first duration sets; the parallel strategy represents the relationship between the multiple processing nodes in the forward and backward propagation process, as well as the training tasks undertaken by each node.

[0048] Parallelism strategies refer to how to decompose the model training task into multiple subtasks in distributed training and execute them in parallel on multiple processing nodes (such as CPUs, GPUs, or TPUs) to accelerate the training process and improve efficiency. Different parallelism strategies are suitable for different scenarios and hardware configurations, and choosing an appropriate parallelism strategy can significantly improve training speed and resource utilization.

[0049] In some implementations, distributed model training can employ multi-dimensional hybrid parallel strategies, such as data parallelism (DP), tensor parallelism (TP), and pipeline parallelism (PP). These multiple parallel strategies can be used in combination to coordinate thousands or even tens of thousands of GPUs to accelerate model training.

[0050] The data parallelism strategy involves dividing the training data into multiple subsets, each of which is assigned to an independent processing node (e.g., a GPU) for training. The parameters of all processing nodes are updated synchronously in some way after each iteration.

[0051] Tensor parallelism involves partitioning the model's tensors (e.g., weight matrices) into multiple sub-tensors and assigning them to different processing nodes (e.g., a GPU) for computation. Each processing node computes its assigned sub-tensor portion and collaborates with other processing nodes through communication operations to complete the full computation.

[0052] Pipeline parallelism involves dividing the model into multiple stages, with each stage assigned to a processing node (or a group of devices) for computation. Data is passed sequentially between these stages, forming a pipelined computation process.

[0053] like Figure 2As shown, the multi-dimensional hybrid parallel strategy can be a three-dimensional hybrid parallel strategy. The first dimension is pipelined parallelism (PP), the second dimension is data parallelism (DP), and the third dimension is tensor parallelism (TP). The coordinates of each GPU in the three-dimensional hybrid parallel strategy can represent the role of each GPU in distributed training, that is, the training task undertaken by each GPU and its relationships in the forward and backward propagation process. For example, for GPU12 in the figure, its relationships include the relationship between GPU12 and GPU4, the relationship between GPU12 and GPU20, and the relationship between GPU12 and GPU8, etc.

[0054] In some implementations, the specific way to determine the first duration set of multiple groups can be: based on the roles of multiple processing nodes in distributed training, the first duration data is grouped to obtain the first duration set of multiple groups.

[0055] In some implementations, the specific way to determine the first duration set of multiple groups can be: based on the roles of multiple processing nodes in distributed training, the first duration data is grouped and preprocessed to obtain the first duration set of multiple groups.

[0056] For example, such as Figure 2 As shown, the role of each GPU is determined based on data parallelism, pipelined parallelism, and tensor parallelism strategies. Therefore, the first duration of data can be grouped according to the data parallelism strategy, resulting in two groups: the first group includes GPU0, GPU8, GPU16, and GPU24; the second group includes GPU4, GPU12, GPU20, and GPU28. Alternatively, the first duration of data can be grouped according to the pipelined parallelism strategy, resulting in four groups: the first group includes GPU0 and GPU4; the second group includes GPU8 and GPU12; the third group includes GPU16 and GPU20; and the fourth group includes GPU24 and GPU28. Alternatively, the first duration of data can be grouped according to the tensor parallelism strategy, resulting in four groups.

[0057] Furthermore, the first duration data can be grouped according to data parallelism and tensor parallelism strategies. Alternatively, the first duration data can be grouped according to data parallelism and pipelined parallelism strategies. Alternatively, the first duration data can be grouped according to tensor parallelism and pipelined parallelism strategies. Alternatively, the first duration data can be grouped according to data parallelism, tensor parallelism, and pipelined parallelism strategies.

[0058] It should be noted that the more grouping criteria there are, the more detailed the division becomes, and the clearer the role of each processing node (GPU) in distributed training.

[0059] Step 103: Based on the probability density function and first threshold of each of the multiple groups, analyze and process the first duration set of each of the multiple groups to obtain the slow nodes among the multiple processing nodes.

[0060] The probability density function (PDF) is used to characterize the distribution of computation time across different GPU groups. The first threshold refers to the threshold corresponding to each group, used to identify slow nodes within that group. Different groups correspond to different probability density functions and first thresholds; therefore, using different probability density functions and first thresholds for different groups can improve the accuracy of slow node detection.

[0061] In some implementations, the specific way to determine the slow node among multiple processing nodes can be as follows: based on the probability density function of each of the multiple groups, the duration data in the first duration set of the multiple groups are density estimated to obtain the probability density of each processing node in the multiple groups; based on the relationship between the probability density of each processing node in the multiple groups and the corresponding first threshold, the slow node among the multiple groups is determined.

[0062] In some implementations, the specific method for determining slow nodes among multiple processing nodes can be as follows: based on the probability density functions of each of the multiple groups, the duration data in the first duration set of the multiple groups are density estimated to obtain the probability density of each processing node in the multiple groups; based on the relationship between the probability density of each processing node in the multiple groups and the corresponding first threshold, the degree of abnormality of each processing node in the multiple groups is determined; based on the degree of abnormality of each processing node in the multiple groups, it is determined whether each processing node is a slow node.

[0063] In this embodiment, the computational performance of each processing node in distributed model training is evaluated using the first duration data of multiple processing nodes in the current iteration. Grouping the first duration data using a parallel strategy for distributed model training reduces the impact of load imbalance on slow node detection and facilitates use under different parallel strategies. Since the probability density function can characterize the overall distribution of computation time of different groups of GPUs, analyzing the first duration sets of multiple groups using their respective probability density functions and first thresholds allows for evaluation and detection of processing nodes based on the overall distribution of computation time. This method is more accurate than detecting slow nodes based on single characteristics such as the mean, median, or variance of computation time. Therefore, the slow node detection method provided in this embodiment can intelligently and accurately detect slow nodes in distributed model training, improving the efficiency and accuracy of slow node detection. Furthermore, this method can be applied to distributed training scenarios with different numbers of devices, different models, and different training parameters, and can be used under different parallel strategies, demonstrating strong compatibility.

[0064] This disclosure provides a slow node detection method, which can be executed by the processor of a computer device. For example... Figure 3 As shown, the method includes the following steps 301 to 307:

[0065] Step 301: Based on the data parallel strategy in the parallel strategy, divide the dataset into multiple batches of sub-datasets.

[0066] Data parallelism strategy refers to dividing a dataset into multiple subsets to make full use of computing resources.

[0067] In some implementations, the data parallelism strategy may be determined based on at least one of the following: dataset complexity, dependencies between data, hardware performance of processing nodes, and software performance. Dependencies between data are used to characterize the spatial and temporal locality of data. For example, for image data, dependencies refer to the spatial relationships between adjacent pixels; for text data, dependencies refer to the logical relationships between preceding and following text.

[0068] In some implementations, the data can be divided into finer-grained parts based on the complexity of the dataset in order to make full use of the parallelism of computing resources.

[0069] In some implementations, based on the dependencies between data, data with strong dependencies can be grouped onto the same processing node to reduce the communication overhead of data transmission.

[0070] In some implementations, based on the hardware and software performance of the processing nodes, highly complex data can be allocated to processing nodes with better hardware and software performance, while less complex data can be allocated to processing nodes with slightly lower hardware and software performance.

[0071] Step 302: Based on the pipeline parallel strategy in the parallel strategy, the network layer of the large model is divided into multiple network stages.

[0072] The pipelined parallel strategy refers to dividing the network layers of a large model into multiple network stages to achieve parallel processing at different stages.

[0073] In some implementations, pipeline parallelism strategies can be determined based on the network architecture of a large model and the sequential relationships between different network layers. For example, in a video processing pipeline, the output of an image frame in the decoding stage is the input of the image frame in the processing stage, and the output of the image frame in the processing stage is the input of the encoding stage.

[0074] Step 303: Based on the tensor parallel strategy in the parallel strategy, divide the model data of the large model into multiple sub-model data.

[0075] Tensor parallelism involves dividing model data (tensors) into multiple sub-model data (sub-tensors). Model data includes at least model parameters and input data for each network layer. Model data can contain billions or even hundreds of billions of parameters, making it difficult for a single processing node (GPU) to complete training within a reasonable timeframe. Tensor parallelism distributes different parts of the model across multiple GPUs for parallel computation, thereby fully utilizing the computing resources within the cluster.

[0076] In some implementations, tensor-parallel data can be determined based on the characteristics of the model data and the characteristics of the training task. The characteristics of the model data can include its dimension and shape. For example, if the model data is two-dimensional, it can be partitioned by rows or columns; if the model data is three-dimensional, it can be partitioned by channels, height, or width. The characteristics of the training task can include the type of training task and the dependencies between training tasks. For example, if the type of training task is matrix multiplication, it can be partitioned by rows or columns; if the type of training task is convolution, it can be partitioned by channels or spatial dimensions. If the output of one operation is the input of the next operation, the partitioning needs to ensure that the data can be correctly transferred between different processing nodes.

[0077] Step 304: Based on the correlation between the subset datasets, network stages, and sub-model data, allocate the subset datasets, network stages, and sub-model data of the multiple batches to the multiple processing nodes, so that the multiple processing nodes can train the corresponding network stages based on the subset datasets of the corresponding batches and the corresponding sub-model data.

[0078] Based on the relationships between sub-data sets, network stages, and sub-model data, and by allocating multiple batches of sub-data sets, multiple network stages, and multiple sub-model data to multiple processing nodes, the training task to be handled by each processing node can be determined, that is, the role of each processing node in distributed model training is determined.

[0079] Step 305: Obtain the first duration data of each of the multiple processing nodes participating in the distributed model training; the first duration data represents the computation time of the corresponding processing node in the current iteration.

[0080] In some implementations, the specific way to obtain the first duration data can be: to obtain the first duration data of each of the multiple processing nodes through an event timing mechanism.

[0081] The event timing mechanism is used to measure the computation time of the processing node in the current iteration.

[0082] In some implementations, obtaining the first duration data can also be achieved by: obtaining the first duration data of each of the plurality of processing nodes through a programming interface; the first duration data includes: the computation time of the corresponding processing node in processing multiple batches with different data amounts during the forward and backward propagation process.

[0083] The programming interface can be either Compute Unified Device Architecture (CUDA) or PyTorch.

[0084] In some implementations, the slow node analysis module of the monitoring node can be used to obtain the computation time of forward and backward propagation of all GPUs over a time series (such as the latest step time), thus obtaining the first duration data. The first duration data can be represented as: {x0,x1,x2…}, where xi represents the computation time of the GPU numbered i.

[0085] In some implementations, after obtaining the first duration data, the first duration data can be preprocessed, including outlier removal and standardization, to ensure the accuracy and consistency of the data.

[0086] Step 306: Based on the parallel strategy of the distributed model training, the first duration data is grouped to obtain multiple groups of first duration sets; the parallel strategy represents the relationship between the multiple processing nodes in the forward and backward propagation process, as well as the training tasks undertaken by each node.

[0087] The first duration set of multiple groups refers to the final duration set determined after grouping the first duration data.

[0088] In some implementations, step 306 can be achieved through steps 3061 to 3063 as follows:

[0089] Step 3061: Based on the parallel strategy of the distributed model training, determine the roles of the multiple processing nodes in the distributed model training; the roles represent the correlation between the corresponding processing nodes in the forward and backward propagation process, as well as the training tasks they undertake.

[0090] In some implementations, the specific implementation of determining the roles of multiple processing nodes in the distributed model training can refer to steps 301 to 304 above, and will not be repeated here in this embodiment.

[0091] Step 3062: Based on the roles of the multiple processing nodes in the training of the distribution model, the first duration data is grouped to obtain a third duration set of multiple groups.

[0092] The third duration set of multiple groups refers to the duration set initially determined after grouping the first duration data.

[0093] In some implementations, the specific way to determine the third time set of multiple groups can be: grouping the first time data according to at least one of the data parallelism strategy, tensor parallelism strategy and pipeline parallelism strategy to obtain the third time set of multiple groups.

[0094] Step 3063: Cluster and reduce the dimensionality of the third duration sets of the multiple groups respectively to obtain the first duration sets of the multiple groups.

[0095] Clustering is used to detect the homogeneity of duration data within a third duration set. It splits third duration sets with large differences and merges those with high similarity to improve data homogeneity within the duration set and enhance the effectiveness of subsequent processing. Dimensionality reduction is used to reduce data complexity and facilitate subsequent analysis and processing.

[0096] Step 307: Based on the probability density function and first threshold of each of the multiple groups, analyze and process the first duration set of each of the multiple groups to obtain the slow nodes among the multiple processing nodes.

[0097] Here, steps 305 to 307 correspond to steps 101 to 103, respectively. When implementing these steps, the specific implementation methods of steps 101 to 103 can be referred to.

[0098] In some implementations, step 307 can be achieved through the following steps 3071 to 3072:

[0099] Step 3071: Based on the probability density functions of each of the multiple groups, perform density estimation on the duration data in the first duration set of the multiple groups respectively to obtain the probability density of each processing node in the multiple groups.

[0100] The probability density of each processing node refers to the computational score performance of each processing node in the current iteration, that is, the training performance of each processing node in the current iteration.

[0101] In some implementations, a nonparametric density estimation algorithm can be used to obtain the probability density functions of multiple groups based on historical iteration duration data. For example, the nonparametric density estimation algorithm can be a kernel density estimation algorithm.

[0102] For example, the probability density function can be expressed as: Where K is the kernel function, P r (x t ) represents the probability density (calculation score performance) of the corresponding GPU in the current iteration.

[0103] Step 3072: The processing nodes whose probability density reaches the corresponding first threshold in the multiple groups are designated as slow nodes; different groups correspond to different first thresholds.

[0104] In some implementations, the specific way to determine the first threshold is to determine the first threshold for each of the multiple groups based on the abnormal duration data of the multiple groups in the historical iterations.

[0105] In some implementations, for each of the multiple groups, if there is a processing node in each group whose probability density reaches the corresponding first threshold, then the processing node is a slow node; processing nodes whose probability density does not reach the corresponding first threshold are normal nodes.

[0106] In some implementations, the probability density function can be achieved through steps A through C as follows:

[0107] Step A: Obtain the second duration data of the plurality of processing nodes; the second duration data represents the computation time of the plurality of processing nodes in the historical iterations before the current iteration.

[0108] The second duration data refers to the computation time consumed by multiple processing nodes in historical iterations. The second duration data can include the computation time consumed by processing nodes during forward and backward propagation in historical iterations.

[0109] In some implementations, the specific way to obtain the second duration data can be: by using an event timing mechanism to obtain the computation time of each of the multiple processing nodes in the historical iteration, and thus obtain the second duration data; or by using a programming interface to obtain the computation time of each of the multiple processing nodes in the historical iteration, and thus obtain the second duration data.

[0110] Step B: Based on the parallel strategy, the second duration data is grouped to obtain multiple groups of second duration sets.

[0111] The second duration set of multiple groups refers to the duration set obtained after grouping the second duration data.

[0112] It should be noted that the operation of "grouping the second duration data based on the parallel strategy to obtain multiple groups of second duration sets" is similar to the operation of "grouping the first duration data based on the parallel strategy of the distributed model training to obtain multiple groups of first duration sets" in step 306 above. When implementing, you can refer to the specific implementation of step 306 above.

[0113] In some implementations, the specific way to determine the second duration set of multiple groups can be: based on the roles of multiple processing nodes in the training of the distributed model, the second duration data is grouped to obtain the fourth duration set of multiple groups; the fourth duration sets of multiple groups are clustered and dimensionality reduced respectively to obtain the second duration set of multiple groups.

[0114] Step C: Perform function fitting on the duration data in the second duration set of the multiple groups respectively to obtain the probability density function of each of the multiple groups.

[0115] Function fitting refers to using mathematical functions (such as linear functions, exponential functions, etc.) to characterize the inherent patterns of the second duration data of each group.

[0116] In some implementations, a nonparametric density estimation algorithm can be used to fit a function to the duration data in the second duration set of multiple groups, obtaining the probability density function for each group. For example, the nonparametric density estimation algorithm can be a kernel density estimation algorithm.

[0117] Specifically, a nonparametric density estimation algorithm is used to estimate the probability density function of each group by taking the second duration data of each group as the dependent variable and the number of iterations as the independent variable.

[0118] In this embodiment, the probability density functions of multiple groups are fitted to the second duration data of multiple processing nodes in historical iterations. This is to make the probability density functions more suitable for the current distributed training scenario, thereby improving the accuracy of subsequent slow node detection. The computational score performance of each processing node in distributed model training is evaluated using the first duration data of multiple processing nodes in the current iteration. Grouping the first duration data using the parallel strategy of distributed model training reduces the impact of load imbalance on slow node detection and facilitates use under different parallel strategies. Analyzing the first duration sets of multiple groups using their respective probability density functions and first thresholds allows for evaluation and detection of processing nodes based on the overall distribution of computational time. Compared to detection based on single characteristics such as the mean, median, or variance of computational time, the accuracy of slow node detection is higher. Therefore, the slow node detection method provided in this embodiment can intelligently and accurately detect slow nodes in distributed model training, improving the efficiency and accuracy of slow node detection. Furthermore, this method can be applied to distributed training scenarios with different numbers of devices, different models, and different training parameters, and can also be used under different parallel strategies, demonstrating strong compatibility.

[0119] Based on the foregoing embodiments, the slow node detection method provided in this disclosure further includes the following steps 308 to 310:

[0120] Step 308: Determine the frequency of occurrence of the slow node in the number of historical iterations closest to the current iteration.

[0121] The target number can be determined based on business needs and practical experience. For example, the target number could be 5.

[0122] In some implementations, a fixed time interval (the number of historical iterations most recently related to the target) can be used via a sliding window method.

[0123] Step 309: If the frequency of occurrence of the slow node reaches the second threshold, the slow node is identified as a first-level slow node.

[0124] The second threshold is used to determine the level of slow nodes. The level of slow nodes can include a first level and a second level. The first level is for long-term slow nodes, and the second level is for short-term slow nodes. In addition, the level of slow nodes can be set according to actual needs, and is not limited to the first level and the second level.

[0125] In some implementations, a slow node is identified as a first-level slow node if the frequency of occurrence of a slow node in the five most recent historical iterations is 5.

[0126] Step 310: If the frequency of occurrence of the slow node does not reach the second threshold, the slow node is identified as a second-level slow node.

[0127] In this embodiment, the frequency of occurrence of slow nodes in the most recent target number of historical iterations is statistically analyzed, and the level of slow nodes is determined based on the relationship between the frequency of occurrence of slow nodes and a second threshold. This level highlights slow nodes, making it easier to identify problems in a targeted manner and decide whether to repair or replace them.

[0128] Based on the foregoing embodiments, the slow node detection method provided in this disclosure further includes the following steps 311 to 312:

[0129] Step 311: Mark the first duration data of the nodes other than the slow node among the multiple processing nodes as historical data and store them in the time series database.

[0130] Nodes other than slow nodes refer to normal nodes. Marking the first duration data of nodes other than slow nodes as historical data and storing it in the time series database is to facilitate subsequent updates to the probability density function.

[0131] Step 312: Retrieve the first duration data of the other nodes in the time series database and update the probability density function of the multiple groups.

[0132] In some implementations, a nonparametric density estimation algorithm can be used to update the probability density function of multiple groups based on the first duration data of other nodes.

[0133] In this embodiment of the disclosure, the first duration data of nodes other than slow nodes are marked as historical data and stored in the time series database. This is to facilitate the updating of the probability density function, so that the probability density function is always in the latest state, and then the slow node detection is performed based on the latest probability density function, making the detected slow nodes more accurate.

[0134] The following describes the application of the slow node detection method provided in this embodiment in a real-world scenario.

[0135] The slow node detection method proposed in this disclosure evaluates the training performance of GPU nodes in real time and quickly locates slow nodes without affecting the original training performance. It aims to rapidly identify slow computational nodes in large-scale model training with tens of thousands of GPUs, solving problems such as low detection accuracy, poor real-time performance, and poor scalability in existing technologies. It not only effectively addresses the difficulties of manual screening at such scales but also adapts to different parallel strategies and training configurations. This contributes to further solutions to the slow node problem, optimizes training performance, and improves cluster stability.

[0136] This disclosure specifically relates to a method for training large models in deep learning, which utilizes unsupervised machine learning methods, especially nonparametric density estimation techniques, to evaluate the computational performance of multiple processing nodes during distributed training, aiming to quickly identify slow nodes in distributed training in large-scale clusters.

[0137] Specifically, it includes:

[0138] 1. Involves large-scale cluster training: kilocalories or tens of thousands of calories.

[0139] 2. Involves distributed training technology: multidimensional hybrid parallel computing, including a combination of data parallelism, tensor parallelism, and pipeline parallelism.

[0140] 3. Involves slow node detection during training: Slow node detection is performed in distributed training tasks, especially referring to laggards that drag down the entire training task.

[0141] 4. Anomaly data analysis and detection: This involves the processing and analysis of data, especially the detection of anomalies and outliers.

[0142] 5. Monitoring of training tasks in large-scale GPU clusters: This involves real-time monitoring and analysis of GPU clusters executing training tasks.

[0143] 6. Optimization of AI Training Reliability and Efficiency: Identifying and addressing slow training nodes helps improve training efficiency and stability. This facilitates efficient and reliable training of AI models, and is relevant to practical applications, pre-training, and real-world applications of AI models.

[0144] like Figure 4 As shown, in distributed training, a portion of the model or data is allocated to the GPU for processing. Model training is a continuous iterative process of forward and backward computation of the neural network. The process of locating slow nodes may include the following steps 401 to 404:

[0145] Step 401: Obtain the computation time of the GPU in the forward and backward propagation of the model in the distributed training task.

[0146] The calculation duration corresponds to the first duration data mentioned above.

[0147] Step 402: Preprocess the computation time and group it according to the role of the GPU in the distributed training task to obtain the grouped time.

[0148] Step 403: Process the grouping duration using the kernel density estimation algorithm to obtain the probability density function for each group.

[0149] Step 404: Evaluate the GPU computing performance based on the probability density function, and identify GPUs that deviate from the probability density as slow nodes.

[0150] In some implementations, the computation time of all GPUs involved in training during forward and backward propagation over a time series is sampled. The computation time is preprocessed, and grouped according to the GPU's parallel strategy. For example, in a data parallel strategy, GPUs are grouped into multiple computational units, with each group processing a different subset of the dataset. Non-parametric density estimation methods from machine learning, such as kernel density estimation algorithms, are applied to the grouped data to estimate the probability density function of each group's computation time. Then, the computation time of forward and backward propagation for the next training time step is detected, and the probability density is calculated using the probability density function of the corresponding group. This probability density serves as a measure of the GPU's computational performance during training, and probability density scores that deviate from the distribution are identified as outliers, i.e., slow nodes are detected.

[0151] The slow node detection device of this disclosure mainly consists of the following modules, such as... Figure 5 As shown:

[0152] 1. Worker Nodes in Distributed Training: Worker nodes in a cluster that participate in distributed training tasks correspond to a single machine, including hardware devices (each hardware device includes at least one GPU (processing node)), underlying drivers, distributed training framework, training tasks, etc. For large-scale distributed training tasks, it is usually necessary to schedule hundreds or thousands of worker nodes to execute the training tasks.

[0153] 2. Monitoring Nodes for Distributed Training: These are monitoring nodes in the cluster used to monitor training tasks. Each node is a single machine and may not have a GPU. They are primarily used to monitor worker nodes. They support using a single GPU within a worker node as the smallest monitoring unit, collecting training status, intermediate data, and training parameters, and performing real-time slow node detection and persistent data transfer.

[0154] a) Training process data acquisition: Acquire the computation time of forward and backward propagation of the GPU during a time sequence or iteration in the training process, distributed training configuration, or other training process data.

[0155] b) Real-time data analysis: Data analysis is performed based on the computation time of the GPU's forward and backward propagation during training. This considers not only the data characteristics but also aims to fit the data distribution. Data analysis includes: preprocessing the computation time, grouping, and probability density estimation.

[0156] c) Slow node detection and localization: Based on the probability distribution, an anomaly threshold (corresponding to the second threshold mentioned above) is set to detect the current GPU computing time. Once the anomaly threshold is exceeded, it is determined to be a slow node, and the specific parallel role and worker node are quickly located according to its data source.

[0157] d) Message reporting and anomaly handling: Once a slow node is located, it should be reported immediately. Based on the sampling duration, it can be further divided into short-term slow nodes and long-term slow nodes, allowing for targeted troubleshooting and decisions on whether to repair or replace the slow node.

[0158] like Figure 6 As shown, the specific implementation of this disclosure includes enabling distributed training, monitoring and collecting data, data preprocessing, slow node detection, and reporting of abnormal results.

[0159] 1. Enable distributed training.

[0160] First, environment preparation is required, including determining the number of GPUs involved in training and setting the parallel strategy. For example, using a dual-machine, 16-GPU training architecture with the Megatron-LM framework and a 3D hybrid parallel strategy, the parallel parameters are set as follows: tp_size = 2, pp_size = 4, dp_size = 2. DP divides the dataset into mini-batches to increase the training scale of the model. PP divides the model into different stages by layer, assigning each stage to one or more GPUs for processing, and then divides the dataset into micro-batches for computation. TP evenly distributes the computational and storage load of the model among the devices within the node. The 16 GPUs are grouped according to their different parallel roles as follows, with the 3D parallel grouping shown below, where gi represents the GPU numbered i:

[0161] [[[g0,g1],[g4,,g5],[g8,g9],[g12,g13]],[[g2,g3],[g6,g7],[g10,g11],[g14,g15]]].

[0162] After model preparation, dataset loading, and training parameter configuration, large-scale distributed training can be started.

[0163] 2. Monitoring and data collection.

[0164] Collect computation time data of the GPU during task execution. This can be obtained through GPU event timing or programming interfaces (such as CUDA or PyTorch). The data should cover the computation time of forward and backward propagation for all GPUs processing micro-batch or mini-batch tasks. Enable the monitoring node for the training task to continuously monitor the training process and collect GPU computation time data into a time-series database.

[0165] 3. Data preprocessing.

[0166] The slow node analysis module of the monitoring node is used to obtain the computation time data of forward and backward propagation of all GPUs within a time series (such as the latest step time): {x0,x1,x2…}, where xi represents the computation time data sampled from GPU number i. First, data preprocessing is performed, such as outlier removal and standardization, to ensure data accuracy and consistency. Then, the GPU time data is grouped and clustered, for example, based on the GPU's three-dimensional parallel strategy. The data grouping can be represented as: [[[x0,x1],[x4,x5],[x8,x9],[x12,x13]],[[x2,x3],[x6,x7],[x10,x11],[x14,x15]]].

[0167] The grouped data can be further clustered and dimensionality reduced through the data center to form the final grouped data.

[0168] 4. Slow node detection.

[0169] For the grouped data, the probability density function is first obtained using historical data (such as previous iterations) through a kernel density estimation algorithm. Slow node detection is then performed on the computation time of the latest time series (such as the current iteration). The probability density is calculated based on the probability density function of the group to which the detected data belongs. The formula for the probability density function is as follows: Where K is the kernel function, P r (x t As the computational score of the corresponding GPU at the latest training moment, it is determined whether the score exceeds the abnormal threshold. If it does, a slow node is found. If it does not exceed the threshold, the data can be marked as historical data and stored in the time series database for probability density function updates and subsequent evaluation calculations.

[0170] 5. Reporting of abnormal results.

[0171] After locating a slow node, the detection results are recorded and reported in a hierarchical manner. A fixed time interval (such as 5 iterations) is used to determine whether the GPU has a continuous occurrence of slow node events. If slow node events occur continuously, the GPU is reported as a long-term slow node; otherwise, it is reported as a short-term slow node. This allows for targeted troubleshooting and decision-making on whether to repair or replace the slow node.

[0172] like Figure 6 As shown, the overall process of the slow node detection method proposed in this embodiment includes the following steps 601 to 610:

[0173] Step 601: Start large-scale distributed training.

[0174] Step 602: Enable training task monitoring.

[0175] Step 603: Collect training process data into the database.

[0176] Step 604: Obtain the historical computation time and current computation time of the GPU during training.

[0177] Step 605: Preprocess the computation time data and group it according to parallel roles.

[0178] Step 606: Estimate the probability density function of the historical group duration using the kernel density estimation algorithm.

[0179] Step 607: Calculate the probability density of the current calculation time.

[0180] Step 608: Determine whether the abnormal threshold is exceeded.

[0181] If the abnormal threshold is exceeded, proceed to step 609; if the abnormal threshold is not exceeded, proceed to step 610.

[0182] Step 609: Discover slow nodes.

[0183] Step 610: Mark the current calculation duration as historical data.

[0184] The probability density function is updated using historical data.

[0185] The slow node detection method proposed in this disclosure can evaluate node training performance in real time and efficiently, and quickly discover and locate slow nodes. It not only effectively solves the difficulties of manual screening at high-scale training environments, but also adapts to different model training configurations and hybrid parallel strategies. The proposed slow node detection device and equipment, from task monitoring, data capture and persistence, real-time data analysis, slow node discovery and location, and anomaly reporting and handling, not only can evaluate training performance in real time, but also ensures efficient and stable distributed training.

[0186] This method characterizes GPU computation time data using a kernel density estimation algorithm. However, kernel density estimation is just one type of unsupervised method in machine learning. For the aforementioned scheme, other unsupervised clustering methods can also identify slow nodes, but kernel density estimation offers better distribution fitting and computational optimization advantages. Therefore, different algorithm implementations will have differences in accuracy and efficiency. Different kernel functions can be chosen for kernel density estimation, such as Gaussian kernels and polynomial kernels; this method is not limited to the choice of kernel function.

[0187] The intended protection point of this disclosure is: focusing on GPU computing performance in distributed training, formulating strategies to automatically discover slow GPU computing nodes during training in real time, quickly locating and reporting them. The computing performance of GPUs in parallel training is evaluated by grouping the forward and backward computation times of GPUs during training according to different parallel roles, and calculating their performance scores using a probability density function. Slow nodes are then identified using probability density and threshold values.

[0188] The technical effects achievable by the embodiments disclosed herein include: 1. Effectiveness: This method, for the first time, proposes a kernel density estimation approach to identify slow nodes during training. Considering the parallel strategy of distributed training, it comprehensively evaluates node training performance and quickly identifies and locates slow nodes. 2. Efficiency: It automatically evaluates GPU computing performance in real time using a density function, effectively solving the difficulty of manual screening at high GPU scales. Furthermore, once the density function is calculated, it can be updated periodically. During training, each iteration only requires judging a threshold to determine whether a node is slow, resulting in minimal computational overhead. 3. Compatibility: This method can be applied to different numbers of devices, models, and training parameter settings, and can be used under different distributed training strategies, demonstrating good strategy compatibility.

[0189] Based on the foregoing embodiments, this disclosure provides a slow node detection device, which includes various units and modules included in each unit, and can be implemented by a processor in a computer device; of course, it can also be implemented by specific logic circuits; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0190] Figure 7 This is a schematic diagram of the composition structure of a slow node detection device provided in an embodiment of this disclosure, as shown below. Figure 7As shown, the slow node detection device 700 includes: an acquisition module 710, a grouping module 720, and a processing module 730, wherein:

[0191] The acquisition module 710 is configured to acquire first duration data for each of the multiple processing nodes participating in the distributed model training; the first duration data represents the computation time of the corresponding processing node in the current iteration.

[0192] The grouping module 720 is configured to group the first duration data based on the parallel strategy trained by the distributed model to obtain a first duration set of multiple groups; the parallel strategy represents the relationship between the multiple processing nodes in the forward and backward propagation process, as well as the training tasks undertaken by each node.

[0193] The processing module 730 is configured to analyze and process the first duration set of the multiple groups based on the probability density function and the first threshold of each group, and obtain the slow node among the multiple processing nodes.

[0194] In some embodiments, the processing module 730 is further configured to: obtain second duration data of the plurality of processing nodes; the second duration data characterizes the computation time of the corresponding processing node in the historical iterations before the current iteration; group the second duration data based on the parallel strategy to obtain a plurality of second duration sets; and perform function fitting on the duration data in the plurality of second duration sets to obtain the probability density functions of each of the plurality of groups.

[0195] In some embodiments, the processing module 730 is further configured to: divide the dataset into multiple batches of subsets based on the data parallelism strategy in the parallelism strategy; divide the network layers of the large model into multiple network stages based on the pipeline parallelism strategy in the parallelism strategy; divide the model data of the large model into multiple sub-model data based on the tensor parallelism strategy in the parallelism strategy; and allocate the multiple batches of subsets, the multiple network stages, and the multiple sub-model data to the multiple processing nodes based on the correlation between the subsets, network stages, and sub-model data, so that the multiple processing nodes respectively train the corresponding network stages based on the corresponding batches of subsets and the corresponding sub-model data.

[0196] In some embodiments, the acquisition module 710 is further configured to: obtain first duration data for each of the plurality of processing nodes through an event timing mechanism; or, obtain first duration data for each of the plurality of processing nodes through a programming interface; the first duration data includes: the computation time of each corresponding processing node in processing multiple batches with different data amounts during forward and backward propagation.

[0197] In some embodiments, the grouping module 720 is further configured to: determine the roles of the plurality of processing nodes in the distributed model training based on the parallel strategy of the distributed model training; the roles represent the correlation between the corresponding processing nodes in the forward and backward propagation process and the training tasks they undertake; group the first duration data based on the roles of the plurality of processing nodes in the distributed model training to obtain a third duration set of multiple groups; and perform clustering and dimensionality reduction on the third duration sets of multiple groups respectively to obtain a first duration set of multiple groups.

[0198] In some embodiments, the processing module 730 is further configured to: perform density estimation on the duration data in the first duration set of the plurality of groups based on the probability density function of each of the plurality of groups, to obtain the probability density of each processing node in the plurality of groups; and designate the processing nodes in the plurality of groups whose probability density reaches the corresponding first threshold as slow nodes; and designate different groups as different first thresholds.

[0199] In some embodiments, the processing module 730 is further configured to: determine the frequency of occurrence of the slow node in the target number of historical iterations most recent to the current iteration; if the frequency of occurrence of the slow node reaches a second threshold, determine the slow node as a first-level slow node; if the frequency of occurrence of the slow node does not reach the second threshold, determine the slow node as a second-level slow node.

[0200] In some embodiments, the processing module 730 is further configured to: mark the first duration data of the other nodes among the plurality of processing nodes, excluding the slow node, as historical data and store them in a time series database; retrieve the first duration data of the other nodes in the time series database and update the probability density function of the plurality of groups.

[0201] The descriptions of the apparatus embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. In some embodiments, the functions or modules included in the apparatus provided in this disclosure can be used to perform the methods described in the method embodiments above. For technical details not disclosed in the apparatus embodiments of this disclosure, please refer to the descriptions of the method embodiments of this disclosure for understanding.

[0202] It should be noted that, in the embodiments of this disclosure, if the above-described slow node detection method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this disclosure, or the part that contributes to related technologies, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this disclosure are not limited to any specific hardware, software, or firmware, or any combination of hardware, software, and firmware.

[0203] This disclosure provides a computer device including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements some or all of the steps in the above-described method.

[0204] This disclosure provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements some or all of the steps in the above-described method. The computer-readable storage medium may be transient or non-transient.

[0205] This disclosure provides a computer program including computer-readable code, wherein when the computer-readable code is executed in a computer device, a processor in the computer device performs some or all of the steps in the above-described method.

[0206] This disclosure provides a computer program product, such as... Figure 9 As shown, the computer program product includes a non-transitory computer-readable storage medium storing a computer program, which, when read and executed by a computer, implements some or all of the steps in the above-described method. This computer program product can be implemented specifically through hardware, software, or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium; in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0207] It should be noted that the descriptions of the various embodiments above tend to emphasize the differences between them, while their similarities or commonalities can be referenced interchangeably. The descriptions of the above embodiments of the device, storage medium, computer program, and computer program product are similar to the descriptions of the above method embodiments and have similar beneficial effects. For technical details not disclosed in the embodiments of the device, storage medium, computer program, and computer program product of this disclosure, please refer to the descriptions of the method embodiments of this disclosure for understanding.

[0208] It should be noted that, Figure 8 This is a schematic diagram of a hardware entity of a computer device in an embodiment of this disclosure, such as... Figure 8 As shown, the hardware entity of the computer device 800 includes: a processor 801, a communication interface 802, and a memory 803, wherein:

[0209] Processor 801 typically controls the overall operation of computer device 800.

[0210] The communication interface 802 enables computer devices to communicate with other terminals or servers over a network.

[0211] The memory 803 is configured to store instructions and applications executable by the processor 801, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data, and video communication data) in the processor 801 and various modules in the computer device 800. It can be implemented using flash memory or random access memory (RAM). Data transfer between the processor 801, the communication interface 802, and the memory 803 can be performed via bus 804.

[0212] It should be understood that the phrase "an embodiment" or "one embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this disclosure. Therefore, "in one embodiment" or "one embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this disclosure, the sequence numbers of the above steps / processes do not imply a sequential order of execution; the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this disclosure. The sequence numbers of the above embodiments of this disclosure are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0213] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0214] In the several embodiments provided in this disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components may be combined, or integrated into another system, or some features may be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0215] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0216] In addition, each functional unit in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0217] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0218] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, or the part that contributes to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, magnetic disks, or optical disks.

[0219] The above description is merely an embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A slow node detection method, characterized in that, The slow node detection method includes: Obtain the first duration data for each of the multiple processing nodes participating in the distributed model training; the first duration data represents the computation time of the corresponding processing node in the current iteration; Based on the parallel strategy for training the distributed model, the first duration data is grouped to obtain multiple groups of first duration sets; the parallel strategy represents the relationship between the multiple processing nodes in the forward and backward propagation process, as well as the training tasks they undertake; the parallel strategy includes at least one of the following: data parallel strategy, pipeline parallel strategy, and tensor parallel strategy. Based on the probability density function and first threshold of each of the multiple groups, the first duration set of the multiple groups is analyzed and processed to obtain the slow nodes among the multiple processing nodes.

2. The slow node detection method according to claim 1, characterized in that, The slow node detection method further includes: Obtain the second duration data for each of the plurality of processing nodes; the second duration data represents the computation time of the corresponding processing node in the historical iterations prior to the current iteration; Based on the parallel strategy, the second duration data is grouped to obtain multiple groups of second duration sets; The probability density functions of each of the multiple groups are obtained by performing function fitting on the duration data in the second duration set of each group.

3. The slow node detection method according to claim 1, characterized in that, The slow node detection method further includes: Based on the data parallelism strategy in the aforementioned parallel strategy, the dataset is divided into multiple batches of subsets. Based on the pipeline parallel strategy in the above parallel strategy, the network layers of the large model are divided into multiple network stages; Based on the tensor parallel strategy in the parallel strategy, the model data of the large model is divided into multiple sub-model data. Based on the correlation between the sub-data sets, network stages, and sub-model data, the multiple batches of sub-data sets, the multiple network stages, and the multiple sub-model data are allocated to the multiple processing nodes, so that the multiple processing nodes can respectively train the corresponding network stages based on the corresponding batch of sub-data sets and the corresponding sub-model data.

4. The slow node detection method according to any one of claims 1 to 3, characterized in that, The process of obtaining the first duration data for each of the multiple processing nodes participating in the distributed model training includes: The first duration data for each of the multiple processing nodes is obtained through an event timing mechanism; or, The first duration data of each of the multiple processing nodes is obtained through the programming interface; the first duration data includes the computation time of the corresponding processing node in processing multiple batches with different data volumes during the forward and backward propagation process.

5. The slow node detection method according to any one of claims 1 to 3, characterized in that, The parallel strategy based on the distributed model training grouped the first duration data to obtain multiple groups of first duration sets, including: Based on the parallel strategy for training the distributed model, the roles of the multiple processing nodes in the training of the distributed model are determined; the roles represent the relationships between the corresponding processing nodes in the forward and backward propagation process, as well as the training tasks they undertake. Based on the roles of the multiple processing nodes in the training of the distribution model, the first duration data is grouped to obtain a third duration set of multiple groups. Clustering and dimensionality reduction are performed on the third duration sets of the multiple groups respectively to obtain the first duration sets of the multiple groups.

6. The slow node detection method according to any one of claims 1 to 3, characterized in that, The step involves analyzing and processing the first duration sets of the multiple groups based on their respective probability density functions and first thresholds to obtain slow nodes among the multiple processing nodes, including: Based on the probability density functions of each of the multiple groups, the density of the duration data in the first duration set of the multiple groups is estimated to obtain the probability density of each processing node in the multiple groups. Processing nodes whose probability density reaches the corresponding first threshold in the multiple groups are designated as slow nodes; different groups correspond to different first thresholds.

7. The slow node detection method according to any one of claims 1 to 3, characterized in that, The slow node detection method further includes: Determine the frequency of occurrence of the slow node in the number of historical iterations closest to the current iteration; If the frequency of occurrence of the slow node reaches the second threshold, the slow node is identified as a first-level slow node. If the frequency of occurrence of the slow node does not reach the second threshold, the slow node is identified as a second-level slow node.

8. The slow node detection method according to any one of claims 1 to 3, characterized in that, The slow node detection method further includes: The first duration data of the nodes other than the slow node among the plurality of processing nodes are marked as historical data and stored in the time series database; The first duration data of the other nodes in the time series database are retrieved, and the probability density function of the multiple groups is updated.

9. A slow node detection device, characterized in that, The slow node detection device includes: The acquisition module is configured to acquire the first duration data of each of the multiple processing nodes participating in the distributed model training; the first duration data represents the computation time of the corresponding processing node in the current iteration; The grouping module is configured to group the first duration data based on the parallel strategy trained by the distributed model, thereby obtaining a first duration set of multiple groups; the parallel strategy represents the relationship between the multiple processing nodes in the forward and backward propagation process, as well as the training tasks undertaken by each node; the parallel strategy includes at least one of the following: data parallel strategy, pipeline parallel strategy, and tensor parallel strategy. The processing module is configured to analyze and process the first duration set of the multiple groups based on the probability density function and the first threshold of each group, and obtain the slow nodes among the multiple processing nodes.

10. A computer device comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 8.

12. A computer program product, characterized in that, The computer program product includes a non-transitory computer-readable storage medium storing a computer program, which, when read and executed by a computer, implements the steps of the method according to any one of claims 1 to 8.

Images