Node detection method, apparatus, device, storage medium, and computer program product

Abstract

The application provides a node detection method, device, equipment, storage medium and computer program product; the method comprises: for each physical node used for executing a distributed training task, acquiring a logical comparison group corresponding to the physical node, the logical comparison group comprising a plurality of first training nodes; for each training step, collecting the training time consumption of each first training node at the training step, and calculating a first average value based on the training time consumption at the training step; based on the training time consumption of each first training node at the training step, screening at least one second training node from the plurality of first training nodes; based on the training time consumption of the at least one second training node at the training step, calculating a second average value, and performing a difference operation on the first average value and the second average value to obtain a difference index of the training step; based on the difference indexes of a plurality of training steps, detecting the type of the second training node to obtain the node type of the second training node.

Description

Technical Field

[0001] This application relates to data processing technology, and more particularly to a node detection method, apparatus, device, storage medium, and computer program product. Background Technology

[0002] In large-scale distributed learning and training scenarios, the training tasks of the training cluster are distributed to multiple physical nodes for execution. Multiple training nodes in each physical node execute training tasks in parallel. The training nodes need to frequently synchronize model parameters through communication mechanisms to ensure effective distributed learning. If there are abnormal nodes such as slow training nodes, the training speed of the entire training cluster will be limited by the slowest training node due to the synchronization waiting mechanism of distributed learning, thereby reducing the overall training efficiency.

[0003] In related technologies, the detection of abnormal nodes is achieved by only monitoring low-level indicators such as the network status of training nodes, which has low accuracy in detecting abnormal nodes. Summary of the Invention

[0004] This application provides a node detection method, apparatus, device, storage medium, and computer program product, which can improve the accuracy of node detection.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a node detection method, the method comprising:

[0007] For each physical node used to perform the distributed training task, a logical comparison group corresponding to the physical node is obtained, wherein the logical comparison group includes multiple first training nodes;

[0008] For each training step, the training time of each first training node in the training step is collected, and a first average value is calculated based on the training time of the plurality of first training nodes in the training step.

[0009] Based on the training time of each first training node in the training step, at least one second training node is selected from the plurality of first training nodes.

[0010] Based on the training time of the at least one second training node in the training step, a second average value is calculated, and the difference between the first average value and the second average value is calculated to obtain the difference index corresponding to the training step.

[0011] Based on the difference indicators corresponding to the multiple training steps, type detection is performed on the at least one second training node to obtain the node type of the at least one second training node.

[0012] This application provides a node detection device, including:

[0013] The acquisition module is used to acquire the logical comparison group corresponding to each physical node for performing the distributed training task, wherein the logical comparison group includes multiple first training nodes.

[0014] The acquisition module is used to acquire the training time of each first training node in the training step for each training step, and to calculate a first average value based on the training time of the plurality of first training nodes in the training step.

[0015] A filtering module is used to filter at least one second training node from the plurality of first training nodes based on the training time of each first training node in the training step.

[0016] The calculation module is used to calculate a second average value based on the training time of the at least one second training node in the training step, and to perform a difference operation on the first average value and the second average value to obtain the difference index corresponding to the training step.

[0017] The detection module is used to perform type detection on the at least one second training node based on the difference indicators corresponding to multiple training steps, so as to obtain the node type of the at least one second training node.

[0018] This application provides an electronic device, the electronic device comprising:

[0019] Memory is used to store executable instructions or computer programs.

[0020] The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the node detection method provided in the embodiments of this application.

[0021] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the node detection method provided in this application when executed by a processor.

[0022] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the node detection method provided in this application.

[0023] The embodiments of this application have the following beneficial effects:

[0024] First, for each physical node used to execute the distributed training task, a logical comparison group corresponding to the physical node is obtained. The training time of each first training node in the training step is collected, and a first average value is calculated based on the training time of multiple first training nodes in the training step, so that the first average value can represent the overall time consumption level of the current distributed training. Next, at least one second training node is selected based on the training time, and a second average value is calculated based on the training time of at least one second training node in the training step. The difference between the first average value and the second average value is calculated to obtain the difference index corresponding to each training step. By selecting second training nodes from the first training nodes, the deviation of the time consumption level of the second training node from the overall average value (difference index) is calculated. Finally, based on the difference index corresponding to multiple training steps, the node type of at least one second training node is obtained by type detection. By using the difference index of multiple training steps, second training nodes with large deviations in time consumption can be effectively identified, thereby improving the accuracy of node detection. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the node detection system architecture provided in the embodiments of this application;

[0026] Figure 2 This is a schematic diagram of the structure of the device provided in the embodiments of this application;

[0027] Figure 3 This is a first flowchart illustrating the node detection method provided in this application embodiment;

[0028] Figure 4 This is a schematic diagram of the second process of the node detection method provided in the embodiments of this application;

[0029] Figure 5 This is a schematic diagram of the third process of the node detection method provided in the embodiments of this application;

[0030] Figure 6 This is a schematic diagram of the fourth process of the node detection method provided in the embodiments of this application;

[0031] Figure 7 This is a first flowchart illustrating the application scenario of the node detection method provided in this application embodiment;

[0032] Figure 8 This is a second flowchart illustrating the application scenario of the node detection method provided in this application embodiment;

[0033] Figure 9 This is a schematic diagram of the third process in an application scenario of the node detection method provided in the embodiments of this application;

[0034] Figure 10This is a schematic diagram of the fourth process in an application scenario of the node detection method provided in this application embodiment;

[0035] Figure 11 This is a schematic diagram of the fifth process in an application scenario of the node detection method provided in this application embodiment.

[0036] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0038] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0039] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0040] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0041] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0042] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0043] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0044] 1) Distributed deep learning training refers to the process of distributing the training task of a deep learning model across multiple computing nodes for parallel execution.

[0045] 2) A straggler is a node that, during distributed training, executes at a significantly slower pace than other computational units involved in the training process due to various reasons. The presence of stragglers slows down the entire distributed training process.

[0046] 3) Synchronization Barrier Waiting Scenario refers to the situation in distributed training using synchronous mode where all computing units must wait for the slowest one to complete its current step before proceeding to the next step. Slow nodes can cause this waiting scenario.

[0047] 4) Graphics Processing Unit (GPU) refers to a processor that supports massively parallel computing and is used to accelerate the execution of training computation tasks.

[0048] 5) Video RAM (VRAM) refers to the amount of memory required on the graphics processing unit to store model parameters, intermediate calculation results, and training data.

[0049] 6) AllReduce refers to a collective communication operation used to reduce data on multiple computing nodes (e.g., summation, averaging) and synchronize the results to all nodes.

[0050] 7) A physical computing node (Node) refers to a physical computing node or equivalent computing unit used to carry out distributed training tasks. It contains one or more graphics processing unit (GPU) accelerator cards and runs multiple training ranks.

[0051] 8) The training rank node identifier (rank_node_id) is a logical identifier used to represent the physical computing node to which a certain training rank belongs. It is used to establish the correspondence between training ranks and physical computing nodes to support node-based intra-group comparison analysis.

[0052] 9) Training Rank refers to the logical number used to identify the computational entity participating in training in distributed training. A training rank is bound to a specific graphics processing unit (GPU).

[0053] 10) A training step (Step) refers to a training iteration process, including the forward computation and backward computation phases.

[0054] 11) The total time taken for a training step (step_time) refers to the total execution time of a single training rank iteration from start to finish.

[0055] 12) Forward computation time refers to the time consumed by the model in performing the forward propagation phase during one training iteration.

[0056] 13) Backward computation time refers to the time consumed by the model in performing the backpropagation phase during one training iteration.

[0057] 14) Performance analysis tool (Profiler) refers to a tool used to perform time statistics and performance analysis on the program execution process at the training framework level. It can collect logical time consumption information at the training step level, the computation stage, or the collective communication stage.

[0058] The technical limitations of anomaly detection methods in distributed learning and training are as follows:

[0059] 1) Inaccurate identification, prone to misjudgment. The detection method uses a global threshold for cross-physical node comparison. However, due to differences in GPU models, PCIe bandwidth fluctuations, or different network topology locations (across switches) between different physical nodes, the baseline performance of the nodes is inherently inconsistent. Directly comparing across nodes will misclassify normal slow nodes due to low hardware configuration as abnormally slow nodes, i.e., the identification of slow nodes is inaccurate.

[0060] 2) Inability to analyze specific causes of anomalies. Related technologies only focus on the total time (step_time) of a node, issuing an alarm once a timeout occurs. However, they cannot distinguish whether the timeout is caused by a decrease in computing power or communication congestion, making targeted optimization impossible. In other words, these technologies cannot accurately analyze the reasons for a slow node.

[0061] This application provides a node detection method, apparatus, device, computer-readable storage medium, and computer program product, which can improve the accuracy of node detection. The following describes exemplary applications of the electronic devices provided in this application. The devices provided in this application can be implemented as various types of terminals such as laptops, tablets, desktop computers, set-top boxes, smartphones, smart speakers, smartwatches, smart TVs, and vehicle terminals, or as servers. Exemplary applications when the device is implemented as a terminal or server will be described below.

[0062] See Figure 1 , Figure 1 This is a schematic diagram of the architecture of the node detection system provided in this application embodiment. To support a node detection application, the node detection system 100 includes at least a database 500, a terminal 600, a network 300, and a server 200. The terminal 600 is connected to the server 200 through the network 300, which can be a wide area network (WAN), a local area network (LAN), or a combination of both.

[0063] In some embodiments, server 200 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminals and servers can be connected directly or indirectly via wired or wireless communication, which is not limited in this embodiment.

[0064] In some embodiments, the present application embodiments can be implemented by the terminal 600 alone. For example, the logical comparison group corresponding to the physical node can be obtained through the client of the terminal 600. After receiving the logical comparison group corresponding to the physical node, the terminal 600 collects the training time of each first training node in the logical comparison group for each training step, and calculates a first average value based on the training time of multiple first training nodes in the training step, and queries the database 500 for the training time of each first training node in the training step; based on the training time of each first training node in the training step, the terminal 600 selects at least one second training node from multiple first training nodes; based on the training time of at least one second training node in the training step, the terminal 600 calculates a second average value, performs a difference operation on the first average value and the second average value to obtain the difference index corresponding to the training step; based on the difference index corresponding to multiple training steps, the terminal 600 performs type detection on at least one second training node to obtain the node type of at least one second training node, and transmits the node type to the client of the terminal 600 for display.

[0065] In some embodiments, the present application embodiments can be implemented collaboratively by a server and a terminal. For example, the logical comparison group corresponding to the physical node can be obtained through the client of the terminal 600. After receiving the logical comparison group corresponding to the physical node, the terminal 600 sends the logical comparison group corresponding to the physical node to the server 200 through the network 300. After receiving the logical comparison group corresponding to the physical node, server 200 collects the training time of each first training node in the logical comparison group for each training step, and calculates a first average based on the training time of multiple first training nodes in the training step. Server 200 then queries the database 500 for the training time of each first training node in the training step. Based on the training time of each first training node in the training step, server 200 selects at least one second training node from the multiple first training nodes. Based on the training time of at least one second training node in the training step, server 200 calculates a second average, performs a difference operation on the first average and the second average to obtain the difference index corresponding to the training step. Based on the difference index corresponding to multiple training steps, server 200 performs type detection on at least one second training node to obtain the node type of at least one second training node, and transmits the node type to the client of server 200 for display.

[0066] The node detection method provided in this application can be applied to any scenario requiring distributed performance monitoring. Specific application scenarios include:

[0067] 1) In a large-scale language model training scenario, engineers manage a training cluster consisting of multiple servers through a monitoring dashboard (terminal 600). One server (physical node) runs eight training processes, which are divided into a group (logical comparison group). For each training batch (training step), server 200 collects the time required for these eight processes (first training nodes) to complete that batch (training time). Next, server 200 calculates the average time of these eight processes (first average), and selects the two processes with the longest execution times (second training nodes), calculating the average time of these two processes (second average). Then, server 200 calculates the difference between the two averages (difference index). If this difference is consistently large for multiple batches, server 200 analyzes these two slow processes (type detection), discovering that their communication operation time is significantly higher than other processes, ultimately determining their node type as a communication-type slow node, and sending the node type result to the engineer's monitoring interface as an alert.

[0068] 2) In the visual image model training scenario, engineers upload video data via terminal 600 and initiate a distributed training task. The task runs in a cluster of multiple high-performance computers (physical nodes), with the processors within each computer divided into multiple parallel processing units (first training nodes). During training, server 200 receives status data from each processing unit. For each batch of images (training steps), it records the time taken for each processing unit to complete convolution operations and parameter synchronization (training time of the training step) and calculates the overall average processing speed of the cluster (first average). Next, server 200 identifies processing units with significantly lagging processing speeds (at least one second training node), calculates their average lag time (second average), and generates a difference index by comparing it with the overall average speed (difference calculation). Based on the difference index generated in multiple consecutive batches of image processing, server 200 analyzes these significantly lagging processing units to determine whether they are computational performance bottlenecks or data transmission bottlenecks (node ​​type). Finally, the node type result is sent to the maintenance personnel's repair equipment for alarm purposes, allowing maintenance personnel to replace faulty hardware in a timely manner.

[0069] See Figure 2 , Figure 2 This is a schematic diagram of the structure of the electronic device 400 provided in the embodiments of this application. Figure 2The illustrated electronic device 400 includes at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. The various components in the electronic device 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 440.

[0070] Processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0071] User interface 430 includes one or more output devices 431 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0072] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 410.

[0073] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.

[0074] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0075] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0076] The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0077] Presentation module 453 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 431 associated with user interface 430 (e.g., a display screen, a speaker, etc.).

[0078] The input processing module 454 is used to detect and translate one or more user inputs or interactions from one or more input devices 432.

[0079] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2 A node detection device 455 stored in memory 450 is shown. This device can be software in the form of programs and plug-ins, and includes the following software modules: an acquisition module 4551, a collection module 4552, a filtering module 4553, a calculation module 4554, and a detection module 4555. These modules are logically connected and can therefore be arbitrarily combined or further separated according to their implemented functions. The functions of each module will be described below.

[0080] In other embodiments, the apparatus provided in this application can be implemented in hardware. For example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the node detection method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0081] In some embodiments, the terminal or server can implement the node detection method provided in this application by running various computer-executable instructions or computer programs. For example, computer-executable instructions can be microprogram-level commands, machine instructions, or software instructions. Computer programs can be native programs or software modules in an operating system; they can be native applications (APPs), i.e., programs that need to be installed in the operating system to run; or they can be applets that can be embedded in any APP, i.e., programs that only need to be downloaded to a browser environment to run. In summary, the aforementioned computer-executable instructions can be any form of instruction, and the aforementioned computer programs can be any form of application, module, or plugin.

[0082] The node detection method provided in this application will be described in conjunction with exemplary applications and implementations of the terminals provided in the embodiments of this application.

[0083] The node detection method provided in the embodiments of this application will be described below. As described above, the electronic device implementing the node detection method in the embodiments of this application can be a terminal, a server, or a combination of both. Therefore, the executing entity of each step will not be described again below.

[0084] See Figure 3 , Figure 3 This is a schematic flowchart of the image processing method provided in the embodiments of this application, which will be combined with... Figure 3 The steps shown are explained.

[0085] In step 101, for each physical node used to perform the distributed training task, the logical comparison group corresponding to the physical node is obtained.

[0086] The logical comparison group comprises multiple first training nodes. A distributed training task refers to the training of a machine learning model using multiple training processes collaboratively. Distributed training tasks can be Large Language Model (LLM) training tasks, Computer Vision (CV) model training tasks, or Natural Language Processing (NLP) training tasks. A physical node refers to the actual hardware computing platform or device that hosts the distributed training task. A physical node can be an independent server, a workstation, or a virtual machine host or physical machine hosting a container deployed in the cloud. Each physical node includes basic hardware such as a processor, memory, and network interface. A logical comparison group refers to a set of related training processes logically divided for performance comparison and analysis within a distributed training task. Training processes within the same logical comparison group run on the same physical node, facilitating horizontal comparison of training processes. A first training node refers to each independent training process participating in the training and executing the distributed training task. Specifically, a first training node can be used to perform tensor operations, gradient calculations, and parameter synchronization operations within the distributed training task. For example, a physical node (a server) is configured with multiple GPU cards. One of the first training nodes in the physical node corresponds to a training process bound to a GPU card. The logical comparison group includes all training processes (first training nodes) on the physical node.

[0087] In some embodiments, step 101 above can be implemented as follows: obtain the physical node identifier corresponding to each first training node; based on the physical node identifier corresponding to each first training node, classify the first training nodes with the same physical node identifier into a logical comparison group.

[0088] The physical node identifier is a unique identifier used to identify a specific physical node. A physical node identifier can be a network address (such as an IP address), a hostname, a unique hardware serial number, or a logical ID assigned by the cluster management system.

[0089] For example, suppose there are three physical nodes: physical node A, physical node B, and physical node C. Physical node A runs training processes 1, 2, and 3 (the first training node); physical node B runs training processes 4 and 5 (the first training node); and physical node C runs training process 6 (the first training node). If the physical node identifiers for training processes 1, 2, and 3 are all the same as the identifier for physical node A, then training processes 1, 2, and 3 are grouped into a single logical comparison group.

[0090] In this embodiment, by classifying the first training node into a logical comparison group based on the physical node identifier, it is possible to ensure that the first training nodes within the logical comparison group share the same physical resources and local network conditions, thereby improving the fairness and accuracy of performance comparison within the group.

[0091] In step 102, for each training step, the training time of each first training node in the training step is collected, and a first average value is calculated based on the training time of multiple first training nodes in the training step.

[0092] In this context, a training step refers to an iteration or batch of data processing during the training of a machine learning model. In each training step, the model processes a portion of the data, calculates gradients, and updates its parameters. For example, a training step can be the forward and backward propagation process handling a mini-batch of data, or it can be a complete update cycle of the model parameters. Training time refers to the time required for a first training node to complete a training step. This time includes the total time spent on various stages such as data loading, forward computation, backpropagation, gradient updates, and communication synchronization. For example, training time can be the timestamp difference. Calculating the first average refers to calculating the arithmetic mean of the training times of all first training nodes in the logical comparison group for the same training step, to obtain the overall average time performance metric for the nodes in the logical comparison group. For example, the first average can be a simple arithmetic mean, a weighted average, or an average after removing outliers.

[0093] Here, the first average value is calculated based on the training time of multiple first training nodes in the training step, which can be referred to in the following formula (1).

[0094] (1)

[0095] in, This represents the first average value; This represents the training time for the i-th first training node; This indicates the number of the first training nodes.

[0096] For example, suppose a logical alignment group contains 8 first training nodes with training times of 120ms, 115ms, 110ms, 105ms, 100ms, 95ms, 90ms, and 85ms respectively. According to the calculation method for the preset number K, K is set to 2. Then the first average (TopK) is the average of the two first training nodes with the highest training times, which is 117.5ms.

[0097] In some embodiments, the first average value can be calculated based on the unit load time of multiple first training nodes. The step 102 above, "calculate the first average value based on the training time of multiple first training nodes in the training step", can be implemented as follows: for each training step, obtain the data load feature value of the training samples processed by each first training node; calculate the unit load time of each first training node based on the ratio of training time to data load feature value; calculate the average value of the unit load time of multiple first training nodes and use it as the first average value.

[0098] The data load feature value refers to a numerical value used to quantify the workload of training samples. For example, in natural language processing tasks, the data load feature value could be the number of tokens contained in the training samples; in computer vision tasks, it could be the image resolution or the number of targets to be processed. Unit workload time refers to the time required for the first training node to process a unit of data load, and it is a metric for measuring computational efficiency.

[0099] For example, suppose that in a training step, the training time of the first training node A is 200ms, and the training samples it processes contain 2000 tokens (data load feature values). Then, the unit load time of the first training node A is 0.1ms / token. The training time of the first training node B is 180ms, and the training samples it processes contain 1500 tokens (data load feature values). Then, the unit load time of the first training node B is 0.12ms / token. When calculating the first average, the unit load time data of 0.1ms / token and 0.12ms / token will be used, instead of the absolute training times of 200ms and 180ms.

[0100] In this embodiment, the data load feature value of the training samples processed by each first training node is obtained for each training step, which is equivalent to adding a task load dimension. The unit load time of each first training node is calculated based on the ratio of training time to the data load feature value, transforming an absolute time indicator into a relative efficiency indicator and reducing interference from uneven training data distribution. Calculating the difference index based on unit load time allows for precise differentiation between increased time due to external uneven data distribution and increased time purely due to decreased node performance, avoiding misclassification of normal nodes processing complex samples as slow nodes, thereby improving the accuracy of node detection.

[0101] In some embodiments, the step 102 above, "collecting the training time of each first training node in the training step", can be implemented in the following way: at the beginning of the training step, record the start timestamp of the training step; at the end of the training step, record the end timestamp of the training step; and calculate the training time of the training step based on the end timestamp and the start timestamp of the training step.

[0102] The start timestamp refers to a point in time when the training step begins execution, such as a system clock value in milliseconds or microseconds. The end timestamp refers to a point in time when the training step completes execution, such as a system clock value in milliseconds or microseconds.

[0103] For example, suppose that training process 1 (the first training node) records a start timestamp of 1000 milliseconds at the beginning of a certain training step and an end timestamp of 1500 milliseconds at the end of the training step, then the training time of training process 1 in that training step is 500 milliseconds.

[0104] In this embodiment of the application, by recording the start and end timestamps of the training steps and calculating the time difference, the training time of each first training node can be accurately obtained with low overhead, thereby providing a data foundation for subsequent time performance analysis.

[0105] In step 103, at least one second training node is selected from multiple first training nodes based on the training time of each first training node in the training step.

[0106] In this context, filtering refers to selecting second training nodes from multiple first training nodes that meet specific requirements based on preset conditions or rules. Filtering can be based on numerical comparisons, such as identifying the second training node corresponding to the maximum or minimum training time; or it can be based on matching specific attributes, such as identifying second training nodes that match a certain pattern. A second training node refers to one or more first training nodes with abnormal performance (e.g., the slowest or fastest) that are filtered out based on their training time performance in a particular training step. The second training node is the object of subsequent node detection. For example, the second training node could be the training process with the longest execution time, one of the top N training processes in terms of execution time, or a training process whose execution time exceeds a certain threshold.

[0107] In some embodiments, step 103 above can be implemented in the following manner: a screening threshold is calculated based on a first average value; for each first training node, when the training time of the first training node in the training step is greater than a preset threshold, the first training node is determined as at least one second training node.

[0108] The screening threshold refers to a pre-defined numerical limit used to determine whether the training time of the first training node is abnormal. The screening threshold can be a fixed value, such as an empirical value, or a value calculated based on statistical methods, such as the first average plus a standard deviation factor.

[0109] For example, suppose the first average value of a logical alignment group is 100ms, and the preset threshold (selection threshold) is set to 1.2 times the first average value, i.e., 120ms. If the training time of a certain first training node is 125ms, then the first training node is selected as the second training node.

[0110] In this embodiment, by selecting the first training node whose training time exceeds a preset threshold as the second training node, nodes with performance significantly lower than the average level can be quickly identified, thereby improving the identification efficiency of abnormal nodes.

[0111] In some embodiments, refer to Figure 4 Step 103 above can be adopted as follows: Figure 4 Steps 1031 to 1032 shown are implemented.

[0112] In step 1031, based on the training time of each first training node in the training step, the multiple first training nodes are sorted to obtain a sorted sequence.

[0113] Sorting refers to the process of rearranging a set of data according to a specific criterion (such as the duration of training). Sorting can be in ascending or descending order. A sorted sequence refers to a set of data arranged in a specific order after sorting operations.

[0114] For example, suppose a logical alignment group contains 5 first training nodes with training times of 110ms, 90ms, 120ms, 95ms, and 105ms respectively. After sorting these training times in descending order, the resulting sorted sequence is: 120ms, 110ms, 105ms, 95ms, and 90ms.

[0115] In step 1032, a preset number of first training nodes are selected from the sorted sequence, and the preset number of first training nodes are determined as at least one second training node.

[0116] In this context, the training time of the second training node is greater than or equal to that of the other training nodes, which are the first training nodes in the sorted sequence excluding the second training node. The preset quantity refers to a value pre-set before step 1032, used to specify the amount of data to be filtered. The preset quantity can be a fixed value or a value dynamically calculated based on parameters. Other training nodes refer to all first training nodes in the logical comparison group, excluding the filtered second training node.

[0117] For example, the sorted sequence is: 120ms, 110ms, 105ms, 95ms, 90ms. If the preset number is 2, then the two first training nodes with training times of 120ms and 110ms are determined as the second training nodes.

[0118] In some embodiments, prior to step 1032 above, the method provided in this application further includes: obtaining the total number of first training nodes included in the logical comparison group; multiplying the total number with a preset ratio coefficient to obtain a candidate number; when the candidate number is greater than a preset upper limit, determining the preset upper limit as the preset number; when the candidate number is less than a preset lower limit, determining the preset lower limit as the preset number, wherein the preset upper limit is greater than the preset lower limit; when the candidate number is less than or equal to the preset upper limit and the candidate number is greater than or equal to the preset lower limit, determining the candidate number as the preset number.

[0119] The total quantity refers to the total number of elements contained in a logical comparison group. The preset ratio coefficient is a pre-set ratio value used for multiplication with the total quantity to calculate a target quantity. The preset ratio coefficient can be a percentage or a decimal. The multiplication operation refers to multiplying the total quantity by the preset ratio coefficient. The candidate quantity refers to a preliminary quantity value calculated as an intermediate result before the final preset quantity is determined. The preset quantity upper limit is a maximum value pre-set before method execution, used to limit the preset quantity from exceeding this value. The preset quantity lower limit is a minimum value pre-set before method execution, used to limit the preset quantity from falling below this value.

[0120] Here, the preset quantity can be calculated using the following formula (2).

[0121] (2)

[0122] Wherein, K is a preset quantity; This represents a function that takes the minimum value. This represents the function that takes the maximum value. This represents the floor function; N represents the total number; 0.25 represents the preset scaling factor; 8 represents the preset upper limit of the number to avoid excessive computational overhead caused by an excessively large K when N is large; 2 represents the preset lower limit of the number to avoid instability caused by degenerating to only consider one point when N is small. If N is less than 2, K can be set to 1.

[0123] For example, suppose a logical alignment group contains 20 first training nodes (total number N). According to the formula, the number of candidates is first calculated to be 5. Then, the number of candidates is compared with a preset lower limit, max(2, 5) is 5. Finally, the number of candidates is compared with a preset upper limit, min(8, 5) is 5. Therefore, the preset number K is determined to be 5.

[0124] In this embodiment, the total number of first training nodes in the logical comparison group is obtained, and the total number is multiplied by a preset ratio coefficient to obtain the candidate number. This allows the number of nodes to be detected to be adjusted based on the training nodes in the logical comparison group. When the number of candidates is greater than a preset upper limit, the preset upper limit is set as the preset number; when the number of candidates is less than a preset lower limit, the preset lower limit is set as the preset number; and when the number of candidates is less than or equal to the preset upper limit and greater than or equal to the preset lower limit, the candidate number is set as the preset number. By adaptively determining the range of the detection number, the coverage and efficiency of anomaly detection are ensured, thereby improving the accuracy of subsequent node detection results.

[0125] In this embodiment, multiple first training nodes are sorted based on their training time in the training step to obtain a sorted sequence, which can intuitively display the time distribution of all nodes in the current step. A preset number of first training nodes are selected from the sorted sequence and identified as at least one second training node, directly locating the tail node with the worst time performance. Specifically, the training time of the second training node is greater than or equal to that of other training nodes, and the other training nodes are the first training nodes in the sorted sequence excluding the second training node. By selecting the node with the longest time, the specific node causing the entire cluster to wait can be accurately located, thereby improving the accuracy of the initial node screening process.

[0126] In step 104, based on the training time of at least one second training node in the training step, a second average value is calculated, and the difference between the first average value and the second average value is calculated to obtain the difference index corresponding to the training step.

[0127] The calculation of the second average refers to averaging the training time of at least one selected second training node in the same training step. If there is only one second training node, the second average is simply the node's training time itself. For example, the second average can be the time of a single second training node or the arithmetic mean of the times of multiple second training nodes. The difference operation refers to subtracting two values ​​to obtain the difference between them. The difference operation can be an absolute difference or a relative difference (e.g., a percentage difference). The difference index refers to the numerical value of the difference between the first average and the second average obtained through the difference operation. The difference index is used to quantify the degree to which the time performance of the second training node deviates from the overall average level.

[0128] In some embodiments, when the first average is the average unit load time of a plurality of first training nodes, the second average is the average unit load time of at least one second training node; when the first average is the average training time of a plurality of first training nodes, the second average is the average training time of at least one second training node.

[0129] In this embodiment of the application, the second average value is calculated based on the training time of at least one second training node in the training step, and can be referred to the following formula (3).

[0130] (3)

[0131] in, This represents the second average value; N represents the total number of the second training nodes. This represents the training time for the i-th second training node.

[0132] For example, suppose a logical alignment group contains 5 first training nodes with training times of 120ms, 110ms, 105ms, 95ms, and 90ms respectively. Then the second average (Mean) is 104ms.

[0133] In this embodiment of the application, the difference between the first average value and the second average value is calculated to obtain the difference index corresponding to the training step, which can be referred to in the following formula (4).

[0134] (4)

[0135] in, Indicates the difference index corresponding to the training steps; This represents the first average value; This represents the second average value.

[0136] For example, assuming the first average (TopK) is 117.5ms and the second average (Mean) is 104ms, then the difference metric (difference index) corresponding to the training step is 13.5ms.

[0137] In step 105, based on the difference indicators corresponding to multiple training steps, type detection is performed on at least one second training node to obtain the node type of at least one second training node.

[0138] Type detection refers to the process of determining the specific problem type of one or more second training nodes based on metrics, rules, or models. The purpose of type detection is to identify the root cause of abnormal node performance. For example, type detection can be based on a rule engine or a machine learning classification model. Node type refers to the description of the problem category to which the second training node belongs after type detection. Node type can represent the specific cause of abnormal node performance. For example, node type can be a computationally slow node, a communication-slow node, a hybrid slow node, a normal node, an I / O (input / output) slow node, a memory-slow node, etc.

[0139] In some embodiments, refer to Figure 5 Step 105 above can be adopted as follows: Figure 5 Steps 1051 to 1052 shown are implemented.

[0140] In step 1051, for each training step, when the difference index corresponding to the training step is greater than the preset index corresponding to the training step, the second training node corresponding to the training step is marked as a candidate node.

[0141] Here, the preset metric refers to a threshold set before execution, used to compare with the difference metric to determine whether an anomaly has occurred in the training step. The preset metric can be a fixed value or dynamically calculated, such as a statistical value or standard deviation based on historical data. Marking a node as a candidate node means that when the difference metric of a training step meets the conditions, the corresponding second training node is temporarily marked as a potential problem node for further statistical analysis. A candidate node refers to a second training node that is marked as potentially problematic during the initial screening stage.

[0142] Here, when the difference index corresponding to the training step is greater than the preset index corresponding to the training step, the second training node corresponding to the training step is marked as a candidate node, as can be referred to in the following formula (5).

[0143] (5)

[0144] in, Indicates the difference index corresponding to the training steps; This indicates the preset metrics corresponding to the training steps.

[0145] For example, suppose the difference metric for a certain training step is 13.5ms, while the preset metric for that training step is 10ms. Since 13.5ms is greater than 10ms, the second training node selected in that training step is marked as a candidate node.

[0146] In some embodiments, prior to step 1051 above, the method provided in this application further includes: obtaining a preset threshold coefficient; for each training step, multiplying the first average value corresponding to the training step with the preset threshold coefficient to obtain a preset index corresponding to the training step.

[0147] The preset threshold coefficient refers to a pre-set proportional value used to adjust the preset index (Threshold). The preset threshold coefficient can be a fixed small value, such as 0.1 or 0.2, or it can be a dynamic value determined based on historical data analysis or expert experience.

[0148] Here, for each training step, the first average value corresponding to the training step is multiplied by the preset threshold coefficient to obtain the preset index corresponding to the training step, which can be referred to in the following formula (6).

[0149] (6)

[0150] in, This indicates the preset metrics corresponding to the training steps; This represents the preset threshold coefficient, which can be pre-configured based on training task characteristics, model size, or historical statistical data. This represents the first average value corresponding to the training step.

[0151] For example, suppose the first average value for a certain training step is 100ms, and the preset threshold coefficient is set to 0.1. Then the preset metric for that training step is 10ms.

[0152] In this embodiment of the application, by obtaining a preset threshold coefficient and multiplying it with a first average value, the judgment threshold can be adjusted according to the overall performance level of the current training step, avoiding misjudgment that may be caused by using a fixed threshold, thereby improving the accuracy of slow node suspicion judgment.

[0153] In step 1052, for each second training node, the number of times the second training node is marked in multiple training steps is counted.

[0154] For example, suppose that in a total of 50 training steps, the second training node A is marked as a candidate node in 15 steps. Then the number of times the second training node A is marked is 15.

[0155] In step 1053, based on the number of times, the type of the second training node is detected to obtain the node type of the second training node.

[0156] In some embodiments, step 1053 above can be implemented as follows: when the number of attempts is less than a preset number of attempts, normal nodes are determined as the node type of the second training node; when the number of attempts is greater than or equal to the preset number of attempts, abnormal candidate nodes are determined as the node type of the second training node.

[0157] Among them, abnormal candidate nodes include computational slow nodes, communication slow nodes, and normal nodes.

[0158] For example, suppose the preset number of attempts is 10. The second training node A is marked 15 times, which is greater than the preset number of attempts of 10. Therefore, the second training node A is identified as an abnormal candidate node.

[0159] In some embodiments, step 1053 above can be implemented as follows: when the number of times is less than a preset number of times, normal nodes are determined as the node type of the second training node; when the number of times is greater than or equal to the preset number of times, the total computation time and total communication time of the second training node in multiple training steps are counted, and the second training node is classified based on the total computation time and total communication time to obtain the node type of the second training node.

[0160] The "preset count" refers to a threshold set before execution, used to compare the number of times the second training node is marked as a candidate node to determine if the candidate node consistently presents a problem. "Normal node" refers to a node type that, after detection, is determined to have no significant anomalies. "Total computation time" refers to the total time spent by the second training node across multiple training steps on purely computational tasks such as model forward computation and backpropagation (gradient calculation). For example, total computation time can be the computation time of the GPU cores or the execution time of the arithmetic logic unit (ALU) of the central processing unit (CPU). "Total communication time" refers to the total time spent by the second training node across multiple training steps on communication tasks such as data exchange, parameter synchronization, and gradient aggregation with other nodes. For example, total communication time can be bus transmission time.

[0161] For example, suppose the second training node B is marked 20 times, which is greater than the preset number of times 10. Further, the total computation time and total communication time of the second training node B in these marked training steps are statistically analyzed, and it is classified as a computationally slow node (node ​​type) based on these time data.

[0162] In some embodiments, the above-mentioned "classifying the second training node based on the total computation time and the total communication time to obtain the node type of the second training node" can be implemented in the following way: the training time of the second training node in multiple training steps is accumulated to obtain the total training time of the second training node; a first proportion of the total computation time in the total training time is calculated, and a second proportion of the total communication time in the total training time is calculated; the node type of the second training node is determined based on the first proportion and the second proportion.

[0163] Here, "cumulative" refers to adding up the training times one by one to obtain their sum. The total training time refers to the total time spent by the second training node across multiple training steps. The first percentage refers to the proportion of the total computation time to the total training time. The first percentage can be a percentage or a decimal. The second percentage refers to the proportion of the total communication time to the total training time. The second percentage can also be a percentage or a decimal.

[0164] For example, suppose the total training time of a second training node across multiple training steps is 1000 seconds, its total computation time is 700 seconds, and its total communication time is 200 seconds. Then, the first proportion of the total computation time in the total training time is 0.7, and the second proportion of the total communication time in the total training time is 0.2.

[0165] In some embodiments, the above-mentioned "determining the node type of the second training node based on the first proportion and the second proportion" can be implemented in the following manner: when the first proportion is greater than the first preset proportion and the second proportion is less than or equal to the second proportion, a computational slow node is determined as the node type of the second training node; when the first proportion is less than or equal to the first preset proportion and the second proportion is greater than the second preset proportion, a communication slow node is determined as the node type of the second training node; when the first proportion is less than or equal to the first preset proportion and the second proportion is less than or equal to the second preset proportion, a normal node is determined as the node type of the second training node; when the first proportion is greater than the first preset proportion and the second proportion is greater than the second preset proportion, a hybrid slow node is determined as the node type of the second training node.

[0166] The first preset percentage refers to a threshold set before execution, used to compare with the first percentage to determine if the computation time is abnormally high. The second preset percentage refers to a threshold set before method execution, used to compare with the second percentage to determine if the communication time is abnormally high. Computational slow nodes are those identified after type detection as having performance degradation primarily due to excessively long computation tasks. For example, computational slow nodes may be caused by hardware computing unit performance degradation. Communication slow nodes are those identified after type detection as having performance degradation primarily due to excessively long data communication or synchronization tasks. For example, communication slow nodes may be caused by inefficient communication protocols. Hybrid slow nodes are those identified after type detection as having both excessively long computation and communication task times, with both contributing to performance degradation. For example, hybrid slow nodes may have performance degradation due to varying degrees in both computation and communication, or there may be a mutual influence between the two.

[0167] For example, suppose the first proportion is 0.7 and the second proportion is 0.2. The preset first proportion is 0.6 and the preset second proportion is 0.3. Since the first proportion of 0.7 is greater than the first preset proportion of 0.6, and the second proportion of 0.2 is less than the second preset proportion of 0.3, the second training node is determined to be a computationally slow node (node ​​type).

[0168] In this embodiment, when the first proportion is greater than the first preset proportion and the second proportion is less than or equal to the second proportion, computationally slow nodes are identified as the node type of the second training node, thus identifying anomalies caused by insufficient computing power. When the first proportion is less than or equal to the first preset proportion and the second proportion is greater than the second preset proportion, communication-type slow nodes are identified as the node type of the second training node, thus identifying anomalies caused by network transmission. When the first proportion is greater than the first preset proportion and the second proportion is greater than the second preset proportion, hybrid-type slow nodes are identified as the node type of the second training node, thus identifying anomalies caused by both computational and communication reasons. When the first proportion is less than or equal to the first preset proportion and the second proportion is less than or equal to the second preset proportion, normal nodes are identified as the node type of the second training node, thus identifying normal nodes whose causes are not computational or communication-related. Through the classification criteria of this application, the specific root causes of anomalies can be clearly distinguished, thereby improving the efficiency of hardware optimization for distributed nodes.

[0169] In this embodiment, the total training time of the second training node is obtained by summing the training times of the second training node in multiple training steps, thus determining the time base for percentage calculation. Analyzing the first percentage of the total training time in the total training time and calculating the second percentage of the total communication time in the total training time, the relative percentages of computation time and communication time can be determined. Based on the first and second percentages, the node type of the second training node is determined. By using normalized relative percentages for node classification, false detections caused by differences in the duration of different training tasks can be eliminated, thereby improving the accuracy of node type detection.

[0170] In this embodiment, when the number of attempts is less than a preset number, normal nodes are identified as the node type of the second training node, thus eliminating nodes that only exhibit occasional anomalies. When the number of attempts is greater than or equal to the preset number, the total computation time and total communication time of the second training node in multiple training steps are statistically analyzed, allowing for the acquisition of detailed time consumption for abnormal nodes at different task stages. Classifying the second training node based on the total computation time and total communication time yields its node type. Analyzing the detailed time consumption provides data support for subsequently identifying the specific causes of abnormal nodes, thereby improving the accuracy of node classification and detection.

[0171] In this embodiment, for each training step, when the difference index corresponding to the training step is greater than the preset index corresponding to the training step, the second training node corresponding to the training step is marked as a candidate node. The preset index filters out nodes with slight time fluctuations within the normal range. For each second training node, the number of times it is marked in multiple training steps is counted. Based on the number of times, type detection is performed on the second training node to obtain its node type. This effectively distinguishes between occasional slight time fluctuations and nodes with continuous time increases, thereby reducing node misclassification and improving the reliability of node detection results.

[0172] In some embodiments, see Figure 6 ,exist Figure 3 After step 105 shown, steps 201 to 203 can also be executed, as explained below.

[0173] In step 201, for each second training node, when the node type of the second training node is a communication-type slow node, a computing-type slow node, or a hybrid slow node, the total hardware resource usage index of the second training node in multiple training steps is calculated.

[0174] The total hardware resource utilization metric refers to a comprehensive measure of the hardware resources (such as GPU utilization, CPU utilization, memory usage, network bandwidth usage, disk I / O, etc.) used by the second training node across multiple training steps. For example, the total hardware resource utilization metric could be average GPU utilization, peak memory usage, or network send or receive rate.

[0175] In some embodiments, step 201 above can be implemented as follows: in multiple training steps, the GPU utilization of the second training node is collected; in multiple training steps, the video memory usage of the second training node is collected; based on the GPU utilization and video memory usage, the total hardware resource usage index of the second training node in multiple training steps is calculated.

[0176] GPU utilization refers to the level of activity of the GPU cores in executing computational tasks over a period of time, expressed as a percentage. Memory usage refers to the amount of GPU memory allocated and used, measured in MB or GB.

[0177] For example, suppose a second training node has an average GPU utilization of 95% and an average VRAM usage of 20GB across multiple training steps. These data constitute the total hardware resource usage metric of the second training node across multiple training steps.

[0178] In this embodiment of the application, by collecting core hardware indicators such as GPU utilization, the resource usage of the second training node during the training process can be comprehensively reflected, thereby providing a multi-dimensional data foundation for subsequent node type matching.

[0179] In step 202, based on the total hardware resource usage index of the second training node in multiple training steps, the node type of the second training node is matched to obtain the matching result of the second training node.

[0180] In this context, matching refers to comparing the total hardware resource usage metric with the node type of the second training node to determine if there is a correlation between them or if they conform to preset rules. Matching can be exact matching or fuzzy matching.

[0181] In some embodiments, step 202 above can be implemented as follows: when the node type of the second training node is a computational slow node, the total hardware resource occupancy index is compared with a preset first resource occupancy index to obtain a first comparison result; the matching result of the second training node is determined based on the first comparison result; when the node type of the second training node is a communication slow node, the total hardware resource occupancy index is compared with a preset second resource occupancy index to obtain a second comparison result; the matching result of the second training node is determined based on the second comparison result.

[0182] Specifically, when the first comparison result indicates that the total hardware resource usage index is greater than the preset first resource usage index, the matching result is unsuccessful; when the first comparison result indicates that the total hardware resource usage index is less than the preset second resource usage index, the matching result is unsuccessful.

[0183] For example, suppose a second training node is identified as a computationally slow node with a total hardware resource utilization (GPU utilization) of 60%. The preset first resource utilization (GPU utilization) is 80%. Since the total hardware resource utilization of 60% is less than the preset second resource utilization of 80%, the first comparison result indicates that the match is unsuccessful, and thus the matching result of the second training node is determined to be unsuccessful.

[0184] In step 203, when the matching result indicates that the total hardware resource usage index does not match the node type of the second training node, the node type of the second training node is corrected to a normal node.

[0185] In this context, "correcting to a normal node" refers to changing the type attribute of the second training node, which was originally marked as a computational slow node or a communication slow node, to a normal node.

[0186] For example, suppose a second training node is initially identified as a computationally slow node, but the matching result is unsuccessful. According to the correction rule, the node type of the second training node will be corrected from a computationally slow node to a normal node.

[0187] In this embodiment, for each second training node, when the node type is a communication-type slow node, a computation-type slow node, or a hybrid slow node, the total hardware resource consumption index of the second training node in multiple training steps is statistically analyzed. Based on the total hardware resource consumption index of the second training node in multiple training steps, the node type of the second training node is matched to obtain the matching result of the second training node. This can verify whether the time-based detection result matches the actual hardware load. When the matching result indicates that the total hardware resource consumption index does not match the node type of the second training node, the node type of the second training node is corrected to a normal node. By verifying the time consumption analysis with the hardware-level resource monitoring, misjudgments caused by time consumption can be eliminated, thereby further improving the accuracy of node detection.

[0188] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0189] In distributed deep learning training, slow nodes (stragglers) are a common problem. A slow node is an entity that, for various reasons, executes significantly slower than other participating computational units (ranks) during distributed training. The presence of slow nodes slows down the entire distributed training process because distributed training uses a synchronous mode; all computational entities must wait for the slowest one to complete its current step before proceeding to the next. This scenario is known as the distributed training synchronization barrier waiting scenario.

[0190] Specifically, the main causes of slow nodes are hardware heterogeneity, communication layer congestion, and data and load imbalance. Hardware heterogeneity refers to differences in hardware configuration between different computing units. For example, different graphics processing units (GPUs) may have different computing power; the interconnect bus connecting GPUs may have bandwidth fluctuations; and video memory may also cause computational latency for individual GPUs. Communication layer congestion refers to network congestion during parameter synchronization operations (such as AllReduce), or the presence of a slow node causing global waiting, with network congestion further exacerbating communication latency. Data and load imbalance refers to differences in training data distributed to different computing nodes. For example, uneven image resolution leads to different processing complexities; or background tasks on some computing nodes preempt GPU or CPU resources, resulting in low processing efficiency on some nodes.

[0191] Regarding the aforementioned technical problems, the applicant believes that related technologies, in solving the slow node problem in distributed training, mainly focus on a single layer, such as only the hardware layer, communication layer, or algorithm layer. These technologies fail to effectively combine hardware and communication indicators for comprehensive detection, and single-layer detection methods cannot comprehensively and accurately identify slow nodes and their root causes. This application provides a node detection method. This application collects computing, communication, and graphics processing unit (GPU) resource and memory-related indicators during training, and groups computing nodes based on physical topology and logical comparison. It then performs intra-group difference analysis on each computing node, thereby achieving: reducing false positives for slow nodes; determining the type of slow node (e.g., computing-driven, communication-driven, or hybrid); and outputting key anomaly information that can be used in monitoring or alarm systems.

[0192] The following section provides a detailed explanation of the application process of a node detection method provided in this application embodiment, within a distributed training scenario using a graphics processing unit (GPU) accelerator card.

[0193] It should be noted that the operating environment used in this embodiment is a distributed training cluster. The distributed training cluster contains multiple physical computing nodes (Nodes, i.e., the physical nodes mentioned above), and each physical computing node is configured with at least one graphics processing unit (GPU) accelerator card. Multiple training ranks (i.e., the first training node mentioned above) run on each physical computing node, and each training rank completes model training through set communication operations. The training process uses training steps as the basic execution unit.

[0194] The node detection method provided in this application embodiment includes the following four stages in a distributed training scenario of a graphics processing unit (GPU) accelerator card: training metric collection stage, slow node suspicion determination stage based on intra-group ranking differences, multi-dimensional comprehensive determination and slow node classification stage, and detection result output and feedback stage.

[0195] The first phase is the training metric collection phase. During distributed training, training metrics are collected for each training rank. The collection of these metrics is accomplished within the training framework layer (Megatron framework) using a lightweight statistical approach combined with time-series recording of monitoring metrics. The lightweight statistical approach refers to collecting performance data using low-overhead performance analysis tools provided by the training framework without significantly increasing the computational or I / O overhead of the training task. Time-series recording of monitoring metrics refers to continuously recording and storing the collected performance metrics in chronological order. Specific metrics collected include: the total time spent on a single training rank in a training step (step_time, i.e., the training time mentioned above), forward computation time (forward_time), backward computation time (backward_time), the time spent calling the set communication operator, and GPU utilization and memory usage (i.e., the total hardware resource usage mentioned above). Specifically, the time spent calling the set communication operator is collected by recording the start and end times of the core communication operator calls using a performance analysis tool (Profiler) to obtain the communication time information of the core communication operator. The key call operators that are the focus of the statistics include AllReduce, ReduceScatter, and AllGather.

[0196] It should be noted that in the first stage, after collecting the metrics for each training step, each training rank organizes the locally collected metrics. The metrics are organized according to the training rank identifier to form a corresponding set of metric records, which are then centrally summarized and provided to subsequent stages for use.

[0197] The second stage is the slow node suspicion determination stage based on the difference in ranking within the group. Based on the collected training indicators, the slow node suspicion screening operation is performed on the training rank.

[0198] First, the training metrics of multiple training ranks belonging to the same physical computing node are compared and analyzed. A logical comparison group consists of all training ranks on the same physical computing node. Grouping training ranks on the same physical computing node into one group ensures that these training ranks share the same physical hardware (CPU, memory, PCIe bandwidth to graphics processing unit) and local network conditions when comparing performance, thus guaranteeing the fairness of the comparison within the group. Within each logical comparison group, if the group contains N training ranks, the set of total training time (step_time) in the same training step can be represented by the following formula (7).

[0199] (7)

[0200] Where T represents the set of total training time for the same training step; This represents the total training time (step_time) of the first training rank in this training step. This represents the total training time (step_time) of the second training rank in that training step, and so on. This represents the total training time (step_time) for the Nth training rank in this training step; N represents the total number of training ranks within the logical alignment group.

[0201] Secondly, sort the time consumption values ​​in set T in descending order to obtain the sorted sequence. The sorted sequence is represented by the following formula (8).

[0202] (8)

[0203] in, This indicates the total training time (step_time) corresponding to the training rank (rank) with the longest training step time within the logical comparison group. This represents the total time (step_time) of the training step that is second in terms of time consumption, and so on. This represents the total time taken for the training step with the least time consumption (step_time).

[0204] Furthermore, based on the above sorting sequence, the TopK metric is defined. The TopK metric is the average of the total time taken for the top K maximum training steps after sorting (i.e., the preset number mentioned above), which is used to characterize the overall execution level of the slow training rank after sorting within the logical comparison group. The specific calculation method is shown in the following formula (9).

[0205] (9)

[0206] in, This represents the average total time taken for the top K largest training steps after sorting; K is a preset integer parameter. Used to characterize the impact of a small number of abnormally slow training ranks on overall training execution; This represents the total training time (step_time) for the i-th training rank after sorting.

[0207] Here, the value of parameter K is preferred proportionally over a fixed value to prevent the first few values ​​from deviating too much. The specific recommended value is given in formula (10) below.

[0208] (10)

[0209] Wherein, K is a preset integer parameter with a value of 1; This represents a function that takes the minimum value. This represents the function that takes the maximum value. The function represents the floor function; N represents the total number of training ranks within the logical alignment group; 0.25 represents the scaling factor (i.e., the preset scaling factor mentioned above), which means averaging the slowest 25% of training ranks within the group to better represent the slow subset; 8 represents the upper limit of K (i.e., the preset upper limit of the number of values ​​mentioned above), to avoid excessive computational overhead when N is large; 2 represents the lower limit of K (i.e., the preset lower limit of the number of values ​​mentioned above), to avoid instability caused by degenerating to only consider one point when N is small. If N is less than 2, K can be set to 1.

[0210] For example, when N equals 8, K is calculated to be 2 using the above formula. The average of the total time (step_time) of the two slowest training ranks is then used as the K value. When N equals 16, K is calculated to be 4 using the above formula. The average of the total time (step_time) of the four slowest training ranks is then used as the K value. .

[0211] Next, the Mean metric is defined. The Mean (i.e. the first Mean mentioned above) is the average of the total training time (step_time) of all training ranks within the logical comparison group. It is used to characterize the overall normal execution level within the logical comparison group. The specific calculation method is shown in the following formula (11).

[0212] (11)

[0213] in, This represents the average total time taken for training steps across all training ranks within a logical alignment group; N represents the total number of training ranks within a logical alignment group. This represents the total training time (step_time) for the i-th training rank.

[0214] Next, based on the above TopK and Mean, we define the ranking difference index within the logical comparison group (i.e. the difference index mentioned above), and the specific calculation method is as follows (12).

[0215] (12)

[0216] in, Indicates the ranking difference index within the logical alignment group; This represents the average total time taken for the first K largest training steps after sorting. This represents the average total training time for all training ranks within the logical comparison group. The ranking difference metric is used to characterize the relative deviation between the slow training rank after ranking within the logical comparison group and the overall average performance level.

[0217] Meanwhile, the specific calculation method for defining the difference judgment threshold (Threshold, i.e. the preset index mentioned above) is as follows (13).

[0218] (13)

[0219] in, Indicates the threshold for determining the difference; This represents the proportional threshold coefficient, which can be pre-configured based on training task characteristics, model size, or historical statistical data. This represents the average total training time for all training ranks within the logical comparison group.

[0220] Finally, when the preset conditions are met, it is determined that there is a suspected abnormally slow node (i.e., the candidate node mentioned above) in the logical comparison group. The judgment method for whether the preset conditions are met can refer to the following formula (14).

[0221] (14)

[0222] in, Indicates the ranking difference index within the logical alignment group; This indicates the threshold for determining the difference.

[0223] The aforementioned preset conditions remain true across multiple consecutive training steps. Specifically, if the difference exceeds a preset threshold, it indicates that the top K training ranks significantly slow down the training progress within the group in the current training step, and these top K training ranks belong to the suspected slow node group in the current training step. After multiple consecutive training steps (preset value), slow nodes that consistently exist in the suspected slow node group, or slow nodes with a probability greater than 90% in the suspected slow node group in each training step, are identified as the final suspected slow nodes.

[0224] The third stage is the multi-dimensional comprehensive judgment and slow node classification stage, which involves further type determination and analysis of the marked suspected slow nodes.

[0225] First, computationally slow nodes (i.e., computationally slow ranks) are detected. For suspected slow nodes, the proportion of forward and backward computation time in each training step is analyzed. This proportion refers to the percentage of forward and backward computation time within the total training time (step time) for a single training rank. This is combined with the GPU utilization and used memory of that training rank for a comprehensive assessment. When the computation time of a suspected training rank is significantly higher than the average of its group, and the GPU utilization and used memory deviate from the group average, the suspected training rank is identified as a computationally slow node. It should be noted that computationally slow nodes can be due to poor hardware performance leading to full load (high GPU utilization) or malfunction causing inoperability (low GPU utilization); both situations can be identified.

[0226] Secondly, detection of communication-related slow nodes (i.e., communication-related slow ranks) is performed. For suspected slow node training ranks, the logical time consumption characteristics of their collective communication are analyzed. The proportion of communication time of a single training rank in a training step is calculated; this proportion refers to the percentage of the communication time of a training rank in the total training step time (step_time) for that single training rank. When the communication time of a suspected training rank is significantly higher than the average of other training ranks within its logical alignment group, that suspected training rank is determined to be a communication slow node.

[0227] Finally, by combining the slow node detection results from computation and communication, the slow nodes are classified in the final classification. The specific final classification results (i.e., the node types mentioned above) include:

[0228] 1) Computational slow nodes: If a training rank is determined to be computationally slow by the computational slow node detection step, but not by the communication slow node detection step, it is classified as a computational slow node.

[0229] 2) Communication-type slow nodes: If a training rank is identified as slow in communication by the slow communication node detection step but not slow in computation by the slow computation node detection step, it is classified as a communication-type slow node. For example, after multiple training steps, if the total proportion of communication time of a suspected slow node's training rank in the multiple training steps exceeds a preset value, or if the average proportion of communication time of a suspected slow node's training rank in the multiple training steps exceeds a preset value, then this suspected slow node's training rank is determined to be a communication-type slow node.

[0230] 3) Hybrid slow nodes: If a training rank is judged as slow in computation by the slow node detection step and as slow in communication by the slow node detection step, it is classified as a hybrid slow node.

[0231] 4) Normal nodes: If a training rank is initially screened as a suspected slow node by the slow node suspicion determination step based on the intra-group ranking difference, but after the above analysis it is found that it is neither slow in computation nor slow in communication, then its slow node suspicion is eliminated and it is classified as a normal node.

[0232] The fourth stage is the output and feedback stage of detection results. The final determination result of the slow node is output to the training monitoring system, which supports performance analysis, anomaly identification, and subsequent processing of the training task. The training monitoring system is an external platform that collects, displays, and analyzes various operational metrics and status information of the distributed training task in real time, and provides a data visualization interface. After receiving the slow node detection result, the monitoring system can highlight the slow node on the visualization interface or pass the information to the training scheduler to enable rescheduling or send alarm notifications.

[0233] To more clearly illustrate the technical solutions of the embodiments of this application in application scenarios, the embodiments of this application will be further described in detail with reference to the accompanying drawings.

[0234] Reference Figure 7 , Figure 7This paper illustrates the overall structure of the distributed slow node detection in training, as described in this embodiment. First, the user submits a task to the training task submission module. The task is executed on the overall architecture of the training cluster, which contains multiple training nodes, from node 0 to node N. On these training nodes, an indicator acquisition layer is used to collect performance data. This layer includes a training time acquisition unit 710, a communication time acquisition unit 720, and a graphics processor resource acquisition unit 730. Specifically, the training time acquisition unit 710 performs forward time acquisition, backward time acquisition, and total training step time acquisition; these acquisition results collectively constitute the total time. The communication time acquisition unit 720 performs communication statistics based on performance analysis tools. The graphics processor resource acquisition unit 730 collects graphics processor utilization and used video memory. All collected indicators are input to a data processing unit 740, which aligns the indicators according to their training rank.

[0235] Reference Figure 8 , Figure 8 This document illustrates the processing flow for outputting a list of suspected slow nodes in an embodiment of this application. The input layer consists of logical data groups. In step 801, node physical topology analysis is performed. The training ranks on different physical nodes, for example, training ranks 0 to 2 located at nodes A and B respectively, are divided into logical comparison group G1 and logical comparison group G2. Next, in step 802, intra-group mean difference analysis and suspicion determination are performed. First, in step 8021, the index sorting is traversed (supporting single or multiple index sorting); then, in step 8022, the intra-group training ranks are sorted according to the total time consumption of a single training rank; then, in step 8023, difference calculation is performed (calculating based on the statistical results algorithm of N training steps); finally, in step 8024, it is determined whether the difference threshold is exceeded. If the determination result is no, in step 803, it is not marked as a slow node; if the determination result is yes, in step 804, it is marked as a suspected slow node, and then in step 805, auxiliary anomaly attribution is performed through graphics processor utilization, used video memory, etc. Finally, in step 806, a list of suspected slow nodes is output. The list of suspected nodes includes the slow node identifier, the logical comparison group to which it belongs, and the type of abnormal indicator (based on ranking differences and auxiliary factors).

[0236] Reference Figure 9 , Figure 9This paper illustrates the generation process of computational slow node analysis results in an embodiment of this application. The input layer of this process is a list of suspected slow node training ranks. First, in step 901, a computation time comparison analysis is performed. The comparison analysis includes statistics on forward computation time, statistics on backward computation time, and calculation of the time ratio of each computation stage. Simultaneously, in step 902, a graphics processor resource utilization analysis is performed. The graphics processor resource utilization analysis includes calculating the average graphics processor utilization rate for each training rank. Furthermore, in step 903, a video memory usage analysis is performed. The video memory usage analysis counts the used video memory. The results of these three analysis steps are combined into the output layer, ultimately outputting the computational slow node analysis results, which are used for comprehensive determination of slow node types.

[0237] Reference Figure 10 , Figure 10 This paper illustrates the generation process of computational slow node analysis results in an embodiment of this application. The input layer of this process is a list of suspected slow node training ranks. First, in step 1001, intra-group analysis of communication time is performed. Intra-group analysis includes statistics on aggregated communication time and calculation of the proportion of time spent in each communication phase. Simultaneously, in step 1002, graphics processor resource utilization analysis is performed. Graphics processor resource utilization analysis includes calculating the average graphics processor utilization rate for each training rank. Furthermore, in step 1003, video memory usage analysis is performed. Video memory usage analysis statistics the used video memory. The results of these three analysis steps are combined into the output layer, ultimately outputting the communication-type slow node analysis results, which are used for comprehensive determination of slow node types.

[0238] Reference Figure 11 , Figure 11 This paper illustrates the process of multi-dimensional integrated judgment of slow node results in this embodiment. The input layer is the judgment result of whether each node belongs to a computational slow node or a communication slow node. The judgment result is input to the multi-dimensional integrated judgment module 1110, which is used to determine the combination relationship between computational and communication slow nodes, thereby determining the type and status of slow nodes and classifying nodes as slow nodes or normal nodes. Next, the output layer outputs the slow node detection results and provides feedback. The feedback results are processed by the slow node information module 1120, which integrates node identifiers, node types, and key anomaly indicators. Finally, the node identifiers, node types, and key anomaly indicators are sent to the training monitoring system 1130 for visualization.

[0239] Compared with related technologies, the node detection method provided in this application has the following advantages:

[0240] First, the embodiments of this application achieve comprehensive detection of slow nodes in distributed training by collecting statistical information on the total time spent in training steps (Step), the time spent in the computation stage, and the time spent in the communication stage at the training framework layer.

[0241] Second, the embodiments of this application, based on the intra-group ranking difference analysis and statistical threshold determination mechanism, compare the execution time of each training rank, effectively avoiding misjudgment of slow nodes.

[0242] Third, the embodiments of this application are based on the node physical topology and logical comparison grouping mechanism. Under the same or equivalent resource conditions, the training process is compared and analyzed within the group. This avoids the interference of hardware configuration or communication topology differences between different nodes on the judgment results, thereby realizing the fairness and comparability judgment across nodes and across training ranks.

[0243] Fourth, the embodiments of this application analyze slow nodes from multiple dimensions by comprehensively considering computation time characteristics, communication time characteristics, graphics processing unit (GPU) utilization rate indicators, and used video memory indicators. This can accurately distinguish between computationally slow nodes, communication slow nodes, and hybrid slow nodes, reducing the reliance on human experience judgment.

[0244] Fifth, the embodiments of this application use a performance analysis tool (Profiler) in a low-overhead manner during the normal training process, which is suitable for online detection scenarios of long-term, large-scale training tasks.

[0245] It is understood that in the embodiments of this application, data such as user information are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with relevant laws, regulations and standards.

[0246] The following description continues to illustrate the exemplary structure of the node detection device 455 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2 As shown, the software modules stored in the node detection device 455 in the memory 450 may include:

[0247] The acquisition module 4551 is used to acquire the logical comparison group corresponding to each physical node for performing the distributed training task, wherein the logical comparison group includes multiple first training nodes.

[0248] The acquisition module 4552 is used to acquire the training time of each first training node in the training step for each training step, and to calculate a first average value based on the training time of the plurality of first training nodes in the training step.

[0249] The filtering module 4553 is used to filter at least one second training node from the plurality of first training nodes based on the training time of each first training node in the training step.

[0250] The calculation module 4554 is used to calculate a second average value based on the training time of the at least one second training node in the training step, and to perform a difference operation on the first average value and the second average value to obtain the difference index corresponding to the training step.

[0251] The detection module 4555 is used to perform type detection on the at least one second training node based on the difference indicators corresponding to multiple training steps, so as to obtain the node type of the at least one second training node.

[0252] In some embodiments, the filtering module 4553 is further configured to sort the plurality of first training nodes based on the training time of each first training node in the training step to obtain a sorted sequence; filter out a preset number of first training nodes from the sorted sequence, and determine the preset number of first training nodes as the at least one second training node, wherein the training time of the second training node is greater than or equal to that of other training nodes, and the other training nodes are the first training nodes in the sorted sequence other than the second training node.

[0253] In some embodiments, the filtering module 4553 is further configured to obtain the total number of the first training nodes included in the logical comparison group; multiply the total number by a preset ratio coefficient to obtain a candidate number; when the candidate number is greater than a preset upper limit, determine the preset upper limit as the preset number; when the candidate number is less than a preset lower limit, determine the preset lower limit as the preset number, wherein the preset upper limit is greater than the preset lower limit; when the candidate number is less than or equal to the preset upper limit and the candidate number is greater than or equal to the preset lower limit, the candidate number is determined as the preset number.

[0254] In some embodiments, the detection module 4555 is further configured to, for each training step, mark the second training node corresponding to the training step as a candidate node when the difference index corresponding to the training step is greater than the preset index corresponding to the training step; for each second training node, count the number of times the second training node is marked in multiple training steps; and perform type detection on the second training node based on the number of times to obtain the node type of the second training node.

[0255] In some embodiments, the detection module 4555 is further configured to determine a normal node as the node type of the second training node when the number of times is less than a preset number of times; and when the number of times is greater than or equal to the preset number of times, to calculate the total computation time and total communication time of the second training node in multiple training steps, and to classify the second training node based on the total computation time and the total communication time to obtain the node type of the second training node.

[0256] In some embodiments, the detection module 4555 is further configured to accumulate the training time of the second training node in multiple training steps to obtain the total training time of the second training node; calculate the first proportion of the total computation time in the total training time, and calculate the second proportion of the total communication time in the total training time; and determine the node type of the second training node based on the first proportion and the second proportion.

[0257] In some embodiments, the detection module 4555 is further configured to: determine computational slow nodes as the node type of the second training node when the first proportion is greater than the first preset proportion and the second proportion is less than or equal to the second proportion; determine communication slow nodes as the node type of the second training node when the first proportion is less than or equal to the first preset proportion and the second proportion is greater than the second preset proportion; determine normal nodes as the node type of the second training node when the first proportion is less than or equal to the first preset proportion and the second proportion is less than or equal to the second preset proportion; and determine hybrid slow nodes as the node type of the second training node when the first proportion is greater than the first preset proportion and the second proportion is greater than the second preset proportion.

[0258] In some embodiments, the above-described apparatus further includes a detection module, configured to, for each of the second training nodes, when the node type of the second training node is a communication-type slow node, a computing-type slow node, or a hybrid slow node, calculate the total hardware resource usage index of the second training node in multiple training steps; based on the total hardware resource usage index of the second training node in multiple training steps, match the node type of the second training node to obtain a matching result for the second training node; when the matching result indicates that the total hardware resource usage index does not match the node type of the second training node, correct the node type of the second training node to a normal node.

[0259] This application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the node detection method described above in this application.

[0260] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the node detection method provided in this application. For example, ... Figure 3 The node detection method is shown.

[0261] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0262] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0263] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0264] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0265] In summary, this application embodiment obtains a logical comparison group corresponding to each physical node used to perform distributed training tasks, collects the training time of each first training node in the training step, and calculates a first average value based on the training time of multiple first training nodes in the training step. This first average value can represent the overall time consumption level of the current distributed training. Next, at least one second training node is selected based on the training time, and a second average value is calculated based on the training time of the at least one second training node in the training step. The difference between the first average value and the second average value is calculated to obtain the difference index corresponding to the training step. By selecting second training nodes from the first training nodes, the deviation of the time consumption level of the second training node from the overall average level (difference index) is calculated. Finally, based on the difference index corresponding to multiple training steps, at least one second training node is type-detected to obtain the node type. Through the multi-step difference index, second training nodes with large deviations in time consumption can be effectively identified, thereby improving the accuracy of node detection.

[0266] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A node detection method, characterized in that, The method includes: For each physical node used to perform a distributed training task, a logical comparison group corresponding to the physical node is obtained, wherein the logical comparison group includes multiple first training nodes, and multiple first training nodes in the same logical comparison group run on the same physical node; For each training step, the training time of each first training node in the training step is collected, and a first average value is calculated based on the training time of the plurality of first training nodes in the training step. Based on the training time of each first training node in the training step, at least one second training node is selected from the plurality of first training nodes. Based on the training time of the at least one second training node in the training step, a second average value is calculated, and the difference between the first average value and the second average value is calculated to obtain the difference index corresponding to the training step. Based on the difference indicators corresponding to the multiple training steps, type detection is performed on the at least one second training node to obtain the node type of the at least one second training node.

2. The method according to claim 1, characterized in that, The step of selecting at least one second training node from the plurality of first training nodes based on the training time of each first training node in the training step includes: Based on the training time of each first training node in the training step, the plurality of first training nodes are sorted to obtain a sorted sequence. A predetermined number of first training nodes are selected from the sorted sequence, and the predetermined number of first training nodes are determined as at least one second training node, wherein the training time of the second training node is greater than or equal to that of other training nodes, and the other training nodes are the first training nodes in the sorted sequence other than the second training node.

3. The method according to claim 2, characterized in that, The method further includes: Obtain the total number of the first training nodes included in the logical comparison group; The total quantity is multiplied by a preset ratio coefficient to obtain the candidate quantity; When the number of candidates is greater than the preset upper limit, the preset upper limit is determined as the preset number; When the number of candidates is less than a preset lower limit, the preset lower limit is determined as the preset number, wherein the preset upper limit is greater than the preset lower limit; When the number of candidates is less than or equal to the preset upper limit and the number of candidates is greater than or equal to the preset lower limit, the number of candidates is determined as the preset number.

4. The method according to claim 1, characterized in that, The step of performing type detection on the at least one second training node based on the difference indicators corresponding to multiple training steps to obtain the node type of the at least one second training node includes: For each training step, when the difference index corresponding to the training step is greater than the preset index corresponding to the training step, the second training node corresponding to the training step is marked as a candidate node. For each second training node, the number of times the second training node is marked in the multiple training steps is counted; Based on the number of times, type detection is performed on the second training node to obtain the node type of the second training node.

5. The method according to claim 4, characterized in that, The step of performing type detection on the second training node based on the number of times to obtain the node type of the second training node includes: When the number of times is less than the preset number, the normal node is determined as the node type of the second training node; When the number of times is greater than or equal to the preset number of times, the total computation time and total communication time of the second training node in multiple training steps are calculated, and the second training node is classified based on the total computation time and the total communication time to obtain the node type of the second training node.

6. The method according to claim 5, characterized in that, The second training node is classified based on the total computation time and the total communication time to obtain the node type of the second training node, including: The training time of the second training node in multiple training steps is accumulated to obtain the total training time of the second training node; Calculate the first percentage of the total computation time in the total training time, and calculate the second percentage of the total communication time in the total training time; Based on the first proportion and the second proportion, the node type of the second training node is determined.

7. The method according to claim 6, characterized in that, The step of determining the node type of the second training node based on the first proportion and the second proportion includes: When the first proportion is greater than the first preset proportion and the second proportion is less than or equal to the second preset proportion, the computational slow node is determined as the node type of the second training node; When the first proportion is less than or equal to the first preset proportion and the second proportion is greater than the second preset proportion, the communication-type slow node is determined as the node type of the second training node; When the first proportion is less than or equal to the first preset proportion and the second proportion is less than or equal to the second preset proportion, the normal node is determined as the node type of the second training node; When the first proportion is greater than the first preset proportion and the second proportion is greater than the second preset proportion, the hybrid slow node is determined as the node type of the second training node.

8. The method according to any one of claims 1-7, characterized in that, The method further includes: For each of the second training nodes, when the node type of the second training node is a communication-type slow node, a computing-type slow node, or a hybrid slow node, the total hardware resource usage index of the second training node in multiple training steps is calculated. Based on the total hardware resource usage index of the second training node in multiple training steps, the node type of the second training node is matched to obtain the matching result of the second training node. When the matching result indicates that the total hardware resource usage index does not match the node type of the second training node, the node type of the second training node is corrected to a normal node.

9. A node detection device, characterized in that, The device includes: The acquisition module is used to acquire a logical comparison group corresponding to each physical node for executing the distributed training task, wherein the logical comparison group includes multiple first training nodes, and the multiple first training nodes in the same logical comparison group run on the same physical node. The acquisition module is used to acquire the training time of each first training node in the training step for each training step, and to calculate a first average value based on the training time of the plurality of first training nodes in the training step. A filtering module is used to filter at least one second training node from the plurality of first training nodes based on the training time of each first training node in the training step. The calculation module is used to calculate a second average value based on the training time of the at least one second training node in the training step, and to perform a difference operation on the first average value and the second average value to obtain the difference index corresponding to the training step. The detection module is used to perform type detection on the at least one second training node based on the difference indicators corresponding to multiple training steps, so as to obtain the node type of the at least one second training node.

10. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the method according to any one of claims 1 to 8.

11. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the method described in any one of claims 1 to 8.

12. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the method according to any one of claims 1 to 8.

Images