Target node screening method and device, electronic equipment and storage medium

Abstract

The invention provides a target node screening method and device, electronic equipment and a storage medium, and relates to the field of artificial intelligence such as deep learning, distributed storage, intelligent chips and computing power. The method comprises the following steps: in a process of executing a model training task, monitoring the performance of the training task in real time; acquiring set communication log information of a set communication group used for executing the model training task in response to determining that the performance of the training task is reduced and determining that the reduction condition meets a screening triggering condition; and according to the set communication log information, determining a target node which causes performance reduction of the training task from the set communication group. By applying the scheme of the invention, the target node with the problem can be found in time, so that the problem can be processed in time, and the performance of the training task is improved.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, particularly to the fields of deep learning, distributed storage, intelligent chips and computing power, and especially to target node screening methods, devices, electronic devices and storage media. Background Technology

[0002] Training task performance is a crucial metric for model training tasks (especially large-scale training tasks). Model training tasks are typically implemented using ensemble communication groups. Due to the synchronization characteristic of ensemble communication, the failure of one node (such as a slow node) will slow down the entire ensemble communication group, thus degrading the performance of the training task. Summary of the Invention

[0003] This disclosure provides a method, apparatus, electronic device, and storage medium for screening target nodes.

[0004] A target node screening method, comprising:

[0005] During the model training task, the performance of the training task is monitored in real time.

[0006] In response to determining that the performance of the training task has degraded, and that the degradation meets the screening triggering conditions, the aggregate communication log information of the aggregate communication group used to execute the model training task is obtained;

[0007] Based on the aggregate communication log information, the target node causing the performance degradation of the training task is identified from the aggregate communication group.

[0008] A target node screening device, comprising:

[0009] The monitoring module is used to monitor the performance of the training task in real time during the execution of the model training task;

[0010] The acquisition module is used to acquire the set communication log information of the set communication group used to execute the model training task in response to determining that the performance of the training task has deteriorated and that the deterioration meets the screening triggering conditions;

[0011] The screening module is used to identify the target node that causes the performance degradation of the training task from the collective communication group based on the collective communication log information.

[0012] An electronic device, comprising:

[0013] At least one processor; and

[0014] A memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described above.

[0016] A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the methods described above.

[0017] A computer program product includes a computer program / instructions that, when executed by a processor, implement the method described above.

[0018] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0019] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0020] Figure 1 This is a flowchart of the first embodiment of the target node screening method described in this disclosure;

[0021] Figure 2 This is a flowchart of the second embodiment of the target node screening method described in this disclosure;

[0022] Figure 3 This is a flowchart of the third embodiment of the target node screening method described in this disclosure;

[0023] Figure 4 This is a flowchart of the fourth embodiment of the target node screening method described in this disclosure;

[0024] Figure 5 This is a schematic diagram of the composition structure of Embodiment 500 of the target node screening device described in this disclosure;

[0025] Figure 6 A schematic block diagram of an electronic device 600 that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0026] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0027] Furthermore, it should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0028] Figure 1 This is a flowchart of the first embodiment of the target node screening method described in this disclosure. Figure 1 As shown, the specific implementation methods are as follows.

[0029] In step 101, the performance of the training task is monitored in real time during the execution of the model training task.

[0030] In step 102, in response to determining that the performance of the training task has deteriorated and that the deterioration meets the screening triggering conditions, the aggregate communication log information of the aggregate communication group used to perform the model training task is obtained.

[0031] In step 103, the target node causing the performance degradation of the training task is determined from the ensemble communication group based on the ensemble communication log information.

[0032] For training task performance degradation caused by slow nodes, traditional methods of monitoring network bandwidth usually do not reveal any problems. That is, the network card bandwidth of all machines is consistent, making it impossible to locate the slow node through inter-machine bandwidth. Moreover, there are no alarms or abnormal logs, making it difficult to find the slow node and thus the problem cannot be solved.

[0033] By employing the scheme described in the above method embodiment, when it is determined that the performance of the training task has deteriorated, the target node with the problem can be quickly and accurately identified based on the acquired collection communication log information. The target node usually refers to the slow node, thereby enabling timely handling of the problem and improvement of training task performance.

[0034] The scheme described in this disclosure does not limit how to monitor the performance of the training task. For example, it can refer to monitoring a predetermined speed indicator to reflect training efficiency. Once it is determined that the speed indicator has decreased (i.e., training has slowed down) and the decrease is greater than a predetermined threshold, it can be determined that the screening trigger condition is met, and the collective communication log information of the collective communication group used to execute the model training task can be obtained.

[0035] Collective communication is a global communication operation mode in which all processes within a process group participate, and multiple processes coordinate to exchange data in a specific way (such as broadcasting, reduction, etc.). Collective communication libraries are software libraries that implement this mode; that is, software libraries designed to enable multi-process collaborative data exchange.

[0036] The fault characteristics of slow nodes can be indirectly reflected by some typical features at the communication layer. Therefore, the solution described in this disclosure can screen out slow nodes in the collective communication group by acquiring and analyzing collective communication log information, thereby improving the efficiency of problem location.

[0037] The collective communication library in the scheme described in this disclosure can be the NVIDIA Collective Communications Library (NCCL) or the Accelerated Computing Unit Collective Communication Library (XCCL), etc., and has wide applicability. However, the collective communication log information obtained and the method of determining the target node based on the collective communication log information will also be different for different collective communication libraries, which will be described separately below.

[0038] 1) NCCL

[0039] NCCL is a collection communication library designed for high-performance communication between multiple graphics processing units (GPUs). It supports parallel computing and communication between multiple GPUs, enabling efficient data transfer and synchronization between multiple GPUs to utilize all GPU resources in the cluster. It is widely used in distributed deep learning training, especially when using multiple GPUs for model training. It provides a consistent interface and communication primitives, making data exchange and synchronization between different GPUs simple and efficient.

[0040] Accordingly, in some embodiments of this disclosure, for NCCL, the aggregated communication group may include at least two machines, and each machine may include at least two GPUs.

[0041] The number of machines and the number of GPUs included in each machine can be determined according to actual needs. Moreover, the number of GPUs included in different machines can be the same or different, depending on actual needs.

[0042] In some embodiments of this disclosure, when the screening trigger condition is met, the acquired aggregate communication log information can be first log information. Specifically, the first log information corresponding to any communication channel can be acquired. The first log information can record the operation record information of the most recently completed M aggregate communication operations, where M is a positive integer greater than 1. The specific value of M can be determined according to actual needs, such as 128.

[0043] In NCCL, a channel is a core abstract concept used to optimize communication performance between multiple GPUs. It can be understood as the physical or logical link between multiple GPUs, similar to multiple lanes on a highway. Parallel communication processing can maximize bandwidth utilization. Accordingly, by properly configuring the number of channels, training efficiency can be significantly improved.

[0044] In the scheme described in this disclosure, after determining that the screening triggering conditions are met, the first log information corresponding to any channel can be obtained, which records the operation record information of the latest M set communication operations. Each set communication operation can be participated in by all GPUs on all machines.

[0045] Accordingly, in some embodiments of this disclosure, for each GPU in the aggregated communication group, the following processing can be performed respectively: based on the operation record information, the node waiting time (waitPeer) of the GPU when executing M aggregated communication operations is determined, and the first evaluation index corresponding to the GPU is determined by combining the M waitPeer times, and then the target node can be determined from each GPU according to the first evaluation index.

[0046] For slow nodes, the issue could be due to slow computation or slow communication. For a single GPU, slow computation means that the GPU submits its set communication operation later than other GPUs. Due to the nature of set communication, other GPUs need to execute `waitPeer` to wait for the slow GPU to submit its operation before continuing processing. Similarly, when other GPUs submit their set communication operations, the slow GPU also needs to execute `waitPeer`, resulting in a significantly shorter `waitPeer` time for the slow GPU. Furthermore, for a single GPU, slow communication means that its reduce-copy process is slower than other GPUs. For example, processing the same data block might require 8 clock cycles for reduce-copy and 2 clock cycles for `waitPeer`, while other GPUs might require 2 clock cycles for reduce-copy and 8 clock cycles for `waitPeer`. Therefore, the slow GPU's `waitPeer` time will also be significantly shorter than other GPUs.

[0047] Therefore, the solution described in this disclosure can quickly locate the target node, i.e., quickly locate the slow node, by analyzing the waitPeer time of different GPUs.

[0048] The first log information can record the operation record information of the latest M set communication operations. Specifically, the operation record information of each set communication operation can include: how many waitPeer operations each GPU performed to complete the set communication operation, and the waitPeer time for each operation.

[0049] Taking any GPU a as an example, assuming that M is 128, that is, assuming that a total of 128 collective communication operations are recorded, for ease of description, these 128 collective communication operations are referred to as collective communication operation 1 to collective communication operation 128, and the corresponding 128 operation records are referred to as operation record information 1 to operation record information 128. In addition, assuming that operation record information 1 records that GPU a executed waitPeer twice, with corresponding waitPeer times of 8 clock cycles and 4 clock cycles respectively, then the sum of the two waitPeer times can be determined as the waitPeer time of GPU a when executing collective communication operation 1. In the same way, the waitPeer times of GPU a when executing collective communication operations 2 to collective communication operations 128 can be obtained respectively. Then, the first evaluation index corresponding to GPU a can be determined by combining the 128 waitPeer times.

[0050] As can be seen, by using the above method, the first evaluation index corresponding to each GPU can be determined simply and efficiently with the help of the first log information. Moreover, the first evaluation index can be determined by combining the waitPeer time corresponding to multiple set communication operations, thereby improving the accuracy of the obtained first evaluation index.

[0051] In some embodiments of this disclosure, for each GPU, the average of M waitPeer times can be obtained, and the average can be determined as the first evaluation index corresponding to the GPU. Then, the GPUs can be sorted in descending order of the value of the first evaluation index, and the GPU in the last L positions after sorting can be determined as the target node, where L is a positive integer.

[0052] The specific value of L can be determined according to actual needs. For example, if it is 1, it means that the GPU that is last in the sorted order can be identified as the target node. If it is 2, it means that the GPU that is last two in the sorted order can be identified as the target node.

[0053] Based on the above introduction, Figure 2This is a flowchart of a second embodiment of the target node screening method described in this disclosure. Figure 2 As shown, the specific implementation methods are as follows.

[0054] In step 201, the performance of the training task is monitored in real time during the execution of the model training task.

[0055] In step 202, in response to determining that the training task performance has deteriorated and that the deterioration meets the screening triggering conditions, the first log information corresponding to any channel is obtained. The first log information records the operation record information of the most recently completed M set communication operations, where M is a positive integer greater than 1.

[0056] In step 203, for each GPU in the ensemble communication group, the following processing is performed: based on the operation record information, the waitPeer time of the GPU when executing M ensemble communication operations is determined, and the average of the M waitPeer times is obtained. The average value is then determined as the first evaluation index corresponding to the GPU.

[0057] In step 204, the GPUs are sorted in descending order of the value of the first evaluation index, and the GPU that is last in the sorted list is determined as the target node.

[0058] Once a target node is identified, corresponding processing measures can be taken, such as fixing problems with the target node or removing the target node.

[0059] (ii) XCCL

[0060] As mentioned earlier, NCCL is a collection communication library designed specifically for high-performance communication of GPUs, while XCCL is a collection communication library designed for accelerated computing units, which usually refer to the Kunlun Core (XPU).

[0061] Accordingly, in some embodiments of this disclosure, for XCCL, the aggregated communication group may include at least two machines, and each machine may include at least two XPUs.

[0062] The number of machines and the number of XPUs in each machine can be determined according to actual needs. Moreover, the number of XPUs in different machines can be the same or different, depending on actual needs.

[0063] In addition, for XCCL, the scheme described in this disclosure proposes three target node screening methods, which are introduced below.

[0064] 1) Method 1

[0065] In some embodiments of this disclosure, when the screening trigger condition is met, the acquired collection communication log information can be second log information. Specifically, the second log information corresponding to any channel of each machine can be acquired. The second log information can record the execution time information when the corresponding machine executes the latest acquired N data blocks (steps). That is, it can record the execution time information corresponding to N steps, where N is a positive integer greater than 1. The execution time information includes the execution start time and execution end time of the corresponding step.

[0066] Accordingly, in some embodiments of this disclosure, the following processing can be performed on each machine: based on the execution start time and execution end time, the in-machine execution time of N steps in the machine is determined, and the second evaluation index corresponding to the machine is determined in combination with the N in-machine execution time, and then the target node can be determined from each machine based on the second evaluation index.

[0067] For XCCL, the communication group may also include a proxy for each machine. The proxy acts as the coordinator of communication tasks, managing the data transmission and reception process for Remote Direct Memory Access (RDMA). RDMA is a network communication technology used in XCCL that can reduce latency and increase throughput. Furthermore, to accelerate data processing, large-scale data can be broken down into fixed-size steps (e.g., 16MB) to achieve batch data processing.

[0068] In practical applications, when using the ring algorithm, the process of a step from entering the machine to being sent is: proxy_recv->rank1……rankn->proxy_send. The ring algorithm is a commonly used collective communication mode that can achieve efficient data exchange.

[0069] Here, `proxy_recv` indicates that the proxy has received a step's data via RDMA and can record the execution start time (args.start_time), where `args.start_time` equals the current time (current_time). `rank1...rankn` indicates that the proxy distributes the steps to the corresponding machines, where each XPU processes them sequentially. `proxy_send` indicates that the proxy has performed a send operation, i.e., sending the processing result to other machines via RDMA. In this case, the execution end time (args.end_time) can be recorded, where `args.end_time` equals the current time.

[0070] For any given step, the in-machine execution time of that step can be determined based on the recorded start and end times. For example, the difference between the start and end times can be obtained and the difference can be determined as the required in-machine execution time.

[0071] Assuming N is 10, then taking any machine x as an example, the in-machine execution time of machine x when executing the latest 10 steps can be obtained in the above manner, and the second evaluation index corresponding to machine x can be determined by combining the 10 in-machine execution times.

[0072] It can be seen that, with the help of the second log information, the second evaluation index corresponding to each machine can be determined simply and efficiently. Moreover, the second evaluation index can be determined by combining the in-machine execution time corresponding to multiple steps, thereby improving the accuracy of the obtained second evaluation index.

[0073] In some embodiments of this disclosure, for any machine, the average of N in-machine execution times corresponding to that machine can be obtained, and the average can be determined as the second evaluation index corresponding to that machine. Then, the machines can be sorted in ascending order of the value of the second evaluation index, and the machine at the last P position after sorting can be determined as the target node, where P is a positive integer.

[0074] Using the above method, the P machines with the longest execution time can be identified, which means the P slowest nodes in the machine's internal communication can be identified. Subsequently, corresponding handling measures can be taken for these slow nodes.

[0075] Based on the above introduction, Figure 3 This is a flowchart of the third embodiment of the target node screening method described in this disclosure. Figure 3 As shown, the specific implementation methods are as follows.

[0076] In step 301, the performance of the training task is monitored in real time during the execution of the model training task.

[0077] In step 302, in response to determining that the training task performance has deteriorated and that the deterioration meets the screening triggering conditions, the second log information corresponding to any channel of each machine is obtained. The second log information records the execution time information of the corresponding machine when it executes the latest N steps, where N is a positive integer greater than 1. The execution time information includes the start time and end time of the corresponding step.

[0078] In step 303, the following processing is performed on each machine: based on the execution start time and execution end time, the in-machine execution time of N steps within the machine is determined, and the average of the N in-machine execution times is obtained. The average value is then determined as the second evaluation index corresponding to the machine.

[0079] In step 304, the machines are sorted in ascending order of the value of the second evaluation index, and the machine at the last P position after sorting is determined as the target node, where P is a positive integer.

[0080] The specific values ​​of N and P can be determined according to actual needs.

[0081] 2) Method Two

[0082] In some embodiments of this disclosure, when the screening trigger condition is met, the acquired aggregate communication log information can be third log information. Specifically, third log information corresponding to any channel can be acquired. The third log information can record the operation record information of the latest W aggregate communication operations, where W is a positive integer greater than 1. The specific value of W can be determined according to actual needs, such as 128.

[0083] Accordingly, in some embodiments of this disclosure, for each XPU in the aggregated communication group, the following processing can be performed respectively: based on the operation record information, the waitPeer time of the XPU when performing W aggregated communication operations is determined, and the third evaluation index corresponding to the XPU is determined in combination with the W waitPeer times. Then, the target node can be determined from each XPU according to the third evaluation index.

[0084] Specifically, in some embodiments of this disclosure, for each XPU, the average of the W waitPeer times corresponding to that XPU can be obtained, and the average can be determined as the third evaluation index corresponding to that XPU. Then, the XPUs can be sorted in descending order of the value of the third evaluation index, and the XPU at the last Q position after sorting can be determined as the target node, where Q is a positive integer. The specific value of Q can be determined according to actual needs.

[0085] As can be seen, Method 2 is similar to the method of determining the target node in NCCL, except that the GPU is replaced by the XPU. For specific implementation details, please refer to the aforementioned related instructions.

[0086] The above approach takes into account both slow internal machine communication and slow inter-machine communication, and identifies the corresponding slow nodes. Subsequently, corresponding measures can be taken to address these slow nodes.

[0087] 3) Method Three

[0088] In some embodiments of this disclosure, when the screening triggering conditions are met, the acquired collection communication log information can be fourth log information. Specifically, the fourth log information corresponding to each machine can be acquired separately. The fourth log information can record: the submission time of the latest R work requests (WR) submitted by the proxy corresponding to the machine and the completion time of the completion queue entries (CQE) corresponding to each WQ, where R is a positive integer greater than 1.

[0089] In RDMA technology, data can be sent (submitted) by submitting WR. After the data is sent, the corresponding CQE can be obtained. The submission time of the latest R WR submission and the completion time of obtaining the CQE corresponding to each WQ can be recorded in the fourth log information.

[0090] The specific value of R can be determined according to actual needs. R can refer to a pre-set threshold (e.g., 5). After confirming that the screening trigger condition is met, when 5 WRs are submitted, a fourth log message can be generated, recording the submission time of the 5 WR submissions and the completion time of obtaining the CQE corresponding to each WQ. Alternatively, after confirming that the screening trigger condition is met, once it is determined that the total data volume submitted in the subsequent R (e.g., 3) WR submissions is greater than or equal to a predetermined data volume (e.g., 32MB), a fourth log message can be generated, recording the submission time of the 3 WR submissions and the completion time of obtaining the CQE corresponding to each WQ.

[0091] In some embodiments of this disclosure, the following processing can be performed on each machine: based on the fourth log information corresponding to the machine, determine the total amount of data submitted when submitting WR for R times, and obtain the time difference between the completion time and submission time corresponding to each WR, and determine the sum of the time differences as the reference time, determine the fourth evaluation index corresponding to the machine based on the total amount of data and the reference time, and determine the target node from each machine based on the fourth evaluation index.

[0092] As can be seen, by using the above method, the fourth evaluation index corresponding to each machine can be determined simply and efficiently with the help of the fourth log information. Moreover, the fourth evaluation index can be determined by combining the relevant information when submitting WR multiple times, thereby improving the accuracy of the obtained fourth evaluation index.

[0093] In some embodiments of this disclosure, for any machine, the ratio of the corresponding total data volume to the reference time can be obtained, and the ratio can be determined as the fourth evaluation index corresponding to the machine. Accordingly, the machines can be sorted in descending order of the value of the fourth evaluation index, and the machine in the last T position after sorting can be determined as the target node, where T is a positive integer.

[0094] As can be seen, the fourth evaluation index determined in the above manner can reflect the bandwidth value of the machine. Accordingly, the T machines with the smallest bandwidth value can be identified as slow nodes. The slow nodes are the T slow nodes with the slowest inter-machine communication. Subsequently, corresponding processing measures can be taken for these slow nodes.

[0095] Based on the above introduction, Figure 4 This is a flowchart of the fourth embodiment of the target node screening method described in this disclosure. Figure 4 As shown, the specific implementation methods are as follows.

[0096] In step 401, the performance of the training task is monitored in real time during the execution of the model training task.

[0097] In step 402, in response to determining that the training task performance has deteriorated and that the deterioration meets the screening triggering conditions, the fourth log information corresponding to each machine is obtained. The fourth log information records the submission time of the latest R WR submission of the proxy corresponding to the machine and the completion time of obtaining the CQE corresponding to each WR, where R is a positive integer greater than 1.

[0098] In step 403, the following processing is performed for each machine: Based on the fourth log information corresponding to the machine, the total amount of data submitted when submitting WR in R times is determined, and the time difference between the completion time and submission time corresponding to each WR is obtained. The sum of the time differences is determined as the reference time, and the ratio of the total amount of data to the reference time is obtained. The ratio is determined as the fourth evaluation index corresponding to the machine.

[0099] In step 404, the machines are sorted in descending order of the value of the fourth evaluation index, and the machine in the last T position after sorting is determined as the target node, where T is a positive integer.

[0100] The specific values ​​of R and T can be determined according to actual needs.

[0101] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this disclosure. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this disclosure. Furthermore, for parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0102] The scheme described in this disclosure provides multiple target node screening methods, and the specific method(s) used can be determined according to actual needs. Furthermore, the scheme described in this disclosure can be executed in real time during model training tasks. When task performance degrades, it can be quickly diagnosed promptly without additional task runtime or the need to restart the task for diagnosis. This is highly flexible and convenient, and improves the execution efficiency and effectiveness of model training tasks.

[0103] The above is an introduction to the method embodiments. The following describes the solution described in this disclosure further through device embodiments.

[0104] Figure 5 This is a schematic diagram of the structural composition of Embodiment 500 of the target node screening device described in this disclosure. Figure 5 As shown, it includes: a monitoring module 501, an acquisition module 502, and a screening module 503.

[0105] The monitoring module 501 is used to monitor the performance of the training task in real time during the execution of the model training task.

[0106] The acquisition module 502 is used to acquire the set communication log information of the set communication group used to execute the model training task in response to determining that the performance of the training task has deteriorated and that the deterioration meets the screening triggering conditions.

[0107] The screening module 503 is used to identify the target nodes that cause the performance degradation of the training task from the ensemble communication group based on the ensemble communication log information.

[0108] The corresponding collection communication library for a collection communication group can be NCCL or XCCL, etc.

[0109] In some embodiments of this disclosure, for NCCL, the aggregated communication group may include at least two machines, each of which may include at least two GPUs.

[0110] In some embodiments of this disclosure, the aggregate communication log information may include: first log information. Accordingly, in response to determining that the screening trigger condition is met, the acquisition module 502 may acquire the first log information corresponding to any channel. The first log information records the operation record information of the latest M aggregate communication operations, where M is a positive integer greater than 1.

[0111] In some embodiments of this disclosure, the screening module 503 can perform the following processing on each GPU in the aggregated communication group: based on the operation record information, determine the waitPeer time of the GPU when executing M aggregated communication operations, and combine the M waitPeer times to determine the first evaluation index corresponding to the GPU, and then determine the target node from each GPU based on the first evaluation index.

[0112] Specifically, in some embodiments of this disclosure, the screening module 503 can obtain the average of M waitPeer times for each GPU, and can determine the average as the first evaluation index corresponding to the GPU. Then, the GPUs can be sorted in descending order of the value of the first evaluation index, and the GPU in the last L position after sorting can be determined as the target node, where L is a positive integer.

[0113] In some embodiments of this disclosure, for XCCL, the aggregated communication group may include at least two machines, and each machine may include at least two XPUs.

[0114] In some embodiments of this disclosure, the aggregated communication log information may include: second log information. Accordingly, in response to determining that the screening trigger condition is met, the acquisition module 502 may acquire the second log information of each machine corresponding to any channel. The second log information records the execution time information of the corresponding machine when it executes the latest acquired N steps, where N is a positive integer greater than 1. The execution time information includes: the execution start time and execution end time of the corresponding step.

[0115] In some embodiments of this disclosure, for each machine, the screening module 503 may perform the following processing: determine the in-machine execution time of N steps in the machine according to the execution start time and execution end time, and determine the second evaluation index corresponding to the machine in combination with the N in-machine execution time, and then determine the target node from each machine according to the second evaluation index.

[0116] In some embodiments of this disclosure, the screening module 503 can obtain the average of N in-machine execution times for any machine, and determine the average as the second evaluation index for that machine. Then, the machines can be sorted in ascending order of the value of the second evaluation index, and the machine at the last P position after sorting can be determined as the target node, where P is a positive integer.

[0117] In some embodiments of this disclosure, the aggregate communication log information may include: third log information. Accordingly, in response to determining that the screening trigger condition is met, the acquisition module 502 may acquire the third log information corresponding to any channel. The third log information records the operation record information of the latest completed W aggregate communication operations, where W is a positive integer greater than 1.

[0118] Accordingly, in some embodiments of this disclosure, the screening module 503 can perform the following processing for each XPU in the aggregated communication group: based on the operation record information, determine the waitPeer time of the XPU when it performs W aggregated communication operations, and combine the W waitPeer times to determine the third evaluation index corresponding to the XPU. Then, the target node can be determined from each XPU based on the third evaluation index.

[0119] Specifically, in some embodiments of this disclosure, the screening module 503 can obtain the average of the W waitPeer times corresponding to each XPU, and can determine the average as the third evaluation index corresponding to the XPU. Then, the XPUs can be sorted in descending order of the value of the third evaluation index, and the XPU in the last Q position after sorting can be determined as the target node, where Q is a positive integer.

[0120] In some embodiments of this disclosure, the aggregated communication group may further include: a proxy corresponding to each machine, and the aggregated communication log information may include: fourth log information. Accordingly, in response to determining that the screening trigger condition is met, the acquisition module 502 may acquire the fourth log information corresponding to each machine. The fourth log information records: the submission time of the latest R-th WR submission by the proxy corresponding to the machine and the completion time of acquiring the CQE corresponding to each WQ, where R is a positive integer greater than 1.

[0121] In some embodiments of this disclosure, for each machine, the screening module 503 may perform the following processing: based on the fourth log information corresponding to the machine, determine the total amount of data submitted when submitting WR for R times, and obtain the time difference between the completion time and submission time corresponding to each WR, and determine the sum of the time differences as the reference time, obtain the ratio of the total amount of data to the reference time, determine the ratio as the fourth evaluation index corresponding to the machine, and determine the target node from each machine based on the fourth evaluation index.

[0122] In some embodiments of this disclosure, for any machine, the screening module 503 can obtain the ratio of the corresponding total data volume to the reference time, and can determine the ratio as the fourth evaluation index corresponding to the machine. Accordingly, the machines can be sorted in descending order of the value of the fourth evaluation index, and the machine in the last T position after sorting can be determined as the target node, where T is a positive integer.

[0123] Figure 5 The specific workflow of the illustrated device embodiment can be found in the relevant descriptions in the foregoing method embodiments.

[0124] The solutions described in this disclosure can be applied to the field of artificial intelligence, particularly deep learning, distributed storage, intelligent chips, and computing power. Artificial intelligence is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It involves both hardware and software technologies. Artificial intelligence hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. Artificial intelligence software technologies mainly include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0125] Furthermore, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0126] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0127] Figure 6 A schematic block diagram of an electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0128] like Figure 6 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. The RAM 603 may also store various programs and data required for the operation of the electronic device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0129] Multiple components in electronic device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of displays, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows electronic device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0130] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as those described in this disclosure. For example, in some embodiments, the methods described in this disclosure can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the methods described in this disclosure can be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the methods described herein by any other suitable means (e.g., by means of firmware).

[0131] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems-on-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0132] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0133] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM), flash memory, optical fiber, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0134] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0135] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0136] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0137] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0138] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A target node screening method, comprising: During the model training task, the performance of the training task is monitored in real time. In response to determining that the performance of the training task has degraded, and that the degradation meets the screening triggering conditions, the aggregate communication log information of the aggregate communication group used to execute the model training task is obtained; Based on the aggregate communication log information, the target node causing the performance degradation of the training task is identified from the aggregate communication group.

2. The method according to claim 1, wherein, The collective communication library corresponding to the collective communication group includes: a graphics processor collective communication library; The collective communication group includes at least two machines, and each machine includes at least two graphics processors.

3. The method according to claim 2, wherein, The aggregate communication log information includes: first log information; The step of obtaining the aggregate communication log information of the aggregate communication group used to perform the model training task includes: Obtain the first log information corresponding to any communication channel. The first log information records the operation record information of the most recently completed M set communication operations, where M is a positive integer greater than 1.

4. The method according to claim 3, wherein, The target nodes identified from the collective communication group that cause the performance degradation of the training task include: For each graphics processor in the collective communication group, the following processing is performed: based on the operation record information, the node waiting time of the graphics processor when executing the M collective communication operations is determined, and the first evaluation index corresponding to the graphics processor is determined by combining the waiting time of the M nodes. The target node is determined from each graphics processor based on the first evaluation metric.

5. The method according to claim 4, wherein, The step of determining the first evaluation index corresponding to the graphics processor by combining the waiting times of M nodes includes: obtaining the average waiting time of the M nodes, and determining the average as the first evaluation index; The step of determining the target node from each graphics processor according to the first evaluation index includes: sorting each graphics processor in descending order of the value of the first evaluation index, and determining the graphics processor in the last L position after sorting as the target node, where L is a positive integer.

6. The method according to claim 1, wherein, The collective communication library corresponding to the collective communication group includes: the accelerated computing unit collective communication library; The collective communication group includes at least two machines, and each machine includes at least two accelerated computing units.

7. The method according to claim 6, wherein, The aggregated communication log information includes: second log information; The step of obtaining the aggregate communication log information of the aggregate communication group used to perform the model training task includes: The second log information corresponding to any communication channel of each machine is obtained respectively. The second log information records the execution time information when the corresponding machine executes the latest N data blocks, where N is a positive integer greater than 1. The execution time information includes: the execution start time and execution end time of the corresponding data block.

8. The method according to claim 7, wherein, The target nodes identified from the collective communication group that cause the performance degradation of the training task include: For each machine, the following processing is performed: Based on the execution start time and the execution end time, the in-machine execution time of N data blocks in the machine is determined, and the second evaluation index corresponding to the machine is determined by combining the N in-machine execution times; The target node is determined from each machine based on the second evaluation metric.

9. The method according to claim 8, wherein, The step of determining the second evaluation index corresponding to the machine by combining N in-machine execution times includes: obtaining the average of the N in-machine execution times, and determining the average as the second evaluation index; The step of determining the target node from each machine according to the second evaluation index includes: sorting each machine in ascending order of the value of the second evaluation index, and determining the machine at the last P position after sorting as the target node, where P is a positive integer.

10. The method according to claim 6, wherein, The aggregated communication log information includes: third log information; The step of obtaining the aggregate communication log information of the aggregate communication group used to perform the model training task includes: Obtain the third log information corresponding to any communication channel. The third log information records the operation record information of the latest W set communication operations, where W is a positive integer greater than 1.

11. The method according to claim 10, wherein, The target nodes identified from the collective communication group that cause the performance degradation of the training task include: For each accelerated computing unit in the aggregated communication group, the following processing is performed: based on the operation record information, the node waiting time of the accelerated computing unit when executing the W aggregated communication operations is determined, and the third evaluation index corresponding to the accelerated computing unit is determined by combining the waiting time of the W nodes. The target node is determined from each accelerated computing unit based on the third evaluation metric.

12. The method according to claim 11, wherein, The step of determining the third evaluation index corresponding to the accelerated computing unit by combining the waiting times of W nodes includes: obtaining the average waiting time of the W nodes and determining the average as the third evaluation index; The step of determining the target node from each accelerated computing unit according to the third evaluation index includes: sorting each accelerated computing unit in descending order of the value of the third evaluation index, and determining the accelerated computing unit in the last Q position after sorting as the target node, where Q is a positive integer.

13. The method according to claim 6, wherein, The collective communication group also includes: agents corresponding to each machine; The aggregate communication log information includes: fourth log information; The step of obtaining the aggregate communication log information of the aggregate communication group used to perform the model training task includes: The fourth log information corresponding to each machine is obtained respectively. The fourth log information records: the submission time of the latest R work requests submitted by the agent corresponding to the machine, and the completion time when the completion queue entries corresponding to each work request are obtained respectively, where R is a positive integer greater than 1.

14. The method according to claim 13, wherein, The target nodes identified from the collective communication group that cause the performance degradation of the training task include: For each machine, the following processing is performed: Based on the fourth log information corresponding to the machine, the total amount of data submitted when submitting the R work requests is determined, and the time difference between the completion time and the submission time corresponding to each work request is obtained. The sum of the time differences is used as the reference time, and the fourth evaluation index corresponding to the machine is determined based on the total amount of data and the reference time. The target node is determined from each machine based on the fourth evaluation metric.

15. The method according to claim 14, wherein, The step of determining the fourth evaluation index corresponding to the machine based on the total data volume and the reference time includes: obtaining the ratio of the total data volume to the reference time, and determining the ratio as the fourth evaluation index; The step of determining the target node from each machine according to the fourth evaluation index includes: sorting each machine in descending order of the value of the fourth evaluation index, and determining the machine in the last T position after sorting as the target node, where T is a positive integer.

16. A target node screening device, comprising: The monitoring module is used to monitor the performance of the training task in real time during the execution of the model training task; The acquisition module is used to acquire the set communication log information of the set communication group used to execute the model training task in response to determining that the performance of the training task has deteriorated and that the deterioration meets the screening triggering conditions; The screening module is used to identify the target node that causes the performance degradation of the training task from the collective communication group based on the collective communication log information.

17. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-15.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-15.

19. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the method of any one of claims 1-15.

Images