Abnormality detection method and device, electronic device, distributed computing system and storage medium

By initiating detection of communication connections in a distributed computing system and analyzing the return results, determining the abnormal computing unit, the abnormal problems of non-CPU computing units and associated devices in a large-scale distributed computing system are solved, and training efficiency and accuracy are improved. The problem of positioning and handling exceptions in the prior art is developed.

CN118012680BActive Publication Date: 2024-11-22BEIJING XIYU JIZHI TECH CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311869853.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-12-31
Publication Date
2024-11-22
Estimated Expiration
2043-12-31

AI Technical Summary

Technical Problem

In distributed computing systems, especially in large-scale neural network model training environments, it is difficult to accurately locate abnormalities in non-CPU computing units such as GPUs and their associated switches and hardware devices, resulting in a decrease in training efficiency.

Method used

By initiating a detection communication connection between the computing nodes of the distributed computing system, using a PCIe switch to bypass the CPU for detection, performing correctness detection and transmission stability detection (bandwidth and delay detection), analyzing and returning results to determine the abnormal computing unit, And disconnect or block the abnormal unit to avoid dragging down system performance.

Benefits of technology

It effectively solves the problem that abnormalities in non-central processor computing units and associated nodes in large-scale distributed computing systems drag down neural network training, improves training efficiency and accuracy, reduces noise, and can quickly locate and deal with PCIe fault scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118012680B_ABST
    Figure CN118012680B_ABST
Patent Text Reader

Abstract

The present application discloses an anomaly detection method for a distributed computing system, comprising: initiating a detection communication connection to at least one of the computing units of at least a part of the computing nodes of the distributed computing system, the detection communication connection is transmitted via the first switch communication link to which the detected computing unit is connected and does not pass through the CPU of the computing node where the detected computing unit is located; receiving the return result of the detection communication connection from each of the detected computing units; through a given detection, analyzing the return result to determine the abnormal computing unit in the detected computing unit. The present application also relates to an anomaly detection device, electronic device, distributed computing system and storage medium for a distributed computing system. The solution of the present application can effectively solve the current problem of the difficulty in locating the anomalies of the computing units of non-central processing units and the switches and hardware devices in the associated nodes in large-scale distributed computing systems, which hinders the distributed training of neural networks.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of distributed computing, and in particular to an anomaly detection method and device for a distributed computing system, an electronic device, a distributed computing system, and a storage medium. Background Art

[0002] In the field of artificial intelligence, due to the urgent need for high-performance computing, non-CPU processing units such as GPUs are often used to optimize the training and inference processes. These units are designed for parallel processing and can effectively handle complex operations, significantly improving efficiency compared to CPUs.

[0003] With the development of artificial intelligence technology, especially with the booming development of large language models (LLM), the scale of neural network models is getting larger and larger, which in turn promotes the widespread adoption of distributed training methods. In this distributed training environment, a considerable part of the calculations and related communications and memory access activities are mainly performed by GPUs rather than CPUs, which involves intensive communication between GPUs and between GPUs and other heterogeneous components.

[0004] Such a distributed computing system often includes multiple computing nodes, each of which may include multiple non-CPU computing units, such as GPUs, for performing distributed training. When there are abnormalities in computing units, such as GPUs and their connecting parts, in such a distributed computing network, it is difficult to accurately locate such abnormal computing units. If the presence of abnormal computing units cannot be determined as soon as possible, the abnormalities in these computing units, such as failures, bottlenecks, downgrades, and other problems, will greatly drag down the training efficiency of the entire distributed computing system.

[0005] As distributed training is in its infancy, the industry currently lacks sufficient understanding of this issue and lacks corresponding solutions.

[0006] The description of the background technology is intended to help understand the relevant technology in the relevant field, and does not constitute an admission that the content of the background technology belongs to the prior art. Summary of the invention

[0007] The present application embodiment aims to provide an anomaly detection method and device for a distributed computing system, an electronic device, a distributed computing system and a storage medium, which can alleviate or solve at least one of the technical problems mentioned above.

[0008] In a first aspect, a method for detecting anomalies in a distributed computing system is provided. The distributed computing system includes a plurality of computing nodes, each computing node includes a CPU, a first switch, and one or more computing units, the computing unit is a non-CPU processing unit, wherein the first switch is connected upstream to an upper-level switch or a CPU interface, and the first switch is connected downstream to a plurality of hardware devices, and the plurality of hardware devices connected downstream to the first switch include at least one of the computing units.

[0009] Here, the anomaly detection method includes:

[0010] Initiate a detection communication connection to at least one of the computing units of at least a portion of the computing nodes, wherein the detection communication connection is transmitted via a first switch communication link to which the detected computing unit is connected and does not pass through a CPU of the computing node where the detected computing unit is located;

[0011] receiving a return result of the detection communication connection from each of the detected computing units;

[0012] Through a given detection, the returned result is analyzed to determine an abnormal computing unit among the detected computing units.

[0013] In some embodiments, the detection communication connection does not pass through an upper-level switch of the first switch to which the detected computing unit is connected.

[0014] In some embodiments, the given detection includes a correctness detection and a transmission stability detection.

[0015] In some embodiments, the transmission stability detection includes bandwidth detection and / or delay detection.

[0016] In some embodiments, the step of analyzing the returned result through a given detection to determine an abnormal computing unit in the detected computing unit includes:

[0017] Perform multiple correctness tests, and determine as abnormal computing units a computing unit that fails one or more of the correctness tests, a first switch that fails one or more of the correctness tests, a computing unit connected to a communication link of the first switch, or a computing unit connected to the same first switch as a hardware device that fails one or more of the correctness tests;

[0018] During the bandwidth detection, the connection bandwidth of all the computing units detected in the detection communication is determined, and the quantile bandwidth is determined based on a given first quantile threshold, and the computing units with a bandwidth less than the quantile bandwidth or less than the threshold bandwidth are determined as abnormal computing units, and the threshold bandwidth is a first given difference less than the quantile bandwidth; and / or

[0019] During the delay detection, the connection delay of all the computing units detected in the detection communication is determined, and the quantile delay is determined based on a given second quantile threshold, and the computing units with a delay greater than the quantile delay or greater than the threshold delay are determined as abnormal computing units, and the threshold delay is slower than the quantile delay by a second given difference.

[0020] In some embodiments, the correctness detection includes a memory allocation correctness detection, a memory release correctness detection, a virtual memory allocation correctness detection and / or a virtual memory release correctness detection of the detected computing unit.

[0021] In some embodiments, the correctness test is selected from at least one of the following tests:

[0022] Basic sub-thread fixed buffer memory allocation detection,

[0023] Basic child thread fixed buffer virtual memory allocation detection,

[0024] Basic memory allocation detection,

[0025] Basic small buffer mapping detection,

[0026] Basic misaligned mapping detection,

[0027] Basic virtual memory allocation detection,

[0028] Basic testing with tokens,

[0029] Data verification memory allocation detection,

[0030] Data verification virtual memory allocation detection,

[0031] Access memory allocation invalidation detection after release,

[0032] Access-after-free virtual memory allocation invalidation detection,

[0033] After GDR is turned off, access memory allocation failure detection,

[0034] After GDR is turned off, access to virtual memory allocation failure detection,

[0035] After release, access the branch operation fork memory allocation failure detection,

[0036] Access fork virtual memory allocation failure detection after release,

[0037] Fork memory allocation failure detection after GDR mapping,

[0038] Fork virtual memory allocation failure detection after GDR mapping,

[0039] Child process GDR maps parent process fork memory allocation failure detection,

[0040] Child process GDR mapping parent process fork virtual memory allocation failure detection,

[0041] The child process GDR fixes the parent process's token memory failure detection.

[0042] Memory allocation failure detection after fork mapping and release,

[0043] Detection of virtual memory allocation failure after fork mapping and release,

[0044] Double mapped memory allocation failure detection,

[0045] Double mapped virtual memory allocation failure detection,

[0046] Unix socket shared file descriptor GDR mapping memory allocation failure detection,

[0047] Unix socket shared file descriptor GDR mapping virtual memory allocation failure detection,

[0048] Unix socket shared file descriptor GDR fixed buffer memory allocation failure detection,

[0049] Unix socket shared file descriptor GDR fixed buffer virtual memory allocation failure detection.

[0050] In some embodiments, the anomaly detection method further includes:

[0051] The computing unit determined to be abnormal, the first switch connected to the computing unit determined to be abnormal, or other hardware devices connected to the same first switch as the computing unit determined to be abnormal are disconnected or shielded.

[0052] In some embodiments, the anomaly detection method includes: newly connecting a plurality of the computing nodes, each of the newly connected computing nodes including one or more computing units; initiating a detection communication connection to at least one of the computing units of at least a portion of the computing nodes, including: initiating the detection communication connection to the computing unit of the newly connected computing node; the anomaly detection method also includes: preventing computing units determined to be abnormal from accessing.

[0053] In some embodiments, the first switch and the upper-level switch are PCIe switches (PCIeSwitch).

[0054] In a second aspect, an abnormality detection device for a distributed computing system is provided. The distributed computing system includes a plurality of computing nodes, each computing node includes a CPU, a first switch, and one or more computing units, the computing unit is a non-CPU processing unit, wherein the first switch is connected to an upper-level switch or a CPU interface in an upstream direction, and the first switch is connected to a plurality of hardware devices in a downstream direction, and the plurality of hardware devices connected in a downstream direction by the first switch include at least one of the computing units.

[0055] Here, the abnormality detection device includes:

[0056] a communication initiating unit, configured to initiate a detection communication connection to at least one of the computing units of at least a portion of the computing nodes, wherein the detection communication connection is transmitted via a first switch communication link to which the detected computing unit is connected and does not pass through a CPU of the computing node where the detected computing unit is located;

[0057] a receiving unit configured to receive a return result of the detection communication connection from each of the detected computing units; and

[0058] The abnormality determination unit is configured to analyze the returned result through a given detection to determine an abnormal computing unit in the detected computing unit.

[0059] In a third aspect, an electronic device is provided, comprising a processor and a memory storing a computer program, wherein the processor is configured to implement the method described in any embodiment of the present application when running the computer program.

[0060] In a fourth aspect, a distributed computing system is provided, comprising a plurality of computing nodes, each computing node comprising a CPU, a first switch and one or more computing units, the computing units being non-CPU processing units, wherein the first switch is upstream connected to an upper-level switch or a CPU interface, the first switch is downstream connected to a plurality of hardware devices, and the plurality of hardware devices downstream connected to the first switch include at least one of the computing units;

[0061] The distributed computing system also includes one or more electronic devices according to any embodiment of the present application.

[0062] In some embodiments, one or more computing nodes of the plurality of computing nodes serve as the one or more electronic devices.

[0063] In some embodiments, the plurality of hardware devices connected to the first switch further include a network interface card NIC, and the computing unit includes a GPU.

[0064] In some embodiments, at least some of the computing nodes include two levels of switches, the first level of switches includes the first switch, and the second level of switches includes a second switch to which the first switch is upstream connected, wherein the first switch has two downstream ports, and the two downstream ports of at least some of the first switches are respectively connected to the NIC and the GPU.

[0065] In some embodiments, at least some of the computing nodes include a first-level switch, the first switch is upstream connected to the root complex RC, wherein the first switch has two downstream ports, and the two downstream ports of at least some of the first switches are respectively connected to the NIC and the GPU.

[0066] In a fifth aspect, a storage medium is provided, wherein the storage medium stores a computer program, and the computer program is configured to implement the method described in any embodiment of the present application when executed.

[0067] The anomaly detection method for a distributed computing system of the present application includes: initiating a detection communication connection to at least one of the computing units of at least a part of the computing nodes of the distributed computing system, the detection communication connection is transmitted via a first switch communication link to which the detected computing unit is connected and does not pass through the CPU of the computing node where the detected computing unit is located; receiving a return result of the detection communication connection from each of the detected computing units; through a given detection, analyzing the return result to determine the abnormal computing unit in the detected computing unit. This solution can effectively solve the current problem of difficulty in locating the anomalies of non-central processing unit computing units and switches and hardware devices in associated nodes in large-scale distributed computing systems, which hinders the distributed training of neural networks.

[0068] Further preferred features and effects of the present application will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use are briefly introduced below. It should be clear that the drawings in the following description only cover some embodiments of the present application, and ordinary technicians in this field can obtain other drawings based on these drawings without performing creative work. The purpose of these drawings is to better illustrate the technical details to help understand the implementation methods of the present application, among which:

[0070] Figure 1 A topology diagram in an exemplary computing node is shown;

[0071] Figure 2 Another topology diagram in an exemplary computing node is shown;

[0072] Figure 3 A schematic structural diagram showing a distributed computing system and computing nodes thereof that can be used for distributed training;

[0073] Figure 4 Another schematic structural diagram of a distributed computing system and computing nodes thereof that can be used for distributed training is shown;

[0074] Figure 5 Another schematic structural diagram showing a distributed computing system and computing nodes thereof that can be used for distributed training;

[0075] Figure 6 A schematic flow chart showing an abnormality detection method according to an embodiment of the present application;

[0076] Figure 7 A schematic flow chart showing bandwidth detection according to an embodiment of the present application is shown;

[0077] Figure 8 A schematic flowchart showing delay detection according to an embodiment of the present application is shown;

[0078] Fig. 9 An exemplary structural diagram showing an abnormality detection device according to an embodiment of the present application;

[0079] Fig.10 A schematic diagram showing the structure of an electronic device that can implement the method of the embodiment of the present application; and

[0080] Fig.11 A schematic diagram showing the structure of a distributed computing system according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0081] The exemplary embodiments of the present application will be described in detail below, and the illustrations of the examples have been shown in the accompanying drawings. When referring to the drawings, unless otherwise specified, the same numbers or marks in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Instead, they are only examples of some aspects of the devices and methods covered by one or more embodiments of this specification as detailed in the claims of this application.

[0082] The term "including" and its variations used in this specification are all intended to be broadly inclusive, i.e., "including but not limited to" the items listed. Unless otherwise stated, the term "or" means "and / or", the term "based on" means relying on, or at least partially relying on, the terms "an example embodiment" and "an embodiment" refer to at least one example embodiment, and the term "another embodiment" means at least one different embodiment. The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0083] Currently, many distributed computing systems have been proposed for various scenarios that require high performance, large computing volume and / or large data volume. Computing scenarios, such as distributed training of neural networks. In these distributed computing systems, multiple computing nodes are connected through a network, and computing nodes often use other computing units other than CPUs to perform part of the computing, or even most or all of the computing tasks. Such computing units are, for example, GPUs. Accordingly, in a distributed training environment, a considerable portion of the computing and related communications and memory access activities are mainly performed by GPUs, rather than CPUs, which involves intensive communications between GPUs and between GPUs and other heterogeneous components.

[0084] Combined with reference Figure 1 and Figure 2 , showing different topological diagrams of exemplary nodes. Figure 1 As shown in FIG, in a computing node, a communication mode between a GPU and other hardware (such as another GPU or an RDMA network interface card RNIC) is to bypass the CPU (and then pass through the node host memory) through a PCIe switch and then transmit to another GPU. Figure 2 As shown in the figure, the second communication mode between the GPU and other hardware (such as another GPU or RDMANIC) is to bypass the CPU dependency through DMA and directly transmit to the RNIC through the PCIe Switch through GDR (GPU DirectRDMA). As mentioned earlier, since the GPU is the main executor of a considerable part of the calculations and related communication and memory access activities in distributed training, the second communication mode is often more advantageous and more important in calculations such as distributed training.

[0085] In addition, due to the huge number of parameters, a large number of computing nodes and their computing units, such as GPUs (e.g., 100 to 10,000 levels), are required in the distributed computing system used for distributed training of large language models (LLMs), and there are inevitably transmission requirements for synchronization and updating of training parameters (e.g., gradient parameters) and model parameters between different GPUs. The failure of a single GPU or even just poor performance, such as bottlenecks or degradation, may still have a significant impact on distributed training. However, due to the large number, it is often difficult to accurately locate these abnormal GPUs or their associated switches or hardware.

[0086] The inventors thus realized the need to provide the ability to quickly locate abnormal non-CPU computing units, such as GPUs, in larger-scale distributed computing systems. However, traditional solutions often focus on fault detection and traditional fault detection solutions are often executed by the host, and are usually only indicator-driven or communication verification is performed at the link layer. These solutions are limited by hardware capabilities and indicator expression capabilities, and these solutions add too much noise and cannot accurately confirm abnormal computing units, such as GPUs, in larger-scale distributed computing systems. In addition, since PCIe failure scenarios are relatively complex, it is even more difficult for traditional fault detection solutions to solve such problems.

[0087] In this regard, an embodiment of the present application provides an anomaly detection method for a distributed computing system, which can preferably be implemented by one or more computing nodes of the computing system. The method can be implemented in computing nodes where multiple non-detected computing units are located (referred to as "non-detected"). Preferably, anomaly detection can be performed multiple times, with different non-detected computing nodes performing anomaly detection each time; anomaly detection can also be performed once or multiple times, with multiple non-detected computing nodes performing anomaly detection in parallel.

[0088] In an embodiment of the present application, the distributed computing system may include multiple computing nodes, each computing node includes a CPU, a first switch and one, preferably multiple computing units. The computing unit is a non-CPU processing unit (such as a GPU). The first switch is connected upstream to an upper-level switch or a CPU interface. The first switch is connected downstream to multiple hardware devices, and the multiple hardware devices connected downstream to the first switch include at least one of the computing units. In a preferred embodiment, the first switch and the upper-level switch are PCIe switches (PCIe Switch).

[0089] Figures 3 to 5 Several schematic structures of a distributed computing system and computing nodes thereof for distributed training are shown, but the computing system to which the anomaly detection solution of the present application can be applied is not limited thereto.

[0090] Combined with reference Figures 3 to 5, distributed computing systems ( Figures 3 to 5 (not shown) may include multiple computing nodes 310, 410, 510, wherein Figures 3 to 5 The ellipsis in the figure schematically represents a plurality of computing nodes. Each computing node includes a CPU (not shown), a first switch 302, 402, 502, and one or more computing units 301, 401, 501, 1101, which are non-CPU processing units. Figure 3 and Figure 4 In the illustrated embodiment, each computing node has 8 computing units, such as GPUs G0-G7. Figure 5 In the illustrated embodiment, each computing node has four computing units, such as GPUs G0-G3. In the illustrated embodiment, the computing unit is preferably a GPU, but may also be other processing units that can be used for machine learning training or inference or other high-performance computing, such as an NPU.

[0091] like Figure 3 and Figure 4 As shown, the first switch 302, 402 is connected to the upper switch in the upstream direction. More specifically, the computing nodes 310, 410 may include two levels of switches, the first level of switches including the above-mentioned first switches 302, 402, and the second level of switches including the second switches 304, 404 connected to the upper switch in the upstream direction. Here, the first switch is the bottom switch, and the second switch is the upper switch. The second switch 304, 404 is connected to the root complex (RC) 305, 405 as the CPU and PCIe bus interface in the upstream direction, and is also presented in the form of a CPU slot, such as Socket 0 and Socket 1. Figure 3 In the illustrated embodiment, the second switch 304 is connected to the first switch 302 and another computing unit, such as a GPU, for example, G1, via two downstream ports. Figure 4 In the embodiment shown, both downlink ports of the second switch 404 are connected to the first switch. Figure 3 In the illustrated embodiment, each computing node has four first switches 302; Figure 4 In the illustrated embodiment, each computing node has eight first switches 402 .

[0092] like Figure 5 As shown, the first switch 502 is upstream connected to the root complex RC 505 as the CPU and PCIe bus interface, and is also in the form of a CPU socket, such as Socket 0 and Socket 1. Here, there is only a single-layer switch in the computing node, namely the first switch.

[0093] like Figures 3 to 5As shown, the multiple hardware devices connected downstream to the first switch 310, 410, 510 include at least one of the computing units 301, 401, 501. In addition, the hardware devices connected downstream to the first switch may also include network interface cards NIC 303, 403, 503, preferably RDMA network interface cards (RNIC). Figures 3 to 5 As shown, the first switches 310, 410, 510 have two downlink ports, and at least some of the first switches 310, 410, 510, preferably all of the first switches have two downlink ports (downlinks 306, 406, 506) connected to computing units, such as GPUs and NICs, such as RNICs, respectively. Figure 3 In the illustrated embodiment, each computing node has four RNICs, namely R0-R3; Figure 4 In the illustrated embodiment, each computing node has 8 RNICs, namely R0-R7; Figure 5 In the illustrated embodiment, each computing node has four RNICs, namely R0-R3.

[0094] exist Figures 3 to 5 The first switch and the possible second switch described in the distributed computing system shown are switches within the node, not switch devices for connecting two nodes. Preferably, the first switch and the possible second switch are PCIe switches. Accordingly, the first switch and the possible second switch described in the embodiment of the present application can have any suitable shape and structure, for example, they can be in the form of a PCIe chip and a corresponding slot, and the present application does not limit this.

[0095] Next, the anomaly detection method according to the embodiment of the present application is described. In this article, "abnormal" will be interpreted broadly, including but not limited to failures, as well as bottlenecks, degradations and poor performance of non-faults (when the relevant components can still work). Further, in a preferred embodiment of the present application, the detection scheme explicitly includes bottleneck, degradation and poor performance of non-faults or errors (when the relevant components can still work) on the computing node and the related first switch and the hardware and related communication links, which is particularly conducive to the distributed training of the neural network model.

[0096] Figure 6 The anomaly detection method according to an embodiment of the present application is shown, which may include:

[0097] S610: Initiate a detection communication connection to at least one of the computing units of at least some of the computing nodes.

[0098] In a further embodiment of the present application, the detection communication connection is transmitted via a first switch communication link to which the detected computing unit is connected and does not pass through a CPU of a computing node where the detected computing unit is located.

[0099] In a further embodiment, the detection communication connection does not pass through the upper level switch of the first switch to which the detected computing unit is connected. In a further embodiment, the downlink port (downlink) of the first switch is respectively connected to computing units, such as GPU and NIC, and the detection communication connection is established so that the NIC connected to the first switch corresponding to the detected computing unit (GPU), such as RNIC, passes through a downlink of the first switch corresponding to the detected computing unit, through the first switch corresponding to the detected computing unit, and through another downlink of the first switch corresponding to the detected computing unit to reach the detected computing unit. Figures 3 to 5 Taking the embodiment shown as an example, if GPU G0 is the computing unit to be detected, the detection communication connection passes through NIC 303, 403, 503, such as R0; the downlink 306, 406, 506 on the left side of the figure, the first switch 302, 402, 502 on the left side of the figure, such as a PCIe switch; the downlink 306, 406, 506 on the right side of the first switch; to reach the computing unit 301, 401, 502 to be detected, such as G0. As mentioned above, the abnormality detection is performed by one or more computing nodes in the computing system where the computing unit to be detected is located, and more specifically, the detection communication connection is initiated. In a preferred embodiment of the present application, only the above-mentioned PCIe link of the computing unit to be detected can be detected.

[0100] In a preferred embodiment of the present application, the gdr_copy tool can be used to verify the correctness, performance and GDR capability of the PCIe link.

[0101] S620: Receive a return result of the detection of the communication connection from each of the detected computing units.

[0102] S630: Analyze the returned result through a given detection to determine an abnormal computing unit among the detected computing units.

[0103] In the embodiment of the present application, the given detection includes correctness detection and transmission stability detection.

[0104] In some embodiments described in the present application, the anomaly detection is implemented at the application layer, and preferably implemented through other computing nodes in the computing system, which can help eliminate noise and quickly locate abnormal computing units.

[0105] In an embodiment of the present application, performing correctness detection for analysis in step S630 may include: performing multiple correctness detections, and determining a computing unit that fails to pass one or more of the correctness detections, a first switch that fails to pass one or more of the correctness detections, a computing unit connected to a communication link of the first switch, or a computing unit connected to the same first switch as a hardware device that fails to pass one or more of the correctness detections as an abnormal computing unit.

[0106] According to a further embodiment, the correctness detection includes a memory allocation correctness detection, a memory release correctness detection, a virtual memory allocation correctness detection and / or a virtual memory release correctness detection of the detected computing unit.

[0107] In a preferred embodiment of the present application, when performing correctness detection for analysis in step S630, correctness verification can be performed according to the scenario, and if an error occurs, the hardware behavior can be clearly located.

[0108] Preferably, step S630 includes: determining different assigned tasks of the detected computing unit; performing a correctness check on the detected computing unit, and determining whether the detected computing unit is abnormal according to the assigned tasks and the correctness check results. Here, for example, different computing units may be assigned different tasks, and then when different related correctness check problems are detected, the computing unit involved in the correctness problem, such as a GPU, can be quickly determined according to the association between the task and the correctness check.

[0109] According to a preferred embodiment, the correctness test is selected from at least one of the following tests, preferably including all of the following tests:

[0110] Basic sub-thread fixed buffer memory allocation detection,

[0111] Basic child thread fixed buffer virtual memory allocation detection,

[0112] Basic memory allocation detection,

[0113] Basic small buffer mapping detection,

[0114] Basic misaligned mapping detection,

[0115] Basic virtual memory allocation detection,

[0116] Basic testing with tokens,

[0117] Data verification memory allocation detection,

[0118] Data verification virtual memory allocation detection,

[0119] Access memory allocation invalidation detection after release,

[0120] Access-after-free virtual memory allocation invalidation detection,

[0121] After GDR is turned off, access memory allocation failure detection,

[0122] After GDR is turned off, access to virtual memory allocation failure detection,

[0123] After release, access the branch operation fork memory allocation failure detection,

[0124] Access fork virtual memory allocation failure detection after release,

[0125] Fork memory allocation failure detection after GDR mapping,

[0126] Fork virtual memory allocation failure detection after GDR mapping,

[0127] Child process GDR maps parent process fork memory allocation failure detection,

[0128] Child process GDR mapping parent process fork virtual memory allocation failure detection,

[0129] The child process GDR fixes the parent process's token memory failure detection.

[0130] Memory allocation failure detection after fork mapping and release,

[0131] Detection of virtual memory allocation failure after fork mapping and release,

[0132] Double mapped memory allocation failure detection,

[0133] Double mapped virtual memory allocation failure detection,

[0134] Unix socket shared file descriptor GDR mapping memory allocation failure detection,

[0135] Unix socket shared file descriptor GDR mapping virtual memory allocation failure detection,

[0136] Unix socket shared file descriptor GDR fixed buffer memory allocation failure detection,

[0137] Unix socket shared file descriptor GDR fixed buffer virtual memory allocation failure detection.

[0138] In a further embodiment, the above-mentioned memory may be CUDA memory.

[0139] In another embodiment of the present application, the transmission stability detection includes bandwidth detection, delay detection, or both.

[0140] In an embodiment of the present application, when performing bandwidth detection for analysis in step S630, it may include: determining the connection bandwidth of all computing units detected in the detection communication, determining it as the quantile bandwidth based on a given first quantile threshold, and determining a computing unit that is less than the quantile bandwidth or less than the threshold bandwidth as an abnormal computing unit, and the threshold bandwidth is a first given difference smaller than the quantile bandwidth.

[0141] refer to Figure 7 , when performing bandwidth detection, the bandwidth of all detected computing units can be determined in step S710. In some specific embodiments, multiple different types of bandwidth can be obtained, such as read bandwidth and write bandwidth; and / or, the bandwidth can be obtained multiple times, for example, the bandwidth corresponding to repeated writing of a given size, such as the same size, increasing size, and decreasing size, can be determined. Accordingly, the return result obtained in step S620 can have relevant information. For example, the cuMemAlloc function can be used to allocate memory in the global memory of the GPU. Through device pointers and mapping addresses, the electronic device that initiates the detection communication connection can directly access the detected GPU memory in other computing nodes, focusing mainly on measuring the bandwidth of data writing and reading. The results show the efficiency of GPU memory operations under different test conditions. Continue to refer to Figure 7 , the quantile bandwidth may be determined in step S720 based on the bandwidths of all detected computing units. In an exemplary embodiment, the 90th quantile bandwidth may be selected, but other quantiles are also conceivable.

[0142] Next, in step S730, abnormal computing nodes, such as abnormal GPUs, bottlenecks, and degraded GPUs can be determined in the detected computing nodes, such as GPUs, based on the quantile bandwidth. In some embodiments, an abnormality, such as a bottleneck or degradation, can be determined by setting a threshold bandwidth that is lower than the quantile bandwidth by a first given difference. The first given difference can be a percentage threshold, such as 10% to 30%. In other embodiments, the quantile bandwidth can be directly used as the threshold bandwidth, and the quantile can be selected to be relatively low, such as 10-30 quantiles.

[0143] In an embodiment of the present application, when performing delay detection for analysis in step S630, it may include: determining the connection delay of all detected computing units in the detection communication, determining it as a quantile delay based on a given second quantile threshold, and determining a computing unit with a delay greater than the quantile delay or greater than the threshold delay as an abnormal computing unit, wherein the threshold delay is a second given difference slower than the quantile delay.

[0144] refer to Figure 8When performing latency detection, the latency of all detected computing units may be determined in step S810. In some specific embodiments, the latency may be obtained multiple times. Accordingly, the returned result obtained in step S620 may have relevant information. The returned result may include a benchmark test for measuring the performance of the GPU direct RDMA (GDR) memory copy function, and preferably an iterative test of different data sizes.

[0145] Continue to refer Figure 8 , the quantile bandwidth may be determined in step S820 based on the delays of all detected computing units. In an exemplary embodiment, the 90th percentile delay may be selected, but other quantiles are also conceivable.

[0146] Next, in step S830, an abnormal computing node, such as an abnormal GPU, such as a bottleneck or a degraded GPU, can be determined in the detected computing node, such as a GPU, according to the quantile delay. In some embodiments, an abnormality, such as a bottleneck or degradation, can be determined by setting a threshold delay that is lower than the quantile delay by a second given difference. The second given difference can be a percentage threshold, such as 10% to 30%. In other embodiments, the quantile delay can be directly used as the threshold delay, and the quantile can be selected to be relatively low, such as 10-30 quantiles.

[0147] In the embodiment of the present application, the anomaly detection scheme can be implemented in the deployed computing nodes of the computing system, for example, before executing the computing task. Accordingly, when an abnormal computing unit is detected, the abnormal computing unit can be disconnected or shielded from the computing system.

[0148] In an embodiment of the present application, the abnormality detection method also includes: disconnecting or shielding the computing unit determined to be abnormal, the first switch connected to the computing unit determined to be abnormal, or other hardware devices connected to the same first switch as the computing unit determined to be abnormal.

[0149] In another embodiment, the anomaly detection scheme can also be applied to implement when a computing node is newly connected in the computing system. In this embodiment, the anomaly detection method includes: newly connecting multiple computing nodes, each of the newly connected computing nodes includes one or more computing units; accordingly, step S610 may include: initiating the detection communication connection to the computing unit of the newly connected computing node; the method also includes: preventing the computing unit determined to be abnormal from accessing.

[0150] By way of explanation and not limitation, in distributed training of large language models (LLMs), efficient parameter synchronization and updates across GPUs are important to maintain model consistency and accuracy. Each GPU in the network may be processing different parts of the model or different batches of data, but they all need to share insights to update the model uniformly. This process is highly dependent on the speed and efficiency of the communication network, typically involving PCIe switches and RDMA devices. As a result, even if the PCIe switches or RDMA devices are not faulty, any performance degradation or bottleneck or degradation can cause significant problems in distributed training, including parameter synchronization delays. Specifically, for LLMs, all nodes in the network may need to synchronize billions of parameters regularly. Even minor delays or bandwidth limitations can result in training with outdated parameters, leading to inconsistencies and potentially reducing the overall accuracy of the model or increasing training time. In this regard, the embodiments of the present application provide an anomaly detection scheme for a distributed computing system, which can specifically solve the problem that it is difficult to locate the anomalies of the computing units of non-central processors and the switches and hardware devices in the associated nodes in large-scale distributed computing systems, which hinder the distributed training of neural networks. For example, on the one hand, correctness verification can be performed according to the scenario, and if an error occurs, the hardware behavior can be clearly located. On the other hand, stability verification can be performed according to performance, and if an error / performance degradation occurs, the PCIe fault / anomaly can be located. In addition, although this article describes the anomaly detection method of the embodiments of the present application as being used in a computing system for distributed training of neural networks, it is conceivable that the anomaly detection scheme of the embodiments of the present application can be used in other application scenarios, such as other distributed computing systems with high performance, large computing volume and / or large data volume computing.

[0151] In another embodiment of the present application, an anomaly detection device for a distributed computing system may be provided, which may be implemented at the software level, for example. The anomaly detection device may be used to detect a distributed computing system described elsewhere in this document, such as a distributed computing system described above with reference to anomaly detection methods, such as Figures 3 to 5 Those shown will not be repeated here.

[0152] Fig. 9An abnormality detection device 900 of an embodiment of the present application is used. The abnormality detection device 900 may include a communication initiating unit 910, a receiving unit 920 and an abnormality determination unit 930. In this embodiment, the communication initiating unit 910 may be configured to initiate a detection communication connection to at least one of the computing units of at least a portion of the computing nodes. Here, the detection communication connection is transmitted via the first switch communication link to which the detected computing unit is connected and does not pass through the CPU of the computing node where the detected computing unit is located. The receiving unit 920 may be configured to receive a return result of the detection communication connection from each of the detected computing units. The abnormality determination unit 930 may be configured to analyze the return result to determine an abnormal computing unit in the detected computing unit through a given detection.

[0153] In a further embodiment of the present application, the abnormality determination unit 930 may include a correctness detection subunit, a bandwidth detection subunit and / or a delay detection subunit, and these subunits may be correspondingly configured to perform the correctness detection, bandwidth detection and / or delay detection as described above, which are not elaborated here.

[0154] The method and its steps, sub-steps and features in the embodiments of the present application can be combined with the device of the embodiments of the present application in a non-contradictory manner, and the device and its components, modules, units and features in the embodiments of the present application can be combined with the method of the embodiments of the present application in a non-contradictory manner.

[0155] In some embodiments of the present application, an electronic device is provided, which includes a processor and a memory storing a computer program, and the processor is configured to implement any method of the embodiments of the present application when running the computer program.

[0156] Fig.10 A schematic diagram of an electronic device 1000 that can be used to implement the method of the embodiment of the present application or to implement the embodiment of the present application is shown. In some embodiments, the number of electronic devices may be more or less than the number shown. In some embodiments, a single or multiple electronic devices may be used for implementation. In some embodiments, cloud or distributed electronic devices may also be used for implementation.

[0157] like Fig.10As shown, the electronic device 1000 includes a processor 1010 and a memory 1020. The processor is used to execute programs stored in the memory, which can implement the methods, steps or functions described in the above embodiments when executed by a computer. The processor 1010 may include various types of processors, such as a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), a digital signal processor (DSP), etc. The processor 1010 and the memory 1020 are interconnected via a bus 1030. An input / output (I / O) interface and the like can also be connected to the bus 1030.

[0158] In a further embodiment of the present application, a distributed computing system incorporating the electronic device and / or anomaly detection device described above may also be included.

[0159] refer to Fig.11 Combined with reference Figures 3 to 5 , distributed computing system 1100 ( Figures 3 to 5 The computing unit 300 may include multiple computing nodes 310, 410, 510, 1110, each of which includes a CPU (not shown), a first switch 302, 402, 502, 1102, and one or more computing units 301, 401, 501, 1101. In the illustrated embodiment, the computing unit is a non-CPU processing unit, preferably a GPU, such as Figures 3 to 5 and Fig.11 The G0...G7 shown in the figure can also be other processing units that can be used for machine learning training or inference or other high-performance computing, such as an NPU.

[0160] like Figure 3 and Figure 4 As shown, the first switch 302, 402 is connected to the upper switch in the upstream direction. More specifically, the computing nodes 310, 410 may include two levels of switches, the first level of switches including the first switches 302, 402, and the second level of switches including the second switches 304, 404 connected to the upper switch in the upstream direction. Here, the first switch is a bottom-layer switch, and the second switch is an upper-layer switch. The second switch 304, 404 is connected to the root complex RC as the CPU and PCIe bus interface in the upstream direction, and is also presented in the form of a CPU slot, such as Socket 0 and Socket 1. Figure 3 In the illustrated embodiment, the second switch 304 is connected to the first switch 302 and another computing unit, such as a GPU, for example, G1, via two downstream ports. Figure 4 In the illustrated embodiment, both downlink ports of the second switch 404 are connected to the first switch.

[0161] like Figure 5As shown, the first switch 502 is upstream connected to the root complex RC serving as the CPU and PCIe bus interface, and is also in the form of a CPU socket, such as Socket 0 and Socket 1. Here, there is only a single-layer switch in the computing node, namely the first switch.

[0162] like Figures 3 to 5 As shown, the multiple hardware devices connected downstream to the first switch 310, 410, 510 include at least one of the computing units 301, 401, 501. In addition, the hardware device connected downstream to the first switch may also include a network interface card NIC, preferably an RDMA network interface card (RNIC), such as Figures 3 to 5 and Fig.11 As shown in R0, R1, R2, R3. Figures 3 to 5 As shown, the first switches 310, 410, 510, 1110 have two downstream ports, and at least some of the first switches 310, 410, 510, 1110, preferably all of the first switches have two downstream ports respectively connected to a NIC, such as an RNIC, such as R0, and a GPU, such as G0.

[0163] refer to Fig.11 The distributed computing system 1100 further includes one or more electronic devices 1120 according to the embodiment of the present application, which are, for example, Fig.10 The electronic device 1000 shown. The electronic device 1120 can initiate anomaly detection to log in to the detected computing node, more specifically through the network, corresponding hardware devices, such as NIC and the corresponding first switch, such as PCIe switch to initiate a cross-machine detection communication connection to the detected computing unit, such as GPU. In a preferred embodiment, one or more computing nodes in the distributed computing system will be used as the one or more electronic devices. By using one or more computing nodes of the computing system as electronic devices for implementing the anomaly detection method of the embodiment of the present application, additional benefits can be provided, that is, the anomaly detection is initiated by the "other" computing nodes inside the computing system to detect the detected computing nodes, better simulating the calculation of the computing system, such as data transmission, synchronization, etc. in the distributed training process of large-scale neural networks. In some embodiments, when anomaly detection is implemented by multiple electronic devices, it can be detected simultaneously or separately. In a preferred embodiment, when performing anomaly detection, all computing nodes in the computing system except the computing nodes used as electronic devices are detected.

[0164] Although not shown in the figures, the embodiments of the present application further provide a storage medium storing computer programs configured to execute a method associated with any embodiment when run.

[0165] It should be understood that all operations in the method described above are merely exemplary, and the present disclosure is not limited to any operation in the method or the order of these operations, but should cover all other equivalent transformations under the same or similar concept.

[0166] It should also be understood that all modules in the above described device can be implemented in various ways. These modules can be implemented as hardware, software, or a combination thereof. In addition, any module in these modules can be further divided into submodules or combined together functionally.

[0167] Processors have been described in conjunction with various devices and methods. These processors can be implemented using electronic hardware, computer software or any combination thereof. Whether these processors are implemented as hardware or software will depend on specific application and the overall design constraints imposed on the system. As an example, the processor provided in the present disclosure, any part of the processor or any combination of processors can be implemented as a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic device (PLD), a state machine, a gate logic, a discrete hardware circuit, and other suitable processing components configured to perform the various functions described in the present disclosure. The function of the processor provided in the present disclosure, any part of the processor or any combination of processors can be implemented as software performed by a microprocessor, a microcontroller, a DSP or other suitable platforms.

[0168] Software should be broadly considered to represent instructions, instruction sets, codes, code segments, program codes, programs, subroutines, software modules, applications, software applications, software packages, routines, subroutines, objects, running threads, processes, functions, etc. Software can reside in a computer-readable medium. A computer-readable medium can include, for example, a memory, which can be, for example, a magnetic storage device (e.g., a hard disk, a floppy disk, a magnetic stripe), an optical disk, a smart card, a flash memory device, a random access memory (RAM), a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a register, or a removable disk. Although the memory is shown as being separated from the processor in the various aspects provided in the present disclosure, the memory can also be located inside the processor (e.g., a cache or a register).

[0169] The above description is provided to enable any person skilled in the art to implement the various aspects described herein. Various modifications to these aspects are apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, the claims are not intended to be limited to the aspects shown herein. All structural and functional equivalents of elements of various aspects described in this disclosure that are known or to be known to those skilled in the art will be expressly included herein by reference and are intended to be covered by the claims.

Claims

1. A method for detecting anomalies in a distributed computing system, characterized in that: The distributed computing system includes a plurality of computing nodes, each computing node includes a CPU, a first switch and one or more computing units, the computing unit is a non-CPU processing unit, wherein the first switch is connected to an upper-level switch or a CPU interface in an upstream direction, and the first switch is connected to a plurality of hardware devices in a downstream direction, and the plurality of hardware devices connected to the first switch in a downstream direction include at least one computing unit; The anomaly detection method comprises: Initiate a detection communication connection to at least one of the computing units of at least a portion of the computing nodes, wherein the detection communication connection is transmitted via a first switch communication link to which the detected computing unit is connected and does not pass through a CPU of the computing node where the detected computing unit is located; receiving a return result of the detection communication connection from each of the detected computing units; and Through a given detection, the returned result is analyzed to determine an abnormal computing unit among the detected computing units.

2. The anomaly detection method according to claim 1, characterized in that: The detection communication connection does not pass through the upper-level switch of the first switch to which the detected computing unit is connected.

3. The anomaly detection method according to claim 1, characterized in that: The given detection includes correctness detection and transmission stability detection.

4. The anomaly detection method according to claim 3, characterized in that: The transmission stability detection includes bandwidth detection and / or delay detection.

5. The anomaly detection method according to claim 4, characterized in that: The step of analyzing the returned result through a given detection to determine an abnormal computing unit in the detected computing unit includes: Determine, through multiple correctness tests, a computing unit that fails one or more of the correctness tests, a first switch that fails one or more of the correctness tests, a computing unit connected to a communication link of the first switch, or a computing unit connected to the same first switch as a hardware device that fails one or more of the correctness tests as an abnormal computing unit; Determine, by the bandwidth detection, the connection bandwidth of all the computing units detected in the detection communication, determine it as the quantile bandwidth based on a given first quantile threshold, and determine the computing units with a bandwidth less than the quantile bandwidth or less than the threshold bandwidth as abnormal computing units, wherein the threshold bandwidth is a first given difference less than the quantile bandwidth; and / or Through the delay detection, the connection delay of all the computing units detected in the detection communication is determined, and the quantile delay is determined based on a given second quantile threshold, and the computing units with a delay greater than the quantile delay or greater than the threshold delay are determined as abnormal computing units, and the threshold delay is slower than the quantile delay by a second given difference.

6. The abnormality detection method according to claim 3, characterized in that: The correctness detection includes a memory allocation correctness detection, a memory release correctness detection, a virtual memory allocation correctness detection and / or a virtual memory release correctness detection of the detected computing unit.

7. The abnormality detection method according to claim 6, characterized in that: The correctness test is selected from at least one of the following tests: Basic sub-thread fixed buffer memory allocation detection, Basic child thread fixed buffer virtual memory allocation detection, Basic memory allocation detection, Basic small buffer mapping detection, Basic misaligned mapping detection, Basic virtual memory allocation detection, Basic testing with tokens, Data verification memory allocation detection, Data verification virtual memory allocation detection, Access memory allocation invalidation detection after release, Access-after-free virtual memory allocation invalidation detection, After GDR is turned off, access memory allocation failure detection, After GDR is turned off, access to virtual memory allocation failure detection, After release, access the branch operation fork memory allocation failure detection, Access fork virtual memory allocation failure detection after release, Fork memory allocation failure detection after GDR mapping, Fork virtual memory allocation failure detection after GDR mapping, Child process GDR maps parent process fork memory allocation failure detection, Child process GDR mapping parent process fork virtual memory allocation failure detection, The child process GDR fixes the parent process's token memory failure detection. Memory allocation failure detection after fork mapping and release, Detection of virtual memory allocation failure after fork mapping and release, Double mapped memory allocation failure detection, Double mapped virtual memory allocation failure detection, Unix socket shared file descriptor GDR mapping memory allocation failure detection, Unix socket shared file descriptor GDR mapping virtual memory allocation failure detection, Unix socket shared file descriptor GDR fixed buffer memory allocation failure detection, Unix socket shared file descriptor GDR fixed buffer virtual memory allocation failure detection.

8. The abnormality detection method according to any one of claims 1 to 7, characterized in that: The anomaly detection method further includes: The computing unit determined to be abnormal, the first switch connected to the computing unit determined to be abnormal, or other hardware devices connected to the same first switch as the computing unit determined to be abnormal are disconnected or shielded.

9. The abnormality detection method according to claim 1, characterized in that: The method comprises: newly connecting a plurality of computing nodes, each of the newly connected computing nodes comprising one or more computing units; The initiating the detection communication connection to at least one of the computing units of at least a part of the computing nodes comprises: initiating the detection communication connection to the computing unit of the newly connected computing node; The method further includes: preventing a computing unit determined to be abnormal from accessing.

10. The abnormality detection method according to claim 1, characterized in that: The first switch and the upper-level switch are PCIe switches (PCIe Switch).

11. An anomaly detection device for a distributed computing system, characterized in that: The distributed computing system includes a plurality of computing nodes, each computing node includes a CPU, a first switch and one or more computing units, the computing unit is a non-CPU processing unit, wherein the first switch is connected to an upper-level switch or a CPU interface in an upstream direction, and the first switch is connected to a plurality of hardware devices in a downstream direction, and the plurality of hardware devices connected to the first switch in a downstream direction include at least one computing unit; The abnormality detection device comprises: a communication initiating unit, configured to initiate a detection communication connection to at least one of the computing units of at least a portion of the computing nodes, wherein the detection communication connection is transmitted via a first switch communication link to which the detected computing unit is connected and does not pass through a CPU of the computing node where the detected computing unit is located; a receiving unit configured to receive a return result of the detection communication connection from each of the detected computing units; and The abnormality determination unit is configured to analyze the returned result through a given detection to determine an abnormal computing unit in the detected computing unit.

12. An electronic device, characterized in that: The invention comprises a processor and a memory storing a computer program, wherein the processor is configured to implement the method according to any one of claims 1 to 10 when running the computer program.

13. A distributed computing system, characterized in that: The method comprises a plurality of computing nodes, each computing node comprises a CPU, a first switch and one or more computing units, wherein the computing unit is a non-CPU processing unit, wherein the first switch is upstream connected to an upper-level switch or a CPU interface, and the first switch is downstream connected to a plurality of hardware devices, and the plurality of hardware devices downstream connected to the first switch include at least one computing unit; The distributed computing system further comprises one or more electronic devices according to claim 12.

14. The distributed computing system according to claim 13, characterized in that: One or more computing nodes among the plurality of computing nodes serve as the one or more electronic devices.

15. The distributed computing system according to claim 13, characterized in that: The plurality of hardware devices connected to the first switch further include a network interface card NIC, and the computing unit includes a GPU.

16. The distributed computing system according to claim 15, characterized in that: At least part of the computing nodes include two-level switches, the first-level switches include the first switch, and the second-level switches include a second switch connected upstream to the first switch, wherein the first switch has two downstream ports, and the two downstream ports of at least part of the first switches are respectively connected to the NIC and the GPU.

17. The distributed computing system according to claim 15, characterized in that: At least some of the computing nodes include a first-level switch, the first switch is upstream connected to the root complex RC, wherein the first switch has two downstream ports, and the two downstream ports of at least some of the first switches are respectively connected to the NIC and the GPU.

18. A storage medium, characterized in that: The storage medium stores a computer program, and the computer program is configured to implement the method according to any one of claims 1 to 10 when executed.