Abnormal reason detection method and device, equipment, storage medium and program product

Abstract

The invention provides an abnormal reason detection method and device, equipment, a storage medium and a program product. The method comprises the following steps: when a node cluster is abnormal in a process of executing a calculation task, acquiring a transmission path of an abnormal data stream in the node cluster; wherein the node cluster comprises a plurality of nodes for cooperatively executing a calculation task, and the plurality of nodes comprise a first node for sending an abnormal data flow and a second node for receiving the abnormal data flow; a first detection instruction is sent to the first node, the first detection instruction is used for indicating the first node to detect connectivity in the direction from the first node to the second node according to a transmission path included in the first detection instruction, and a first detection result is obtained; receiving a first detection result returned by the first node based on the first detection instruction; and based on the first detection result, determining an abnormity reason of the abnormity. According to the invention, the detection efficiency and the detection precision of the abnormal reasons can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to an abnormal reason detection method and device, equipment, a storage medium and a program product. BACKGROUND

[0002] When a computing task is performed based on a node cluster, various devices such as a framework, a server, a storage node, a network and a computing power node are usually involved. Therefore, it is common that an exception occurs during the execution process. In the related art, all nodes are checked in a full link (including software, hardware and a network layer) to determine an abnormal reason. However, since the nodes in the cluster are connected by a network, a single exception can be quickly transmitted through the network, so many nodes need to be checked at the same time, a large amount of manpower and time are consumed for layer-by-layer checking, and checking accuracy is difficult to guarantee.

[0003] SUMMARY

[0004] The embodiments of the present application provide an abnormal reason detection method, device, electronic equipment, computer readable storage medium and computer program product, which can improve the detection efficiency and accuracy of the abnormal reason.

[0005] The technical scheme of the embodiments of the present application is implemented as follows:

[0006] The embodiments of the present application provide an abnormal reason detection method, which comprises the following steps.

[0007] When an exception occurs in the process of performing a computing task by a node cluster, a transmission path of an abnormal data stream in the node cluster is acquired.

[0008] The node cluster comprises a plurality of nodes that cooperatively perform the computing task, and the plurality of nodes comprise a first node that sends the abnormal data stream and a second node that receives the abnormal data stream.

[0009] A first detection instruction is sent to the first node, and the first detection instruction is used to instruct the first node to detect connectivity in a direction from the first node to the second node according to the transmission path included in the first detection instruction, and obtain a first detection result.

[0010] The first detection result returned by the first node based on the first detection instruction is received.

[0011] An abnormal reason of the exception is determined based on the first detection result.

[0012] The embodiments of the present application also provide an abnormal reason detection method applied to a first node, which comprises the following steps.

[0013] The first node receives a first detection instruction.

[0014] The first detection instruction is sent by the node cluster when an exception occurs in the process of executing the computing task, and the first detection instruction includes a transmission path of an abnormal data stream in the node cluster.

[0015] In response to the first detection instruction, connectivity in a direction from the first node to the second node is detected according to the transmission path, and a first detection result is obtained.

[0016] The first detection result is sent, and the first detection result is used to determine an abnormal reason for the exception.

[0017] Embodiments of the present application also provide an abnormal reason detection device, comprising:

[0018] The acquisition module is configured to acquire a transmission path of an abnormal data stream in a node cluster when an exception occurs in the process of executing a computing task.

[0019] The node cluster includes a plurality of nodes that cooperatively execute the computing task, and the plurality of nodes include a first node that sends the abnormal data stream and a second node that receives the abnormal data stream.

[0020] The first sending module is configured to send a first detection instruction to the first node, and the first detection instruction is used to instruct the first node to detect connectivity in a direction from the first node to the second node according to the transmission path included in the first detection instruction, and obtain a first detection result.

[0021] The first receiving module is configured to receive the first detection result returned by the first node based on the first detection instruction.

[0022] The determination module is configured to determine an abnormal reason for the exception based on the first detection result.

[0023] In the above scheme, the acquisition module is further configured to acquire an abnormal log of the abnormal data stream in the node cluster, and determine an abnormal communication queue pair bound with the abnormal log; acquire a correspondence relationship between a plurality of communication queue pairs and a plurality of communication transmission paths, the communication queue pair and the communication transmission path being in one-to-one correspondence; determine a target communication transmission path corresponding to the abnormal communication queue pair based on the correspondence relationship, and take the target communication transmission path as the transmission path of the abnormal data stream.

[0024] In the scheme, the determining module is further configured to: when the first detection result indicates that the connectivity detection passes, determine that the abnormal reason for the abnormality is that software of at least one of the first node and the second node is abnormal, the software being used to perform the computing task; when the first detection result indicates that the connectivity detection fails, send a second detection instruction to the first node and a third detection instruction to the second node; the second detection instruction is used to instruct the first node to detect connectivity in a direction from the first node to a third node to obtain a second detection result, and the third detection instruction is used to instruct the second node to detect connectivity in a direction from the second node to the third node to obtain a third detection result, the network link between the third node and the first node being normal, and the network link between the third node and the second node being normal; receive the second detection result returned by the first node based on the second detection instruction and the third detection result returned by the second node based on the third detection instruction; and determine the abnormal reason for the abnormality based on the second detection result and the third detection result.

[0025] In the scheme, the determining module is further configured to: when the second detection result and the third detection result both indicate that the connectivity detection passes, determine that the abnormal reason for the abnormality is that the network link between the first node and the second node is abnormal; when the second detection result indicates that the connectivity detection fails and the third detection result indicates that the connectivity detection passes, determine that the abnormal reason for the abnormality is that the hardware of the first node is abnormal; when the second detection result indicates that the connectivity detection passes and the third detection result indicates that the connectivity detection fails, determine that the abnormal reason for the abnormality is that the hardware of the second node is abnormal; and when the second detection result indicates that the connectivity detection fails and the third detection result indicates that the connectivity detection fails, determine that the abnormal reason for the abnormality is that the hardware of the first node and the hardware of the second node are both abnormal.

[0026] In the scheme, the determining module is further configured to: when the first detection result indicates that the connectivity detection passes, obtain an abnormal log of the abnormal data flow; when the abnormal log belongs to the first node, determine that the abnormal reason for the abnormality is that the software of the first node is abnormal; when the abnormal log belongs to the second node, determine that the abnormal reason for the abnormality is that the software of the second node is abnormal; and when the abnormal log includes a first abnormal log of the first node and a second abnormal log of the second node, determine that the abnormal reason for the abnormality is that the software of at least one of the first node and the second node is abnormal.

[0027] In the scheme, the determining module is further configured to, when the abnormal log comprises a first abnormal log of the first node and a second abnormal log of the second node, perform the following processing: when a first generation time of the first abnormal log is earlier than a second generation time of the second abnormal log, determining that the abnormal reason for the abnormality is that the software of the first node is abnormal; when the first generation time of the first abnormal log is later than the second generation time of the second abnormal log, determining that the abnormal reason for the abnormality is that the software of the second node is abnormal; and when the first generation time of the first abnormal log is equal to the second generation time of the second abnormal log, determining that the abnormal reason for the abnormality is that the software of the first node and the second node is abnormal.

[0028] In the scheme, the determining module is further configured to, after determining that the abnormal reason for the abnormality is that the software of at least one of the first node and the second node is abnormal, acquire an abnormal node log of a target node, the target node being at least one of the first node and the second node; identify a target error code in the abnormal node log; acquire a correspondence between a plurality of error codes and a plurality of software abnormal reasons, the error codes and the software abnormal reasons corresponding to each other in one-to-one manner; determine a target software abnormal reason corresponding to the target error code based on the correspondence, and take the target software abnormal reason as the reason for the software of the target node being abnormal.

[0029] In the scheme, the acquiring module is further configured to, when the node cluster does not have an abnormality during execution of the computing task, acquire a normal transmission path of a normal data flow in the node cluster, the plurality of nodes comprising a fourth node that sends the normal data flow and a fifth node that receives the normal data flow; send a pre-detection instruction to the fourth node, the pre-detection instruction being used to instruct the fourth node to detect connectivity in a direction from the fourth node to the fifth node according to the normal transmission path and acquire a measurement index of the connectivity; receive the measurement index returned by the fourth node based on the pre-detection instruction; and when the measurement index does not match a baseline measurement index, generate abnormal early warning information.

[0030] In the solution, the determining module is further configured to: when the abnormal cause is that a network link between the first node and the second node is abnormal, switch the network link between the first node and the second node to a target network link; when the abnormal cause is that software of a target node is abnormal, restart the target node, the target node being at least one of the first node and the second node; and when the abnormal cause is that hardware of the target node is abnormal, remove the target node and call a sixth node to execute a target computing task on the target node, the computing task including the target computing task.

[0031] In the solution, the number of abnormal data streams is multiple; the first sending module is further configured to: before the first sending module sends the first detection instruction to the first node, acquire a network link feature of each abnormal data stream; based on the network link feature of each abnormal data stream, cluster the multiple abnormal data streams to obtain at least one data stream cluster; and for each data stream cluster, select a target abnormal data stream from abnormal data streams included in the data stream cluster; and the first sending module is further configured to send the first detection instruction to each first node that sends the target abnormal data stream.

[0032] In the solution, the number of computing tasks is multiple, and the number of abnormal data streams is multiple; the first sending module is further configured to, before the first sending module sends the first detection instruction to the first node, for each abnormal data stream, perform the following processing respectively: from the multiple computing tasks, determine a target computing task to which the abnormal data stream belongs; based on a task identifier of the target computing task and a transmission path of the abnormal data stream, generate a first detection instruction of the abnormal data stream; and add the first detection instruction of the abnormal data stream into an instruction buffer queue; and the first sending module is further configured to send the first detection instruction to a first node indicated by the task identifier in the first detection instruction according to a sending order of each first detection instruction in the instruction buffer queue.

[0033] Embodiments of the present application further provide an abnormal cause detection apparatus applied to a first node, comprising:

[0034] A second receiving module configured to receive a first detection instruction.

[0035] The first detection instruction is sent when a node cluster appears abnormal during execution of a computing task, the first detection instruction includes a transmission path of an abnormal data stream in the node cluster, the node cluster includes multiple nodes that cooperatively execute the computing task, and the multiple nodes include the first node that sends the abnormal data stream and a second node that receives the abnormal data stream.

[0036] a detection module, configured to detect connectivity in a direction from the first node to the second node according to the transmission path in response to the first detection instruction, to obtain a first detection result;

[0037] a second sending module, configured to send the first detection result, the first detection result being used for determining an abnormal reason of the abnormality.

[0038] In the scheme, the detection module is further configured to, in response to the first detection instruction, establish a communication connection between the first node and the second node according to a source address and a destination address in the transmission path, and obtain a source port number and a destination port number of the communication connection; obtain a target source port number in the transmission path, and modify the source port number to the target source port number; send a detection data stream to the destination port number of the second node according to the target source port number, the detection data stream being used for detecting connectivity in the direction from the first node to the second node; when receiving a response data stream returned by the second node based on the detection data stream, generate a first detection result representing that connectivity detection is passed; and when not receiving the response data stream returned by the second node based on the detection data stream, generate a first detection result representing that connectivity detection is not passed.

[0039] The embodiment of the present application further provides an electronic device, comprising:

[0040] a memory, configured to store computer executable instructions;

[0041] a processor, configured to execute the computer executable instructions stored in the memory, to implement the abnormal reason detection method provided by the embodiment of the present application.

[0042] The embodiment of the present application further provides a computer readable storage medium, which stores computer executable instructions or computer programs, and the computer executable instructions or computer programs are executed by a processor to implement the abnormal reason detection method provided by the embodiment of the present application.

[0043] The embodiment of the present application further provides a computer program product, which comprises computer executable instructions or computer programs, and the computer executable instructions or computer programs are executed by a processor to implement the abnormal reason detection method provided by the embodiment of the present application.

[0044] The embodiment of the present application has the following beneficial effects:

[0045] According to the above-mentioned embodiments of the present application, when the node cluster has an exception during the execution of the computing task, the transmission path of the abnormal data stream in the node cluster is obtained, so that the first detection instruction is sent to the first node (i.e., the sending end of the abnormal data stream), so that the first node detects the connectivity in the direction from the first node to the second node (i.e., the receiving end of the abnormal data stream) according to the transmission path included in the first detection instruction, and obtains the first detection result, so that after receiving the first detection result, the abnormal reason is determined based on the first detection result. Here, according to the transmission path of the abnormal data stream, the connectivity detection in the direction from the first node to the second node is realized by restoring the transmission of the abnormal data stream, so that the abnormal reason is determined based on the obtained first detection result. In this way, 1) the reliability of the connectivity detection is ensured, thereby improving the detection accuracy of the abnormal reason; and only the nodes involved in the abnormal data stream need to be detected, which can improve the detection efficiency of the abnormal reason and reduce the detection cost. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 FIG. 1 is a first architecture schematic diagram of an abnormal reason detection system provided by an embodiment of the present application;

[0047] Figure 2 FIG. 2 is a structural schematic diagram of an electronic device provided by an embodiment of the present application;

[0048] Figure 3A FIG. 3 is a first flow schematic diagram of an abnormal reason detection method provided by an embodiment of the present application;

[0049] Figure 3B FIG. 4 is a second flow schematic diagram of an abnormal reason detection method provided by an embodiment of the present application;

[0050] Figure 3C FIG. 5 is a third flow schematic diagram of an abnormal reason detection method provided by an embodiment of the present application;

[0051] Figure 3D FIG. 6 is a fourth flow schematic diagram of an abnormal reason detection method provided by an embodiment of the present application;

[0052] Figure 4 FIG. 7 is a schematic diagram of an abnormal reason detection process provided in the related art;

[0053] Figure 5 FIG. 8 is a fifth flow schematic diagram of an abnormal reason detection method provided by an embodiment of the present application;

[0054] Figure 6 FIG. 9 is a second architecture schematic diagram of an abnormal reason detection system provided by an embodiment of the present application;

[0055] Figure 7 FIG. 10 is a schematic diagram of an abnormal flow five-tuple collection platform provided by an embodiment of the present application;

[0056] Figure 8 FIG. 1 is a schematic diagram of a detection task issuing platform according to an embodiment of the present application;

[0057] Figure 9 FIG. 2 is a schematic diagram of a detection process of a detection tool according to an embodiment of the present application;

[0058] Figure 10 FIG. 3 is a schematic diagram of secondary detection by means of a third-party node according to an embodiment of the present application.

[0059] It should be noted that the above-mentioned "first", "second" are only used to distinguish different schemes, and do not represent the degree of advantage or priority in the implementation process. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application will be described in further detail below with reference to the drawings, and the described embodiments should not be regarded as limiting the present application, and all other embodiments obtained by those skilled in the art without creative labor shall fall within the scope of protection of the present application.

[0061] In the following description, "some embodiments" are referred to, which describe a subset of all possible embodiments, but it can be understood that "some embodiments" can be the same subset or different subset of all possible embodiments, and can be combined with each other without conflict.

[0062] In the following description, the terms "first\second\third" are only used to distinguish similar objects, and do not represent a specific order of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence as allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0063] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works with other related parts to achieve a predetermined target, and can be implemented entirely or partially by using software, hardware (such as processing circuitry or memory) or a combination thereof. Similarly, one processor (or multiple processors or memory) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an integral module or unit that includes the functions of the module or unit.

[0064] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0065] The related data collection and processing in the embodiments of the present application should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing behavior within the scope of authorization of laws and regulations and the personal information subject.

[0066] Before further detailing the embodiments of the present application, the terms and phrases involved in the embodiments of the present application are explained, which are applicable to the following explanations.

[0067] 1) Client, an application program running in an electronic device for providing various services, such as a client supporting abnormal reason detection.

[0068] 2) In response to, used to indicate the condition or state on which the operation is performed, when the dependent condition or state is met, the performed one or more operations can be real-time or have a set delay; in the absence of special instructions, there is no restriction on the execution sequence of the multiple operations performed.

[0069] 3) Transmission path, i.e. five-tuple information, including: source Internet Protocol (IP) address, destination IP address, source port, destination port, and transmission protocol. The transmission path is used to identify the corresponding data flow.

[0070] The embodiments of the present application provide an abnormal reason detection method, device, electronic equipment, computer readable storage medium and computer program product, which can improve the detection efficiency and accuracy of abnormal reasons. Next, based on the above description of the terms and phrases involved in the embodiments of the present application, the embodiments of the present application are described in detail.

[0071] The abnormal reason detection system provided by the embodiments of the present application is described below. Referring to Figure 1 , Figure 1 is the first architecture schematic diagram of the abnormal reason detection system provided by the embodiments of the present application. To realize supporting an exemplary application, the abnormal reason detection system 100 includes: a server 200, a network 300, a terminal 400, a node cluster (including multiple nodes, such as a first node 600 and a second node 700). Among them, the server 200, the network 300, the terminal 400, the first node 600 and the second node 700 are connected through the network 300, and the network 300 can be a wide area network or a local area network, or a combination of the two, using wireless or wired links to realize data transmission.

[0072] Here, the terminal 400 (for example, running a client supporting abnormal reason detection) sends an abnormal reason detection request to the server 200 in response to the abnormal reason detection instruction, where the abnormal reason detection request is used to instruct the server 200 to perform abnormal reason detection on an abnormality when the node cluster has an abnormality in the process of executing a computing task; the server 200 receives the abnormal reason detection request; in response to the abnormal reason detection request, when the node cluster has an abnormality in the process of executing a computing task, the transmission path of the abnormal data stream in the node cluster is obtained; wherein the node cluster includes a plurality of nodes that cooperatively execute a computing task, and the plurality of nodes include: a first node 600 that sends an abnormal data stream and a second node 700 that receives an abnormal data stream; a first detection instruction is sent to the first node 600; the first node 600 receives the first detection instruction, and in response to the first detection instruction, detects the connectivity in the direction from the first node 600 to the second node 700 according to the transmission path included in the first detection instruction, to obtain a first detection result; the first detection result is returned to the server 200; the server 200 receives the first detection result returned by the first node based on the first detection instruction; based on the first detection result, the abnormal reason of the abnormality is determined; notification information of the abnormal reason is returned to the terminal 400; and the terminal 400 receives the abnormal reason returned by the server 200, and displays the notification information of the abnormal reason.

[0073] The abnormal reason detection method provided by the embodiments of the present application is implemented by an electronic device, for example, can be implemented by a terminal alone, can be implemented by a server alone, and can be implemented by a terminal and a server cooperatively. The electronic device implementing the abnormal reason detection method provided by the embodiments of the present application can be various types of terminals or servers. Among them, the server (for example, the server 200) can be a stand-alone physical server, can be a server cluster or a distributed system composed of multiple physical servers, and can also be a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms. The terminal (for example, the terminal 400) can be a notebook computer, a tablet computer, a desktop computer, a smart phone, a smart voice interaction device (for example, a smart speaker), a smart home appliance (for example, a smart television), a smart watch, a vehicle-mounted terminal, a wearable device, a virtual reality (VR) device, a flying device, etc., but is not limited thereto. The terminal and the server can be connected directly or indirectly through wired or wireless communication, and the embodiments of the present application do not limit this.

[0074] In some embodiments, a terminal or a server can implement the exception cause detection method provided by the embodiments of the present application by running various computer-executable instructions or computer programs. For example, the computer-executable instructions can be microprogram-level commands, machine instructions, or software instructions. The computer programs can be native programs or software modules in an operating system; can be native applications (APPs), i.e., programs that need to be installed in an operating system to run; or can be applets that can be embedded into any APP, i.e., programs that only need to be downloaded into a browser environment to run. In summary, the above computer-executable instructions can be any form of instructions, and the above computer programs can be any form of applications, modules, or plug-ins.

[0075] The electronic device implementing the exception cause detection method provided by the embodiments of the present application is described below. Referring to Figure 2 , Figure 2 is a structural schematic diagram of the electronic device provided by the embodiments of the present application. The electronic device 500 provided by the embodiments of the present application can be a terminal or a server. As shown in Figure 2 , the electronic device 500 includes at least one processor 510, a memory 550, at least one network interface 520, and a user interface 530. The various components in the electronic device 500 are coupled together by a bus system 540. It can be understood that the bus system 540 is used to realize the connection and communication between the components. In addition to a data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for the purpose of clear illustration, all kinds of buses are marked as the bus system 540 in Figure 2 .

[0076] The processor 510 can be an integrated circuit chip with a signal processing capability, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0077] The user interface 530 includes one or more output devices 531 that enable the presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, a mouse, a microphone, a touch screen display, a camera, other input buttons and controls.

[0078] The memory 550 can be removable, non-removable, or a combination thereof. The memory 550 can include one or more storage devices physically located in proximity to the processor 510. The memory 550 includes volatile memory or non-volatile memory, and can also include both volatile and non-volatile memory. Non-volatile memory can be read only memory (ROM), and volatile memory can be random access memory (RAM). The memory 550 described in embodiments of the present application is intended to include any suitable type of memory.

[0079] In some embodiments, the memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or a subset or superset thereof, which are exemplarily illustrated below.

[0080] The operating system 551 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks.

[0081] The network communication module 552 is used to communicate with other electronic devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including Bluetooth, wireless compatibility authentication (WiFi), and universal serial bus (USB), etc.

[0082] The presentation module 553 is used to enable the presentation of information via one or more output devices 531 associated with the user interface 530 (e.g., a display screen, a speaker, etc.), such as a user interface for operating peripheral devices and displaying content and information.

[0083] The input processing module 554 is used to detect and interpret one or more user inputs or interactions from one or more input devices 532.

[0084] In some embodiments, the abnormality cause detection apparatus provided by embodiments of the present application can be implemented in a software manner, Figure 2 An abnormality cause detection apparatus 555 stored in the memory 550 is shown, which can be software in the form of programs and plug-ins, including the following software modules: an acquisition module 5551, a first sending module 5552, a first receiving module 5553, and a determination module 5554. These modules are logical, and thus can be combined or further split according to the functions implemented, and the functions of each module will be described below.

[0085] The abnormal reason detection method provided by the embodiments of the present application will be described below. As described above, the abnormal reason detection method provided by the embodiments of the present application is implemented by an electronic device, for example, can be implemented by a server or a terminal alone, or by a server and a terminal cooperatively. Therefore, the execution subject of each step will not be repeated below. See Figure 3A , Figure 3A FIG. 1 is a first flowchart of the abnormal reason detection method provided by the embodiments of the present application. The abnormal reason detection method provided by the embodiments of the present application includes the following steps.

[0086] Step 101: When an exception occurs in the process of executing a computing task by a node cluster, a transmission path of an abnormal data stream in the node cluster is obtained.

[0087] The node cluster includes a plurality of nodes that cooperatively execute the computing task, and the plurality of nodes include a first node that sends the abnormal data stream and a second node that receives the abnormal data stream.

[0088] For step 101, the node cluster includes a plurality of nodes that are connected together through a network to cooperatively execute a computing task. One node cluster can execute at least one computing task at the same time. The nodes used for different computing tasks can be different or can be intersected, which is not limited here. For example, the computing task includes but is not limited to a training task of a machine learning model, an inference task of a machine learning model (such as weather simulation, molecular dynamics calculation, knowledge question answering, etc.), a data read / write task in a distributed storage system, etc. The computing task is a task cooperatively executed by a plurality of nodes.

[0089] During the execution of the computing task by the node cluster, an exception (i.e., a fault) can occur, which can affect the execution efficiency, correctness, stability, etc. of the computing task, and can cause problems such as interruption of the computing task, performance degradation, incorrect computing result, resource waste, progress stagnation, etc. For example, the exception includes but is not limited to interruption of the execution of the computing task, repeated timeout retries of the computing task, data consistency failure between the plurality of nodes executing the computing task, which causes incorrect computing result, etc.

[0090] Specifically, the exception that does not completely interrupt the computing task can include:

[0091] (1) Hardware-related failures: A) Partial hardware performance degradation of node: For example, a small number of bad blocks (not completely failed) appear in the video memory of the graphics processing unit (GPU), causing occasional data check errors during calculation; the central processing unit (CPU) core frequency abnormally decreases, reducing the computing efficiency of the node; the network card has intermittent packet loss (not completely disconnected), causing an increase in data transmission delay. Such failures will not directly interrupt the task, but will cause the task to run slowly, the result to be abnormal, or the need for repeated retries. B) Instability of cooling or power supply: node fan failure causes CPU / GPU temperature to be too high, triggering frequency reduction protection (performance decreases but does not shut down); power module aging causes voltage fluctuations, causing occasional data calculation errors (such as floating-point operation precision abnormalities). The task is not interrupted, but the calculation result may be distorted, and there is a risk of sudden interruption during long-term operation. C) Partial failure of storage medium: partial damage to the disk of the distributed storage node causes slow or occasional failure of data reading and writing; the SSD has a write amplification anomaly, which reduces the data persistence efficiency. The task may continue to run, but the overall progress will be delayed due to frequent input / output retries.

[0092] (2) Software and task scheduling failures: A) Task jamming or deadlock: When multiple nodes cooperate, a part of the nodes enter an infinite waiting state due to synchronization logic defects (such as distributed locks not being released), and the task is not interrupted but completely stalled; the computing framework has memory leaks, causing the node memory to gradually run out, and the task runs slower and slower but is not immediately interrupted. B) Data consistency failure: In distributed computing, some nodes have inconsistent local cache data and global data due to network delay or clock synchronization deviation, resulting in incorrect final calculation results (such as parameter update asynchronous anomalies in large model training, causing model convergence failure). The task is not interrupted, but the output result is invalid. C) Resource contention and scheduling anomalies: The cluster resource manager (such as K8s) has scheduling logic errors, causing multiple tasks to compete for the same node's GPU / CPU resources, resulting in resource over-allocation (such as actual allocation of resources exceeding the node limit), and the task runs frequently but does not terminate. Task priority configuration error, low-priority tasks preempt high-priority task resources, causing core task progress delay.

[0093] (3) Network related non-interruptive failures: A) Network performance degradation: Intermittent congestion occurs on the cluster network link (e.g., core switch buffer overflow), causing communication delays to rise from microseconds to milliseconds; packet loss exists between some nodes (e.g., 0.1% packet loss rate), although communication is not interrupted, the efficiency of distributed synchronization is greatly reduced (e.g., model operation time increases by 10 times). B) Network partitioning: The cluster network is divided into multiple independent subnets (e.g., access layer switch failure causes some nodes to be isolated from the core layer), nodes in different subnets can still communicate and run tasks within the subnet, but task synchronization across subnets fails, causing the global task to "split" (some nodes continue to calculate, some nodes wait, the overall task cannot be completed but is not completely interrupted).

[0094] (4) Storage and data related failures: A) Data read and write anomalies: The distributed file system has a metadata server response delay, causing the task to read input data to frequently timeout and retry (but not completely fail); data checksum error, causing some data blocks to need to be re-copied, task running progress stagnates but does not interrupt. B) Storage capacity warning and quota limit: The remaining capacity of the storage cluster is insufficient (e.g., less than 10%), causing the data write speed to drop sharply (due to frequent fragmentation); the task exceeds the preset storage quota, the write operation is limited, causing data persistence delay during calculation, affecting the overall efficiency of the task.

[0095] (5) Security and environmental failures: a) Nodes are maliciously invaded or infected: Some nodes are infected with malicious programs that occupy CPU / GPU resources, causing the available resources for computing tasks to decrease, and the running to slow down; malicious programs tamper with intermediate data during calculation, causing the task result to be incorrect (not interrupted but the output is invalid). B) Environment configuration inconsistency: The software dependency versions of the cluster nodes are inconsistent, causing some nodes to run normally and some nodes to have compatibility errors (e.g., function call failure), the task appears "partially successful, partially failed" non-interruptive exception in a heterogeneous environment.

[0096] Specifically, the abnormality that the computing task is completely interrupted can include: (1) an end-side software failure: an upper-layer business software problem, such as a training framework process hanging, a communication library error of a node cluster, an abnormality in configuration of a driving parameter of Remote Direct Memory Access (RDMA), and the like, can cause a communication task to be unable to be completed, and trigger an ERRCQE log (that is, an abnormality log); (2) an end-side hardware failure: a server physical hardware abnormality, such as a GPU failure, an RDMA network card damage, a server mainboard problem, and the like, can cause a node to be unable to normally participate in communication; and (3) a network link failure: involving an Access Layer Switch (LA), a Convergence Layer Switch (LC), a Core Layer Switch (SGLC), and a link, such as a switch port congestion, an optical fiber interruption, a data path abnormality, and the like, and showing that communication between nodes is interrupted.

[0097] When an abnormality occurs in the node cluster in the process of executing the computing task, an abnormal data flow in the node cluster can be determined, and a transmission path of the abnormal data flow can be obtained, the transmission path can be five-tuple information, and the five-tuple information can include source IP address, destination IP address, source port, destination port, transmission protocol, and the like. The abnormal data flow is related to the abnormality, and the computing task depends on multiple nodes to communicate through a network, and data (such as model parameters, intermediate calculation results, and the like in a model training task) needs to be continuously transmitted between nodes. These data transmissions form stable “communication flows (that is, data flows)”, each data flow is uniquely identified by five-tuple information, and accurate data interaction is ensured. Because an abnormality occurs in the node cluster in the process of executing the computing task, normal transmission of the data flow is broken (data interaction is not completed as expected), and the data flow becomes an abnormal data flow, and a trigger factor can correspond to three categories of failures: 1) a software failure: such as a training framework process hanging, a communication library logic error, causing a data sending / receiving instruction to be unable to be executed, and a communication flow to be interrupted; 2) a hardware failure: such as a GPU / network card hardware failure (such as a graphics memory error, a network card packet loss), causing data processing or transmission to be abnormal, a communication flow to be stuck or data to be damaged; and 3) a network link failure: such as a switch port failure, an optical fiber link interruption, causing a data transmission path to be blocked, and a communication flow to be unable to reach a specific node.

[0098] In some embodiments, the step 101 of obtaining the transmission path of the abnormal data flow in the node cluster can be implemented by performing the following steps: obtaining an abnormal log of the abnormal data flow in the node cluster, and determining an abnormal communication queue pair bound with the abnormal log; obtaining a correspondence between a plurality of communication queue pairs and a plurality of communication transmission paths, the communication queue pair and the communication transmission path corresponding one-to-one; determining a target communication transmission path corresponding to the abnormal communication queue pair based on the correspondence, and taking the target communication transmission path as the transmission path of the abnormal data flow.

[0099] Here, when the abnormal data flow occurs in the node cluster, the abnormal log of the abnormal data flow is automatically generated, which is a landmark record of the abnormal data flow. For example, when the abnormal data flow occurs in the node cluster based on RDMA communication, the ERRCQE log (i.e., the abnormal log) is generated. The abnormal log is bound with the communication queue pair (i.e., QP) triggering the abnormal data flow, and the abnormal log is used to mark the communication queue pair in which the abnormal data flow occurs, which can be referred to as the abnormal communication queue pair. The communication queue pair is a core structure for managing the sending and receiving of the data flow, and is directly associated with a specific data flow. When the communication library of the node cluster initializes the data flow, the correspondence between the communication queue pair of each data flow and the communication transmission path (such as the five-tuple information) is recorded in advance, and the communication queue pair and the communication transmission path correspond one-to-one. Based on this, since the abnormal log is bound with the abnormal communication queue pair, and the communication queue pair and the communication transmission path correspond one-to-one, the transmission path of the abnormal data flow can be determined through the abnormal log. Specifically, the abnormal communication queue pair can be first found from a plurality of communication queue pairs, and then the target communication transmission path corresponding to the abnormal communication queue pair is determined based on the correspondence, so that the target communication transmission path is taken as the transmission path of the abnormal data flow.

[0100] By applying the above embodiments, 1) the transmission path of the abnormal data flow can be quickly determined through the correspondence between the plurality of communication queue pairs and the plurality of communication transmission paths, and the abnormal communication queue pair bound with the abnormal log, avoiding blind investigation and wasting time cost and labor cost, thereby improving the positioning efficiency of the abnormal reason; 2) without restarting the computing task, the transmission path of the abnormal data flow can be quickly obtained through the abnormal log, narrowing the range of suspicious nodes with the abnormality, and further improving the positioning efficiency of the abnormal reason.

[0101] In some embodiments, an application for anomaly cause detection can be deployed on a server or a terminal, and the server or the terminal obtains the transmission path of the abnormal data flow in the node cluster by running the application. Specifically, an abnormal collection platform of the transmission path of the abnormal data flow can be set in the application, and the abnormal collection platform can be used to collect the transmission path of the abnormal data flow in the node cluster, so as to realize unified collection and unified management of the transmission path of the abnormal data flow, and improve the collection and management efficiency and resource utilization.

[0102] In some embodiments, referring to Figure 3B The following steps 201-204 can also be performed: step 201, when the node cluster does not appear abnormal in the process of executing the computing task, obtaining a normal transmission path of a normal data flow in the node cluster, the plurality of nodes including a fourth node for sending the normal data flow and a fifth node for receiving the normal data flow; step 202, sending a pre-detection instruction to the fourth node, the pre-detection instruction being used to instruct the fourth node to detect connectivity in a direction from the fourth node to the fifth node according to the normal transmission path, and obtain a measurement index of the connectivity; step 203, receiving the measurement index returned by the fourth node based on the pre-detection instruction; and step 204, generating an abnormal early warning information when the measurement index does not match a baseline measurement index.

[0103] Here, the potential data transmission risk can also be identified in advance when the node cluster does not appear abnormal in the process of executing the computing task, so as to realize abnormal early warning and risk avoidance, and reduce the influence of the abnormality on the execution of the computing task. Therefore, for step 201, when the node cluster does not appear abnormal in the process of executing the computing task, the normal transmission path of the normal data flow in the node cluster is obtained. Specifically, since the correspondence between the communication queue pair of each data flow and the communication transmission path (such as the five-tuple information) is recorded in advance when the communication library of the node cluster initializes the data flow, the communication transmission path corresponding to the communication queue pair of the normal data flow can be determined based on the correspondence, so as to take the communication transmission path corresponding to the communication queue pair of the normal data flow as the normal transmission path of the normal data flow. Meanwhile, the fourth node for sending the normal data flow and the fifth node for receiving the normal data flow are also determined.

[0104] For step 202, a pre-detection instruction is sent to the fourth node, the pre-detection instruction being used to instruct the fourth node to detect the connectivity in the direction from the fourth node to the fifth node according to a normal transmission path, the pre-detection instruction being generated based on the normal transmission path. After receiving the pre-detection instruction, the fourth node sends a pre-detection data stream to the fifth node according to the normal transmission path included in the pre-detection instruction, to detect the connectivity in the direction from the fourth node to the fifth node, so as to obtain a measurement index of the connectivity in the detection process. For example, the measurement index includes, but is not limited to, a transmission delay, a packet loss rate, and other indexes for measuring the communication quality. In actual application, the pre-detection instruction can be periodically sent to the fourth node to realize periodic pre-detection.

[0105] For step 203, the measurement index returned by the fourth node based on the pre-detection instruction is received. For step 204, a baseline measurement index is first obtained, which can be pre-set. The baseline measurement index and the measurement index are one-to-one corresponding. For example, the baseline measurement index can be: a baseline delay less than or equal to 50 μs, a baseline packet loss rate of 0, and the like. The measurement index and the baseline measurement index are matched, and if the measurement index does not match the baseline measurement index (for example, the delay in the measurement index is greater than the baseline delay, and the packet loss rate in the measurement index is greater than the baseline packet loss rate), an abnormal early warning information is generated. In actual application, when the number of times that the measurement index does not match the baseline measurement index is greater than a threshold number of times, the abnormal early warning information is generated.

[0106] By applying the above embodiment, potential data transmission risks are identified in advance when no abnormality occurs in the process of executing a computing task by the node cluster, abnormal early warning and risk avoidance are realized, the influence of the abnormality on the execution of the computing task is reduced, and the stability of the node cluster in executing the computing task is improved.

[0107] Step 102: a first detection instruction is sent to the first node.

[0108] The first detection instruction is used to instruct the first node to detect the connectivity in the direction from the first node to the second node according to a transmission path included in the first detection instruction, to obtain a first detection result.

[0109] After obtaining the transmission path of the abnormal data stream, the node transmitting the abnormal data stream and the node receiving the abnormal data stream are determined from the plurality of nodes, i.e., the first node and the second node. Specifically, the first node and the second node can be determined according to the transmission path of the abnormal data stream. For example, the transmission path is five-tuple information, the first node can be located based on the source IP in the five-tuple information, and the second node can be located based on the destination IP in the five-tuple information.

[0110] Therefore, the first node sending data streams to the second node can be restored to detect the abnormal reason for determining the abnormality. At this time, the first detection instruction is generated, and the transmission path of the abnormal data stream can be included in the first detection instruction. For step 102, the first detection instruction is sent to the first node. After receiving the first detection instruction, the first node detects the connectivity in the direction from the first node to the second node according to the transmission path included in the first detection instruction, and obtains the first detection result. Specifically, the first node can send a detection data stream to the second node according to the transmission path, so as to detect the connectivity in the direction from the first node to the second node by using the detection data stream. When the response data stream returned by the second node based on the detection data stream is received, the first detection result indicating that the connectivity detection is passed is generated, and when the response data stream returned by the second node based on the detection data stream is not received, the first detection result indicating that the connectivity detection is failed is generated. The first node returns the first detection result to the device end sending the first detection instruction.

[0111] In actual application, a detection instruction issuing platform can be set in the application program for abnormal reason detection, and the abnormality collection platform sends the transmission path of the abnormal data stream to the detection instruction issuing platform; the detection instruction issuing platform generates the first detection instruction based on the transmission path and sends the first detection instruction to the first node. In this way, the unified generation and unified issuance of the first detection instruction in the node cluster can be realized, the management efficiency and resource utilization rate of the first detection instruction are improved, the stability and correctness of the generation and issuance of the first detection instruction are improved, and the correctness of the abnormal reason detection is ensured.

[0112] In some embodiments, the number of abnormal data streams is multiple; before step 102 "sending the first detection instruction to the first node" is executed, the following steps can also be executed: obtaining the network link characteristics of each abnormal data stream; clustering the multiple abnormal data streams based on the network link characteristics of each abnormal data stream to obtain at least one data stream cluster; for each data stream cluster, selecting a target abnormal data stream from the abnormal data streams included in the data stream cluster; based on this, step 102 "sending the first detection instruction to the first node" can be implemented by executing the following steps: sending the first detection instruction to each first node sending the target abnormal data stream.

[0113] Here, the network link feature of each abnormal data flow can be first acquired, which can be understood as the feature of the network link through which the abnormal data flow is transmitted from the first node to the second node, and can include features such as switch access layer, switch aggregation layer, switch core layer, switch link, and the like involved in the network link. Then, according to the network link feature of each abnormal data flow, the plurality of abnormal data flows are clustered to obtain at least one data flow cluster, and each data flow cluster includes at least one abnormal data flow. When clustering, a clustering algorithm can be used, such as a K-means clustering algorithm, which is not limited here. Since clustering is based on network link features, the network link features of each data flow cluster are the same, and based on this, for each data flow cluster, an abnormal data flow is determined from the data flow cluster as a target abnormal data flow, which can be any abnormal data flow in the data flow cluster. By detecting the transmission connectivity of the target abnormal data flow, the transmission connectivity of each abnormal data flow in the data flow cluster is detected. In this way, when sending the first detection instruction, the first detection instruction is only sent to the first node that sends each target abnormal data flow.

[0114] According to the above embodiment, the plurality of abnormal data flows are clustered according to the network link feature of each abnormal data flow, so that the first detection instruction is only sent to one abnormal data flow in each data flow cluster. In this way, the resource pressure of generating and issuing the first detection instruction can be reduced, and since the network link feature of the abnormal data flow is focused, the first detection result based on the first detection instruction can be used to quickly determine whether the abnormal reason of the abnormality is a network link abnormality, and the suspected network failure is quickly delimited.

[0115] In some embodiments, the number of computing tasks is multiple, and the number of abnormal data flows is multiple; before step 102 "sending the first detection instruction to the first node", the following steps can also be performed: for each abnormal data flow, the following processing is performed respectively: from the plurality of computing tasks, a target computing task to which the abnormal data flow belongs is determined; based on the task identifier of the target computing task and the transmission path of the abnormal data flow, the first detection instruction of the abnormal data flow is generated; the first detection instruction of the abnormal data flow is added to the instruction buffer queue; based on this, step 102 "sending the first detection instruction to the first node" can be implemented by performing the following steps: according to the issuing order of each first detection instruction in the instruction buffer queue, the first detection instruction is sent to the first node indicated by the task identifier in the first detection instruction.

[0116] Here, since the number of computing tasks is multiple and the number of abnormal data streams is multiple, before sending the first detection instruction to the first node, the target computing task to which the abnormal data stream belongs is first determined from the multiple computing tasks, so that the first detection instruction of the abnormal data stream is generated based on the task identifier of the target computing task and the transmission path of the abnormal data stream. In actual application, the first detection instruction of the abnormal data stream can also be added to the instruction buffer queue. In this way, according to the delivery order of each first detection instruction in the instruction buffer queue, the first detection instruction is sent to the first node indicated by the task identifier in the first detection instruction. The delivery order can be the addition order of the first detection instruction, or the order determined according to the delivery priority, which is not limited here

[0117] By applying the above embodiment, 1) by delivering the first detection instruction according to the task identifier of the computing task, the accuracy of the first detection instruction can be ensured, including one-to-one correspondence between the first detection instruction and the abnormal data stream, and alignment between the first detection instruction and the computing task; 2) with the help of the instruction buffer queue, the delivery stability of the first detection instruction is improved; in this way, a stable basis is provided for subsequent execution of the first node to perform connectivity detection.

[0118] Step 103: receiving the first detection result returned by the first node based on the first detection instruction.

[0119] For step 103, after obtaining the first detection result, the first node returns the first detection result to the device end sending the first detection instruction. At this time, the first detection result returned by the first node based on the first detection instruction is received, and the first detection result is used to represent that the connectivity detection passes or the connectivity detection fails.

[0120] Step 104: determining the abnormal reason for the abnormality based on the first detection result.

[0121] For step 104, after obtaining the first detection result, according to the information of connectivity detection passing or connectivity detection failing represented by the first detection result, the abnormal reason for the abnormality occurring in the process of executing the computing task by the node cluster is determined.

[0122] In actual application, an analysis platform can also be set in the application program for abnormal reason detection. The analysis platform receives the first detection result and analyzes the first detection result to obtain the abnormal reason for the abnormality. In this way, unified management and analysis of the first detection result in the node cluster can be realized, the management efficiency and analysis utilization rate of the first detection result are improved, and the efficiency of abnormal reason detection is improved.

[0123] In some embodiments, the step 104 "determining the abnormal cause of the abnormality based on the first detection result" can be implemented by performing the following steps: when the first detection result represents that the connectivity detection passes, determining that the abnormal cause of the abnormality is that the software of at least one of the first node and the second node is abnormal, the software being used to perform the computing task; when the first detection result represents that the connectivity detection fails, sending a second detection instruction to the first node and a third detection instruction to the second node; wherein the second detection instruction is used to instruct the first node to detect the connectivity in the direction from the first node to the third node to obtain a second detection result, and the third detection instruction is used to instruct the second node to detect the connectivity in the direction from the second node to the third node to obtain a third detection result, the network link between the third node and the first node being normal, and the network link between the third node and the second node being normal; receiving the second detection result returned by the first node based on the second detection instruction and the third detection result returned by the second node based on the third detection instruction; and determining the abnormal cause of the abnormality based on the second detection result and the third detection result.

[0124] Here, the detection data flow for the connectivity detection is completely consistent with the abnormal data flow (specifically, the transmission path is completely consistent), i.e., the detection scenario = the fault scenario; the device used for the connectivity detection is independent of the first node and the second node, which can avoid software state interference, and thus can ensure the accuracy of the determination of the abnormal cause.

[0125] When the first detection result represents that the connectivity detection passes, it is determined that the abnormal reason is an end-side software fault, i.e., the software of at least one of the first node and the second node is abnormal. This is because: the cause of the abnormality includes an end-side software fault, an end-side hardware fault, and a network link fault. When the connectivity detection passes, the end-side hardware fault and the network link fault are essentially excluded by "detecting normal transmission of the data stream", and only the end-side software fault remains. The specific derivation is as follows: 1) The connectivity detection passes, which excludes the end-side hardware fault: the end-side hardware (the core communication hardware of the node at the source end / destination end, such as a network card, a GPU communication module, and a motherboard link) is the "physical carrier" of the data stream transmission. If the hardware has a fault (such as a damaged network card or a loose motherboard link), any data stream (including the detection data stream) cannot be normally transmitted. Therefore, the connectivity detection passing means that the source end hardware can normally send the data stream, and the destination end hardware can normally receive the data stream, so the end-side hardware is normal. 2) The connectivity detection passes, which excludes the network link fault: the network link (the LA / LC / SGLC switch, the optical fiber link, and the switch port) is the "transmission channel" of the data stream from the source end to the destination end. If the network link has a fault (such as a damaged LC switch port, an interrupted optical fiber, or a congested link), the detection data stream will fail to be transmitted due to "path interruption / packet loss". However, the connectivity detection passing means that the entire link (LA→LC→SGLC→LC→LA) from the source end to the destination end can normally forward the data stream, so the network link is normal. 3) Only the end-side software fault remains: since the end-side hardware and the network link have been excluded, the cause of the abnormality is the end-side software fault. The core feature of the end-side software fault is that it does not affect the operation of the independent connectivity detection, but causes the data stream of the computing task to be interrupted, which completely matches the scenario of "connectivity detection passing but computing task being abnormal". At this time, it is quickly determined that the cause of the abnormality is not the network link fault.

[0126] When the first detection result represents that the connectivity detection does not pass, it means that the cause of the abnormality is not the end-side software fault, because the connectivity detection is independent of the operation of the computing task, so the end-side software fault does not affect the connectivity detection. However, since the connectivity detection does not pass, it means that the cause of the abnormality is not the end-side software fault. The end-side hardware fault is limited to "the node itself at the source end / destination end" (such as packet loss of the source end network card or damage of the destination end GPU communication module), which causes "the connectivity detection between the node and all other nodes to fail (i.e., the connectivity detection does not pass)". The network link fault is limited to "a specific path from the source end to the destination end" (such as a fault of the LC-05 switch port), which only causes "the connectivity detection of the path to fail", but the connectivity detection between the node and other nodes that are not on the path is still normal (i.e., the connectivity detection passes). At this time, the cause of the abnormality may be the end-side hardware fault or the network link fault. Therefore, the third-party node is needed to further determine whether the cause of the abnormality is the end-side hardware fault or the network link fault.

[0127] Specifically, the second detection instruction is sent to the first node, and the third detection instruction is sent to the second node; at this time, the first node detects the connectivity in the direction from the first node to the third node to obtain a second detection result (indicating that the connectivity detection passes or the connectivity detection fails); the second node detects the connectivity in the direction from the second node to the third node to obtain a third detection result (indicating that the connectivity detection passes or the connectivity detection fails), wherein the detection process of the connectivity in the direction from the first node to the third node and the detection process of the connectivity in the direction from the second node to the third node by the second node are the same as the detection process of the connectivity in the direction from the first node to the second node, which will not be described here. It should be noted that the network link between the third node and the first node is normal, and the network link between the third node and the second node is normal, so as to ensure the reliability of the abnormal reason positioning by means of the third node and exclude interference.

[0128] After obtaining the second detection result returned by the first node based on the second detection instruction and the third detection result returned by the second node based on the second detection instruction, the abnormal reason of the abnormality is determined based on the second detection result and the third detection result.

[0129] By applying the above embodiment, whether the abnormal reason is an end-side software failure can be determined based on whether the connectivity detection passes or not as indicated by the first detection result, if it is an end-side software failure, the network link failure and the end-side hardware failure are quickly excluded; if it is not an end-side software failure, whether the abnormal reason is a network link failure or an end-side hardware failure can be further determined by means of the third node, thereby improving the positioning accuracy of the abnormal reason.

[0130] In some embodiments, the step of determining the abnormal reason of the abnormality based on the second detection result and the third detection result can be implemented by performing the following steps: when the second detection result and the third detection result both indicate that the connectivity detection passes, it is determined that the abnormal reason of the abnormality is that the network link between the first node and the second node is abnormal; when the second detection result indicates that the connectivity detection fails and the third detection result indicates that the connectivity detection passes, it is determined that the abnormal reason of the abnormality is that the hardware of the first node is abnormal; when the second detection result indicates that the connectivity detection passes and the third detection result indicates that the connectivity detection fails, it is determined that the abnormal reason of the abnormality is that the hardware of the second node is abnormal; and when the second detection result indicates that the connectivity detection fails and the third detection result indicates that the connectivity detection fails, it is determined that the abnormal reason of the abnormality is that the hardware of the first node and the second node is abnormal.

[0131] Here, the specific abnormal reason is determined from the end-side hardware failure and the network link failure. When the second detection result and the third detection result both represent that the connectivity detection passes, it indicates that the connectivity from the first node to the third node is normal, and the connectivity from the second node to the third node is also normal. Since the above analysis shows that the second detection result and the third detection result both represent that the connectivity detection passes, the possibility of the end-side hardware failure of the first node and the second node is excluded. Therefore, the abnormal reason is that the network link between the first node and the second node is abnormal (i.e., network link failure).

[0132] When the second detection result represents that the connectivity detection does not pass, and the third detection result represents that the connectivity detection passes, since the above analysis shows that the third detection result represents that the connectivity detection passes, the possibility of the end-side hardware failure of the second node is excluded. Since the network link between the third node and the second node is normal, the possibility of the network link failure of the second node is also excluded. Therefore, the second node is normal. The second detection result represents that the connectivity detection does not pass, and since the network link between the third node and the first node is normal, in combination with the above analysis, it is known that the hardware of the first node is abnormal (i.e., end-side hardware failure).

[0133] When the second detection result represents that the connectivity detection passes, and the third detection result represents that the connectivity detection does not pass, since the above analysis shows that the second detection result represents that the connectivity detection passes, the possibility of the end-side hardware failure of the first node is excluded. Since the network link between the third node and the first node is normal, the possibility of the network link failure of the first node is also excluded. Therefore, the first node is normal. The third detection result represents that the connectivity detection does not pass, and since the network link between the third node and the second node is normal, in combination with the above analysis, it is known that the hardware of the second node is abnormal (i.e., end-side hardware failure).

[0134] When the second detection result represents that the connectivity detection does not pass, and the third detection result represents that the connectivity detection does not pass, the above analysis shows that the abnormal reason is that the hardware of the first node and the second node is both abnormal (i.e., end-side hardware failure).

[0135] By means of the third node, the above embodiment further locates the abnormal reason of the abnormality, and improves the positioning accuracy of the abnormal reason.

[0136] In some embodiments, when the first detection result indicates that the connectivity detection passes, it is determined that the abnormal reason of the occurrence of the abnormality is that the software of at least one of the first node and the second node is abnormal, and the following steps can be further performed to distinguish the node in which the software is abnormal: when the first detection result indicates that the connectivity detection passes, the abnormal log of the abnormal data flow is obtained; when the abnormal log belongs to the first node, it is determined that the abnormal reason of the occurrence of the abnormality is that the software of the first node is abnormal; when the abnormal log belongs to the second node, it is determined that the abnormal reason of the occurrence of the abnormality is that the software of the second node is abnormal; and when the abnormal log includes the first abnormal log of the first node and the second abnormal log of the second node, it is determined that the abnormal reason of the occurrence of the abnormality is that the software of at least one of the first node and the second node is abnormal.

[0137] Here, when it is determined that the software of at least one of the first node and the second node is abnormal, the node in which the software is abnormal can be further identified. Specifically, when the first detection result indicates that the connectivity detection passes, the abnormal log of the abnormal data flow is obtained. If the abnormal log belongs to the first node, it indicates that the software of the first node is abnormal, because the abnormal log is generated when the software of the first node is abnormal. If the abnormal log belongs to the second node, it indicates that the software of the second node is abnormal, because the abnormal log is generated when the software of the second node is abnormal. If the abnormal log includes the first abnormal log of the first node and the second abnormal log of the second node, it indicates that the software of the first node and / or the second node is abnormal. In actual applications, whether the software of the target node (at least one of the first node and the second node) is abnormal can be confirmed. For example, if the abnormal log of the target node includes the abnormal record of the occurrence of the abnormality in the process of performing the computing task, it is further verified that the software of the target node is abnormal, thereby improving the positioning accuracy of the abnormal reason.

[0138] By using the above embodiments, the node in which the software is abnormal can be further identified based on the abnormal log of the abnormal data flow, thereby improving the positioning accuracy of the abnormal reason.

[0139] In some embodiments, when the abnormal log comprises the first abnormal log of the first node and the second abnormal log of the second node, it is determined that the abnormal cause of the abnormality is that the software of at least one of the first node and the second node is abnormal, and the following step can be further performed to distinguish the node where the software is abnormal: when the abnormal log comprises the first abnormal log of the first node and the second abnormal log of the second node, the following processing is performed: when the first generation time of the first abnormal log is earlier than the second generation time of the second abnormal log, it is determined that the abnormal cause of the abnormality is that the software of the first node is abnormal; when the first generation time of the first abnormal log is later than the second generation time of the second abnormal log, it is determined that the abnormal cause of the abnormality is that the software of the second node is abnormal; and when the first generation time of the first abnormal log is equal to the second generation time of the second abnormal log, it is determined that the abnormal cause of the abnormality is that the software of the first node and the second node is abnormal.

[0140] Here, when the specific node where the software is abnormal is not identified based on the abnormal log, the first generation time of the first abnormal log and the second generation time of the second abnormal log can be obtained, so as to compare the first generation time with the second generation time to identify the specific node where the software is abnormal. Specifically, when the first generation time is earlier than the second generation time, it indicates that the first node is abnormal first, and the second node generates the second abnormal log under the influence of the first node, so it is considered that the software of the first node is abnormal. When the first generation time is later than the second generation time, it indicates that the second node is abnormal first, and the first node generates the first abnormal log under the influence of the second node, so it is considered that the software of the second node is abnormal. When the first generation time of the first abnormal log is equal to the second generation time of the second abnormal log, it indicates that the software of the first node and the second node is abnormal.

[0141] By applying the above embodiments, the specific node where the software is abnormal can be identified when the first node and the second node both generate abnormal logs, and the positioning accuracy of the abnormal cause is improved.

[0142] In some embodiments, after it is determined that the abnormal cause of the abnormality is that the software of at least one of the first node and the second node is abnormal, the following steps can be further performed: obtaining an abnormal node log of a target node, the target node being at least one of the first node and the second node; identifying a target error code in the abnormal node log; obtaining a corresponding relationship between a plurality of error codes and a plurality of software abnormal causes, the error codes and the software abnormal causes corresponding one by one; determining a target software abnormal cause corresponding to the target error code based on the corresponding relationship, and taking the target software abnormal cause as the cause of the abnormality of the software of the target node.

[0143] Here, when it is determined that the abnormal reason is that the software of the target node (at least one of the first node and the second node) is abnormal, further identification of the software abnormal reason can be performed. Specifically, first, the abnormal node log of the target node is obtained, which can record error codes such as communication library error code NCCL_ERR_CONN_TIMEOUT, NCCL_ERR_QP_CREATE, RDMA driver error code IBV_ERR_PORT_DOWN, etc. Then, a correspondence between a plurality of error codes and a plurality of software abnormal reasons is obtained, which is one-to-one between the error codes and the software abnormal reasons. The correspondence can be pre-established, and the correspondence can support dynamic updating, such as the existence of a new error code and a corresponding new software abnormal reason, which can be added to expand the coverage of error codes and software abnormal reasons. For example, the software abnormal reason corresponding to the error code "NCCL_ERR_CONN_TIMEOUT" is "software process memory leak, causing communication thread blocking"; the software abnormal reason corresponding to the error code "NCCL_ERR_QP_CREATE" is "communication library (such as NCCL) version incompatibility"; and the software abnormal reason corresponding to the error code "IBV_ERR_PORT_DOWN" is "RDMA driver parameter configuration error (port not enabled)". Based on this, the target software abnormal reason corresponding to the target error code can be determined based on the correspondence, and the target software abnormal reason is taken as the reason for the software of the target node being abnormal. After identifying the software abnormal reason, a notification message of the software abnormal reason can be output, and repair suggestion information (such as upgrading the version of the communication library (such as NCCL) to the target version) of the software abnormality can be pushed.

[0144] By applying the above embodiments, the correspondence between a plurality of error codes and a plurality of software abnormal reasons is used to realize automatic analysis of the software abnormal reason, and the positioning of the software abnormality is penetrated from the "end side level" to the "code / configuration level", which improves the positioning accuracy of the abnormal reason and reduces the network operation threshold.

[0145] In some embodiments, when the abnormal reason is determined to be abnormal, the following steps can be performed: when the abnormal reason is that the network link between the first node and the second node is abnormal, the network link between the first node and the second node is switched to the target network link; when the abnormal reason is that the software of the target node is abnormal, the target node is restarted, and the target node is at least one of the first node and the second node; and when the abnormal reason is that the hardware of the target node is abnormal, the target node is removed, and a sixth node is called to perform a target computing task on the target node, and the computing task includes the target computing task.

[0146] Here, after determining the abnormal reason of the abnormality, a corresponding processing method can also be provided for the abnormality to automatically process the abnormality in the node cluster. Specifically, (1) when the abnormal reason is that the network link (such as the LA-01→LC-02 link) between the first node and the second node is abnormal, the network link between the first node and the second node can be switched to the target network link (the LA-01→LC-03 backup link), so as to realize the transmission of the data stream between the first node and the second node through the target network link. At the same time, a network operation notification message can also be output to notify the network operation personnel to timely process the network link with the abnormality. (2) When the abnormal reason is that the software of the target node (at least one of the first node and the second node) is abnormal, the target node can be restarted, which can specifically include: automatically restarting the process and communication library involved in the target node when performing the computing task; if the target node fails to restart, the “process pulling script” can also be called to restore the communication service between the first node and the second node. (3) When the abnormal reason is that the hardware of the target node (at least one of the first node and the second node) is abnormal, the target node can be removed, such as being marked as unavailable to be excluded from the node cluster; at the same time, the sixth node is called to execute the target computing task (part of the computing task) on the target node. In actual application, after the automatic processing of the abnormality, the transmitted data stream can be referred to as a repaired data stream, so the connectivity of the repaired data stream can also be detected again, if the detection passes, it indicates that the computing task continues to be executed; if the detection fails, the automatic processing of the abnormality can be repeatedly executed, and if the number of times of the detection failing is greater than a threshold, an alarm information is generated to notify the network operation personnel to timely process.

[0147] By applying the above embodiments, after determining the abnormal reason of the abnormality, the abnormality in the node cluster can be automatically processed, so as to improve the speed and efficiency of abnormality repair, and perfect the abnormality processing process.

[0148] According to the transmission path of the abnormal data stream, the connectivity detection in the direction from the first node to the second node is implemented by restoring the transmission of the abnormal data stream, and the abnormal cause is determined based on the obtained first detection result. In this way, 1) the reliability of the connectivity detection is ensured, thereby improving the detection accuracy of the abnormal cause; and only the nodes involved in the abnormal data stream need to be detected, which can improve the detection efficiency of the abnormal cause and reduce the detection cost.

[0149] In some examples, the present application can be applied to the field of cloud technology. For example, each node in the node cluster can be a cloud server, and the plurality of nodes are used to cooperatively execute a computing task (such as a model training task, a cloud computing task, etc.). When an exception occurs in the node cluster during the execution of the computing task, the transmission path of the abnormal data stream in the node cluster is obtained, and a first detection instruction is sent to the first node (i.e., the sending end of the abnormal data stream) to enable the first node to detect the connectivity in the direction from the first node to the second node (i.e., the receiving end of the abnormal data stream) according to the transmission path included in the first detection instruction, to obtain a first detection result, and to determine the abnormal cause of the exception based on the first detection result after receiving the first detection result. Here, according to the transmission path of the abnormal data stream, the connectivity detection in the direction from the first node to the second node is implemented by restoring the transmission of the abnormal data stream, and the abnormal cause is determined based on the obtained first detection result. In this way, 1) the reliability of the connectivity detection is ensured, thereby improving the detection accuracy of the abnormal cause of the cloud service; and only the nodes involved in the abnormal data stream need to be detected, which can improve the detection efficiency of the abnormal cause of the cloud service and reduce the detection cost.

[0150] The following describes an abnormal cause detection method provided by an embodiment of the present application. As described above, the abnormal cause detection method provided by the embodiment of the present application is implemented by an electronic device, for example, can be implemented by a server or a terminal alone, or cooperatively by a server and a terminal. Therefore, the execution subject of each step will not be repeated in the following description. See Figure 3C , Figure 3C FIG. 3 is a third flowchart of an abnormal cause detection method provided by an embodiment of the present application. The abnormal cause detection method provided by the embodiment of the present application includes the following steps.

[0151] Step 301: The first node receives the first detection instruction.

[0152] The first detection instruction is sent by the node cluster when an exception occurs in the process of executing the computing task, and the first detection instruction includes a transmission path of an abnormal data stream in the node cluster. The node cluster includes a plurality of nodes that cooperatively execute the computing task, and the plurality of nodes include a first node that sends the abnormal data stream and a second node that receives the abnormal data stream.

[0153] Here, after the node cluster has an exception in the process of executing the computing task, the abnormal data stream in the node cluster can be determined, and the transmission path of the abnormal data stream can be obtained. The transmission path can be five-tuple information, which can include source IP address, destination IP address, source port, destination port, and transmission protocol information. The generation of the abnormal data stream is related to the exception, and the computing task relies on multiple nodes to communicate through the network. Data needs to be continuously transmitted between nodes (such as model parameters, intermediate calculation results, etc. in a model training task). These data transmissions form stable "communication flows (i.e., data streams)", each data stream is uniquely identified by five-tuple information to ensure accurate data interaction. Since the node cluster has an exception in the process of executing the computing task, the normal transmission of the data stream is broken at this time, so the data stream becomes an abnormal data stream, and the transmission path of the abnormal data stream is obtained.

[0154] After obtaining the transmission path of the abnormal data stream, the node that transmits the abnormal data stream is determined from the plurality of nodes, i.e., the first node that sends the abnormal data stream and the second node that receives the abnormal data stream. Specifically, the first node and the second node can be determined according to the transmission path of the abnormal data stream. For example, the transmission path is five-tuple information, the source IP in the five-tuple information can be located to the first node, and the destination IP in the five-tuple information can be located to the second node. Therefore, the first node sending the data stream to the second node can be restored and detected to determine the abnormal reason of the exception. At this time, the first detection instruction is generated, which can include the transmission path of the abnormal data stream; and the first detection instruction is sent to the first node. The first node receives the first detection instruction.

[0155] Step 302: In response to the first detection instruction, the connectivity in the direction from the first node to the second node is detected according to the transmission path, and a first detection result is obtained.

[0156] Here, the first node, after receiving the first detection instruction, detects the connectivity in the direction from the first node to the second node according to the transmission path included in the first detection instruction, to obtain a first detection result. Specifically, the first node can send a detection data stream to the second node according to the transmission path, so as to detect the connectivity in the direction from the first node to the second node by using the detection data stream. When receiving a response data stream returned by the second node based on the detection data stream, a first detection result indicating that the connectivity detection is passed is generated; when not receiving the response data stream returned by the second node based on the detection data stream, a first detection result indicating that the connectivity detection is failed is generated.

[0157] In some embodiments, referring to Figure 3D , the step 302 of "detecting the connectivity in the direction from the first node to the second node according to the transmission path in response to the first detection instruction, to obtain a first detection result" can be implemented by performing the following steps 3021-3025: the step 3021 of, in response to the first detection instruction, establishing a communication connection between the first node and the second node according to the source address and the destination address in the transmission path, and obtaining a source port number and a destination port number of the communication connection; the step 3022 of obtaining a target source port number in the transmission path, and modifying the source port number to the target source port number; the step 3023 of sending a detection data stream to the destination port number of the second node according to the target source port number, the detection data stream being used to detect the connectivity in the direction from the first node to the second node; the step 3024 of, when receiving a response data stream returned by the second node based on the detection data stream, generating a first detection result indicating that the connectivity detection is passed; and the step 3025 of, when not receiving the response data stream returned by the second node based on the detection data stream, generating a first detection result indicating that the connectivity detection is failed.

[0158] For the step 3021, in response to the first detection instruction, a communication connection between the first node and the second node is established according to the source address (i.e., a source IP address) and the destination address (i.e., a destination IP address) in the transmission path, and a source port number and a destination port number of the communication connection are obtained. In this way, the first node can send a data stream to the second node.

[0159] For the step 3022, since the source port number is randomly allocated when the communication connection is established, the target source port number in the transmission path is obtained, and the source port number is modified to the target source port number. In this way, it is ensured that the transmission path of the detection data stream and the transmission path of the abnormal data stream are completely consistent, and the reliability of the connectivity detection is improved. It should be noted that the destination port number is fixed; and if the destination port number is also randomly allocated, the allocated destination port number also needs to be modified to the destination port number in the transmission path.

[0160] For step 3023, the detection data stream is sent to the destination port number of the second node according to the target source port number, so as to detect the connectivity in the direction from the first node to the second node. When sending the detection data stream, the detection parameters (such as the number of data packets of the detection data stream, the sending frequency of the detection data stream, etc.) can be determined according to the communication type of the abnormal data stream (such as long frame high frequency type, short frame low frequency type, short frame medium frequency type, etc.), so as to send the detection data stream based on the detection parameters.

[0161] For step 3024, when the response data stream returned by the second node based on the detection data stream is received, the first detection result representing that the connectivity detection is passed is generated. For step 3025, when the response data stream returned by the second node based on the detection data stream is not received, the first detection result representing that the connectivity detection is not passed is generated.

[0162] By applying the above embodiment, by sending the detection data stream according to the transmission path, it can be ensured that the transmission path of the detection data stream and the transmission path of the abnormal data stream are completely consistent, so as to ensure the reliability of the connectivity detection, and further, through the first detection result of the connectivity detection, the abnormal reason can be quickly located.

[0163] Step 303: sending the first detection result.

[0164] The first detection result is used for determining the abnormal reason of the abnormality.

[0165] Here, the first node returns the first detection result to the device end sending the first detection instruction. After obtaining the first detection result, the device end sending the first detection instruction determines the abnormal reason of the abnormality in the process of executing the computing task of the node cluster according to the information representing that the connectivity detection is passed or the connectivity detection is not passed in the first detection result.

[0166] In actual application, a detection tool can also be set in the first node, which is used to execute steps 301-303, realize the connectivity detection, and obtain the first detection result.

[0167] With the above embodiments of the application, when an exception occurs in the process of the node cluster performing the computing task, the transmission path of the abnormal data stream in the node cluster is obtained, a first detection instruction is sent to the first node (i.e., the sending end of the abnormal data stream), so that the first node detects the connectivity in the direction from the first node to the second node (i.e., the receiving end of the abnormal data stream) according to the transmission path included in the first detection instruction, and a first detection result is obtained. Then, based on the first detection result, the abnormal cause of the exception is determined. Here, according to the transmission path of the abnormal data stream, the connectivity detection in the direction from the first node to the second node is realized by restoring the transmission of the abnormal data stream, and the abnormal cause is determined based on the obtained first detection result. In this way, 1) the reliability of the connectivity detection is ensured, thereby improving the detection accuracy of the abnormal cause; and only the nodes involved in the abnormal data stream need to be detected, which can improve the detection efficiency of the abnormal cause and reduce the detection cost.

[0168] In the following, taking a training task of a computing task as a training task of a machine learning model (such as a large model) as an example, an exemplary application of the embodiments of the application in an actual application scenario is described.

[0169] In the related art, when a training task is interrupted (i.e., an exception occurs in the process of performing the computing task), the abnormal nodes that may exist are determined based on the node information on the training log. The fault root cause is determined by checking all nodes in the full link (including software, hardware, and network layer). Since the large model has strong synchronization, a single fault can be quickly transmitted through the network, and many nodes, frameworks, servers, networks, storage, and computing power modules need to be simultaneously checked by the operation and maintenance personnel. In fact, the composition of the entire training cluster involves many software and hardware parts, many of which are black boxes, so the checking is very difficult and requires a lot of manpower and time to check layer by layer until the root cause is finally confirmed. Therefore, the related art has the following problems: 1) Some training tasks are very large and may involve thousands of cards or even tens of thousands of cards, and many modules may report exceptions at the same time, resulting in many suspicious points. To quickly locate the root cause, many operation and maintenance personnel need to be involved, and a lot of time is consumed. 2) In the GPU training cluster, a large number of software / hardware are black boxes, such as GPU, commercial network card, CUDA, etc. The checking can only be done by judging the explicit alarm log. If a silent fault is encountered, it is difficult to start, which further prolongs the positioning time. 3) When the training is interrupted, there is usually network-related abnormal information. The existing positioning method cannot quickly determine whether it is a suspected network problem. 4) The traditional network positioning method is based on the pingmesh of the tcp protocol. However, in the training scene, the data stream is an RDMA stream, so the existing network detection scheme cannot effectively position the training scene.

[0170] In the process of training a large model, usually contains several thousand cards or even tens of thousands of cards, and trains for tens of days or even months, because of the various devices involved, such as GPU, CPU, server, storage, network and so on. Therefore, it is common for training interruptions to occur during the training process. However, since these training nodes are connected by a network, when a training interruption occurs, it is often accompanied by a communication interruption, resulting in an abnormal flow alarm or network-related error logs on the training log, resulting in almost all training interruption faults being suspected network faults. Since network communication is mainly composed of three parts: end-side software, end-side hardware, and network link. Therefore, the root cause of this problem can be divided into three categories: the first category is the problem of upper-layer business software, the second category is the problem of server physical hardware, and the third category is the problem of network link abnormality; Due to the synchronization of large model training, the fault can be transmitted to many nodes in a very short time, because it appears to be a suspected network fault, so it is often from the network to start troubleshooting. If the root cause of the problem is not caused by the network, the troubleshooting direction is wrong, which greatly increases the recovery time of the training fault.

[0171] Based on this, the embodiment of the present application provides a method for quickly bounding suspected network faults in the large model training interruption scenario. By adding five-tuple information of communication connection (i.e. data flow of communication) in the collection communication library, when the training is interrupted, collect the five-tuple information of all abnormal flows affected, and then restore the detection of the abnormal flow through the Netquint Rping tool (i.e. based on the five-tuple information of the abnormal flow, detect whether the transmission between the sending end and the receiving end of the abnormal flow is normal through the probe flow). If the abnormal flow restoration test is normal (i.e. the probe flow transmission is normal), it means that the problem has nothing to do with the network, and the first category of software problems can be quickly bounded. If the abnormal flow restoration test is not normal (i.e. the probe flow transmission is not normal), it means that the training interruption is suspected to be caused by network abnormalities, but it needs to be further confirmed whether it is the second or third category of problems. And on this basis, a third-party normal node is introduced to form a one-to-one Netquint Rping test, if the new test is normal (i.e. the third-party node and the sending end and the receiving end of the abnormal flow are all normal), since the paths do not intersect, it can be judged that the network data path is abnormal, and the network problem is bounded, that is, the third category, which needs to be located by the network side. If there is an abnormal test case in the two tests (i.e. the transmission between the third-party node and the sending end and / or receiving end of the abnormal flow is not normal), it means that the problem is caused by the end-side hardware, that is, the second category, which is bounded as a server hardware problem.

[0172] It should be noted that the main focus of the present application is not to restart the training task, when the training is interrupted, the abnormal quintuple information in the collection communication library is obtained, all possible abnormal nodes and network links are obtained, and the Netquint Rping detection flow (i.e. the above detection data flow) is synchronized and issued, and then according to the detection result (i.e. the above first detection result) the software problem, hardware problem or network problem can be quickly delimited, greatly reducing the operation and maintenance manpower investment and shortening the positioning time. In practical application, the present application is mainly used in the AI large model training interruption exception scene. Through the method, the fault in the training process can be quickly delimited, and it can be accurately distinguished whether it is a software problem, a hardware problem or a network problem, the fault positioning efficiency is improved, and the training cost is saved.

[0173] As shown in Figure 4 When the training is interrupted during large-scale training, the related art first investigates suspicious nodes, specifically, full-link node investigation is performed on the modules involved (including training framework, network, storage, GPU, computing power, etc.), suspicious information is collected, and suspicious nodes are analyzed and summarized based on the suspicious information. Then, by replacing the nodes, the card kicking and training are performed. At this time, a large number of suspicious nodes are screened out, and the specific root cause needs to be further located, because if only the fault nodes are screened out and the root cause is not located, these nodes cannot be used continuously. The current GPU resources are relatively effective, if the real problem node cannot be quickly found, and the root cause is found and repaired, the available GPU standby resources will soon be exhausted, and a large amount of GPU resources will be wasted.

[0174] To solve the above problems, the present application adds quintuple information to all data flows in the collection communication library, when the training is interrupted, the communication of these nodes will also be interrupted, at this time the abnormal flow affected by the interruption is generated. The communication library records the quintuple information of the affected abnormal flow. Since the data flow in large model training is all RDMA traffic, a feasible solution is to determine whether the problem is related to the network by constructing an RDMA flow based on the recorded information. However, traditional RDMA detection does not support custom quintuple, and there is no way to completely restore the path when the business fails, therefore the present application designs and develops the Netquint Rping tool which can freely set the flow quintuple, this tool can freely define the quintuple information of the data flow to achieve the purpose of restoring the abnormal flow.

[0175] As shown in Figure 5As shown, an abnormal flow is generated between node A and node B, so in the first step, node A and node B are subjected to abnormal flow detection, and a detection result is obtained; then the detection result is analyzed, if all the detection flows are normally transmitted (i.e., the detection is normal), it indicates that the abnormal flow is not caused by the network and hardware (because the detection flow is completely simulated by the five-tuple information of the abnormal flow, if the hardware or network has a problem, then the detection flow must be problematic), then the abnormal flow is caused by software (such as process hanging, causing data flow interruption, etc.). In this way, it can be quickly judged whether the abnormal flow is caused by the end-side software problem. Continue, on the basis of the first step, since the detection flow is abnormally transmitted (i.e., the detection is not normal), it indicates that the components (i.e., node A, node B, network path) of the transmission detection flow must have abnormal parts. At this time, a third-party normal node C is introduced to assist in judgment. Specifically, detection flows from node A to node C and from node B to node C are constructed, if the two newly added detection flows are normal, it indicates that the abnormal flow is not related to the node (i.e., not related to the hardware), and the abnormal reason is limited to the network link (path) problem; otherwise, it is a node abnormality, and which detection flow is abnormal, the corresponding node has an end-side hardware problem.

[0176] To achieve the above object, the embodiment of the present application provides a suspected network failure fast localization scheme when a large model training is interrupted, referring to Figure 6 , for a training cluster, the fault reason positioning system provided by the embodiment of the present application mainly includes: a Netquint Rping detection tool (i.e., the above detection tool), an abnormal flow five-tuple collection platform (i.e., the above abnormal collection platform), a detection task issuing platform (i.e., the above detection instruction issuing platform), and an analysis platform. Next, each part in Figure 6 will be described.

[0177] Step S1: as shown in Figure 7 (as shown in the abnormal flow five-tuple collection platform in Figure 6 ), when the training cluster is interrupted, since there are still communication tasks on the training process at this time, the training process interruption causes these communication tasks to be unable to be completed. Because the communication task will generate an ERRCQE log due to timeout, and the ERRCQE is bound to the queue pair QP, the five-tuple information (including source IP, destination IP, source port, destination port, and protocol) of the QP is recorded on the communication library, so the five-tuple information of the abnormal flow of the affected communication task can be obtained. Because the affected communication nodes can be more than one, multiple GPU nodes may appear abnormal flow, so an abnormal flow five-tuple collection platform is designed on the end side to detect and collect the five-tuple information of the abnormal flow. After the training interruption and the generation of the abnormal flow, the abnormal flow five-tuple collection platform collects all the five-tuple information of the abnormal flow, and sends it to the detection task issuing platform.

[0178] Step S2: As shown in the probe task issuing platform shown in Figure 8 Figure 6 As shown in the probe task issuing platform shown in

[0179] Step S3: The probe task issuing platform finds the GPU server that needs to issue the probe task according to the source IP and destination IP in the five-tuple information of the abnormal flow, and then issues the Netquint Rping probe tool on the GPU server. As shown in the Netquint Rping probe tool shown in Figure 9 Figure 6 As shown in the Netquint Rping probe tool shown in

[0180] Step S4: The analysis platform obtains the detection result, and decides whether further detection is needed according to the detection result. If the S3 detection is passed, it means that the end-side hardware and network link involved in the detection flow in S3 are normal, and the root cause of the training interruption is the end-side software.

[0181] If the detection fails, it means that the end-side software is not the cause of the training interruption. Then it may be the end-side hardware or the network link. In order to further confirm the problem attribution, a third-party node C is introduced for secondary detection issuing, and the detection flows of node A and node C and the detection flows of node B and node C are issued respectively. In order to ensure the effectiveness of the detection, it is necessary to ensure that the newly added node C is a normal node, and that node A and node C, and node B and node C are normal network paths.

[0182] As shown in Figure 10 ​​As shown in the middle (1), when the newly added node C is normally detected with the suspected fault node A and the node B, it is indicated that the end-side hardware of the node A and the node B is normal, and the root cause of the problem is the network link between the node A and the node B, and the fault is delimited as a network link problem. For example, Figure 10 As shown in the middle (2), when the newly added node C is normally detected with one of the nodes (for example, the node A) and abnormally detected with the other node (for example, the node B), it is indicated that the end-side hardware of the node B is faulty, and the fault is delimited as an end-side hardware problem of the node B; for example, Figure 10 As shown in the middle (3), when all the newly added detection flows are abnormal, it is indicated that the end-side hardware of the node A and the node B is abnormal.

[0183] By applying the above embodiments of the present application, the suspected network fault can be quickly delimited, and the time for delimiting the suspected network fault is reduced. In actual implementation, after the training interruption occurs, the time for delimiting the suspected network fault is shortened from an average of 12 hours to within 1 minute.

[0184] The following continues to describe an exemplary structure of the implementation of the abnormal reason detection apparatus 555 provided by the embodiments of the present application as a software module. In some embodiments, as shown in the middle (1), the software module stored in the abnormal reason detection apparatus 555 in the memory 550 can include: Figure 2 The acquisition module 5551 is configured to acquire a transmission path of an abnormal data flow in a node cluster when the node cluster occurs an abnormality in the process of executing a computing task; wherein the node cluster includes a plurality of nodes that cooperatively execute the computing task, and the plurality of nodes include a first node that sends the abnormal data flow and a second node that receives the abnormal data flow; the first sending module 5552 is configured to send a first detection instruction to the first node, and the first detection instruction is used to instruct the first node to detect the connectivity in the direction from the first node to the second node according to the transmission path included in the first detection instruction, and obtain a first detection result; the first receiving module 5553 is configured to receive the first detection result returned by the first node based on the first detection instruction; and the determination module 5554 is configured to determine an abnormal reason of the abnormality based on the first detection result.

[0185] In some embodiments, the acquisition module 5551 is further configured to acquire an abnormal log of the abnormal data flow in the node cluster, and determine an abnormal communication queue pair bound with the abnormal log; acquire a correspondence relationship between a plurality of communication queue pairs and a plurality of communication transmission paths, the communication queue pair and the communication transmission path are in one-to-one correspondence; determine a target communication transmission path corresponding to the abnormal communication queue pair based on the correspondence relationship, and take the target communication transmission path as the transmission path of the abnormal data flow.

[0186] In some embodiments, the determining module 5554 is further configured to: when the first detection result indicates that the connectivity detection passes, determine that the abnormal cause of the abnormality is that software of at least one of the first node and the second node is abnormal, the software being used to perform the computing task; when the first detection result indicates that the connectivity detection fails, send a second detection instruction to the first node and a third detection instruction to the second node; the second detection instruction is used to instruct the first node to detect connectivity in a direction from the first node to a third node to obtain a second detection result, and the third detection instruction is used to instruct the second node to detect connectivity in a direction from the second node to the third node to obtain a third detection result, the network link between the third node and the first node is normal, and the network link between the third node and the second node is normal; receive the second detection result returned by the first node based on the second detection instruction and receive the third detection result returned by the second node based on the third detection instruction; and determine the abnormal cause of the abnormality based on the second detection result and the third detection result.

[0187] In some embodiments, the determining module 5554 is further configured to: when the second detection result and the third detection result both indicate that the connectivity detection passes, determine that the abnormal cause of the abnormality is that the network link between the first node and the second node is abnormal; when the second detection result indicates that the connectivity detection fails and the third detection result indicates that the connectivity detection passes, determine that the abnormal cause of the abnormality is that the hardware of the first node is abnormal; when the second detection result indicates that the connectivity detection passes and the third detection result indicates that the connectivity detection fails, determine that the abnormal cause of the abnormality is that the hardware of the second node is abnormal; and when the second detection result indicates that the connectivity detection fails and the third detection result indicates that the connectivity detection fails, determine that the abnormal cause of the abnormality is that the hardware of the first node and the second node is abnormal.

[0188] In some embodiments, the determining module 5554 is further configured to: when the first detection result indicates that the connectivity detection passes, obtain an abnormal log of the abnormal data flow; when the abnormal log is attributed to the first node, determine that the abnormal cause of the abnormality is that the software of the first node is abnormal; when the abnormal log is attributed to the second node, determine that the abnormal cause of the abnormality is that the software of the second node is abnormal; and when the abnormal log includes a first abnormal log of the first node and a second abnormal log of the second node, determine that the abnormal cause of the abnormality is that the software of at least one of the first node and the second node is abnormal.

[0189] In some embodiments, the determining module 5554 is further configured to, when the exception log comprises a first exception log of the first node and a second exception log of the second node, determine that the abnormal cause of the exception is that the software of the first node is abnormal when a first generation time of the first exception log is earlier than a second generation time of the second exception log, determine that the abnormal cause of the exception is that the software of the second node is abnormal when the first generation time of the first exception log is later than the second generation time of the second exception log, and determine that the abnormal cause of the exception is that the software of the first node and the second node is abnormal when the first generation time of the first exception log is equal to the second generation time of the second exception log.

[0190] In some embodiments, the determining module 5554 is further configured to, after determining that the abnormal cause of the exception is that the software of at least one of the first node and the second node is abnormal, acquire an exception node log of a target node, the target node being at least one of the first node and the second node, identify a target error code in the exception node log, acquire a correspondence between a plurality of error codes and a plurality of software abnormal causes, the error codes and the software abnormal causes corresponding to each other in one-to-one manner, determine a target software abnormal cause corresponding to the target error code based on the correspondence, and determine the target software abnormal cause as the cause of the abnormal software of the target node.

[0191] In some embodiments, the acquiring module 5551 is further configured to, when the node cluster does not have an exception during execution of the computing task, acquire a normal transmission path of a normal data flow in the node cluster, the plurality of nodes comprising a fourth node that sends the normal data flow and a fifth node that receives the normal data flow, send a pre-detection instruction to the fourth node, the pre-detection instruction being used to instruct the fourth node to detect connectivity in a direction from the fourth node to the fifth node according to the normal transmission path and acquire a measurement index of the connectivity, receive the measurement index returned by the fourth node based on the pre-detection instruction, and generate an exception early warning information when the measurement index does not match a baseline measurement index.

[0192] In some embodiments, the determining module 5554 is further configured to, when the abnormal cause is that a network link between the first node and the second node is abnormal, switch the network link between the first node and the second node as a target network link; when the abnormal cause is that software of a target node is abnormal, restart the target node, the target node being at least one of the first node and the second node; and when the abnormal cause is that hardware of the target node is abnormal, remove the target node and invoke a sixth node to perform a target computing task on the target node, the computing task including the target computing task.

[0193] In some embodiments, the number of abnormal data streams is a plurality; the first sending module 5552 is further configured to, before the first sending module 5552 sends the first detection instruction to the first node, acquire a network link feature of each abnormal data stream; cluster a plurality of abnormal data streams based on the network link feature of each abnormal data stream to obtain at least one data stream cluster; and for each data stream cluster, select a target abnormal data stream from abnormal data streams included in the data stream cluster; and the first sending module 5552 is further configured to send the first detection instruction to each first node that sends the target abnormal data stream.

[0194] In some embodiments, the number of computing tasks is a plurality, and the number of abnormal data streams is a plurality; the first sending module 5552 is further configured to, before the first sending module 5552 sends the first detection instruction to the first node, for each abnormal data stream, perform the following processing respectively: determine a target computing task to which the abnormal data stream belongs from a plurality of computing tasks; generate a first detection instruction of the abnormal data stream based on a task identifier of the target computing task and a transmission path of the abnormal data stream; and add the first detection instruction of the abnormal data stream into an instruction buffer queue; and the first sending module 5552 is further configured to send the first detection instruction to a first node indicated by the task identifier in the first detection instruction according to a sending order of each first detection instruction in the instruction buffer queue.

[0195] The embodiment of the present application further provides an abnormal reason detection device, which is applied to a first node and comprises: a second receiving module, configured to receive a first detection instruction; wherein the first detection instruction is sent by a node cluster when an abnormality occurs in the process of executing a computing task, the first detection instruction comprises a transmission path of an abnormal data stream in the node cluster, the node cluster comprises a plurality of nodes that cooperatively execute the computing task, the plurality of nodes comprise the first node that sends the abnormal data stream and a second node that receives the abnormal data stream; a detection module, configured to respond to the first detection instruction, detect connectivity in a direction from the first node to the second node according to the transmission path, and obtain a first detection result; and a second sending module, configured to send the first detection result, wherein the first detection result is used for determining an abnormal reason of the abnormality.

[0196] In some embodiments, the detection module is further configured to, in response to the first detection instruction, establish a communication connection between the first node and the second node according to a source address and a destination address in the transmission path, and obtain a source port number and a destination port number of the communication connection; obtain a target source port number in the transmission path, and modify the source port number to the target source port number; send a detection data stream to the destination port number of the second node according to the target source port number, wherein the detection data stream is used for detecting the connectivity in the direction from the first node to the second node; when receiving a response data stream returned by the second node based on the detection data stream, generate a first detection result indicating that the connectivity detection is passed; and when not receiving the response data stream returned by the second node based on the detection data stream, generate a first detection result indicating that the connectivity detection is not passed.

[0197] It should be noted that the description of the device embodiments in the present application is similar to the description of the above-mentioned method embodiments, and has similar beneficial effects as the method embodiments, which will not be repeated here. For the technical details not described in the abnormal reason detection device provided by the embodiment of the present application, the technical details can be understood based on the description of the above-mentioned method embodiments.

[0198] The embodiment of the present application further provides a computer program product, which comprises computer executable instructions or a computer program, and the computer executable instructions or the computer program are stored in a computer readable storage medium. A processor of an electronic device reads the computer executable instructions or the computer program from the computer readable storage medium, and the processor executes the computer executable instructions or the computer program, so that the electronic device executes the abnormal reason detection method provided by the embodiment of the present application.

[0199] The embodiment of the present application further provides a computer readable storage medium, and the computer readable storage medium stores computer executable instructions or a computer program. When the computer executable instructions or the computer program are executed by a processor, the processor executes the abnormal reason detection method provided by the embodiment of the present application.

[0200] In some embodiments, the computer readable storage medium can be a RAM, a ROM, a flash memory, a magnetic surface memory, an optical disc, or a CD-ROM memory, etc. The computer readable storage medium can also be various devices including one or any combination of the above storage mediums.

[0201] In some embodiments, the computer executable instructions can be in the form of programs, software, software modules, scripts or codes, written in any form of programming language (including a compiled or interpreted language, or a declarative or procedural language), and can be deployed in any form, including being deployed as an independent program or being deployed as a module, a component, a subroutine or other unit suitable for use in a computing environment.

[0202] As an example, the computer executable instructions can but not necessarily correspond to a file in a file system, can be stored in a part of a file storing other programs or data, for example, stored in one or more scripts in a Hyper Text Markup Language (HTML) document, stored in a single file dedicated to the program in question, or stored in multiple cooperative files (for example, files storing one or more modules, subprograms or code parts).

[0203] As an example, the computer executable instructions can be deployed to be executed on one electronic device, or executed on multiple electronic devices located in one place, or executed on multiple electronic devices distributed in multiple places and interconnected through a communication network.

[0204] The above description is only for the embodiments of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement and improvement made within the spirit and scope of the present application shall be included in the protection scope of the present application.

Claims

1. A method for detecting the cause of anomalies, characterized in that, The method includes: When an anomaly occurs in the node cluster during the execution of computing tasks, the transmission path of the abnormal data stream in the node cluster is obtained; The node cluster includes multiple nodes that collaboratively execute the computing task, and the multiple nodes include: a first node that sends the abnormal data stream and a second node that receives the abnormal data stream; Send a first detection instruction to the first node. The first detection instruction is used to instruct the first node to detect the connectivity in the direction from the first node to the second node according to the transmission path included in the first detection instruction, and obtain a first detection result. Receive the first detection result returned by the first node based on the first detection instruction; Based on the first detection result, the cause of the anomaly is determined.

2. The method as described in claim 1, characterized in that, The step of obtaining the transmission path of the abnormal data stream in the node cluster includes: Obtain the abnormal logs of abnormal data streams in the node cluster, and determine the abnormal communication queue pairs bound to the abnormal logs; Obtain the correspondence between multiple communication queue pairs and multiple communication transmission paths, wherein the communication queue pairs and the communication transmission paths correspond one-to-one; Based on the correspondence, the target communication transmission path corresponding to the abnormal communication queue is determined, and the target communication transmission path is used as the transmission path of the abnormal data stream.

3. The method as described in claim 1, characterized in that, The step of determining the cause of the anomaly based on the first detection result includes: When the first detection result indicates that the connectivity detection is passed, the cause of the anomaly is determined to be: the software of at least one of the first node and the second node is abnormal, and the software is used to perform the computing task; When the first detection result indicates that the connectivity detection has failed, a second detection command is sent to the first node, and a third detection command is sent to the second node; Wherein, the second detection instruction is used to instruct the first node to detect the connectivity in the direction from the first node to the third node and obtain a second detection result; the third detection instruction is used to instruct the second node to detect the connectivity in the direction from the second node to the third node and obtain a third detection result; the network link between the third node and the first node is normal, and the network link between the third node and the second node is normal. Receive the second detection result returned by the first node based on the second detection instruction, and receive the third detection result returned by the second node based on the second detection instruction; Based on the second and third detection results, the cause of the anomaly is determined.

4. The method as described in claim 3, characterized in that, The step of determining the cause of the anomaly based on the second detection result and the third detection result includes: When both the second and third detection results indicate that the connectivity detection has passed, the cause of the anomaly is determined to be: there is an anomaly in the network link between the first node and the second node; When the second detection result indicates that the connectivity test has failed, and the third detection result indicates that the connectivity test has passed, the cause of the abnormality is determined to be: the hardware of the first node is abnormal; When the second detection result indicates that the connectivity detection has passed and the third detection result indicates that the connectivity detection has failed, the cause of the anomaly is determined to be: the hardware of the second node is faulty; When the second detection result indicates that the connectivity test has failed, and the third detection result indicates that the connectivity test has failed, the cause of the abnormality is determined to be that the hardware of both the first node and the second node is abnormal.

5. The method as described in claim 3, characterized in that, When the first detection result indicates that the connectivity detection has passed, the cause of the anomaly is determined to be: at least one of the software of the first node and the second node has an anomaly, including: When the first detection result indicates that the connectivity detection is passed, the abnormal log of the abnormal data stream is obtained; When the exception log belongs to the first node, the cause of the exception is determined to be: the software of the first node has an exception; When the exception log belongs to the second node, the cause of the exception is determined to be: the software of the second node is abnormal; When the exception log includes the first exception log of the first node and the second exception log of the second node, the cause of the exception is determined to be that at least one of the software of the first node and the second node has an exception.

6. The method as described in claim 5, characterized in that, When the exception log includes the first exception log of the first node and the second exception log of the second node, the cause of the exception is determined to be: at least one of the software of the first node and the second node has an exception, including: When the exception log includes the first exception log of the first node and the second exception log of the second node, the following processing is performed: When the first generation time of the first abnormal log is earlier than the second generation time of the second abnormal log, the cause of the abnormality is determined to be: the software of the first node has an abnormality; When the first generation time of the first abnormal log is later than the second generation time of the second abnormal log, the cause of the abnormality is determined to be: the software of the second node is abnormal; When the first generation time of the first abnormal log is equal to the second generation time of the second abnormal log, the cause of the abnormality is determined to be: the software of both the first node and the second node has an abnormality.

7. The method as described in claim 3, characterized in that, After determining that the cause of the anomaly is that at least one of the software in the first node and the second node is abnormal, the method further includes: Obtain the abnormal node logs of the target node, wherein the target node is at least one of the first node and the second node; Identify the target error code in the log of the abnormal node; Obtain the correspondence between multiple error codes and multiple software exception causes, wherein the error codes and the software exception causes are in one-to-one correspondence; Based on the correspondence, the cause of the target software anomaly corresponding to the target error code is determined, and the cause of the target software anomaly is used as the reason for the software anomaly of the target node.

8. The method according to any one of claims 1-7, characterized in that, The method further includes: When the node cluster does not encounter any abnormalities during the execution of the computing task, the normal transmission path of the normal data stream in the node cluster is obtained. The plurality of nodes include a fourth node that sends the normal data stream and a fifth node that receives the normal data stream. A pre-detection instruction is sent to the fourth node, which instructs the fourth node to detect the connectivity from the fourth node to the fifth node according to the normal transmission path and obtain a connectivity measurement index. Receive the measurement metrics returned by the fourth node based on the pre-detection instruction; An anomaly warning is generated when the measured metric does not match the baseline measured metric.

9. The method according to any one of claims 1-7, characterized in that, The method further includes: When the cause of the anomaly is an anomaly in the network link between the first node and the second node, the network link between the first node and the second node is switched to the target network link; When the cause of the anomaly is a software malfunction in the target node, the target node is restarted, wherein the target node is at least one of the first node and the second node; When the cause of the anomaly is a hardware malfunction of the target node, the target node is removed, and the sixth node is invoked to execute the target computing task on the target node, the computing task including the target computing task.

10. The method according to any one of claims 1-7, characterized in that, The number of abnormal data streams is multiple; before sending the first detection command to the first node, the method further includes: Obtain the network link characteristics of each of the abnormal data streams; Based on the network link characteristics of each of the abnormal data streams, the multiple abnormal data streams are clustered to obtain at least one data stream cluster; For each data stream cluster, a target abnormal data stream is selected from the abnormal data streams included in the data stream cluster; Sending the first detection command to the first node includes: The first detection instruction is sent to each of the first nodes that sends the target abnormal data stream.

11. The method according to any one of claims 1-7, characterized in that, The number of computational tasks is multiple, and the number of abnormal data streams is multiple; before sending the first detection instruction to the first node, the method further includes: For each of the aforementioned abnormal data streams, the following processing is performed: From the multiple computing tasks, determine the target computing task to which the abnormal data stream belongs; Based on the task identifier of the target computing task and the transmission path of the abnormal data stream, a first detection instruction for the abnormal data stream is generated. Add the first detection instruction of the abnormal data stream to the instruction buffer queue; Sending the first detection command to the first node includes: The first detection instruction is sent to the first node indicated by the task identifier in the first detection instruction according to the order in which each first detection instruction is issued in the instruction buffer queue.

12. A method for detecting the cause of an anomaly, characterized in that, The method includes: The first node receives the first detection command; The first detection instruction is sent when an anomaly occurs in the node cluster during the execution of the computing task. The first detection instruction includes the transmission path of the abnormal data stream in the node cluster. The node cluster includes multiple nodes that cooperate in executing the computing task. The multiple nodes include the first node that sends the abnormal data stream and the second node that receives the abnormal data stream. In response to the first detection command, the connectivity in the direction from the first node to the second node is detected according to the transmission path to obtain a first detection result; Send the first detection result, which is used to determine the cause of the anomaly.

13. The method as described in claim 12, characterized in that, The step of responding to the first detection command and detecting the connectivity in the direction from the first node to the second node according to the transmission path to obtain a first detection result includes: In response to the first detection command, a communication connection is established between the first node and the second node according to the source address and destination address in the transmission path, and the source port number and destination port number of the communication connection are obtained; Obtain the target source port number in the transmission path, and modify the source port number to the target source port number; According to the target source port number, a detection data stream is sent to the destination port number of the second node. The detection data stream is used to detect the connectivity in the direction from the first node to the second node. When a response data stream is received from the second node based on the detection data stream, a first detection result representing that the connectivity detection has passed is generated; When no response data stream is received from the second node based on the detection data stream, a first detection result representing that the connectivity detection failed is generated.

14. An anomaly cause detection device, characterized in that, The device includes: The acquisition module is used to acquire the transmission path of the abnormal data stream in the node cluster when an abnormality occurs during the execution of the computing task. The node cluster includes multiple nodes that collaboratively execute the computing task, and the multiple nodes include: a first node that sends the abnormal data stream and a second node that receives the abnormal data stream; The first sending module is used to send a first detection instruction to the first node. The first detection instruction is used to instruct the first node to detect the connectivity in the direction from the first node to the second node according to the transmission path included in the first detection instruction, and to obtain a first detection result. The first receiving module is configured to receive the first detection result returned by the first node based on the first detection instruction; The determination module is used to determine the cause of the anomaly based on the first detection result.

15. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions for a computer; A processor, when executing computer-executable instructions stored in the memory, implements the anomaly cause detection method according to any one of claims 1 to 13.

16. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by the processor, the abnormal cause detection method according to any one of claims 1 to 13 is implemented.

17. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by the processor, the abnormal cause detection method according to any one of claims 1 to 13 is implemented.

Images