Abnormality cause detection method, device, apparatus, storage medium, and program product

Abstract

The application provides an abnormal reason detection method, device, equipment, storage medium and program product; the method comprises the following steps: when an exception occurs in the process of executing a computing task by a node cluster, a transmission path of an abnormal data stream in the node cluster is acquired; wherein the node cluster comprises a plurality of nodes for cooperatively executing the computing task, and the plurality of nodes comprise a first node for sending the abnormal data stream and a second node for receiving the abnormal data stream; a first detection instruction is sent to the first node, the first detection instruction is used for instructing the first node to detect the connectivity in the direction from the first node to the second node according to the transmission path included in the first detection instruction, and a first detection result is obtained; the first detection result returned by the first node based on the first detection instruction is received; and the abnormal reason of the exception is determined based on the first detection result. Through the application, the detection efficiency and detection accuracy of the abnormal reason can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, device, storage medium, and program product for detecting abnormal causes. Background Technology

[0002] When performing computing tasks based on a node cluster, various devices are typically involved, such as frameworks, servers, storage nodes, networks, and computing nodes. Therefore, anomalies are common during execution. Related technologies involve performing a full-link investigation on all nodes (including software, hardware, and network layers) to determine the cause of the anomaly. However, since the nodes within the cluster are connected by a network, a single anomaly can be rapidly propagated through the network. Therefore, it is often necessary to investigate many nodes simultaneously, requiring significant manpower and time for layer-by-layer investigation, and the accuracy of the investigation is difficult to guarantee. Summary of the Invention

[0003] This application provides an anomaly cause detection method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve the detection efficiency and accuracy of anomaly causes.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides an anomaly cause detection method, including: When an anomaly occurs in the node cluster during the execution of computing tasks, the transmission path of the abnormal data stream in the node cluster is obtained; The node cluster includes multiple nodes that collaboratively execute the computing task, and the multiple nodes include: a first node that sends the abnormal data stream and a second node that receives the abnormal data stream; Send a first detection instruction to the first node. The first detection instruction is used to instruct the first node to detect the connectivity in the direction from the first node to the second node according to the transmission path included in the first detection instruction, and obtain a first detection result. Receive the first detection result returned by the first node based on the first detection instruction; Based on the first detection result, the cause of the anomaly is determined.

[0005] This application embodiment also provides an anomaly cause detection method, applied to a first node, including: The first node receives the first detection command; The first detection instruction is sent when an anomaly occurs in the node cluster during the execution of the computing task. The first detection instruction includes the transmission path of the abnormal data stream in the node cluster. The node cluster includes multiple nodes that cooperate in executing the computing task. The multiple nodes include the first node that sends the abnormal data stream and the second node that receives the abnormal data stream. In response to the first detection command, the connectivity from the first node to the second node is detected according to the transmission path to obtain a first detection result; Send the first detection result, which is used to determine the cause of the anomaly.

[0006] This application embodiment also provides an anomaly cause detection device, including: The acquisition module is used to acquire the transmission path of the abnormal data stream in the node cluster when an abnormality occurs during the execution of the computing task. The node cluster includes multiple nodes that collaboratively execute the computing task, and the multiple nodes include: a first node that sends the abnormal data stream and a second node that receives the abnormal data stream; The first sending module is used to send a first detection instruction to the first node. The first detection instruction is used to instruct the first node to detect the connectivity in the direction from the first node to the second node according to the transmission path included in the first detection instruction, and to obtain a first detection result. The first receiving module is configured to receive the first detection result returned by the first node based on the first detection instruction; The determination module is used to determine the cause of the anomaly based on the first detection result.

[0007] In the above scheme, the acquisition module is further configured to acquire the abnormal logs of the abnormal data stream in the node cluster, and determine the abnormal communication queue pairs bound to the abnormal logs; acquire the correspondence between multiple communication queue pairs and multiple communication transmission paths, wherein the communication queue pairs and the communication transmission paths correspond one-to-one; based on the correspondence, determine the target communication transmission path corresponding to the abnormal communication queue pair, and use the target communication transmission path as the transmission path of the abnormal data stream.

[0008] In the above scheme, the determining module is further configured to: when the first detection result indicates that the connectivity detection has passed, determine that the cause of the anomaly is: at least one of the software of the first node and the second node is abnormal, and the software is used to execute the computation task; when the first detection result indicates that the connectivity detection has failed, send a second detection instruction to the first node and a third detection instruction to the second node; wherein, the second detection instruction is configured to instruct the first node to detect the connectivity in the direction from the first node to the third node to obtain a second detection result, and the third detection instruction is configured to instruct the second node to detect the connectivity in the direction from the second node to the third node to obtain a third detection result, wherein the network link between the third node and the first node is normal, and the network link between the third node and the second node is normal; receive the second detection result returned by the first node based on the second detection instruction, and receive the third detection result returned by the second node based on the third detection instruction; and determine the cause of the anomaly based on the second detection result and the third detection result.

[0009] In the above scheme, the determining module is further configured to: determine that the cause of the anomaly is an anomaly in the network link between the first node and the second node when both the second detection result and the third detection result indicate that the connectivity detection has passed; determine that the cause of the anomaly is an anomaly in the hardware of the first node when the second detection result indicates that the connectivity detection has failed and the third detection result indicates that the connectivity detection has passed; determine that the cause of the anomaly is an anomaly in the hardware of the second node when the second detection result indicates that the connectivity detection has passed and the third detection result indicates that the connectivity detection has failed; and determine that the cause of the anomaly is an anomaly in the hardware of both the first node and the second node when both the second detection result indicates that the connectivity detection has failed and the third detection result indicates that the connectivity detection has failed.

[0010] In the above scheme, the determining module is further configured to: acquire the abnormal log of the abnormal data stream when the first detection result indicates that the connectivity detection has passed; determine the cause of the abnormality when the abnormal log belongs to the first node: the software of the first node is abnormal; determine the cause of the abnormality when the abnormal log belongs to the second node: the software of the second node is abnormal; determine the cause of the abnormality when the abnormal log includes the first abnormal log of the first node and the second abnormal log of the second node: the software of at least one of the first node and the second node is abnormal.

[0011] In the above scheme, the determining module is further configured to perform the following processing when the abnormal log includes the first abnormal log of the first node and the second abnormal log of the second node: when the first generation time of the first abnormal log is earlier than the second generation time of the second abnormal log, determine that the abnormal cause of the abnormality is: the software of the first node has an abnormality; when the first generation time of the first abnormal log is later than the second generation time of the second abnormal log, determine that the abnormal cause of the abnormality is: the software of the second node has an abnormality; when the first generation time of the first abnormal log is equal to the second generation time of the second abnormal log, determine that the abnormal cause of the abnormality is: the software of both the first node and the second node has an abnormality.

[0012] In the above scheme, the determining module is further configured to: after determining that the cause of the anomaly is that at least one of the first node and the second node has a software anomaly, obtain the anomaly node log of the target node, wherein the target node is at least one of the first node and the second node; identify the target error code in the anomaly node log; obtain the correspondence between multiple error codes and multiple software anomaly causes, wherein the error codes and the software anomaly causes correspond one-to-one; and based on the correspondence, determine the target software anomaly cause corresponding to the target error code, and use the target software anomaly cause as the reason for the software anomaly of the target node.

[0013] In the above scheme, the acquisition module is further configured to acquire the normal transmission path of the normal data stream in the node cluster when no abnormality occurs during the execution of the computing task, wherein the plurality of nodes include a fourth node that sends the normal data stream and a fifth node that receives the normal data stream; send a pre-detection instruction to the fourth node, wherein the pre-detection instruction is configured to instruct the fourth node to detect the connectivity in the direction from the fourth node to the fifth node according to the normal transmission path, and acquire a connectivity measurement index; receive the measurement index returned by the fourth node based on the pre-detection instruction; and generate an abnormal warning information when the measurement index does not match the baseline measurement index.

[0014] In the above scheme, the determining module is further configured to: switch the network link between the first node and the second node to the target network link when the cause of the anomaly is an anomaly in the network link between the first node and the second node; restart the target node when the cause of the anomaly is a software anomaly in the target node, wherein the target node is at least one of the first node and the second node; and remove the target node and call the sixth node to execute the target computing task on the target node when the cause of the anomaly is a hardware anomaly in the target node, wherein the computing task includes the target computing task.

[0015] In the above scheme, there are multiple abnormal data streams; the first sending module is further configured to acquire the network link characteristics of each abnormal data stream before sending the first detection instruction to the first node; based on the network link characteristics of each abnormal data stream, cluster the multiple abnormal data streams to obtain at least one data stream cluster; for each data stream cluster, select a target abnormal data stream from the abnormal data streams included in the data stream cluster; the first sending module is further configured to send the first detection instruction to each first node that sends the target abnormal data stream.

[0016] In the above scheme, there are multiple computing tasks and multiple abnormal data streams; the first sending module is further configured to perform the following processing for each abnormal data stream before sending the first detection instruction to the first node: determine the target computing task to which the abnormal data stream belongs from the multiple computing tasks; generate a first detection instruction for the abnormal data stream based on the task identifier of the target computing task and the transmission path of the abnormal data stream; add the first detection instruction for the abnormal data stream to the instruction buffer queue; the first sending module is further configured to send the first detection instruction to the first node indicated by the task identifier in the first detection instruction according to the issuance order of each first detection instruction in the instruction buffer queue.

[0017] This application embodiment also provides an anomaly cause detection device, applied to a first node, including: The second receiving module is used to receive the first detection command; The first detection instruction is sent when an anomaly occurs in the node cluster during the execution of the computing task. The first detection instruction includes the transmission path of the abnormal data stream in the node cluster. The node cluster includes multiple nodes that cooperate in executing the computing task. The multiple nodes include the first node that sends the abnormal data stream and the second node that receives the abnormal data stream. The detection module is used to respond to the first detection command, detect the connectivity in the direction from the first node to the second node according to the transmission path, and obtain a first detection result; The second sending module is used to send the first detection result, which is used to determine the cause of the anomaly.

[0018] In the above scheme, the detection module is further configured to, in response to the first detection command, establish a communication connection between the first node and the second node according to the source address and destination address in the transmission path, and obtain the source port number and destination port number of the communication connection; obtain the target source port number in the transmission path and modify the source port number to the target source port number; send a detection data stream to the destination port number of the second node according to the target source port number, the detection data stream being used to detect connectivity in the direction from the first node to the second node; when a response data stream returned by the second node based on the detection data stream is received, generate a first detection result indicating that the connectivity detection has passed; when a response data stream returned by the second node based on the detection data stream is not received, generate a first detection result indicating that the connectivity detection has failed.

[0019] This application also provides an electronic device, including: Memory is used to store executable instructions for a computer; When the processor executes computer-executable instructions stored in the memory, it implements the anomaly cause detection method provided in the embodiments of this application.

[0020] This application also provides a computer-readable storage medium storing computer-executable instructions or computer programs, which, when executed by a processor, implement the anomaly cause detection method provided in this application.

[0021] This application also provides a computer program product, including computer-executable instructions or a computer program, which, when executed by a processor, implements the anomaly cause detection method provided in this application.

[0022] The embodiments of this application have the following beneficial effects: Applying the above embodiments of this application, when an anomaly occurs in the node cluster during the execution of a computing task, the transmission path of the abnormal data stream in the node cluster is obtained, and a first detection instruction is sent to the first node (i.e., the sender of the abnormal data stream). This causes the first node to detect the connectivity from the first node to the second node (i.e., the receiver of the abnormal data stream) according to the transmission path included in the first detection instruction, obtaining a first detection result. Upon receiving the first detection result, the cause of the anomaly is determined based on the first detection result. Here, by reconstructing the transmission path of the abnormal data stream, connectivity detection from the first node to the second node is achieved, thereby determining the cause of the anomaly based on the obtained first detection result. Thus, 1) the reliability of connectivity detection is guaranteed, thereby improving the detection accuracy of the cause of the anomaly; and only the nodes involved in the abnormal data stream need to be detected, which improves the detection efficiency of the cause of the anomaly and reduces the detection cost. Attached Figure Description

[0023] Figure 1 This is a first architecture diagram of the anomaly cause detection system provided in the embodiments of this application; Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application; Figure 3A This is a first flowchart illustrating the anomaly cause detection method provided in this application embodiment; Figure 3B This is a schematic diagram of the second process of the anomaly cause detection method provided in the embodiments of this application; Figure 3C This is a schematic diagram of the third process of the anomaly cause detection method provided in the embodiments of this application; Figure 3D This is a schematic diagram of the fourth process of the anomaly cause detection method provided in the embodiments of this application; Figure 4 This is a schematic diagram of the anomaly cause detection process provided in related technologies; Figure 5 This is a schematic diagram of the fifth process of the anomaly cause detection method provided in the embodiments of this application; Figure 6 This is a schematic diagram of the second architecture of the anomaly cause detection system provided in the embodiments of this application; Figure 7 This is a schematic diagram of the abnormal flow quintuple acquisition platform provided in the embodiments of this application; Figure 8 This is a schematic diagram of the detection task distribution platform provided in an embodiment of this application; Figure 9 This is a schematic diagram of the detection process of the detection tool provided in the embodiments of this application; Figure 10This is a schematic diagram illustrating secondary probing using a third-party node, provided in an embodiment of this application.

[0024] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0027] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0028] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of a larger module or unit that includes the functionality of the module or unit.

[0029] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0030] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0031] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0032] 1) Client: An application running on an electronic device that provides various services, such as a client that supports the detection of the cause of anomalies.

[0033] 2) In response to, used to indicate the conditions or states on which the operation performed depends. When the conditions or states on which it depends are met, one or more operations performed may be performed in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations are performed.

[0034] 3) Transmission path, also known as the 5-tuple information, includes: source Internet Protocol (IP) address, destination IP address, source port, destination port, and transport protocol. The transmission path is used to identify the corresponding data stream.

[0035] This application provides an anomaly cause detection method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve the detection efficiency and accuracy of anomaly causes. The following is a detailed description of the embodiments of this application based on the above explanation of the terms and concepts used.

[0036] The following describes the anomaly detection system provided in the embodiments of this application. See also: Figure 1 , Figure 1 This is a first architecture diagram of the anomaly cause detection system provided in this application embodiment. To support an exemplary application, the anomaly cause detection system 100 includes: a server 200, a network 300, a terminal 400, and a node cluster (including multiple nodes, such as a first node 600 and a second node 700). The server 200, network 300, terminal 400, first node 600, and second node 700 are connected via network 300, which can be a wide area network (WAN), a local area network (LAN), or a combination of both, using wireless or wired links for data transmission.

[0037] Here, terminal 400 (e.g., running a client that supports anomaly cause detection) responds to anomaly cause detection instructions by sending anomaly cause detection requests to server 200. These requests instruct server 200 to perform anomaly cause detection when an anomaly occurs during the execution of a computing task in the node cluster. Server 200 receives the anomaly cause detection request and, in response, obtains the transmission path of the abnormal data stream in the node cluster when an anomaly occurs during the execution of a computing task. The node cluster includes multiple nodes collaboratively executing computing tasks, and these nodes include: a first node 600 that sends the abnormal data stream and a receiving node 600. The second node 700 of the abnormal data stream sends a first detection command to the first node 600; the first node 600 receives the first detection command, responds to the first detection command, and detects the connectivity in the direction from the first node 600 to the second node 700 according to the transmission path included in the first detection command, and obtains a first detection result; returns the first detection result to the server 200; the server 200 receives the first detection result returned by the first node based on the first detection command; based on the first detection result, determines the cause of the abnormality; returns notification information of the cause of the abnormality to the terminal 400; the terminal 400 receives the cause of the abnormality returned by the server 200 and displays the notification information of the cause of the abnormality.

[0038] The anomaly detection method provided in this application is implemented by an electronic device. For example, it can be implemented by a terminal alone, by a server alone, or by a terminal and a server working together. The electronic device implementing the anomaly detection method provided in this application can be various types of terminals or servers. The server (e.g., server 200) can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal (e.g., terminal 400) can be a laptop, tablet, desktop computer, smartphone, smart voice interaction device (e.g., smart speaker), smart home appliance (e.g., smart TV), smartwatch, vehicle terminal, wearable device, virtual reality (VR) device, aircraft, etc., but is not limited to these. The terminal and server can be connected directly or indirectly through wired or wireless communication, and this application does not impose any restrictions on this.

[0039] In some embodiments, the terminal or server can implement the anomaly detection method provided in this application by running various computer-executable instructions or computer programs. For example, computer-executable instructions can be microprogram-level commands, machine instructions, or software instructions. Computer programs can be native programs or software modules in an operating system; they can be native applications (APPs), i.e., programs that need to be installed in the operating system to run; or they can be applets that can be embedded in any APP, i.e., programs that only need to be downloaded to a browser environment to run. In summary, the aforementioned computer-executable instructions can be any form of instruction, and the aforementioned computer programs can be any form of application, module, or plugin.

[0040] The following describes an electronic device that implements the anomaly detection method according to an embodiment of this application. See also... Figure 2 , Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 500 provided in this embodiment can be a terminal or a server. Figure 2 As shown, electronic device 500 includes at least one processor 510, memory 550, at least one network interface 520, and user interface 530. The various components in electronic device 500 are coupled together via a bus system 540. It is understood that the bus system 540 is used to implement communication between these components. In addition to a data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 2 The general labeled all buses as Bus System 540.

[0041] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0042] User interface 530 includes one or more output devices 531 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0043] Memory 550 may be removable, non-removable, or a combination thereof. Memory 550 may include one or more storage devices physically located away from processor 510. Memory 550 may include volatile memory or non-volatile memory, or both. Non-volatile memory may be read-only memory (ROM), and volatile memory may be random access memory (RAM). The memory 550 described in this application embodiment is intended to include any suitable type of memory.

[0044] In some embodiments, memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0045] Operating system 551 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks; The network communication module 552 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc. Presentation module 553 is used to enable the presentation of information (e.g., user interface for operating peripheral devices and displaying content and information) via one or more output devices 531 (e.g., display screen, speaker, etc.) associated with user interface 530. The input processing module 554 is used to detect and translate one or more user inputs or interactions from one or more input devices 532.

[0046] In some embodiments, the anomaly detection device provided in this application can be implemented in software. Figure 2 An anomaly cause detection device 555 stored in memory 550 is shown. It can be software in the form of programs and plug-ins, including the following software modules: acquisition module 5551, first transmission module 5552, first reception module 5553 and determination module 5554. These modules are logical and can therefore be arbitrarily combined or further split according to the functions they implement. The functions of each module will be described below.

[0047] The following describes the anomaly detection method provided in the embodiments of this application. As mentioned above, the anomaly detection method provided in the embodiments of this application is implemented by an electronic device, such as a server or terminal alone, or a server and terminal working together. Therefore, the executing entity of each step will not be described again below. See Figure 3A , Figure 3A This is a first flowchart illustrating the anomaly cause detection method provided in this application embodiment. The anomaly cause detection method provided in this application embodiment includes: Step 101: When an anomaly occurs in the node cluster during the execution of a computing task, obtain the transmission path of the abnormal data stream in the node cluster.

[0048] The node cluster consists of multiple nodes that collaboratively execute computational tasks. These multiple nodes include a first node that sends abnormal data streams and a second node that receives abnormal data streams.

[0049] For step 101, the node cluster comprises multiple nodes connected via a network to collaboratively execute computational tasks. A node cluster can execute at least one computational task simultaneously. Different computational tasks may use different nodes or overlap in their usage; this is not limited here. Examples of such computational tasks include, but are not limited to: training tasks for machine learning models, inference tasks for machine learning models (such as weather simulation, molecular dynamics calculations, knowledge question answering, etc.), and data read / write tasks in a distributed storage system. This computational task is executed collaboratively by multiple nodes.

[0050] During the execution of computing tasks in a node cluster, anomalies (i.e., failures) may occur. These anomalies can affect the execution efficiency, correctness, and stability of the computing tasks, potentially leading to problems such as task interruption, performance degradation, incorrect calculation results, resource waste, and progress stalling. Examples of such anomalies include, but are not limited to: interruption of computing task execution, multiple timeout retries of computing tasks, and data consistency failures between multiple nodes executing the computing task, resulting in incorrect calculation results.

[0051] Specifically, exceptions that cause incomplete interruptions to computing tasks may include: (1) Hardware-related failures: A) Degradation of node hardware performance: For example, a small number of bad blocks (not completely failed) appear in the graphics processing unit (GPU) memory, causing occasional data verification errors during calculation; the core frequency of the central processing unit (CPU) drops abnormally, reducing the node's computing efficiency; the network card experiences intermittent packet loss (not completely disconnected), leading to increased data transmission latency. These failures do not directly interrupt the task, but can cause the task to run slowly, produce abnormal results, or require repeated retries. B) Unstable heat dissipation or power supply: A faulty node fan causes the CPU / GPU temperature to be too high, triggering frequency reduction protection (performance degrades but the system does not shut down); aging power supply modules cause voltage fluctuations, causing occasional data calculation errors (such as abnormal floating-point precision). The task is not interrupted, but the calculation results may be distorted, and there is a risk of sudden interruption during long-term operation. C) Partial failure of storage media: Partial damage to the disks of the distributed storage nodes causes slow or occasional failures in reading and writing some data; abnormal write amplification of the SSD reduces the efficiency of data persistence. The task may continue to run, but the overall progress will be delayed due to frequent input / output retries.

[0052] (2) Software and task scheduling failures: A) Task stalling or deadlock: During multi-node collaboration, due to synchronization logic defects, some nodes enter an infinite wait (e.g., the distributed lock is not released), the task is not interrupted but completely stalled; the computing framework has memory leaks, causing the node memory to gradually run out, the task runs slower and slower but is not immediately interrupted. B) Data consistency failure: In distributed computing, due to network latency or clock synchronization deviation, some nodes have inconsistent local cached data with global data, resulting in incorrect calculation results (e.g., asynchronous parameter updates during large model training, causing model convergence failure). The task is not interrupted, but the output results are invalid. C) Resource contention and scheduling anomalies: The cluster resource manager (e.g., K8s) has scheduling logic errors, causing multiple tasks to compete for the GPU / CPU resources of the same node, resulting in resource over-allocation (e.g., the actual allocated resources exceed the node limit), the task frequently stalls but does not terminate; the task priority configuration is incorrect, low-priority tasks preempt high-priority task resources, causing the core task progress to be delayed.

[0053] (3) Network-related non-disruptive faults: A) Network performance degradation: Intermittent congestion occurs in the cluster network links (such as core switch buffer overflow), causing communication latency to increase from microseconds to milliseconds; packet loss occurs between some nodes (such as a packet loss rate of 0.1%), although communication is not interrupted, the distributed synchronization efficiency drops significantly (such as model operation time increases by 10 times). B) Network partitioning: The cluster network is divided into multiple independent subnets (such as access layer switch failure causing some nodes to be isolated from the core layer). Nodes in different subnets can still communicate and run tasks within the subnet, but cross-subnet task synchronization fails, causing the global task to "split" (some nodes continue to calculate, some nodes wait, the overall task cannot be completed but is not completely interrupted).

[0054] (4) Storage and data-related faults: A) Data read / write anomalies: The distributed file system experiences metadata server response delays, causing frequent timeouts and retries when reading input data (but not complete failures); incorrect data checksums cause some data blocks to need to be re-copied, halting task progress but not interrupting it. B) Storage capacity warnings and quota limits: Insufficient remaining storage cluster capacity (e.g., below 10%) causes a sharp drop in data write speed (due to frequent defragmentation); tasks exceed preset storage quotas, resulting in write operations being rate-limited, causing data persistence delays during computation and affecting overall task efficiency.

[0055] (5) Security and environmental failures: a) Nodes are maliciously compromised or infected: Some nodes are infected with malicious programs that consume CPU / GPU resources, resulting in reduced available resources for computing tasks and slower operation; malicious programs tamper with intermediate computing data, resulting in incorrect task results (uninterrupted but invalid output). b) Inconsistent environment configuration: Inconsistent software dependency versions among cluster nodes cause some nodes to run normally while others experience compatibility errors (such as function call failures), and tasks in heterogeneous environments may exhibit non-interrupted exceptions of "partial success and partial failure".

[0056] Specifically, anomalies that completely interrupt computing tasks may include: (1) End-side software failure: This is a problem with the upper-layer business software, such as the training framework process hanging, the communication library of the node cluster being incorrect, or the driver parameter configuration of Remote Direct Memory Access (RDMA) being abnormal. These will cause the communication task to fail and trigger the ERRCQE log (i.e., the exception log); (2) End-side hardware failure: This refers to abnormal physical hardware of the server, such as GPU failure, RDMA network card damage, or server motherboard problems. These will prevent the nodes from participating in communication normally; (3) Network link failure: This involves Access Layer Switch (LA), Convergence Layer Switch (LC), Core Layer Switch (SGLC) and links, such as switch port congestion, fiber optic interruption, or abnormal data path. These will manifest as communication interruption between nodes.

[0057] When an anomaly occurs in a node cluster during computational tasks, the abnormal data stream within the cluster can be identified, allowing the retrieval of its transmission path. This path can be represented by a 5-tuple, which includes information such as the source IP address, destination IP address, source port, destination port, and transmission protocol. The generation of abnormal data streams is related to anomalies. Computational tasks rely on multiple nodes communicating over a network, requiring continuous data transmission between nodes (such as model parameters and intermediate computation results in model training tasks). These data transmissions form a stable "communication stream (i.e., data stream)," with each stream uniquely identified by the 5-tuple to ensure accurate data exchange. When an anomaly occurs during the execution of computational tasks by the node cluster, the normal transmission of the data stream is disrupted (data interaction is not completed as expected). This data stream becomes an abnormal data stream. The triggering factors can correspond to three main categories of faults: 1) Software faults: such as the training framework process hanging or communication library logic errors, which prevent the execution of data sending / receiving instructions and interrupt the communication stream; 2) Hardware faults: such as GPU / network card hardware faults (e.g., video memory errors, network card packet loss), which cause abnormal data processing or transmission, resulting in communication stream stagnation or data corruption; 3) Network link faults: such as switch port faults or fiber optic link interruptions, which block the data transmission path and prevent the communication stream from reaching specific nodes.

[0058] In some embodiments, step 101, "obtaining the transmission path of the abnormal data stream in the node cluster," can be achieved by performing the following steps: obtaining the abnormal logs of the abnormal data stream in the node cluster and determining the abnormal communication queue pairs bound to the abnormal logs; obtaining the correspondence between multiple communication queue pairs and multiple communication transmission paths, with each communication queue pair and communication transmission path corresponding one-to-one; based on the correspondence, determining the target communication transmission path corresponding to the abnormal communication queue pair, and using the target communication transmission path as the transmission path of the abnormal data stream.

[0059] Here, when an abnormal data stream occurs in the node cluster, an abnormal log for that data stream is automatically generated. This log serves as a marker of the abnormal data stream. For example, when an abnormal data stream occurs in a node cluster based on RDMA communication, an ERRCQE log (i.e., an abnormal log) will be generated. The abnormal log is bound to the communication queue pair (i.e., QP) that triggered the abnormal data stream. The abnormal log is used to mark the communication queue pair that triggered the abnormal data stream; this pair can be called an abnormal communication queue pair. The communication queue pair is the core structure used to manage the sending and receiving of data streams and is directly associated with the specific data stream. When the node cluster's communication library initializes the data stream, it pre-records the correspondence between the communication queue pair and the communication transmission path (such as 5-tuple information) for each data stream. The communication queue pair and the communication transmission path correspond one-to-one. Based on this, since the abnormal log is bound to the abnormal communication queue pair, and the communication queue pair corresponds one-to-one with the communication transmission path, the transmission path of the abnormal data stream can be determined through the abnormal log. Specifically, the abnormal communication queue pair can be found from multiple communication queue pairs first, and then the target communication transmission path corresponding to the abnormal communication queue pair can be determined based on the correspondence, so as to use the target communication transmission path as the transmission path of the abnormal data stream.

[0060] By applying the above embodiments, 1) through the correspondence between multiple communication queue pairs and multiple communication transmission paths, as well as the abnormal communication queue pairs bound to the abnormal log, the transmission path of the abnormal data stream can be quickly determined, avoiding blind investigation and wasting time and manpower costs, thereby improving the efficiency of locating the cause of the abnormality; 2) without restarting the computing task, the transmission path of the abnormal data stream can be quickly obtained through the abnormal log, narrowing down the scope of suspicious nodes with abnormalities, and further improving the efficiency of locating the cause of the abnormality.

[0061] In some embodiments, an application for anomaly detection can be deployed on a server or terminal. By running this application, the server or terminal can obtain the transmission paths of abnormal data streams in the node cluster. Specifically, an anomaly collection platform for the transmission paths of abnormal data streams can be set up in the application. This platform can collect the transmission paths of abnormal data streams appearing in the node cluster, achieving unified collection and management of the transmission paths of abnormal data streams, thereby improving collection and management efficiency and resource utilization.

[0062] In some embodiments, see Figure 3B The following steps 201-204 can also be executed: Step 201: When no abnormality occurs during the execution of the computing task in the node cluster, obtain the normal transmission path of the normal data flow in the node cluster, including multiple nodes, including a fourth node that sends the normal data flow and a fifth node that receives the normal data flow; Step 202: Send a pre-detection instruction to the fourth node, which is used to instruct the fourth node to detect the connectivity in the direction from the fourth node to the fifth node according to the normal transmission path, and obtain the connectivity measurement index; Step 203: Receive the measurement index returned by the fourth node based on the pre-detection instruction; Step 204: When the measurement index does not match the baseline measurement index, generate an abnormal warning information.

[0063] Here, potential data transmission risks can be identified in advance even when no anomalies occur during the execution of computing tasks by the node cluster, enabling anomaly warning and risk avoidance, and reducing the impact of anomalies on the execution of computing tasks. Therefore, for step 201, when no anomalies occur during the execution of computing tasks by the node cluster, the normal transmission path of the normal data stream in the node cluster is obtained. Specifically, since the correspondence between the communication queue pairs and communication transmission paths (such as quintuple information) of each data stream is pre-recorded when the communication library of the node cluster initializes the data stream, the communication transmission path corresponding to the communication queue pairs of the normal data stream can be determined based on this correspondence, and thus the communication transmission path corresponding to the communication queue pairs of the normal data stream is used as the normal transmission path of the normal data stream. At the same time, the fourth node that sends the normal data stream and the fifth node that receives the normal data stream are also determined.

[0064] In step 202, a pre-detection command is sent to the fourth node. This command instructs the fourth node to detect connectivity from the fourth node to the fifth node along the normal transmission path. The pre-detection command is generated based on the normal transmission path. After receiving the pre-detection command, the fourth node sends a pre-detection data stream to the fifth node along the normal transmission path specified in the command to detect connectivity from the fourth node to the fifth node, thereby obtaining connectivity metrics during the detection process. For example, these metrics may include, but are not limited to, transmission latency, packet loss rate, and other indicators of communication quality. In practical applications, the pre-detection command can be periodically sent to the fourth node to achieve periodic pre-detection.

[0065] For step 203, the measurement metrics returned by the fourth node based on the pre-detection command are received. For step 204, the baseline measurement metrics are first obtained. These baseline measurement metrics can be pre-set, and there is a one-to-one correspondence between the baseline measurement metrics and the baseline measurement metrics. For example, the baseline measurement metrics could be: baseline latency less than or equal to 50μs, baseline packet loss rate 0, etc. The measurement metrics are matched with the baseline measurement metrics. If the measurement metrics do not match the baseline measurement metrics (e.g., latency in the measurement metrics is greater than the baseline latency, packet loss rate in the measurement metrics is greater than the baseline packet loss rate), an anomaly warning message is generated. In practical applications, an anomaly warning message is generated when the number of times the measurement metrics do not match the baseline measurement metrics exceeds a threshold.

[0066] By applying the above embodiments, potential data transmission risks can be identified in advance when no anomalies occur during the execution of computing tasks by the node cluster, thereby achieving anomaly warning and risk avoidance, reducing the impact of anomalies on the execution of computing tasks, and improving the stability of the node cluster in executing computing tasks.

[0067] Step 102: Send the first detection command to the first node.

[0068] The first detection instruction is used to instruct the first node to detect the connectivity in the direction from the first node to the second node according to the transmission path included in the first detection instruction, and to obtain the first detection result.

[0069] After obtaining the transmission path of the abnormal data stream, the node that transmitted the abnormal data stream is determined from multiple nodes: the first node that sent the abnormal data stream and the second node that received the abnormal data stream. Specifically, the first and second nodes can be determined based on the transmission path of the abnormal data stream. For example, if the transmission path is a 5-tuple, the first node can be located based on the source IP in the 5-tuple, and the second node can be located based on the destination IP in the 5-tuple.

[0070] Therefore, the data stream sent from the first node to the second node can be reconstructed and detected to determine the cause of the anomaly. At this time, a first detection instruction is generated, which may include the transmission path of the abnormal data stream. For step 102, the first detection instruction is sent to the first node. After receiving the first detection instruction, the first node detects the connectivity from the first node to the second node according to the transmission path included in the first detection instruction, obtaining a first detection result. Specifically, the first node can send a detection data stream to the second node according to the transmission path to detect the connectivity from the first node to the second node. When a response data stream is received from the second node based on the detection data stream, a first detection result indicating that the connectivity detection has passed is generated; when no response data stream is received from the second node based on the detection data stream, a first detection result indicating that the connectivity detection has failed is generated. The first node returns the first detection result to the device that sent the first detection instruction.

[0071] In practical applications, a detection command distribution platform can be set up in the application used for anomaly cause detection. The anomaly acquisition platform sends the transmission path of the anomaly data stream to the detection command distribution platform; the detection command distribution platform generates a first detection command based on the transmission path and sends the first detection command to the first node. In this way, the unified generation and distribution of the first detection command in the node cluster can be realized, improving the management efficiency and resource utilization of the first detection command, improving the stability and correctness of the generation and distribution of the first detection command, and ensuring the correctness of anomaly cause detection.

[0072] In some embodiments, there are multiple abnormal data streams. Before executing step 102, "send a first detection instruction to the first node", the following steps may also be performed: obtain the network link characteristics of each abnormal data stream; based on the network link characteristics of each abnormal data stream, cluster the multiple abnormal data streams to obtain at least one data stream cluster; for each data stream cluster, select a target abnormal data stream from the abnormal data streams included in the data stream cluster; based on this, executing step 102, "send a first detection instruction to the first node", can be achieved by executing the following steps: send a first detection instruction to each first node that sends the target abnormal data stream.

[0073] Here, we can first obtain the network link characteristics of each abnormal data stream. These network link characteristics can be understood as the features of the network links traversed by the abnormal data stream from the first node to the second node. For example, these characteristics may include those of the switch access layer, switch aggregation layer, switch core layer, and switch links involved in the network link. Then, based on the network link characteristics of each abnormal data stream, multiple abnormal data streams are clustered to obtain at least one data stream cluster, and each data stream cluster includes at least one abnormal data stream. Clustering can be performed using a clustering algorithm, such as K-means clustering, which is not limited here. Since clustering is based on network link characteristics, the network link characteristics of each data stream cluster are the same. Based on this, for each data stream cluster, an abnormal data stream is determined as the target abnormal data stream. This target abnormal data stream can be any abnormal data stream in the data stream cluster. By detecting the transmission connectivity of this target abnormal data stream, it is equivalent to detecting the transmission connectivity of each abnormal data stream in the data stream cluster. Thus, when sending the first detection command, the first detection command is only sent to the first node that sent the target abnormal data stream.

[0074] By applying the above embodiments, multiple abnormal data streams are clustered according to the network link characteristics of each abnormal data stream, so that the first detection instruction is sent only to one abnormal data stream in each data stream cluster. This can reduce the resource pressure of generating and issuing the first detection instruction, and since it focuses on the network link characteristics of the abnormal data stream, it can also quickly determine whether the cause of the abnormality is a network link abnormality based on the first detection result of the first detection instruction, thus realizing the rapid delineation of suspected network faults.

[0075] In some embodiments, there are multiple computing tasks and multiple abnormal data streams. Before executing step 102 "send the first detection instruction to the first node", the following steps may also be performed: For each abnormal data stream, the following processing is performed: From multiple computing tasks, determine the target computing task to which the abnormal data stream belongs; Based on the task identifier of the target computing task and the transmission path of the abnormal data stream, generate the first detection instruction for the abnormal data stream; Add the first detection instruction for the abnormal data stream to the instruction buffer queue. Based on this, executing step 102 "send the first detection instruction to the first node" can be achieved by performing the following steps: According to the order of issuance of each first detection instruction in the instruction buffer queue, send the first detection instruction to the first node indicated by the task identifier in the first detection instruction.

[0076] Here, since there are multiple computational tasks and multiple abnormal data streams, before sending the first detection instruction to the first node, the target computational task to which the abnormal data stream belongs is first determined from among the multiple computational tasks. Based on the task identifier of the target computational task and the transmission path of the abnormal data stream, a first detection instruction for the abnormal data stream is generated. In practical applications, the first detection instruction for the abnormal data stream can also be added to an instruction buffer queue. Then, according to the order in which each first detection instruction in the instruction buffer queue is issued, the first detection instruction is sent to the first node indicated by the task identifier in the first detection instruction. The issuance order can be the order in which the first detection instructions were added, or an order determined by the issuance priority; this is not limited here.

[0077] By applying the above embodiments, 1) by issuing the first detection instruction according to the task identifier of the computing task, the accuracy of the first detection instruction can be guaranteed, including that the first detection instruction corresponds one-to-one with the abnormal data stream and that the first detection instruction is aligned with the computing task; 2) by using the instruction buffer queue, the stability of issuing the first detection instruction is improved; thus, a stable foundation is provided for the subsequent first node to perform connectivity detection.

[0078] Step 103: Receive the first detection result returned by the first node based on the first detection command.

[0079] For step 103, after obtaining the first detection result, the first node returns the first detection result to the device that sent the first detection instruction. At this time, the device receives the first detection result returned by the first node based on the first detection instruction. The first detection result is used to characterize whether the connectivity detection is passed or the connectivity detection is not passed.

[0080] Step 104: Based on the first detection result, determine the cause of the abnormality.

[0081] For step 104, after obtaining the first detection result, the cause of the abnormality in the node cluster during the execution of the computing task is determined based on the information represented by the first detection result indicating whether the connectivity detection passed or failed.

[0082] In practical applications, an analysis platform can be set up within the application used for anomaly cause detection. This platform receives the initial detection results, analyzes them, and identifies the causes of the anomalies. This enables unified management and analysis of the initial detection results across the node cluster, improving management efficiency and analytical utilization, thereby enhancing the efficiency of anomaly cause detection.

[0083] In some embodiments, step 104, "determining the cause of the anomaly based on the first detection result," can be implemented by performing the following steps: when the first detection result indicates that the connectivity detection has passed, the cause of the anomaly is determined to be: at least one of the software of the first node and the second node is faulty, and the software is used to perform computational tasks; when the first detection result indicates that the connectivity detection has failed, a second detection instruction is sent to the first node, and a third detection instruction is sent to the second node; wherein, the second detection instruction is used to instruct the first node to detect the connectivity in the direction from the first node to the third node to obtain a second detection result, and the third detection instruction is used to instruct the second node to detect the connectivity in the direction from the second node to the third node to obtain a third detection result, and the network link between the third node and the first node is normal, and the network link between the third node and the second node is normal; the second detection result returned by the first node based on the second detection instruction is received, and the third detection result returned by the second node based on the second detection instruction is received; based on the second detection result and the third detection result, the cause of the anomaly is determined.

[0084] Here, the detection data stream used for connectivity detection is completely consistent with the abnormal data stream (specifically, the transmission path is completely consistent), that is, the detection scenario = the fault scenario; the device used for connectivity detection is independent of the first node and the second node, which can avoid software state interference, thus ensuring the accuracy of the determination of the cause of the abnormality.

[0085] When the first detection result indicates that the connectivity test passes, the cause of the anomaly is determined to be a software fault on the endpoint, meaning that at least one of the software in the first and second nodes is faulty. This is because the causes of the anomaly include software faults, hardware faults, and network link faults. When the connectivity test passes, it essentially eliminates hardware faults and network link faults by "detecting normal data flow transmission," leaving only software faults on the endpoint. The specific derivation is as follows: 1) If the connectivity test passes, hardware faults on the endpoint are excluded: The endpoint hardware (the source / destination end is the core communication hardware of the node, such as the network card, GPU communication module, and motherboard link) is the "physical carrier" for data flow transmission. If the hardware is faulty (such as a damaged network card or a loose motherboard link), then any data flow (including the detection data flow) cannot be transmitted normally. Therefore, passing the connectivity test means that the source hardware can send data flow normally and the destination hardware can receive data flow normally, so there is no fault in the endpoint hardware. 2) If connectivity test passes, network link failure is ruled out: The network link (LA / LC / SGLC switch, fiber optic link, switch port) is the "transmission channel" for data flow from the source to the destination. If there is a fault in the network link (such as LC switch port failure, fiber optic interruption, link congestion), the detection data flow will fail to transmit due to "path interruption / packet loss". A successful connectivity test means that the entire link from the source to the destination (LA→LC→SGLC→LC→LA) can forward the data flow normally, and there is no abnormality in the network link. 3) Only end-side software failure remains: Since both end-side hardware and network link have been ruled out, the cause of the anomaly is an end-side software failure. The core characteristic of an end-side software failure is that it does not affect the operation of independent connectivity testing, but it will cause the data flow of the computing task to be interrupted. This perfectly matches the scenario of "connectivity test passes but computing task is abnormal". At this time, it is quickly determined that the cause of the anomaly is not a network link failure.

[0086] When the first test result indicates that the connectivity test failed, it means that the cause of the anomaly is not an end-side software failure. This is because connectivity testing is independent of the execution of the computing task; therefore, end-side software failures do not affect connectivity testing. However, the failure of the connectivity test indicates that it is not an end-side software failure. End-side hardware failures are limited to the "source / destination node itself" (such as packet loss on the source network card or damage to the destination GPU communication module), which will cause "connectivity tests between this node and all other nodes to fail (i.e., connectivity tests fail)." Network link failures are limited to the "specific path from the source to the destination" (such as a port failure on the LC-05 switch), which will only cause "connectivity tests for that path to fail," but connectivity tests between the node and other nodes outside that path will still be normal (i.e., connectivity tests pass). In this case, the cause of the anomaly may be an end-side hardware failure or a network link failure. Therefore, it is necessary to use a third-party node to further determine whether the cause of the anomaly is an end-side hardware failure or a network link failure.

[0087] Specifically, a second detection command is sent to the first node, and a third detection command is sent to the second node. At this time, the first node detects the connectivity in the direction from the first node to the third node, obtaining a second detection result (indicating whether the connectivity detection passed or failed). The second node detects the connectivity in the direction from the second node to the third node, obtaining a third detection result (indicating whether the connectivity detection passed or failed). The detection process for connectivity in the direction from the first node to the third node and for the second node to detect connectivity in the direction from the second node to the third node are the same as the detection process for connectivity in the direction from the first node to the second node, and will not be repeated here. It should be noted that the network link between the third node and the first node is normal, and the network link between the third node and the second node is normal. This ensures the reliability of anomaly localization using the third node and eliminates interference.

[0088] After obtaining the second detection result returned by the first node based on the second detection command and the third detection result returned by the second node based on the second detection command, the cause of the anomaly is determined based on the second and third detection results.

[0089] By applying the above embodiments, based on whether the connectivity test represented by the first detection result passes, it is possible to determine whether the cause of the anomaly is a software failure on the end side. If it is a software failure on the end side, network link failure and hardware failure on the end side can be quickly ruled out. If it is not a software failure on the end side, the third node can be used to further determine whether the cause of the anomaly is a network link failure or a hardware failure on the end side, thereby improving the accuracy of locating the cause of the anomaly.

[0090] In some embodiments, "determining the cause of the anomaly based on the second and third detection results" can be achieved by performing the following steps: when both the second and third detection results indicate that the connectivity detection has passed, the cause of the anomaly is determined to be: an anomaly exists in the network link between the first node and the second node; when the second detection result indicates that the connectivity detection has failed and the third detection result indicates that the connectivity detection has passed, the cause of the anomaly is determined to be: an anomaly exists in the hardware of the first node; when the second detection result indicates that the connectivity detection has passed and the third detection result indicates that the connectivity detection has failed, the cause of the anomaly is determined to be: an anomaly exists in the hardware of the second node; when both the second and third detection results indicate that the connectivity detection has failed, the cause of the anomaly is determined to be: an anomaly exists in the hardware of both the first node and the second node.

[0091] Here, the specific cause of the anomaly is determined from both end-side hardware failure and network link failure. When both the second and third detection results indicate that the connectivity test has passed, it means that the connectivity from the first node to the third node is normal, and the connectivity from the second node to the third node is also normal. As the above analysis shows, since both the second and third detection results indicate that the connectivity test has passed, the possibility of end-side hardware failure in the first and second nodes is ruled out. Therefore, the cause of the anomaly is an anomaly (i.e., network link failure) in the network link between the first and second nodes.

[0092] When the second detection result indicates that the connectivity test has failed, and the third detection result indicates that the connectivity test has passed, as can be seen from the above analysis, the possibility of a hardware failure on the second node is ruled out. Furthermore, since the network link between the third node and the second node is normal, the possibility of a network link failure on the second node is also ruled out. Therefore, the second node is normal. When the second detection result indicates that the connectivity test has failed, and since the network link between the third node and the first node is normal, combined with the above analysis, it can be concluded that the hardware of the first node is abnormal (i.e., a hardware failure on the end).

[0093] When the second detection result indicates that the connectivity test has passed and the third detection result indicates that the connectivity test has failed, as can be seen from the above analysis, the possibility of a hardware failure on the first node is ruled out, and since the network link between the third node and the first node is normal, the possibility of a network link failure on the first node is also ruled out. Therefore, the first node is normal. The third detection result indicates that the connectivity test has failed, and since the network link between the third node and the second node is normal, combined with the above analysis, it can be concluded that the hardware of the second node is abnormal (i.e., a hardware failure on the first node).

[0094] When the second detection result indicates that the connectivity test has failed, and the third detection result indicates that the connectivity test has failed, the above analysis shows that the cause of the anomaly is that the hardware of both the first node and the second node is abnormal (i.e., the end-side hardware is faulty).

[0095] By applying the above embodiments and with the help of a third node, the cause of the anomaly was further located, thus improving the accuracy of the anomaly cause location.

[0096] In some embodiments, when the first detection result indicates that the connectivity detection has passed, the cause of the anomaly is determined to be: at least one of the first node and the second node has a software anomaly. The following steps can be performed to further distinguish the node with the software anomaly: when the first detection result indicates that the connectivity detection has passed, an anomaly log of the abnormal data stream is obtained; when the anomaly log belongs to the first node, the cause of the anomaly is determined to be: the software of the first node has an anomaly; when the anomaly log belongs to the second node, the cause of the anomaly is determined to be: the software of the second node has an anomaly; when the anomaly log includes a first anomaly log of the first node and a second anomaly log of the second node, the cause of the anomaly is determined to be: at least one of the first node and the second node has a software anomaly.

[0097] Here, when it is determined that at least one of the first and second nodes has a software anomaly, the specific node with the software anomaly can be further identified. Specifically, when the first detection result indicates that the connectivity detection has passed, the anomaly log of the abnormal data stream is acquired. If the anomaly log belongs to the first node, it indicates that the software of the first node has an anomaly, because anomaly log is generated when the software of the first node has an anomaly. If the anomaly log belongs to the second node, it indicates that the software of the second node has an anomaly, because anomaly log is generated when the software of the second node has an anomaly. If the anomaly log includes the first anomaly log of the first node and the second anomaly log of the second node, then the software of the first node and / or the second node has an anomaly. In practical applications, it is also possible to confirm whether the target node (at least one of the first and second nodes) has a software anomaly. For example, if the anomaly log of the target node includes anomaly records that occurred during the execution of the computation task, then the software anomaly of the target node is further verified, thereby improving the accuracy of software anomaly location.

[0098] By applying the above embodiments, the abnormal logs based on the abnormal data stream can further identify the specific nodes where software abnormalities exist, thereby improving the accuracy of locating the cause of the abnormality.

[0099] In some embodiments, when the exception log includes a first exception log of a first node and a second exception log of a second node, the cause of the exception is determined to be: at least one of the first node and the second node has a software exception. The following steps can be performed to further distinguish the node with the software exception: When the exception log includes a first exception log of a first node and a second exception log of a second node, the following processing is performed: when the first generation time of the first exception log is earlier than the second generation time of the second exception log, the cause of the exception is determined to be: the software of the first node has an exception; when the first generation time of the first exception log is later than the second generation time of the second exception log, the cause of the exception is determined to be: the software of the second node has an exception; when the first generation time of the first exception log is equal to the second generation time of the second exception log, the cause of the exception is determined to be: the software of both the first node and the second node has an exception.

[0100] Here, if the specific node with the software anomaly cannot be identified based on the anomaly logs, the first generation time of the first anomaly log and the second generation time of the second anomaly log can be obtained and compared to identify the specific node with the software anomaly. Specifically, if the first generation time is earlier than the second generation time, it indicates that the first node experienced the anomaly first, and the second node generated the second anomaly log under the influence of the first node; therefore, the software of the first node is considered to have an anomaly. If the first generation time is later than the second generation time, it indicates that the second node experienced the anomaly first, and the first node generated the first anomaly log under the influence of the second node; therefore, the software of the second node is considered to have an anomaly. When the first generation time of the first anomaly log is equal to the second generation time of the second anomaly log, it indicates that the software of both the first and second nodes has an anomaly.

[0101] By applying the above embodiments, even when both the first and second nodes generate abnormal logs, the node with the specific software abnormality can be identified, thus improving the accuracy of locating the cause of the abnormality.

[0102] In some embodiments, after determining that the cause of the anomaly is that the software of at least one of the first node and the second node is abnormal, the following steps may also be performed: obtaining the abnormal node log of the target node, wherein the target node is at least one of the first node and the second node; identifying the target error code in the abnormal node log; obtaining the correspondence between multiple error codes and multiple software anomaly causes, wherein the error codes and software anomaly causes correspond one-to-one; and determining the target software anomaly cause corresponding to the target error code based on the correspondence, and taking the target software anomaly cause as the reason for the software anomaly of the target node.

[0103] Here, after determining that the cause of the anomaly is a software anomaly in the target node (at least one of the first and second nodes), further identification of the cause of the software anomaly is possible. Specifically, firstly, the anomaly node log of the target node is obtained. This log may record error codes, such as communication library error codes NCCL_ERR_CONN_TIMEOUT, NCCL_ERR_QP_CREATE, RDMA driver error code IBV_ERR_PORT_DOWN, etc. Then, the correspondence between multiple error codes and multiple software anomaly causes is obtained. There is a one-to-one correspondence between the error codes and the software anomaly causes. This correspondence may be pre-established and can be dynamically updated. For example, if new error codes and corresponding new software anomaly causes exist, the new error codes and corresponding new software anomaly causes can be added to expand the coverage of error codes and software anomaly causes. For example, the software exception cause corresponding to the error code "NCCL_ERR_CONN_TIMEOUT" is "software process memory leak, causing communication thread blocking"; the software exception cause corresponding to the error code "NCCL_ERR_QP_CREATE" is "incompatible communication library (such as NCCL) version"; and the software exception cause corresponding to the error code "IBV_ERR_PORT_DOWN" is "incorrect RDMA driver parameter configuration (port not enabled)". Based on this, the target software exception cause corresponding to the target error code can be determined based on the correspondence, and the target software exception cause can be used as the reason for the software exception on the target node. After identifying the software exception cause, a notification message of the software exception cause can be output, and repair suggestions for the software exception can be pushed (such as upgrading the communication library (such as NCCL) version to the target version, etc.).

[0104] By applying the above embodiments, the correspondence between multiple error codes and multiple software anomaly causes can be used to achieve automated analysis of software anomaly causes, and the location of software anomalies can be penetrated from the "end-side level" to the "code / configuration level", which improves the accuracy of anomaly cause location and lowers the threshold of network operation and maintenance.

[0105] In some embodiments, after determining the cause of the anomaly, the following steps may also be performed: when the cause of the anomaly is an anomaly in the network link between the first node and the second node, the network link between the first node and the second node is switched to the target network link; when the cause of the anomaly is an anomaly in the software of the target node, the target node is restarted, and the target node is at least one of the first node and the second node; when the cause of the anomaly is an anomaly in the hardware of the target node, the target node is removed, and the sixth node is invoked to execute the target computing task on the target node, the computing task including the target computing task.

[0106] Here, after determining the cause of the anomaly, corresponding handling methods can be provided to automate the handling of anomalies in the node cluster. Specifically, (1) when the cause of the anomaly is an anomaly in the network link between the first node and the second node (such as the LA-01→LC-02 link), the network link between the first node and the second node can be switched to the target network link (LA-01→LC-03 backup link), thereby realizing the transmission of data flow between the first node and the second node through the target network link. At the same time, network operation and maintenance notification messages can also be output to notify network operation and maintenance personnel to handle the abnormal network link in a timely manner. (2) when the cause of the anomaly is an anomaly in the software of the target node (at least one of the first node and the second node), the target node can be restarted, which can specifically include: automatically restarting the processes and communication libraries involved in the execution of computing tasks by the target node; if the target node fails to restart, the "process restart script" can also be called to restore the communication service between the first node and the second node. (3) When the cause of the anomaly is a hardware malfunction of the target node (at least one of the first and second nodes), the target node can be removed, for example, by marking it as unavailable to remove it from the node cluster. Simultaneously, the sixth node is invoked to execute the target computation task on the target node (a portion of the computation task). In practical applications, after the anomaly is automatically processed, the transmitted data stream can be called the repaired data stream. Therefore, the connectivity of the repaired data stream can be re-checked. If the check passes, the computation task can be continued. If the check fails, the automated anomaly processing can be repeated. If the number of failed checks exceeds a threshold, an alarm is generated to notify network maintenance personnel to handle the issue promptly.

[0107] By applying the above embodiments, after determining the cause of the anomaly, the anomalies occurring in the node cluster can be handled automatically, thereby improving the speed and efficiency of anomaly repair and perfecting the anomaly handling process.

[0108] Applying the above embodiments of this application, when an anomaly occurs in the node cluster during the execution of a computing task, the transmission path of the abnormal data stream in the node cluster is obtained, and a first detection instruction is sent to the first node (i.e., the sender of the abnormal data stream). This causes the first node to detect the connectivity from the first node to the second node (i.e., the receiver of the abnormal data stream) according to the transmission path included in the first detection instruction, obtaining a first detection result. Upon receiving the first detection result, the cause of the anomaly is determined based on the first detection result. Here, by reconstructing the transmission path of the abnormal data stream, connectivity detection from the first node to the second node is achieved, thereby determining the cause of the anomaly based on the obtained first detection result. Thus, 1) the reliability of connectivity detection is guaranteed, thereby improving the detection accuracy of the cause of the anomaly; and only the nodes involved in the abnormal data stream need to be detected, which improves the detection efficiency of the cause of the anomaly and reduces the detection cost.

[0109] In some examples, this application can be applied to the field of cloud technology. For instance, each node in a node cluster can be a cloud server, and multiple nodes are used to collaboratively execute computing tasks (such as model training tasks, cloud computing tasks, etc.). When an anomaly occurs in the node cluster during the execution of a computing task, the transmission path of the abnormal data stream in the node cluster is obtained, and a first detection instruction is sent to the first node (i.e., the sender of the abnormal data stream). This causes the first node to detect the connectivity in the direction from the first node to the second node (i.e., the receiver of the abnormal data stream) according to the transmission path included in the first detection instruction, and obtain a first detection result. After receiving the first detection result, the cause of the anomaly is determined based on the first detection result. Here, by restoring the transmission of the abnormal data stream according to the transmission path of the abnormal data stream, the connectivity detection in the direction from the first node to the second node is realized, and the cause of the anomaly is determined based on the obtained first detection result. Thus, 1) the reliability of connectivity detection is guaranteed, thereby improving the detection accuracy of the cause of anomalies in cloud services; and only the nodes involved in the abnormal data stream need to be detected, which can improve the detection efficiency of the cause of anomalies in cloud services and reduce the detection cost.

[0110] The following describes the anomaly detection method provided in the embodiments of this application. As mentioned above, the anomaly detection method provided in the embodiments of this application is implemented by an electronic device, such as a server or terminal alone, or a server and terminal working together. Therefore, the executing entity of each step will not be described again below. See Figure 3C , Figure 3C This is a third flowchart illustrating the anomaly cause detection method provided in this application embodiment. The anomaly cause detection method provided in this application embodiment includes: Step 301: The first node receives the first detection command.

[0111] The first detection instruction is sent when an anomaly occurs in the node cluster during the execution of the computing task. The first detection instruction includes the transmission path of the abnormal data stream in the node cluster. The node cluster includes multiple nodes that cooperate to execute the computing task. The multiple nodes include a first node that sends the abnormal data stream and a second node that receives the abnormal data stream.

[0112] Here, when an anomaly occurs in the node cluster during the execution of a computation task, the abnormal data stream within the node cluster can be identified, thereby obtaining the transmission path of the abnormal data stream. This transmission path can be a 5-tuple of information, which may include source IP address, destination IP address, source port, destination port, and transmission protocol. The generation of abnormal data streams is related to anomalies. Computation tasks rely on multiple nodes communicating over a network, and nodes need to continuously transmit data (such as model parameters and intermediate calculation results in model training tasks). These data transmissions form a stable "communication stream (i.e., data stream)," and each data stream is uniquely identified by the 5-tuple of information to ensure accurate data exchange. Because an anomaly occurs in the node cluster during the execution of a computation task, the normal transmission of the data stream is disrupted, thus this data stream becomes an abnormal data stream, and the transmission path of the abnormal data stream is obtained at this point.

[0113] After obtaining the transmission path of the abnormal data stream, the node transmitting the abnormal data stream is determined from multiple nodes: the first node sending the abnormal data stream and the second node receiving the abnormal data stream. Specifically, the first and second nodes can be determined based on the transmission path of the abnormal data stream. For example, if the transmission path is a 5-tuple, the source IP in the 5-tuple can be used to locate the first node, and the destination IP in the 5-tuple can be used to locate the second node. Therefore, the data stream sent from the first node to the second node can be reconstructed and detected to determine the cause of the anomaly. At this time, a first detection command is generated, which may include the transmission path of the abnormal data stream; and the first detection command is sent to the first node. The first node receives the first detection command.

[0114] Step 302: In response to the first detection command, detect the connectivity in the direction from the first node to the second node according to the transmission path, and obtain the first detection result.

[0115] Here, after receiving the first detection command, the first node detects the connectivity in the direction from the first node to the second node according to the transmission path included in the first detection command, and obtains a first detection result. Specifically, the first node can send a detection data stream to the second node according to the transmission path, so as to use the detection data stream to detect the connectivity in the direction from the first node to the second node. When a response data stream based on the detection data stream is received from the second node, a first detection result indicating that the connectivity detection has passed is generated; when no response data stream based on the detection data stream is received from the second node, a first detection result indicating that the connectivity detection has failed is generated.

[0116] In some embodiments, see Figure 3D Step 302, "In response to the first detection command, detect the connectivity in the direction from the first node to the second node according to the transmission path, and obtain the first detection result," can be achieved by executing the following steps 3021-3025: Step 3021, In response to the first detection command, establish a communication connection between the first node and the second node according to the source address and destination address in the transmission path, and obtain the source port number and destination port number of the communication connection; Step 3022, Obtain the target source port number in the transmission path, and modify the source port number to the target source port number; Step 3023, Send a detection data stream to the destination port number of the second node according to the target source port number. The detection data stream is used to detect the connectivity in the direction from the first node to the second node; Step 3024, When a response data stream is received from the second node based on the detection data stream, generate a first detection result indicating that the connectivity detection has passed; Step 3025, When no response data stream is received from the second node based on the detection data stream, generate a first detection result indicating that the connectivity detection has failed.

[0117] In step 3021, in response to the first detection command, a communication connection is established between the first node and the second node according to the source address (i.e., source IP address) and destination address (i.e., destination IP address) in the transmission path, and the source port number and destination port number of the communication connection are obtained. Thus, the first node can send a data stream to the second node.

[0118] For step 3022, since the source port number is randomly assigned when establishing a communication connection, the target source port number in the transmission path is obtained, and the source port number is modified to the target source port number. This ensures that the transmission path of the detected data stream and the transmission path of the abnormal data stream are completely consistent, improving the reliability of connectivity detection. It should be noted that the destination port number here is fixed; however, if the destination port number is also randomly assigned, then the assigned destination port number also needs to be modified to the destination port number in the transmission path.

[0119] For step 3023, a detection data stream is sent to the destination port number of the second node according to the target source port number to detect the connectivity in the direction from the first node to the second node. When sending the detection data stream, the detection parameters (such as the number of data packets in the detection data stream, the sending frequency of the detection data stream, etc.) can be determined according to the communication type of the abnormal data stream (such as long frame high frequency type, short frame low frequency type, short frame medium frequency type, etc.), so that the detection data stream is sent based on the detection parameters.

[0120] For step 3024, when a response data stream based on the detection data stream is received from the second node, a first detection result indicating that the connectivity detection has passed is generated. For step 3025, when a response data stream based on the detection data stream is not received from the second node, a first detection result indicating that the connectivity detection has failed is generated.

[0121] By applying the above embodiments, by sending the detection data stream according to the transmission path, it can be ensured that the transmission path of the detection data stream and the transmission path of the abnormal data stream are completely consistent, thereby ensuring the reliability of connectivity detection. Furthermore, the cause of the anomaly can be quickly located through the first detection result of connectivity detection.

[0122] Step 303: Send the first test result.

[0123] The first test result is used to determine the cause of the abnormality.

[0124] Here, the first node returns the first detection result to the device that sent the first detection command. After receiving the first detection result, the device that sent the first detection command determines the cause of the anomaly that occurred during the execution of the computing task by the node cluster based on the information represented by the first detection result indicating whether the connectivity test passed or failed.

[0125] In practical applications, a detection tool can also be set at the first node. This detection tool is used to perform steps 301-303 to achieve connectivity detection and obtain the first detection result.

[0126] Applying the above embodiments of this application, when an anomaly occurs in the node cluster during the execution of a computing task, the transmission path of the abnormal data stream in the node cluster is obtained, and a first detection instruction is sent to the first node (i.e., the sender of the abnormal data stream). This causes the first node to detect the connectivity from the first node to the second node (i.e., the receiver of the abnormal data stream) according to the transmission path included in the first detection instruction, obtaining a first detection result. Upon receiving the first detection result, the cause of the anomaly is determined based on the first detection result. Here, by reconstructing the transmission path of the abnormal data stream, connectivity detection from the first node to the second node is achieved, thereby determining the cause of the anomaly based on the obtained first detection result. Thus, 1) the reliability of connectivity detection is guaranteed, thereby improving the detection accuracy of the cause of the anomaly; and only the nodes involved in the abnormal data stream need to be detected, which improves the detection efficiency of the cause of the anomaly and reduces the detection cost.

[0127] The following example illustrates an exemplary application of the embodiments of this application in a real-world application scenario, using the computational task of training a machine learning model (such as a large model).

[0128] In related technologies, when a training task is interrupted (i.e., an anomaly occurs during the execution of the aforementioned computational task), the node information on the training log is used to identify potentially abnormal nodes. The root cause of the failure is determined by performing a full-link investigation on all nodes (including software, hardware, and network layers). Because large models have strong synchronization characteristics, a single failure can be rapidly transmitted through the network, often requiring simultaneous investigation of many nodes. Maintenance personnel from different modules such as the framework, servers, network, storage, and computing power need to collaborate synchronously. In reality, the entire training cluster involves many software and hardware components, many of which are black boxes, making investigation very difficult and requiring significant manpower and time for layer-by-layer investigation until the root cause is finally identified. Therefore, related technologies have the following problems: 1) Some training tasks are very large, potentially involving thousands or even tens of thousands of GPUs. Many modules may report anomalies simultaneously, leading to numerous suspects. Quickly locating the root cause requires the simultaneous intervention of many maintenance personnel, consuming a significant amount of time. 2) In GPU training clusters, a large amount of software / hardware is black-box, such as GPUs, commercial network cards, CUDA, etc. Troubleshooting can only be done through explicit alarm logs. If a silent fault is encountered, it is even more difficult to find a solution, further prolonging the time for delineation and localization. 3) Since training nodes are connected via a network, when training is interrupted, there are usually network-related abnormal information. Existing localization methods cannot quickly determine whether it is a suspected network problem. 4) Traditional network delineation methods are based on ping mesh using the TCP protocol. However, the data stream in the training scenario is an RDMA stream. Therefore, existing network probing schemes cannot effectively delineate the training scenario.

[0129] During the training of large models, which typically involves thousands or even tens of thousands of GPUs and last for tens of days or even months, training interruptions are common due to the various devices involved, such as GPUs, CPUs, servers, storage, and networks. Since these training nodes are connected by a network, training interruptions are often accompanied by communication disruptions, generating abnormal flow alarms or network-related error logs in the training logs. This means that almost all training interruption-related failures are suspected to be network faults. Network communication mainly consists of three parts: client-side software, client-side hardware, and the network link. Therefore, the root causes of this problem can be divided into three categories: first, upper-layer business software issues; second, server physical hardware issues; and third, network link anomalies. Due to the synchronous nature of large model training, faults can propagate to many nodes in a very short time. Because they superficially appear as suspected network faults, troubleshooting often begins with the network. If the root cause is not network-related, the troubleshooting direction is incorrect, significantly increasing the recovery time for training failures.

[0130] Based on this, this application provides a rapid delimitation method for suspected network faults in large model training interruption scenarios. By adding 5-tuple information of communication connections (i.e., communication data streams) to the collection communication library, when training is interrupted, the 5-tuple information of all affected abnormal streams is collected. Then, the Netquint Rping tool is used to reconstruct and probe the abnormal streams (i.e., based on the 5-tuple information of the abnormal streams, the transmission between the sender and receiver of the abnormal stream is probed to see if it is normal). If the abnormal stream reconstruction test is normal (i.e., the probe stream transmission is normal), it indicates that the problem is unrelated to the network and can be quickly delimited as a first-type software problem. If the abnormal stream reconstruction test is abnormal (i.e., the probe stream transmission is abnormal), it indicates that the training interruption is suspected to be caused by a network anomaly, but further confirmation is needed to determine whether it is a second-type or third-type problem. Based on this, a normal third-party node is introduced to form a one-to-one Netquint Rping test. If the new test is normal (i.e., the third-party node transmits normally to both the sender and receiver of the abnormal flow), since the paths do not intersect, it can be judged that the network data path is abnormal, which is classified as a network problem, i.e., the third type, and the network side needs to locate the data path problem. If there are abnormal test cases in the two tests (i.e., the transmission between the third-party node and the sender and / or receiver of the abnormal flow is abnormal), it indicates that the problem is caused by the end-side hardware, i.e., the second type, which is classified as a server hardware problem.

[0131] It should be noted that the main focus of this application is that it eliminates the need to restart the training task. When training is interrupted, it can obtain all possible abnormal nodes and network links by using the abnormal five-tuple information in the communication library, and simultaneously send out the Netquint Rping probe stream (i.e., the detection data stream mentioned above). Then, based on the probe results (i.e., the first detection result mentioned above), it can quickly determine whether the problem is a software, hardware, or network issue, greatly reducing the manpower required for operation and maintenance and shortening the troubleshooting time. In practical applications, this application is mainly used in abnormal scenarios where AI large model training is interrupted. This method can quickly delineate faults during the training process, accurately distinguish whether they are software, hardware, or network problems, improve fault location efficiency, and save training costs.

[0132] like Figure 4 As shown, during large-scale training, when a training interruption occurs, the relevant technologies first investigate suspicious nodes. Specifically, this involves a full-link node investigation of the affected modules (including the training framework, network, storage, GPU, computing power, etc.), collecting suspicious information, and analyzing and summarizing the suspicious nodes based on this information. Then, training resumes by replacing the faulty nodes. At this point, only a large number of suspicious nodes have been identified; the root cause still needs to be located. If only faulty nodes are identified without pinpointing the root cause, these nodes cannot be reused. Current GPU resources are relatively limited. If the real problem nodes cannot be found quickly, and the root cause cannot be located and fixed, the available GPU backup resources will soon be exhausted, resulting in a significant waste of GPU resources.

[0133] To address the aforementioned issues, this application adds 5-tuple information to all data streams in the aggregated communication library. When training is interrupted, communication between these nodes is also interrupted, resulting in abnormal streams affected by the interruption. The communication library records the 5-tuple information of the affected abnormal streams. Since data streams in large model training are RDMA traffic, a feasible solution is to determine whether the problem is network-related by constructing RDMA streams based on the recorded information. However, traditional RDMA probing does not support custom 5-tuples and cannot fully reconstruct the path during service failures. Therefore, this application designs and develops the Netquint Rping tool, which allows for freely setting the 5-tuple information of streams to reconstruct abnormal streams.

[0134] like Figure 5As shown, an abnormal flow occurred between node A and node B. Therefore, in the first step, abnormal flow detection was performed on nodes A and B to obtain the detection results. Then, the detection results were analyzed. If all detection flows were transmitted normally (i.e., the detection was normal), it meant that the abnormal flow was not caused by the network or hardware (because the detection flow completely simulates the five-tuple information of the abnormal flow; if there is a problem with the hardware or network, then the detection flow must be problematic). In this case, the abnormal flow was caused by software (such as a process hanging causing data flow interruption). In this way, it is possible to quickly determine whether the abnormal flow is caused by a problem with the end-side software. Continuing, based on the first step, since the detection flow was transmitted abnormally (i.e., the detection was abnormal), it means that there must be an abnormal part in the components of the transmitted detection flow (i.e., node A, node B, and network path). At this time, a third-party normal node C was introduced to assist in the judgment. Specifically, construct probe flows from node A to node C and from node B to node C. If both newly added probe flows are normal, it means that the abnormal flow is unrelated to the node (i.e., unrelated to the hardware), and the cause of the abnormality is determined to be a network link (path) problem; otherwise, it is a node abnormality. The node with the abnormal probe flow has an end-side hardware problem.

[0135] To achieve the above objectives, embodiments of this application provide a fast delimitation scheme for suspected network faults when large model training is interrupted, see [link to relevant documentation]. Figure 6 For training clusters, the fault location system provided in this application mainly includes: the Netquint Rping detection tool (i.e., the aforementioned detection tool), the abnormal flow 5-tuple collection platform (i.e., the aforementioned abnormal collection platform), the detection task distribution platform (i.e., the aforementioned detection command distribution platform), and the analysis platform. The following will combine... Figure 6 Each part of the document will be explained.

[0136] Step S1: As Figure 7 As shown (for) Figure 6 The abnormal flow 5-tuple acquisition platform shown here, when a training cluster experiences a training interruption, communication tasks still exist within the training process. The interruption prevents these communication tasks from completing. Since these communication tasks generate ERRCQE logs due to timeouts, and ERRCQE is bound to queue pairs (QPs), the 5-tuple information of the QPs (including source IP, destination IP, source port, destination port, and protocol) can be recorded in the communication library. Therefore, the 5-tuple information of the abnormal flows of the affected communication tasks can be obtained. Because more than one communication node may be affected, multiple GPU nodes may experience abnormal flows. Therefore, an abnormal flow 5-tuple acquisition platform is designed on the edge side to detect and collect the 5-tuple information of abnormal flows. After a training interruption occurs and abnormal flows are generated, the abnormal flow 5-tuple acquisition platform collects the 5-tuple information of all abnormal flows and sends it to the detection task distribution platform.

[0137] Step S2: As Figure 8 As shown (for) Figure 6 The probe task distribution platform shown receives the five-tuple information of these abnormal streams, classifies them according to the dimensions of the training tasks, generates corresponding probe tasks, and distributes the probe tasks to the probe task queue, thereby distributing the probe tasks from the probe task queue to the GPU cluster.

[0138] Step S3: The probe task distribution platform uses the source IP and destination IP from the five-tuple information of the abnormal flow to locate the GPU server to which the probe task needs to be distributed, and then distributes it to the Netquint Rping probe tool on the GPU server. For example... Figure 9 As shown (for) Figure 6 The Netquint Rping probing tool (shown) executes corresponding probing tasks based on probing commands. The Netquint Rping probing tool includes a Netquint Rping control plane and a Netquint Rping execution plane. The execution plane of the client GPU and the execution plane of the server GPU each create queue pairs (QPs) and establish a link between them. The client GPU obtains the 5-tuple information of the abnormal flow. To ensure complete consistency between the 5-tuple information of the probing flow and the abnormal flow, it modifies the source port number of the QP using the `modify_src_port` interface to match the source port number in the abnormal flow's 5-tuple information. Then, based on the 5-tuple information, it sends the probing flow to the server GPU and generates probing results based on whether a response flow is received. Finally, the probing results are returned to the analysis platform.

[0139] Step S4: The analysis platform obtains the detection results and decides whether to continue detection based on the results. If the S3 detection is successful, it means that the end-side hardware and network links involved in the detection flow in S3 are normal, and the root cause of the training interruption is the end-side software; If the probe fails, it indicates that the training interruption is not caused by the edge software. The problem could be with the edge hardware or the network link. To further confirm the cause, a third-party node C is introduced for secondary probes. Probe streams are sent between node A and node C, and between node B and node C. To ensure the effectiveness of the probes, the newly added node C must be a normal node, and the network paths between node A and node C, and between node B and node C, must be normal.

[0140] like Figure 10As shown in (1), when the detection of the newly added node C and the suspected faulty nodes A and B are all normal, it indicates that the end-side hardware of nodes A and B are normal, the root cause of the problem is the network link between nodes A and B, and the fault is determined to be a network link problem. Figure 10 As shown in (2), when the newly added node C detects normally with one of the nodes (e.g., node A) but abnormally with another node (e.g., node B), it indicates a hardware failure on the end side of node B, and the fault is defined as a hardware problem on the end side of node B; Figure 10 As shown in (3), when all newly added probe streams are abnormal, it indicates that the end-side hardware of both node A and node B is abnormal.

[0141] By applying the above embodiments of this application, suspected network faults can be quickly delimited, reducing the delimitation time. In actual implementation, after the ERCQE log appears during training interruption, the delimitation time for suspected network faults is shortened from an average of 12 hours to less than 1 minute.

[0142] The following description continues to illustrate the exemplary structure of the anomaly detection device 555 provided in the embodiments of this application as a software module. In some embodiments, such as... Figure 2 As shown, the software modules in the anomaly cause detection device 555 stored in the memory 550 may include: an acquisition module 5551, used to acquire the transmission path of the abnormal data stream in the node cluster when an anomaly occurs during the execution of a computing task; wherein, the node cluster includes multiple nodes cooperating in executing the computing task, the multiple nodes including: a first node sending the abnormal data stream and a second node receiving the abnormal data stream; a first sending module 5552, used to send a first detection instruction to the first node, the first detection instruction being used to instruct the first node to detect the connectivity in the direction from the first node to the second node according to the transmission path included in the first detection instruction, and obtain a first detection result; a first receiving module 5553, used to receive the first detection result returned by the first node based on the first detection instruction; and a determination module 5554, used to determine the anomaly cause of the anomaly based on the first detection result.

[0143] In some embodiments, the acquisition module 5551 is further configured to acquire abnormal logs of abnormal data streams in the node cluster and determine abnormal communication queue pairs bound to the abnormal logs; acquire the correspondence between multiple communication queue pairs and multiple communication transmission paths, wherein the communication queue pairs and the communication transmission paths correspond one-to-one; based on the correspondence, determine the target communication transmission path corresponding to the abnormal communication queue pair, and use the target communication transmission path as the transmission path of the abnormal data stream.

[0144] In some embodiments, the determining module 5554 is further configured to: when the first detection result indicates that the connectivity detection has passed, determine that the cause of the anomaly is: at least one of the software of the first node and the second node is abnormal, the software being used to execute the computation task; when the first detection result indicates that the connectivity detection has failed, send a second detection instruction to the first node and a third detection instruction to the second node; wherein the second detection instruction is configured to instruct the first node to detect the connectivity in the direction from the first node to the third node to obtain a second detection result, and the third detection instruction is configured to instruct the second node to detect the connectivity in the direction from the second node to the third node to obtain a third detection result, wherein the network link between the third node and the first node is normal, and the network link between the third node and the second node is normal; receive the second detection result returned by the first node based on the second detection instruction, and receive the third detection result returned by the second node based on the third detection instruction; and determine the cause of the anomaly based on the second detection result and the third detection result.

[0145] In some embodiments, the determining module 5554 is further configured to: determine that the cause of the anomaly is an anomaly in the network link between the first node and the second node when both the second detection result and the third detection result indicate that the connectivity detection has passed; determine that the cause of the anomaly is an anomaly in the hardware of the first node when the second detection result indicates that the connectivity detection has failed and the third detection result indicates that the connectivity detection has passed; determine that the cause of the anomaly is an anomaly in the hardware of the second node when the second detection result indicates that the connectivity detection has passed and the third detection result indicates that the connectivity detection has failed; and determine that the cause of the anomaly is an anomaly in the hardware of both the first node and the second node when both the second detection result indicates that the connectivity detection has failed and the third detection result indicates that the connectivity detection has failed.

[0146] In some embodiments, the determining module 5554 is further configured to: acquire an anomaly log of the abnormal data stream when the first detection result indicates that the connectivity detection has passed; determine the cause of the anomaly when the anomaly log belongs to the first node: the software of the first node is abnormal; determine the cause of the anomaly when the anomaly log belongs to the second node: the software of the second node is abnormal; and determine the cause of the anomaly when the anomaly log includes a first anomaly log of the first node and a second anomaly log of the second node: at least one of the software of the first node and the second node is abnormal.

[0147] In some embodiments, the determining module 5554 is further configured to perform the following processing when the exception log includes a first exception log of the first node and a second exception log of the second node: when the first generation time of the first exception log is earlier than the second generation time of the second exception log, determine that the exception cause is: the software of the first node has an exception; when the first generation time of the first exception log is later than the second generation time of the second exception log, determine that the exception cause is: the software of the second node has an exception; when the first generation time of the first exception log is equal to the second generation time of the second exception log, determine that the exception cause is: the software of both the first node and the second node has an exception.

[0148] In some embodiments, the determining module 5554 is further configured to: after determining that the cause of the anomaly is that at least one of the first node and the second node has a software anomaly, obtain the anomaly node log of the target node, wherein the target node is at least one of the first node and the second node; identify the target error code in the anomaly node log; obtain the correspondence between multiple error codes and multiple software anomaly causes, wherein the error codes and the software anomaly causes correspond one-to-one; determine the target software anomaly cause corresponding to the target error code based on the correspondence, and use the target software anomaly cause as the reason for the software anomaly of the target node.

[0149] In some embodiments, the acquisition module 5551 is further configured to: acquire the normal transmission path of the normal data stream in the node cluster when no abnormality occurs during the execution of the computing task; the plurality of nodes include a fourth node that sends the normal data stream and a fifth node that receives the normal data stream; send a pre-detection instruction to the fourth node, the pre-detection instruction being configured to instruct the fourth node to detect the connectivity in the direction from the fourth node to the fifth node according to the normal transmission path, and acquire a connectivity measurement index; receive the measurement index returned by the fourth node based on the pre-detection instruction; and generate an abnormal warning information when the measurement index does not match the baseline measurement index.

[0150] In some embodiments, the determining module 5554 is further configured to: switch the network link between the first node and the second node to a target network link when the cause of the anomaly is an anomaly in the network link between the first node and the second node; restart the target node when the cause of the anomaly is a software anomaly in the target node, wherein the target node is at least one of the first node and the second node; and remove the target node and invoke a sixth node to execute a target computing task on the target node when the cause of the anomaly is a hardware anomaly in the target node, wherein the computing task includes the target computing task.

[0151] In some embodiments, the number of abnormal data streams is multiple; the first sending module 5552 is further configured to, before sending the first detection instruction to the first node, acquire the network link characteristics of each abnormal data stream; based on the network link characteristics of each abnormal data stream, cluster the multiple abnormal data streams to obtain at least one data stream cluster; for each data stream cluster, select a target abnormal data stream from the abnormal data streams included in the data stream cluster; the first sending module 5552 is further configured to send the first detection instruction to each of the first nodes that sends the target abnormal data stream.

[0152] In some embodiments, the number of computing tasks is multiple, and the number of abnormal data streams is multiple; the first sending module 5552 is further configured to perform the following processing for each abnormal data stream before sending the first detection instruction to the first node: determining the target computing task to which the abnormal data stream belongs from the multiple computing tasks; generating a first detection instruction for the abnormal data stream based on the task identifier of the target computing task and the transmission path of the abnormal data stream; adding the first detection instruction for the abnormal data stream to an instruction buffer queue; the first sending module 5552 is further configured to send the first detection instruction to the first node indicated by the task identifier in the first detection instruction according to the issuance order of each first detection instruction in the instruction buffer queue.

[0153] This application embodiment also provides an anomaly cause detection device applied to a first node, comprising: a second receiving module, configured to receive a first detection instruction; wherein the first detection instruction is sent when an anomaly occurs during the execution of a computing task by a node cluster, the first detection instruction including the transmission path of the abnormal data stream in the node cluster, the node cluster including multiple nodes cooperating in executing the computing task, the multiple nodes including a first node that sends the abnormal data stream and a second node that receives the abnormal data stream; a detection module, configured to, in response to the first detection instruction, detect the connectivity in the direction from the first node to the second node according to the transmission path, and obtain a first detection result; and a second sending module, configured to send the first detection result, the first detection result being used to determine the anomaly cause of the anomaly.

[0154] In some embodiments, the detection module is further configured to, in response to the first detection instruction, establish a communication connection between the first node and the second node according to the source address and destination address in the transmission path, and obtain the source port number and destination port number of the communication connection; obtain the target source port number in the transmission path and modify the source port number to the target source port number; send a detection data stream to the destination port number of the second node according to the target source port number, the detection data stream being used to detect connectivity in the direction from the first node to the second node; when a response data stream returned by the second node based on the detection data stream is received, generate a first detection result indicating that the connectivity detection has passed; when a response data stream returned by the second node based on the detection data stream is not received, generate a first detection result indicating that the connectivity detection has failed.

[0155] It should be noted that the description of the device embodiments in this application is similar to the description of the method embodiments described above, and has similar beneficial effects as the method embodiments, so it will not be repeated here. Any technical details not covered in the anomaly cause detection device provided in the embodiments of this application can be understood based on the description of the technical details in the above method embodiments.

[0156] This application also provides a computer program product, which includes computer-executable instructions or a computer program stored in a computer-readable storage medium. The processor of an electronic device reads the computer-executable instructions or computer program from the computer-readable storage medium and executes the computer-executable instructions or computer program, causing the electronic device to perform the anomaly detection method provided in this application.

[0157] This application also provides a computer-readable storage medium storing computer-executable instructions or computer programs. When the computer-executable instructions or computer programs are executed by a processor, the processor will execute the abnormal cause detection method provided in this application.

[0158] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0159] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0160] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0161] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0162] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A method for detecting the cause of anomalies, characterized in that, The method includes: When an anomaly occurs in the node cluster during the execution of computing tasks, the anomaly log of the abnormal data stream in the node cluster is obtained, and the abnormal communication queue pair bound to the anomaly log is determined. Obtain the correspondence between multiple communication queue pairs and multiple communication transmission paths, wherein the communication queue pairs and the communication transmission paths correspond one-to-one; Based on the correspondence, the target communication transmission path corresponding to the abnormal communication queue is determined, and the target communication transmission path is used as the transmission path of the abnormal data stream; wherein, the node cluster includes multiple nodes that collaboratively execute the computing task, and the multiple nodes include: a first node that sends the abnormal data stream and a second node that receives the abnormal data stream; Send a first detection instruction to the first node. The first detection instruction is used to instruct the first node to detect the connectivity in the direction from the first node to the second node according to the transmission path included in the first detection instruction, and obtain a first detection result. The system receives the first detection result returned by the first node based on the first detection instruction; wherein the first detection instruction is used to cause the first node to perform the following processing: A communication connection is established between the first node and the second node according to the source address and destination address in the transmission path, and the source port number and destination port number of the communication connection are obtained; the source port number is modified to the target source port number in the transmission path; a detection data stream is sent to the destination port number of the second node according to the target source port number, the detection data stream is used to detect the connectivity in the direction from the first node to the second node, and the transmission path of the detection data stream is completely consistent with the transmission path of the abnormal data stream; when a response data stream is received from the second node based on the detection data stream, a first detection result indicating that the connectivity detection has passed is generated; when no response data stream is received from the second node based on the detection data stream, a first detection result indicating that the connectivity detection has failed is generated. Based on the first detection result, the cause of the anomaly is determined.

2. The method as described in claim 1, characterized in that, The step of determining the cause of the anomaly based on the first detection result includes: When the first detection result indicates that the connectivity detection is passed, the cause of the anomaly is determined to be: the software of at least one of the first node and the second node is abnormal, and the software is used to perform the computing task; When the first detection result indicates that the connectivity detection has failed, a second detection command is sent to the first node, and a third detection command is sent to the second node; Wherein, the second detection instruction is used to instruct the first node to detect the connectivity in the direction from the first node to the third node and obtain a second detection result; the third detection instruction is used to instruct the second node to detect the connectivity in the direction from the second node to the third node and obtain a third detection result; the network link between the third node and the first node is normal, and the network link between the third node and the second node is normal. Receive the second detection result returned by the first node based on the second detection instruction, and receive the third detection result returned by the second node based on the third detection instruction; Based on the second and third detection results, the cause of the anomaly is determined.

3. The method as described in claim 2, characterized in that, The step of determining the cause of the anomaly based on the second detection result and the third detection result includes: When both the second and third detection results indicate that the connectivity detection has passed, the cause of the anomaly is determined to be: there is an anomaly in the network link between the first node and the second node; When the second detection result indicates that the connectivity test has failed, and the third detection result indicates that the connectivity test has passed, the cause of the abnormality is determined to be: the hardware of the first node is abnormal; When the second detection result indicates that the connectivity detection has passed and the third detection result indicates that the connectivity detection has failed, the cause of the anomaly is determined to be: the hardware of the second node is faulty; When the second detection result indicates that the connectivity test has failed, and the third detection result indicates that the connectivity test has failed, the cause of the abnormality is determined to be that the hardware of both the first node and the second node is abnormal.

4. The method as described in claim 2, characterized in that, When the first detection result indicates that the connectivity detection has passed, the cause of the anomaly is determined to be: at least one of the software of the first node and the second node has an anomaly, including: When the first detection result indicates that the connectivity detection is passed, the abnormal log of the abnormal data stream is obtained; When the exception log belongs to the first node, the cause of the exception is determined to be: the software of the first node has an exception; When the exception log belongs to the second node, the cause of the exception is determined to be: the software of the second node is abnormal; When the exception log includes the first exception log of the first node and the second exception log of the second node, the cause of the exception is determined to be that at least one of the software of the first node and the second node has an exception.

5. The method as described in claim 4, characterized in that, When the exception log includes the first exception log of the first node and the second exception log of the second node, the cause of the exception is determined to be: at least one of the software of the first node and the second node has an exception, including: When the exception log includes the first exception log of the first node and the second exception log of the second node, the following processing is performed: When the first generation time of the first abnormal log is earlier than the second generation time of the second abnormal log, the cause of the abnormality is determined to be: the software of the first node has an abnormality; When the first generation time of the first abnormal log is later than the second generation time of the second abnormal log, the cause of the abnormality is determined to be: the software of the second node is abnormal; When the first generation time of the first abnormal log is equal to the second generation time of the second abnormal log, the cause of the abnormality is determined to be: the software of both the first node and the second node has an abnormality.

6. The method as described in claim 2, characterized in that, After determining that the cause of the anomaly is that at least one of the software in the first node and the second node is abnormal, the method further includes: Obtain the abnormal node logs of the target node, wherein the target node is at least one of the first node and the second node; Identify the target error code in the log of the abnormal node; Obtain the correspondence between multiple error codes and multiple software exception causes, wherein the error codes and the software exception causes are in one-to-one correspondence; Based on the correspondence, the cause of the target software anomaly corresponding to the target error code is determined, and the cause of the target software anomaly is used as the reason for the software anomaly of the target node.

7. The method according to any one of claims 1-6, characterized in that, The method further includes: When the node cluster does not encounter any abnormalities during the execution of the computing task, the normal transmission path of the normal data stream in the node cluster is obtained. The plurality of nodes include a fourth node that sends the normal data stream and a fifth node that receives the normal data stream. A pre-detection instruction is sent to the fourth node, which instructs the fourth node to detect the connectivity from the fourth node to the fifth node according to the normal transmission path and obtain a connectivity measurement index. Receive the measurement metrics returned by the fourth node based on the pre-detection instruction; An anomaly warning is generated when the measured metric does not match the baseline measured metric.

8. The method according to any one of claims 2-6, characterized in that, The method further includes: When the cause of the anomaly is an anomaly in the network link between the first node and the second node, the network link between the first node and the second node is switched to the target network link; When the cause of the anomaly is a software malfunction in the target node, the target node is restarted, wherein the target node is at least one of the first node and the second node; When the cause of the anomaly is a hardware malfunction of the target node, the target node is removed, and the sixth node is invoked to execute the target computing task on the target node, the computing task including the target computing task.

9. The method according to any one of claims 1-6, characterized in that, The number of abnormal data streams is multiple; before sending the first detection command to the first node, the method further includes: Obtain the network link characteristics of each of the abnormal data streams; Based on the network link characteristics of each of the abnormal data streams, the multiple abnormal data streams are clustered to obtain at least one data stream cluster; For each data stream cluster, a target abnormal data stream is selected from the abnormal data streams included in the data stream cluster; Sending the first detection command to the first node includes: The first detection instruction is sent to each of the first nodes that sends the target abnormal data stream.

10. The method according to any one of claims 1-6, characterized in that, The number of computational tasks is multiple, and the number of abnormal data streams is multiple; before sending the first detection instruction to the first node, the method further includes: For each of the aforementioned abnormal data streams, the following processing is performed: From the multiple computing tasks, determine the target computing task to which the abnormal data stream belongs; Based on the task identifier of the target computing task and the transmission path of the abnormal data stream, a first detection instruction for the abnormal data stream is generated. Add the first detection instruction of the abnormal data stream to the instruction buffer queue; Sending the first detection command to the first node includes: The first detection instruction is sent to the first node indicated by the task identifier in the first detection instruction according to the order in which each first detection instruction is issued in the instruction buffer queue.

11. A method for detecting the cause of an anomaly, characterized in that, The method includes: The first node receives the first detection command; The first detection instruction is sent when an anomaly occurs during the execution of a computing task by the node cluster. The first detection instruction includes the transmission path of the abnormal data stream in the node cluster. The transmission path is a target communication transmission path determined based on the abnormal communication queue pairs bound to the abnormal logs of the abnormal data stream, and the correspondence between multiple communication queue pairs and multiple communication transmission paths. The communication queue pairs and the communication transmission paths correspond one-to-one. The node cluster includes multiple nodes that collaboratively execute the computing task. The multiple nodes include the first node that sends the abnormal data stream and the second node that receives the abnormal data stream. In response to the first detection command, a communication connection is established between the first node and the second node according to the source address and destination address in the transmission path, and the source port number and destination port number of the communication connection are obtained; the target source port number in the transmission path is obtained and modified to the target source port number; a detection data stream is sent to the destination port number of the second node according to the target source port number, the detection data stream is used to detect connectivity in the direction from the first node to the second node, and the transmission path of the detection data stream is completely consistent with the transmission path of the abnormal data stream; when a response data stream returned by the second node based on the detection data stream is received, a first detection result indicating that the connectivity detection has passed is generated; when no response data stream returned by the second node based on the detection data stream is received, a first detection result indicating that the connectivity detection has failed is generated. Send the first detection result, which is used to determine the cause of the anomaly.

12. An anomaly cause detection device, characterized in that, The device includes: The acquisition module is used to acquire, when an anomaly occurs in the node cluster during the execution of a computing task, the anomaly log of the abnormal data stream in the node cluster, and determine the abnormal communication queue pair bound to the anomaly log; acquire the correspondence between multiple communication queue pairs and multiple communication transmission paths, wherein the communication queue pairs and the communication transmission paths correspond one-to-one; based on the correspondence, determine the target communication transmission path corresponding to the abnormal communication queue pair, and use the target communication transmission path as the transmission path of the abnormal data stream; wherein the node cluster includes multiple nodes that collaboratively execute the computing task, and the multiple nodes include: a first node that sends the abnormal data stream and a second node that receives the abnormal data stream; The first sending module is used to send a first detection instruction to the first node. The first detection instruction is used to instruct the first node to detect the connectivity in the direction from the first node to the second node according to the transmission path included in the first detection instruction, and to obtain a first detection result. The first receiving module is configured to receive the first detection result returned by the first node based on the first detection instruction, wherein the first detection instruction is configured to cause the first node to perform the following processing: A communication connection is established between the first node and the second node according to the source address and destination address in the transmission path, and the source port number and destination port number of the communication connection are obtained; the source port number is modified to the target source port number in the transmission path; a detection data stream is sent to the destination port number of the second node according to the target source port number, the detection data stream is used to detect the connectivity in the direction from the first node to the second node, and the transmission path of the detection data stream is completely consistent with the transmission path of the abnormal data stream; when a response data stream is received from the second node based on the detection data stream, a first detection result indicating that the connectivity detection has passed is generated; when no response data stream is received from the second node based on the detection data stream, a first detection result indicating that the connectivity detection has failed is generated. The determination module is used to determine the cause of the anomaly based on the first detection result.

13. The apparatus as claimed in claim 12, characterized in that, The determining module is further configured to: When the first detection result indicates that the connectivity detection is passed, the cause of the anomaly is determined to be: the software of at least one of the first node and the second node is abnormal, and the software is used to perform the computing task; When the first detection result indicates that the connectivity detection has failed, a second detection command is sent to the first node, and a third detection command is sent to the second node; Wherein, the second detection instruction is used to instruct the first node to detect the connectivity in the direction from the first node to the third node and obtain a second detection result; the third detection instruction is used to instruct the second node to detect the connectivity in the direction from the second node to the third node and obtain a third detection result; the network link between the third node and the first node is normal, and the network link between the third node and the second node is normal. Receive the second detection result returned by the first node based on the second detection instruction, and receive the third detection result returned by the second node based on the third detection instruction; Based on the second and third detection results, the cause of the anomaly is determined.

14. The apparatus as claimed in claim 13, characterized in that, The determining module is further configured to: When both the second and third detection results indicate that the connectivity detection has passed, the cause of the anomaly is determined to be: there is an anomaly in the network link between the first node and the second node; When the second detection result indicates that the connectivity test has failed, and the third detection result indicates that the connectivity test has passed, the cause of the abnormality is determined to be: the hardware of the first node is abnormal; When the second detection result indicates that the connectivity detection has passed and the third detection result indicates that the connectivity detection has failed, the cause of the anomaly is determined to be: the hardware of the second node is faulty; When the second detection result indicates that the connectivity test has failed, and the third detection result indicates that the connectivity test has failed, the cause of the abnormality is determined to be that the hardware of both the first node and the second node is abnormal.

15. The apparatus as claimed in claim 13, characterized in that, The determining module is further configured to: When the first detection result indicates that the connectivity detection is passed, the abnormal log of the abnormal data stream is obtained; When the exception log belongs to the first node, the cause of the exception is determined to be: the software of the first node has an exception; When the exception log belongs to the second node, the cause of the exception is determined to be: the software of the second node is abnormal; When the exception log includes the first exception log of the first node and the second exception log of the second node, the cause of the exception is determined to be that at least one of the software of the first node and the second node has an exception.

16. The apparatus as claimed in claim 15, characterized in that, The determining module is further configured to perform the following processing when the abnormal log includes the first abnormal log of the first node and the second abnormal log of the second node: When the first generation time of the first abnormal log is earlier than the second generation time of the second abnormal log, the cause of the abnormality is determined to be: the software of the first node has an abnormality; When the first generation time of the first abnormal log is later than the second generation time of the second abnormal log, the cause of the abnormality is determined to be: the software of the second node is abnormal; When the first generation time of the first abnormal log is equal to the second generation time of the second abnormal log, the cause of the abnormality is determined to be: the software of both the first node and the second node has an abnormality.

17. The apparatus as claimed in claim 13, characterized in that, The determining module is further configured to: After determining that the cause of the anomaly is an anomaly in the software of at least one of the first node and the second node, the anomaly node log of the target node is obtained, wherein the target node is at least one of the first node and the second node; Identify the target error code in the log of the abnormal node; Obtain the correspondence between multiple error codes and multiple software exception causes, wherein the error codes and the software exception causes are in one-to-one correspondence; Based on the correspondence, the cause of the target software anomaly corresponding to the target error code is determined, and the cause of the target software anomaly is used as the reason for the software anomaly of the target node.

18. The apparatus according to any one of claims 12-17, characterized in that, The acquisition module is also used for: When the node cluster does not encounter any abnormalities during the execution of the computing task, the normal transmission path of the normal data stream in the node cluster is obtained. The plurality of nodes include a fourth node that sends the normal data stream and a fifth node that receives the normal data stream. A pre-detection instruction is sent to the fourth node, which instructs the fourth node to detect the connectivity from the fourth node to the fifth node according to the normal transmission path and obtain a connectivity measurement index. Receive the measurement metrics returned by the fourth node based on the pre-detection instruction; An anomaly warning is generated when the measured metric does not match the baseline measured metric.

19. The apparatus according to any one of claims 13-17, characterized in that, The determining module is further configured to: When the cause of the anomaly is an anomaly in the network link between the first node and the second node, the network link between the first node and the second node is switched to the target network link; When the cause of the anomaly is a software malfunction in the target node, the target node is restarted, wherein the target node is at least one of the first node and the second node; When the cause of the anomaly is a hardware malfunction of the target node, the target node is removed, and the sixth node is invoked to execute the target computing task on the target node, the computing task including the target computing task.

20. The apparatus according to any one of claims 12-17, characterized in that, The number of abnormal data streams is multiple; The first sending module is further configured to: acquire network link characteristics of each abnormal data stream before sending the first detection instruction to the first node; cluster the multiple abnormal data streams based on the network link characteristics of each abnormal data stream to obtain at least one data stream cluster; select a target abnormal data stream from the abnormal data streams included in each data stream cluster; and send the first detection instruction to each first node that sends the target abnormal data stream.

21. The apparatus according to any one of claims 12-17, characterized in that, The number of computing tasks is multiple, and the number of abnormal data streams is multiple; The first sending module is further configured to: Before sending the first detection command to the first node, the following processing is performed for each of the abnormal data streams: From the multiple computing tasks, determine the target computing task to which the abnormal data stream belongs; Based on the task identifier of the target computing task and the transmission path of the abnormal data stream, a first detection instruction for the abnormal data stream is generated. Add the first detection instruction of the abnormal data stream to the instruction buffer queue; The first sending module is further configured to send the first detection instruction to the first node indicated by the task identifier in the first detection instruction according to the issuance order of each first detection instruction in the instruction buffer queue.

22. An anomaly cause detection device, applied to a first node, characterized in that, The device includes: The second receiving module is used to receive a first detection instruction; wherein the first detection instruction is sent when an anomaly occurs in the node cluster during the execution of a computing task, the first detection instruction includes the transmission path of the abnormal data stream in the node cluster, the transmission path is a target communication transmission path determined based on the abnormal communication queue pairs bound to the abnormal logs of the abnormal data stream, and the correspondence between multiple communication queue pairs and multiple communication transmission paths, the communication queue pairs and the communication transmission paths correspond one-to-one, the node cluster includes multiple nodes that collaboratively execute the computing task, the multiple nodes include the first node that sends the abnormal data stream and the second node that receives the abnormal data stream; The detection module is configured to, in response to the first detection command, detect the connectivity in the direction from the first node to the second node according to the transmission path, and obtain a first detection result; specifically, in response to the first detection command, establish a communication connection between the first node and the second node according to the source address and destination address in the transmission path, and obtain the source port number and destination port number of the communication connection; obtain the target source port number in the transmission path, and modify the source port number to the target source port number; send a detection data stream to the destination port number of the second node according to the target source port number, the detection data stream being used to detect the connectivity in the direction from the first node to the second node, the transmission path of the detection data stream being completely consistent with the transmission path of the abnormal data stream; when a response data stream returned by the second node based on the detection data stream is received, generate the first detection result indicating that the connectivity detection has passed; when no response data stream returned by the second node based on the detection data stream is received, generate the first detection result indicating that the connectivity detection has failed. The second sending module is used to send the first detection result, which is used to determine the cause of the anomaly.

23. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions for a computer; The processor, when executing computer-executable instructions stored in the memory, implements the anomaly cause detection method according to any one of claims 1 to 11.

24. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, the abnormal cause detection method according to any one of claims 1 to 11 is implemented.

25. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, the abnormal cause detection method according to any one of claims 1 to 11 is implemented.

Images