Anomaly detection method, apparatus and device

Abstract

The embodiment of the application provides an abnormality detection method, a first computing node obtains running state information of a first computing process through a monitoring thread; and the monitoring thread sends the running state information to a first management process. The monitoring thread is a thread in the first computing process, and the running state information comprises abnormality information in a running process of the first computing process. Through the scheme, the first computing node can collect running state information corresponding to execution of a distributed computing task of the first computing process in time through the monitoring thread, so as to quickly determine an abnormality root cause according to the running state information. Compared with the traditional method that can only identify a fault after the computing process is abnormally stopped, the abnormality detection efficiency is improved. Moreover, when the abnormality root cause is analyzed, the running state information is collected in real time, the running state information retains context information at an abnormal moment compared with a log, provides more comprehensive information for positioning of the abnormality root cause, and improves positioning accuracy.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an anomaly detection method, apparatus, and device. Background Technology

[0002] To improve data throughput, computationally intensive tasks can be broken down into smaller, independently executable tasks, which are then distributed across different computing nodes in the cluster. During execution, these computing nodes interact with each other via data exchange to ensure the tasks are executed sequentially. For example, the training and application of artificial intelligence (AI) models often require multiple computing nodes for distributed training of the AI ​​model, or the use of multiple computing nodes to run the AI ​​model to provide inference services.

[0003] Because distributed computing tasks involve a massive hardware and software stack, a hardware or software failure in any computing node will propagate to other nodes, causing task interruption. For example, training an AI model with trillions of parameters might use tens of thousands of graphics processing units (GPUs), with these GPUs working together in a distributed manner. Since training such a large model involves a very large hardware and software stack, and due to the strongly distributed training model, a hardware or software failure in any part will spread to other training processes, causing a complete training interruption. Currently, there are hardware status-based detection technologies that can quickly and accurately detect the location and cause of faulty devices. When a hardware failure occurs, the faulty device can be quickly isolated and troubleshooted based on the type of failure, and training can continue using a healthy device after the faulty device is replaced.

[0004] However, current software fault detection technologies can only detect software faults after an anomaly occurs in the process executing the task. Furthermore, this method requires querying all logs from all computing nodes to diagnose the root cause, resulting in a large workload and long detection time. Log loss can also affect the accuracy of the detection. Summary of the Invention

[0005] Based on this, this application provides an anomaly detection method, apparatus, and equipment to improve the efficiency and accuracy of anomaly detection.

[0006] Firstly, an anomaly detection method is provided. This method is applied to a first computing node, which includes a first computing process and a first management process. The first computing node runs the first computing process to execute a first computing task. The method includes: the first computing node acquiring the running status information of the first computing process through a monitoring thread; and the first computing node sending the running status information to the first management process through the monitoring thread. The monitoring thread is a thread within the first computing process, and the running status information includes anomaly information during the execution of the first computing process. Through this scheme, the first computing node can collect the running status information corresponding to the distributed computing task executed by the first computing process in a timely manner through the monitoring thread. This allows for rapid identification of the root cause of the anomaly based on the running status information, improving anomaly detection efficiency compared to the traditional method of only identifying faults after the computing process abnormally stops. Moreover, the analysis of the root cause of the anomaly is based on the real-time collected running status information. Compared with logs, this running status information retains the context information at the time of the anomaly, thus providing more comprehensive information for locating the root cause and improving the accuracy of the location.

[0007] Therefore, this information can include software anomaly information, indicator anomaly information, etc. This allows for the identification of not only software anomalies but also abnormal indicator phenomena, increasing the diversity of anomaly detection.

[0008] In one possible implementation, the first computing node sends runtime status information to the second management process of the second computing node through a first management process. The second computing node is a computing node that jointly executes distributed computing tasks with the first computing node. This second computing node may include a second computing process and a second management process. The second computing node can obtain the runtime status information of the second computing process through a monitoring thread within the second computing process, and then send this runtime status information to the second management process through the monitoring thread. This runtime status information includes any abnormal information during the execution of the second computing process.

[0009] In this implementation, the second computing node, acting as the central node in the computing cluster, is responsible for collecting the operational status information of other computing nodes in the cluster. All computing nodes in the cluster collaboratively execute distributed computing tasks. After collecting the operational status information of all computing nodes, this information can be sent to a higher-level node, such as the management node. The management node then analyzes the root causes of anomalies based on this information, reducing the consumption of computing resources on the second computing node. Alternatively, the second computing node can analyze the root causes of anomalies based on the collected operational status information and report them to the management node. This way, only the management node needs to be notified of the root causes, reducing the amount of data reported.

[0010] In one possible implementation, the first computing node sends runtime status information to a management node through a first management process. This management node manages the execution of distributed computing tasks. Alternatively, the first computing node can directly report runtime status information to the management node.

[0011] In one possible implementation, the first computing node can also receive runtime status information sent by the third management process of the third computing node through the first management process. This runtime status information includes exception information during the execution of the third computing process of the third computing node, where the third computing node is a computing node that jointly executes distributed computing tasks with the first computing node. The runtime status information corresponding to the third computing node is obtained through a monitoring thread within the third computing process.

[0012] In this implementation, the first computing node can act as the central node of the computing cluster, responsible for collecting the running status information of other computing nodes in the computing cluster, and can report the collected running status information to the management node. The management node analyzes the root cause of the anomaly based on the running status information, thereby reducing the occupation of computing resources in the first computing node.

[0013] In one possible implementation, the first computing node determines the root cause of the anomaly in the distributed computing task based on the anomaly information of the first computing process and the anomaly information of the third computing process; the first computing node then sends the root cause to the management node. In this implementation, the first computing node can report only the root cause of the anomaly to the management node, without reporting the running status information, thus reducing the amount of data reported.

[0014] In one possible implementation, the first computing node can also determine the exception identifier corresponding to an anomaly that occurs during the execution of the first computing process based on the running status information. In this implementation, different anomalies can be identified to quickly locate the root cause of the anomaly based on the exception identifier.

[0015] In one possible implementation, the first computing node may include at least two logical layers that jointly execute the first computing task. These at least two logical layers include a first logical layer and a second logical layer that need to be executed sequentially. The first computing node determines anomaly identifiers for exceptions occurring during the execution of the first computing process based on runtime status information. This includes: the first computing node determining a first anomaly identifier corresponding to an anomaly occurring in the first logical layer based on the runtime status information; and the first computing node determining a second anomaly identifier corresponding to an anomaly occurring in the second logical layer based on the runtime status information and the first anomaly identifier. In this implementation, anomalies across all logical layers can be uniformly identified, so that each anomaly corresponds to a unique anomaly identifier. This allows for the acquisition of anomaly identifiers for each logical layer, avoiding information loss during transmission and improving the accuracy of anomaly root cause localization.

[0016] In one possible implementation, the first computing node can send an exception flag to the second management process of the second computing node through the first management process.

[0017] In one possible implementation, the first computing node sends an exception flag to the management node through the first management process.

[0018] Secondly, an anomaly detection device is provided, which is applied to a first computing node. The first computing node includes a first computing process and a first management process. The first computing node executes a first computing task by running the first computing process. The device includes: an acquisition unit, used to acquire the running status information of the first computing process through a monitoring thread, wherein the monitoring thread is a thread in the first computing process, and the running status information includes anomaly information in the first computing process; and a first sending unit, used to send the running status information to the first management process through the monitoring thread.

[0019] In one possible implementation, the apparatus further includes: a second sending unit, configured to send running status information to a second management process of a second computing node through a first management process, wherein the second computing node is a computing node that performs distributed computing tasks together with the first computing node.

[0020] In one possible implementation, the apparatus further includes: a third sending unit, used to send running status information to a management node through a first management process, wherein the management node is used to manage the running of distributed computing tasks.

[0021] The second transmitting unit and the second transmitting unit can be the same transmitting unit.

[0022] In one possible implementation, the device further includes:

[0023] The receiving unit is used to receive running status information sent by the third management process of the third computing node through the first management process. The running status information includes abnormal information of the running process of the third computing process of the third computing node. The third computing node is a computing node that performs distributed computing tasks together with the first computing node.

[0024] In one possible implementation, the device further includes:

[0025] The processing unit is used to determine the root cause of the abnormality corresponding to the distributed computing task based on the abnormal information of the first computing process and the abnormal information of the third computing process.

[0026] The third sending unit is also used to send the root cause to the management node.

[0027] In one possible implementation, the device further includes:

[0028] The processing unit is used to determine the exception identifier corresponding to the exception that occurs during the operation of the first computing process based on the running status information.

[0029] In one possible implementation, the first computing node includes at least two logical layers that jointly execute the first computing task. The at least two logical layers include a first logical layer and a second logical layer that need to be executed sequentially. The processing unit is specifically used to determine a first exception identifier corresponding to an exception occurring in the first logical layer based on the running status information; and to determine a second exception identifier corresponding to an exception occurring in the second logical layer based on the running status information and the first exception identifier.

[0030] In one possible implementation, the apparatus further includes: a fourth sending unit, configured to send an exception identifier to a second management process of a second computing node via a first management process, wherein the second computing node is a computing node that performs distributed computing tasks together with the first computing node.

[0031] In one possible implementation, the apparatus further includes: a fifth sending unit, used to send an exception identifier to a management node through a first management process, the management node being used to manage the operation of distributed computing tasks.

[0032] The fourth transmitting unit and the second transmitting unit can be the same transmitting unit. Alternatively, the second transmitting unit, the third transmitting unit, the fourth transmitting unit, and the fifth transmitting unit can be the same transmitting unit.

[0033] In one possible implementation, the exception information includes at least one of software exception information and indicator exception information.

[0034] Thirdly, a distributed computing system is provided, the system including a first computing node, the first computing node including a first computing process and a first management process, the first computing node executing a first computing task by running the first computing process;

[0035] The first computing node is used to obtain the running status information of the first computing process through a monitoring thread, wherein the monitoring thread is a thread in the first computing process, and the running status information includes abnormal information during the running of the first computing process; and to send the running status information to the first management process through the monitoring thread.

[0036] In one possible implementation, the system further includes a second computing node, which is a computing node that performs distributed computing tasks together with the first computing node, and the second computing node includes a second management process;

[0037] The second computing node is used to receive the running status information sent by the first computing node through the first management process via the second management process.

[0038] In one possible implementation, the system further includes a management node for managing the execution of distributed computing tasks;

[0039] The management node is used to receive the running status information sent by the first computing node through the first management process.

[0040] In one possible implementation, the system further includes a third computing node, which is a computing node that performs distributed computing tasks together with the first computing node, and the third computing includes a third management process and a third computing process;

[0041] The third computing node is used to send running status information to the first management process through the third management process. The running status information includes abnormal information during the operation of the third computing process.

[0042] In one possible implementation, the first computing node is further configured to determine the root cause of the abnormality corresponding to the distributed computing task based on the abnormality information of the first computing process and the abnormality information of the third computing process; and send the root cause to the management node.

[0043] In one possible implementation, the first computing node is further configured to determine, based on the running status information, an exception identifier corresponding to an exception that occurs during the operation of the first computing process.

[0044] In one possible implementation, the first computing node includes at least two logical layers that jointly execute the first computing task. The at least two logical layers include a first logical layer and a second logical layer that need to be executed sequentially. Specifically, the first computing node is used to determine a first exception identifier corresponding to an exception occurring in the first logical layer based on the running status information; and to determine a second exception identifier corresponding to an exception occurring in the second logical layer based on the running status information and the first exception identifier.

[0045] In one possible implementation, the system further includes a second computing node, which is a computing node that performs distributed computing tasks together with the first computing node, and the second computing node includes a second management process;

[0046] The second computing node is used to receive the exception identifier sent by the first management process through the second management process.

[0047] In one possible implementation, the system further includes a management node for managing the execution of distributed computing tasks; the management node is also used to receive an exception identifier sent by the first management process.

[0048] In one possible implementation, the abnormal information includes at least one of software abnormal information and indicator abnormal information.

[0049] Fourthly, this application provides a computing device including a processor and a memory, which is used to perform distributed computing tasks. The processor and the memory communicate with each other. The processor executes instructions stored in the memory to cause the computing device to perform the method steps executed by the first computing node in the first aspect or any implementation thereof. It should be noted that the memory can be integrated into the processor or can be independent of the processor. The computing device may also include a bus. The processor is connected to the memory via the bus. The memory may include readable storage and random access memory.

[0050] Fifthly, this application provides a computer-readable storage medium storing instructions that, when executed on at least one computing device, cause the at least one computing device to perform the method described in the first aspect or any implementation thereof.

[0051] In a sixth aspect, this application provides a computer program product containing instructions that, when run on at least one computing device, causes the at least one computing device to perform the method described in the first aspect or any implementation thereof.

[0052] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0054] Figure 1 A communication system structure diagram provided in this application embodiment;

[0055] Figure 2 This is a schematic flowchart of an anomaly detection method provided in an embodiment of this application;

[0056] Figure 3This is a schematic diagram of an application scenario provided by an embodiment of this application;

[0057] Figure 4 A layered schematic diagram provided for an embodiment of this application;

[0058] Figure 5 This application provides a schematic diagram of an abnormal addressing scheme.

[0059] Figure 6 A schematic diagram of a computing node structure provided in an embodiment of this application;

[0060] Figure 7 This is a schematic diagram of the hardware structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0061] To enable those skilled in the art to better understand the solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0062] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate; this is merely a method of distinction used in describing objects with the same attributes in the embodiments of this application.

[0063] The technical solutions in this application will now be described with reference to the accompanying drawings. For ease of understanding, subsequent embodiments will be illustrated using distributed computing tasks as AI tasks as an example.

[0064] See Figure 1 This is a schematic diagram of the structure of an exemplary computing cluster 10. Figure 1 As shown, the example of computing cluster 10 consisting of 4 computing nodes (computing node 1 to computing node 4) will be used for illustration.

[0065] In computing cluster 10, a computing node refers to a node with data computing capabilities, such as a server or other computing device. Each computing node may include one or more computing units. Figure 1 This example uses a compute node comprising four compute units, such as compute node 1 including compute units 1 through 4. In practical applications, different compute units deployed within the same compute node can communicate via a bus or switch, such as through a compute express link (CXL) bus.

[0066] The computing units in a computing node can be implemented using a processor. For example, a processor can be any type of processor or any combination thereof, such as a central processing unit (CPU), an accelerator, an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a system-on-chip (SoC), a software-defined infrastructure (SDI) chip, an artificial intelligence (AI) chip, or a data processing unit (DPU). An accelerator can be, for example, a graphics processing unit (GPU), a neural network processing unit (NPU), or a tensor processing unit (TPU).

[0067] In practical applications, computing cluster 10 can run AI tasks, and these AI tasks can be executed in parallel by multiple computing nodes within the cluster. For example, the AI ​​task could be a training task or an inference task for an AI model. When the AI ​​model has a large number of parameters, computing cluster 10 can utilize multiple computing nodes for distributed training of the AI ​​model, or utilize multiple computing nodes to run the AI ​​model to provide corresponding inference services. For example, the AI ​​model could be a large language model (LLM), a bidirectional encoder representations from transformers (BERT) model, or a generative pre-trained Transformer 3 (GPT-3) model, or other types of models such as GPT-4; there is no limitation on the specific type.

[0068] Currently, computing cluster 10 may encounter failures, such as software or hardware failures, when executing AI tasks. A failure in any computing node within cluster 10 will propagate to other computing nodes, causing a complete interruption of training or inference. Current hardware fault detection technology is relatively mature and can quickly and accurately detect faulty nodes and their causes. However, software fault detection technology can only identify software faults when a process becomes abnormal, and cannot identify faults before a process becomes abnormal. This detection method has three problems affecting fault tolerance: first, after a failure, all logs from all computing nodes need to be queried to diagnose the root cause; second, relying on the logs from executing AI tasks is usually insufficient to pinpoint the root cause; and third, for freeze-type failures, a certain timeout period is required, and manual judgment is necessary. These three problems will cause the entire task to stall for a period of time after an AI task fails before it can be resumed.

[0069] Based on this, this application provides an anomaly detection method for any computing node in a computing cluster 10. This computing node includes a computing process, a management process, and a monitoring thread within the computing process. The computing node executes distributed computing tasks by running the computing process. Specifically, the computing node obtains the running status information of the computing process through the monitoring thread. This running status information includes anomaly information during the execution of the computing process. The computing node sends the running status information to the management process through the monitoring thread. In this method, the computing node can collect anomaly information occurring during the execution of distributed computing tasks in real time through the monitoring thread. This allows for rapid identification of anomalies in the distributed computing task based on this anomaly information, rather than waiting for the computing process to malfunction before detecting anomalies, thus improving anomaly detection efficiency. Moreover, when determining the root cause of anomalies, the running status information collected during the process can be used, providing more reference information for locating the root cause compared to the traditional method of relying solely on logs, thereby improving the accuracy of the location.

[0070] It is worth noting that the above Figure 1The computing cluster 10 shown is merely an illustrative example and is not intended to be limiting. For instance, in other possible implementations, the computing cluster 10 may include more types of nodes, such as management nodes or scheduling nodes. Management nodes can be used to manage the computing power of the computing nodes in the computing cluster 10, such as putting some computing nodes in the computing cluster 10 into hibernation to reduce power consumption. Scheduling nodes can be used to schedule AI models to at least one computing node in the computing cluster 10, and different AI models can be scheduled to different computing nodes in the computing cluster 10. Furthermore, the number of computing nodes in the computing cluster 10 can be other numbers, or different computing nodes can be connected via a bus, thus eliminating the need for data communication through switching nodes; this is not limited. Alternatively, the computing cluster 10 may consist of supernodes, where a supernode is a collection of multiple computing nodes, i.e., each supernode includes multiple computing nodes.

[0071] For ease of understanding, embodiments of the information processing method provided in this application will be described below with reference to the accompanying drawings.

[0072] See Figure 2 , Figure 2 This is a flowchart illustrating an anomaly detection method provided in an embodiment of this application. This method can be applied to... Figure 1 The computing cluster 10 described above can also be applied to other suitable computing clusters. For ease of explanation, this embodiment uses an application... Figure 1 The computing cluster 10 shown is used as an example for illustration.

[0073] in, Figure 2 The anomaly detection method shown may specifically include:

[0074] S201: The first computing node obtains the running status information of the first computing process through the monitoring thread.

[0075] The first computing node includes a first computing process and a first management process. The first computing process includes a monitoring thread. The first computing node executes a first computing task by running the first computing process. The first computing node monitors the running status of the first computing process by monitoring the monitoring thread. The first management process is used to report running status information to the superior node.

[0076] In this embodiment, after obtaining the distributed computing task to be executed (i.e., the first computing task), the first computing node can create a first computing process and execute the first computing task by running the first computing process. The first computing process acts as the bearer of the distributed computing task, performing the actual computation. To avoid interference between multiple computing processes within the same computing node, the first computing process can be deployed in a container on the first computing node. Optionally, a first management process is deployed in the container to which the first computing process belongs.

[0077] When the first computing node executes the first computing task through the first computing process, a monitoring thread collects the running status information of the first computing process. This running status information includes abnormal information that occurs during the execution of the first computing process. The abnormal information may include, but is not limited to, software abnormal information and indicator abnormal information. Indicator abnormal information may include performance abnormal information and accuracy abnormal information, thereby enabling rapid identification of abnormal indicator situations.

[0078] For example, the first computing node can collect the aforementioned operational status information in real time through a monitoring thread, or it can collect the information at regular intervals or when a typical event occurs. Typical events may include, but are not limited to, fluctuations in the computational precision or performance of the first computing process during the execution of the first computational task, or software anomalies within the first computing process. After a typical event occurs, the first computing node is triggered to collect the aforementioned operational status information through the monitoring thread. The first computing node can be... Figure 1 Any computing node in the process.

[0079] In one possible implementation, the runtime status information may include not only anomaly information but also other information, such as computation information and resource allocation information corresponding to the first computation task executed by the first computation process. The computation information may include computation time, computation precision, etc.; the resource allocation information may include the computational resources, storage resources, and network resources allocated to the first computation process, which can be used to corroborate the root cause of the anomaly and improve the accuracy of the location.

[0080] It should be noted that, for Figure 1 Any computing node in the computing cluster 10 shown can perform the above operations to obtain the running status information of the computing process in that computing node when executing distributed computing tasks.

[0081] S202: The first computing node sends running status information to the first management process through the monitoring thread.

[0082] In this embodiment, an information channel is added from the monitoring thread in the first computing thread to the first management process, through which running status information is sent to the first management process.

[0083] After the first management process receives the running status information, the first computing node can also perform the following operations:

[0084] One approach involves the first computing node sending runtime status information to the second management process of the second computing node through a first management process. The second computing node is a computing node that jointly executes distributed computing tasks with the first computing node. This second computing node may include a second computing process, which executes a second computing task. The second computing node can obtain the runtime status information of the second computing process through a monitoring thread within the second computing thread. This runtime status information may include any exceptions encountered during the execution of the second computing process. The distributed computing task includes a first computing task and a second computing task.

[0085] In this embodiment, each computing node can deploy a management process, increasing the information channel between management processes in different computing nodes. After receiving the running status from the monitoring thread, the first management process in the first computing node sends the running status information to the second computing node through the information channel between the first management process and the second management process of the second computing node. In this implementation, the second computing node acts as the central node in the computing cluster. For any other computing node in the cluster besides the second computing node, after obtaining the running status information of the computing process through its own monitoring thread, it sends the running status information to the second management process of the second computing node through its own management process.

[0086] After receiving the runtime status information from each computing node, the second computing node can report the runtime status information of all computing nodes to the management node. The management node then determines the root cause of the distributed computing task anomaly based on this information. If a faulty node exists, the management node can also identify it based on the runtime status information, allowing for timely isolation and restart of the distributed computing task. Alternatively, the second computing node can determine the root cause of the distributed computing task anomaly based on the collected runtime status information of each computing node, and simultaneously report the runtime status information of each computing node and the root cause to the management node. The second computing node can also identify the faulty node based on the runtime status information of all computing nodes, allowing for timely isolation and restart of the distributed computing task. After obtaining the runtime status information and / or the root cause of the anomaly, the management node can also determine the corresponding handling strategy to resolve the anomaly. For example, if the anomaly is a precision anomaly, the corresponding handling strategy is to stop computation; if the anomaly is a performance anomaly, the corresponding handling strategy is to issue an alarm; if the anomaly is a software anomaly, the corresponding handling strategy is to isolate the fault point and reschedule a new computing node to execute the distributed computing task.

[0087] It should be noted that, in practice, the second computing node can collect the running status information of other computing nodes at regular intervals or when typical events occur, and report it to the management node.

[0088] For example, the first computing node is Figure 1 The first and second computation nodes are... Figure 1 Computing node 2 in the cluster. In specific implementation, a computing node can be selected from the multiple computing nodes included in the computing cluster 10, or a computing node can be pre-designated as the second computing node, which is responsible for collecting the running status information of other computing nodes in the cluster.

[0089] In one possible implementation, the first compute node sends runtime status information to the management node through a first management process. The management node manages the execution of distributed computing tasks. In this implementation, the first compute node, as the central node in the computing cluster, reports the runtime status information directly to the management node after receiving it.

[0090] Optionally, when the first computing node acts as the central node, it can also receive runtime status information sent by the third management process of the third computing node through the first management process. This runtime status information may include exception information during the execution of the third computing process of the third computing node. The runtime status information is obtained by the third computing node through a monitoring thread, which is a thread within the third computing process. The third computing node is a computing node that jointly executes distributed computing tasks with the first computing node. That is, the third computing node obtains the runtime status information of the third computing process through the monitoring thread within the third computing process, sends this runtime status information to the third management process through the monitoring thread, and then the third management process sends the corresponding runtime status information of the third computing process to the first management process.

[0091] In this implementation, after collecting the runtime status information reported by other computing nodes, the first computing node can report the collected runtime status information of all computing nodes to the management node. This allows the management node to determine the root cause of any anomalies in the distributed computing task based on the collected runtime status information. It should be noted that the first computing node aggregates the runtime status information reported by other computing nodes at regular intervals or when typical events occur, and reports this information to the management node through the first management process.

[0092] In one possible implementation, when the first computing node acts as the central node, it can also determine the root cause of the anomaly corresponding to the distributed computing task based on the anomaly information of the first computing process and the third computing process, and send the root cause to the management node. In this implementation, the first computing node can determine the root cause of the anomaly based on the collected runtime status information and send the root cause to the management node, so that the management node can directly determine the matching processing strategy based on the root cause.

[0093] In this embodiment, after receiving the operating status information and / or root cause, the management node can send alarm information to the user equipment. Simultaneously, the management node can also display the root cause of the anomaly to the user, allowing the user to intuitively understand the anomaly.

[0094] In this embodiment, to improve the speed of root cause location, after obtaining the running status information, the first computing node determines the anomaly identifier corresponding to the anomaly that occurred during the operation of the first computing process based on the running status. Furthermore, after obtaining the anomaly identifier, the first computing node can also send the anomaly identifier to an upper-level node (e.g., a second computing node or management node acting as the central node), so that the upper-level node can determine the root cause of the anomaly based on the anomaly identifier. Specifically, the first computing node sends the anomaly identifier to the second management process of the second computing node through the first management process, or the first computing node sends the anomaly identifier to the management node through the first management process.

[0095] In other words, the first computing node can determine the anomaly identifier corresponding to the anomaly that occurs during the operation of the first computing process based on the running status information locally, and send the anomaly identifier to the upper-level node to provide reference information for the upper-level node to locate the root cause of the anomaly, thereby enabling the rapid location of the root cause of the anomaly.

[0096] In one possible implementation, considering the complex process of executing distributed computing tasks, the computing process can be divided into multiple logical layers for easier management, with each layer performing different operations. Specifically, the first computing node includes at least two logical layers that jointly execute the first computing task. These at least two logical layers include a first logical layer and a second logical layer that need to be executed sequentially. The first computing node determines an exception identifier based on relevant information, including: determining a first exception identifier corresponding to an exception occurring in the first logical layer based on runtime status information; and determining a second exception identifier corresponding to an exception occurring in the second logical layer based on runtime status information and the first exception identifier. In this implementation, the first computing node sends the first and second exception identifiers to an upper-level node (e.g., a second computing node or a management node).

[0097] In this embodiment, the first computing node determines the anomalies that occur in the first computing process at each logical layer and the anomaly identifiers corresponding to the anomalies based on the running status information, and reports the anomaly identifiers to the upper-level node so that the upper-level node can quickly locate the root cause of the anomalies based on the anomaly identifiers of each logical layer.

[0098] In one possible implementation, after receiving the running status information, the first anomaly identifier, and the second anomaly identifier, the upper-level node first determines the third anomaly identifier corresponding to the anomaly that occurred at the upper-level node based on the running status information. The upper-level node then determines the root cause of the anomaly in the first computing process based on the first, second, and third anomaly identifiers. In this implementation, the upper-level node combines different anomaly identifiers to quickly locate the root cause of the anomaly.

[0099] As can be seen, for the first computing node executing the distributed computing task, the running status information of the first computing process is obtained through a monitoring thread; the first computing node then sends the running status information to the first management process through the monitoring thread. The monitoring thread is a thread within the first computing process, and the running status information includes anomalies encountered during the process's execution. This scheme allows the first computing node to collect the running status information corresponding to the distributed computing task in a timely manner through the monitoring thread, enabling rapid identification of the root cause of anomalies. This significantly improves anomaly detection efficiency compared to the traditional method of identifying faults only after the computing process has abnormally stopped. Furthermore, the analysis of the root cause is based on real-time collected running status information, which, compared to logs, retains the contextual information at the time of the anomaly, providing more comprehensive information for locating the root cause and improving accuracy.

[0100] To facilitate understanding of the technical framework of this application, participants Figure 3 The diagram illustrates an anomaly detection technology framework. In this scenario, the second computing node collects the running status of other computing nodes, determines the root cause of the anomaly based on this information, and then reports the root cause to the management node. For ease of understanding, the distributed computing task is used as an example of an AI model training task. In this scenario, the computing process is the process executing the training task.

[0101] (1) When the computing process is training the AI ​​model, the relevant training software stack will record training information in memory, such as the performance and accuracy of the AI ​​model. At the same time, when an exception occurs within the computing process, exception information will also be recorded in memory.

[0102] (2) The monitoring thread is deployed within the computing process and triggers recording (the monitoring thread performs recording operations) of relevant information at regular intervals or when typical events occur. After collecting the running status information, the monitoring thread reports it to the management process to which the computing process in which it resides belongs. For example, Figure 3 The monitoring thread in the first computing node reports to the first management process to which the first computing process belongs; the monitoring thread in the third computing node reports to the third management process to which the third computing process belongs.

[0103] (3) The first or third management process checks the survival status of the target process at regular intervals, for example, checking whether the computing process exists and its current status (e.g., whether it is in a dormant or working state). At the same time, at regular intervals or when typical events occur (internal status update of the computing process, internal abnormal information update of the computing process, etc.), other information and abnormal information are collected, and the running status information is sent to the central management process (i.e., the second management process).

[0104] (4) The central management process collects the running status information reported by all computing nodes, locates the root cause of the anomaly and the abnormal node based on the running status information, and reports it to the management node.

[0105] (5) The management node analyzes and decides on the handling strategy corresponding to the current anomaly based on the information reported by the task center management process. If a faulty node exists, it is isolated and the training task is restarted. At the same time, the root cause of the anomaly is reported to the AI ​​platform to inform the user.

[0106] As discussed above, the process of executing distributed computing tasks can be divided into multiple logical layers, each with different functions. For example... Figure 4 As shown, taking distributed computing tasks as AI model training tasks as an example, the AI ​​model training software stack is divided into a device layer, a heterogeneous computing layer, an AI framework layer, a training management layer, and an AI platform layer corresponding to the management node.

[0107] The Device layer, also known as the computing device layer, executes related computing and communication tasks. On the Device side, it records operator execution time, exceptions related to computing and communication tasks, and exceptions in software stack calls. Above the Device layer is the heterogeneous computing layer, which sends relevant information to the heterogeneous computing layer through device-to-host (D2H) and host-to-device (H2D) communication, as well as process interface calls on the Host side.

[0108] Heterogeneous Computing Layer: Through the abstraction of computing node resource management, this layer allocates relevant computing and communication resources and executes corresponding computing and communication actions. The heterogeneous computing layer uploads relevant information to the AI ​​framework layer via process interface calls and exception handling mechanisms.

[0109] AI Framework Layer: This layer abstracts training behavior and handles communication and computational resource calls. It records the time spent on training-related steps and captures anomalies in the training process, reporting these anomalies to the training management layer or AI platform layer via the standard output (standardout, stdout) and standard error (standard error, stderr) of the training logs.

[0110] Training Management Layer: By managing the training process (computation process), it monitors the survival status and logs of the training process, judges the health status of the training process, and reports it to the AI ​​platform layer through logs or exit codes.

[0111] AI Platform Layer: By monitoring the exit status of the overall task and logs to determine the survival status of the training business, the health status of the training business is judged, and corresponding fault tolerance is implemented based on the health status of the training business.

[0112] It should be noted that, Figure 4 The AI ​​platform layer in the middle can be Figure 3 The management node shown, or the AI ​​platform layer deployed on... Figure 3 In the management node shown.

[0113] exist Figure 4 The following features have been added to the layered framework shown:

[0114] (1) A monitoring thread is added to the training process. This monitoring thread, located within the training process, can collect real-time running status information during training, and statistically analyze training accuracy, performance, and other information to perceive the training status in real time. Simultaneously, the monitoring thread can also detect software anomalies occurring during the training process in real time. Furthermore, it can also collect resource information allocated by the software within the training process; this resource information is used to corroborate the root cause of the fault and improve the accuracy of fault location.

[0115] (2) Add an information channel from the monitoring thread to the management process. The training status information, training exception information and resource allocation information perceived by the monitoring thread can be transmitted to the management process in real time.

[0116] (3) The management process now includes the function of collecting training status information, training anomaly information, and resource allocation information. At the same time, an information channel between the management process and the central management process has been added, allowing the training status information, training anomaly information, and resource allocation information obtained by the management process to be directly sent to the central management process.

[0117] (4) The central management process has added the function of collecting training status information, training anomaly information and resource allocation information of each node, and determines the root cause of the current task based on the information sent by all computing nodes of the current training task. It also adds the information channel from the central management process to the AI ​​platform, and reports the root cause of the training task to the AI ​​platform in real time, triggering the relevant fault tolerance processing of the AI ​​platform.

[0118] To quickly pinpoint the root cause of exceptions, exceptions across all layers of the software stack can be uniformly addressed, with each layer using a globally unique exception code (exception identifier). Monitoring threads and management processes can acquire exceptions occurring at each layer and collect them centrally. The central management process then performs exception propagation chain matching on all exception codes, quickly matching them against an exception pattern library to locate the root cause of the current task's exception. For example, Figure 5As shown, different layers encode potential anomalies for that layer, obtaining corresponding anomaly codes. For example, the Device layer corresponds to 4 anomaly codes, the heterogeneous computing layer to 3, the AI ​​framework layer to 4, the training management layer to 4, and the AI ​​platform layer to 3. In the specific implementation, each layer uploads its determined anomaly code to the layer above it. This allows the layer above to determine the corresponding anomaly code for its current layer based on the runtime status information and the anomaly code. The runtime status information and the anomaly codes of each layer are then sent to the upper layer sequentially, enabling the AI ​​platform layer to determine the root cause of the anomaly based on the anomaly codes of each layer.

[0119] The reason why the lower layer needs to pass exception codes to the upper layer is that different exceptions at the lower layer may manifest as the same exception at the upper layer, making it impossible to determine the root cause of the lower layer exception. For example, the Device layer exception codes 0x0001 and 0x0002 map to the heterogeneous computing layer exception code 0x1001. If the Device layer does not send exception codes to the heterogeneous computing layer, the heterogeneous computing layer determines the exception code to be 0x1001 based on the runtime status information. When determining the root cause of the exception, it cannot determine whether the root cause corresponds to 0x0001 or 0x0002.

[0120] It should be noted that, Figure 5 This is merely a schematic diagram for unified addressing across different logical layers. It does not impose any limitations on the specific implementation of unified addressing. Each layer can determine the number of exception codes and the form of the exception codes based on the actual situation.

[0121] The above combination Figures 1 to 5 The anomaly detection method provided in the embodiments of this application will be introduced. Next, the structure of the computing node provided in the embodiments of this application will be described with reference to the accompanying drawings.

[0122] See Figure 6 This figure is a structural diagram of an anomaly detection device provided in an embodiment of this application. Figure 6 The anomaly detection device 600 is applied to a first computing node, which includes a first computing process and a first management process. The first computing node runs the first computing process to perform a first computing task, including:

[0123] The acquisition unit 601 is used to acquire the running status information of the first computing process through monitoring, wherein the monitoring thread is a thread in the first computing process, and the running status information includes abnormal information of the first computing process.

[0124] The first sending unit 602 is used to send the running status information to the first management process through the monitoring thread.

[0125] In one possible implementation, the apparatus further includes: a second sending unit, configured to send the running status information to a second management process of a second computing node through the first management process, wherein the second computing node is a computing node that performs distributed computing tasks together with the first computing node.

[0126] In one possible implementation, the apparatus further includes: a third sending unit, configured to send the running status information to a management node via the first management process, wherein the management node is configured to manage the running of distributed computing tasks.

[0127] In one possible implementation, the device further includes:

[0128] The receiving unit is configured to receive running status information sent by the third management process of the third computing node through the first management process. The running status information includes abnormal information of the running process of the third computing process of the third computing node. The third computing node is a computing node that performs distributed computing tasks together with the first computing node.

[0129] In one possible implementation, the device further includes:

[0130] The processing unit is configured to determine the root cause of the abnormality corresponding to the distributed computing task based on the abnormality information of the first computing process and the abnormality information of the third computing process.

[0131] The third sending unit is also used to send the root cause to the management node.

[0132] In one possible implementation, the device further includes:

[0133] The processing unit is used to determine the exception identifier corresponding to the exception that occurs during the operation of the first computing process based on the running status information.

[0134] In one possible implementation, the first computing node includes at least two logical layers that jointly execute the first computing task. The at least two logical layers include a first logical layer and a second logical layer that need to be executed sequentially. The processing unit is specifically used to determine a first exception identifier corresponding to an exception occurring in the first logical layer based on the running status information; and to determine a second exception identifier corresponding to an exception occurring in the second logical layer based on the running status information and the first exception identifier.

[0135] In one possible implementation, the apparatus further includes: a fourth sending unit, configured to send the exception identifier to a second management process of a second computing node through the first management process, wherein the second computing node is a computing node that performs distributed computing tasks together with the first computing node.

[0136] In one possible implementation, the apparatus further includes: a fifth sending unit, configured to send the exception identifier to a management node via the first management process, the management node being configured to manage the operation of distributed computing tasks.

[0137] In one possible implementation, the abnormal information includes at least one of software abnormal information and indicator abnormal information.

[0138] because Figure 6 The anomaly detection device 600 shown corresponds to the above. Figure 2 The first computing node in the illustrated embodiment, therefore Figure 6 For the specific implementation method and technical effects of the anomaly detection device 600 shown above, please refer to the above. Figure 2 The relevant details in the illustrated embodiments are described in detail here, and will not be repeated here.

[0139] Figure 7 This application provides a schematic diagram of the hardware structure of a computing device 700, which, for example, can implement the above-described... Figure 2 The first computing node in the illustrated embodiment, etc.

[0140] like Figure 7 As shown, the computing device 700 includes a processor 701, a memory 702, and a communication interface 703. The processor 701, memory 702, and communication interface 703 communicate via a bus 704, or via other means such as wireless transmission. The memory 702 stores instructions, and the processor 701 executes the instructions stored in the memory 702. Further, the computing device 700 may also include a memory unit 705, which is connected to the processor 701, the storage medium 702, and the communication interface 703 via the bus 704. The memory 702 stores program code, and the processor 701 can read the program code stored in the memory 702 into the memory unit 705 and execute the program code in the memory unit 705, causing the processor 701 to perform the following operations:

[0141] The running status information of the first computing process is obtained by monitoring the thread, wherein the monitoring thread is a thread in the first computing process, and the running status information includes abnormal information during the running process of the first computing process.

[0142] The monitoring thread sends the running status information to the first management process.

[0143] It should be understood that in this embodiment, the processor 701 can be a CPU, but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete device assemblies, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0144] The memory 702 may include read-only memory and random access memory, and provides instructions and data to the processor 701. The memory 702 may also include non-volatile random access memory.

[0145] The memory 702 can be volatile memory or non-volatile memory, or it can include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0146] The communication interface 703 is used to communicate with other devices connected to the computing device 700. The bus 704 may include a data bus, a power bus, a control bus, and a status signal bus, etc. However, for clarity, all buses are labeled as bus 704 in the figure.

[0147] It should be understood that the computing device 700 according to the embodiments of this application may correspond to the first computing unit in the embodiments of this application, and may correspond to the execution of the embodiment of this application. Figure 2The method executed by the first computing node in the illustrated method, and the above and other operations and / or functions implemented by the computing device 700, are respectively for the purpose of implementing... Figure 2 The process of the corresponding methods in [the document] will not be elaborated here for the sake of brevity.

[0148] This application embodiment also provides a distributed computing system, the system including a first computing node, the first computing node including a first computing process and a first management process, the first computing node executes a first computing task by running the first computing process;

[0149] The first computing node is used to obtain the running status information of the first computing process through a monitoring thread, wherein the monitoring thread is a thread in the first computing process, and the running status information includes abnormal information during the running of the first computing process; and to send the running status information to the first management process through the monitoring thread.

[0150] In one possible implementation, the system further includes a second computing node, which is a computing node that performs distributed computing tasks together with the first computing node, and the second computing node includes a second management process;

[0151] The second computing node is used to receive the running status information sent by the first computing node through the first management process via the second management process.

[0152] In one possible implementation, the system further includes a management node for managing the execution of distributed computing tasks;

[0153] The management node is used to receive the running status information sent by the first computing node through the first management process.

[0154] In one possible implementation, the system further includes a third computing node, which is a computing node that performs distributed computing tasks together with the first computing node, and the third computing includes a third management process and a third computing process;

[0155] The third computing node is used to send running status information to the first management process through the third management process. The running status information includes abnormal information during the operation of the third computing process.

[0156] In one possible implementation, the first computing node is further configured to determine the root cause of the abnormality corresponding to the distributed computing task based on the abnormality information of the first computing process and the abnormality information of the third computing process; and send the root cause to the management node.

[0157] In one possible implementation, the first computing node is further configured to determine, based on the running status information, an exception identifier corresponding to an exception that occurs during the operation of the first computing process.

[0158] In one possible implementation, the first computing node includes at least two logical layers that jointly execute the first computing task. The at least two logical layers include a first logical layer and a second logical layer that need to be executed sequentially. Specifically, the first computing node is used to determine a first exception identifier corresponding to an exception occurring in the first logical layer based on the running status information; and to determine a second exception identifier corresponding to an exception occurring in the second logical layer based on the running status information and the first exception identifier.

[0159] In one possible implementation, the system further includes a second computing node, which is a computing node that performs distributed computing tasks together with the first computing node, and the second computing node includes a second management process;

[0160] The second computing node is used to receive the exception identifier sent by the first management process through the second management process.

[0161] In one possible implementation, the system further includes a management node for managing the execution of distributed computing tasks; the management node is also used to receive an exception identifier sent by the first management process.

[0162] In one possible implementation, the abnormal information includes at least one of software abnormal information and indicator abnormal information.

[0163] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform the above-described anomaly detection method.

[0164] This application also provides a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the processes or functions described in this application are generated.

[0165] The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, or data center to another website, computer, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0166] The computer program product can be a software installation package. When any of the aforementioned anomaly detection methods is required, the computer program product can be downloaded and executed on a computing device.

[0167] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0168] The terminology used in the above embodiments is for the purpose of describing specific embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to also include expressions such as “one or more,” unless the context clearly indicates otherwise. It should also be understood that in the embodiments of this application, “one or more” refers to one, two, or more; the character “ / ” generally indicates that the preceding and following objects are in an “or” relationship. In the embodiments of this application, “simultaneously” means within the same time period, including situations where they are at the same moment.

[0169] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0170] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An anomaly detection method, characterized in that, The method is applied to a first computing node, the first computing node including a first computing process and a first management process, the first computing node running the first computing process to execute a first computing task, the method including: The first computing node obtains the running status information of the first computing process through a monitoring thread, wherein the monitoring thread is a thread in the first computing process, and the running status information includes abnormal information during the running process of the first computing process; The first computing node sends the running status information to the first management process through the monitoring thread.

2. The method according to claim 1, characterized in that, The method further includes: The first computing node sends the running status information to the second management process of the second computing node through the first management process. The second computing node is a computing node that performs distributed computing tasks together with the first computing node.

3. The method according to claim 1, characterized in that, The method further includes: The first computing node sends the running status information to the management node through the first management process, and the management node is used to manage the running of distributed computing tasks.

4. The method according to claim 3, characterized in that, The method further includes: The first computing node receives running status information sent by the third management process of the third computing node through the first management process. The running status information includes abnormal information of the running process of the third computing process of the third computing node. The third computing node is a computing node that performs distributed computing tasks together with the first computing node.

5. The method according to claim 4, characterized in that, The method further includes: The first computing node determines the root cause of the abnormality corresponding to the distributed computing task based on the abnormal information of the first computing process and the abnormal information of the third computing process. The first computing node sends the root cause to the management node.

6. The method according to any one of claims 1-5, characterized in that, The method further includes: The first computing node determines the exception identifier corresponding to the exception that occurs during the operation of the first computing process based on the running status information.

7. The method according to claim 6, characterized in that, The first computing node includes at least two logical layers that jointly execute the first computing task. These at least two logical layers include a first logical layer and a second logical layer that need to be executed sequentially. The first computing node determines an exception identifier corresponding to an anomaly that occurs during the execution of the first computing process based on the running status information, including: The first computing node determines the first anomaly identifier corresponding to the anomaly that occurred in the first logical layer based on the running status information; The first computing node determines the second exception identifier corresponding to the exception that occurred in the second logic layer based on the running status information and the first exception identifier.

8. The method according to claim 6 or 7, characterized in that, The method further includes: The first computing node sends the exception identifier to the second management process of the second computing node through the first management process. The second computing node is a computing node that performs distributed computing tasks together with the first computing node.

9. The method according to claim 6 or 7, characterized in that, The method further includes: The first computing node sends the exception identifier to the management node through the first management process. The management node is used to manage the operation of distributed computing tasks.

10. The method according to any one of claims 1-9, characterized in that, The abnormal information includes at least one of software abnormal information and indicator abnormal information.

11. An anomaly detection device, characterized in that, The device is applied to a first computing node, which includes a first computing process and a first management process. The first computing node runs the first computing process to perform a first computing task, including: The acquisition unit is used to acquire the running status information of the first computing process through monitoring, wherein the monitoring thread is a thread in the first computing process, and the running status information includes abnormal information of the first computing process. The first sending unit is used to send the running status information to the first management process through the monitoring thread.

12. The apparatus according to claim 11, characterized in that, The device further includes: The second sending unit is used to send the running status information to the second management process of the second computing node through the first management process, wherein the second computing node is a computing node that performs distributed computing tasks together with the first computing node.

13. The apparatus according to claim 11, characterized in that, The device further includes: The third sending unit is used to send the running status information to the management node through the first management process. The management node is used to manage the running of distributed computing tasks.

14. The apparatus according to claim 13, characterized in that, The device further includes: The receiving unit is configured to receive running status information sent by the third management process of the third computing node through the first management process. The running status information includes abnormal information of the running process of the third computing process of the third computing node. The third computing node is a computing node that performs distributed computing tasks together with the first computing node.

15. The apparatus according to claim 14, characterized in that, The device further includes: The processing unit is configured to determine the root cause of the abnormality corresponding to the distributed computing task based on the abnormality information of the first computing process and the abnormality information of the third computing process. The third sending unit is used to send the root cause to the management node.

16. The apparatus according to any one of claims 11-15, characterized in that, The device further includes: The processing unit is used to determine the exception identifier corresponding to the exception that occurs during the operation of the first computing process based on the running status information.

17. The apparatus according to claim 16, characterized in that, The first computing node includes at least two logical layers that jointly execute the first computing task. The at least two logical layers include a first logical layer and a second logical layer that need to be executed sequentially. The processing unit is specifically used to determine a first exception identifier corresponding to an exception occurring in the first logical layer based on the running status information; and to determine a second exception identifier corresponding to an exception occurring in the second logical layer based on the running status information and the first exception identifier.

18. The apparatus according to claim 16 or 17, characterized in that, The device further includes: The fourth sending unit is used to send the exception identifier to the second management process of the second computing node through the first management process, wherein the second computing node is a computing node that performs distributed computing tasks together with the first computing node.

19. The apparatus according to claim 16 or 17, characterized in that, The device further includes: The fifth sending unit is used to send the exception identifier to the management node through the first management process. The management node is used to manage the operation of distributed computing tasks.

20. The apparatus according to any one of claims 11-19, characterized in that, The abnormal information includes at least one of software abnormal information and indicator abnormal information.

21. A distributed computing system, characterized in that, The system includes a first computing node, the first computing node includes a first computing process and a first management process, and the first computing node runs the first computing process to perform a first computing task. The first computing node is used to obtain the running status information of the first computing process through a monitoring thread, wherein the monitoring thread is a thread in the first computing process, and the running status information includes abnormal information during the running of the first computing process; and sends the running status information to the first management process through the monitoring thread.

22. The system according to claim 21, characterized in that, The system further includes a second computing node, which is a computing node that performs distributed computing tasks together with the first computing node, and the second computing node includes a second management process; The second computing node is used to receive the running status information sent by the first computing node through the first management process via the second management process.

23. The system according to claim 21, characterized in that, The system also includes a management node, which is used to manage the execution of distributed computing tasks; The management node is used to receive the running status information sent by the first computing node through the first management process.

24. The system according to claim 23, characterized in that, The system further includes a third computing node, which is a computing node that performs distributed computing tasks together with the first computing node. The third computing includes a third management process and a third computing process. The third computing node is used to send running status information to the first management process through the third management process. The running status information includes abnormal information during the operation of the third computing process.

25. The system according to claim 24, characterized in that, The first computing node is also used to determine the root cause of the abnormality corresponding to the distributed computing task based on the abnormal information of the first computing process and the abnormal information of the third computing process. The first computing node is also used to send the root cause to the management node.

26. A computing device, characterized in that, Including processor and memory; The processor is configured to execute instructions stored in the memory to cause the computing device to perform the steps of the method as described in any one of claims 1 to 10.

27. A computer-readable storage medium, characterized in that, Includes instructions that, when executed on at least one computing device, cause the at least one computing device to perform the steps of the method as described in any one of claims 1 to 10.

28. A computer program product containing instructions, characterized in that, When it is run on at least one computing device, it causes the at least one computing device to perform the method as described in any one of claims 1 to 10.

Images