Fault detection method for server cluster, system, electronic device, and storage medium

Abstract

Disclosed are a fault detection method for a server cluster, a system, an electronic device, and a storage medium. The method relates to the technical field of cloud networks, and comprises: when it is detected that the operation of a server cluster is interrupted, acquiring an operation log before the operation of the server cluster is interrupted; on the basis of the operation log, determining a faulty node in the server cluster, and a fault type of the server cluster; and by invoking a diagnosis module corresponding to the fault type, performing fault diagnosis on the faulty node to obtain a fault detection result of the faulty node, wherein the fault detection result is used for representing the cause of the fault of the faulty node. The present disclosure solves the technical problem in the related art of low fault detection efficiency and accuracy of a server cluster.

Description

Fault detection method and system of server cluster, electronic device and storage medium TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of cloud networks, in particular to a fault detection method and system of a server cluster, an electronic device and a storage medium. BACKGROUND

[0002] Due to the huge resource consumption of large models, distributed training is often required through multiple machines and multiple cards. Considering that large model training has a long time span and involves a large number of nodes and GPUs (Graphics Processing Units), the probability of encountering a fault during training is much higher than that of traditional single-machine training. Because node hardware failure often leads to frequent interruption of large model training, resulting in the waste of expensive cluster computing resources. After the cluster is interrupted, the specific cause of the fault can be located through fault detection, and the power vacancy time caused by hardware failure can be further shortened by automatically isolating the fault node and automatically restarting the failed task. However, the current fault detection is carried out by manual means, resulting in low efficiency and accuracy of fault detection of the cluster.

[0003] In view of the above problems, no effective solutions have been proposed so far. SUMMARY

[0004] The embodiments of the present disclosure provide a fault detection method and system of a server cluster, an electronic device and a storage medium to at least solve the technical problem of low efficiency and accuracy of fault detection of a server cluster in related technologies.

[0005] According to an aspect of an embodiment of the present disclosure, a fault detection method of a server cluster is provided, comprising: in the case of monitoring that the server cluster is interrupted, obtaining a running log before the server cluster is interrupted; based on the running log, determining a fault node in the server cluster and a fault type of the server cluster; using a diagnostic module corresponding to the fault type to diagnose the fault node, to obtain a fault detection result of the fault node, wherein the fault detection result is used to represent the cause of the fault of the fault node.

[0006] According to another aspect of the embodiments of the present disclosure, a method for detecting faults of a server cluster is also provided. The server cluster is configured to perform distributed training on a neural network model. The method comprises: in a case where it is monitored that the distributed training of the neural network model is interrupted, obtaining a training log before the distributed training is interrupted; determining, based on the training log, a fault node in the server cluster and a fault type of the server cluster; performing fault diagnosis on the fault node by calling a diagnosis module corresponding to the fault type, to obtain a fault detection result of the fault node, wherein the fault detection result is used to represent a cause of the fault of the fault node.

[0007] According to another aspect of the embodiments of the present disclosure, a method for detecting faults of a server cluster is also provided. The server cluster is configured to perform distributed training on a neural network model. The method comprises: in a case where it is monitored that the distributed training of the neural network model is interrupted, obtaining a training log before the distributed training is interrupted; determining, based on the training log, a fault node in the server cluster and a fault type of the server cluster; performing fault diagnosis on the fault node by calling a diagnosis module corresponding to the fault type, to obtain a fault detection result of the fault node, wherein the fault detection result is used to represent a cause of the fault of the fault node.

[0008] According to an aspect of the embodiments of the present disclosure, a device for detecting faults of a server cluster is provided. The device comprises: a first obtaining component configured to, in a case where it is monitored that the server cluster is interrupted, obtain a running log before the server cluster is interrupted; a first determining component configured to determine, based on the running log, a fault node in the server cluster and a fault type of the server cluster; and a first calling component configured to call a diagnosis module corresponding to the fault type to perform fault diagnosis on the fault node, to obtain a fault detection result of the fault node, wherein the fault detection result is used to represent a cause of the fault of the fault node.

[0009] According to another aspect of the embodiments of the present disclosure, a device for detecting faults of a server cluster is also provided. The server cluster is configured to perform distributed training on a neural network model. The device comprises: a second obtaining component configured to, in a case where it is monitored that the distributed training of the neural network model is interrupted, obtain a training log before the distributed training is interrupted; a second determining component configured to determine, based on the training log, a fault node in the server cluster and a fault type of the distributed training; and a second calling component configured to call a diagnosis module corresponding to the fault type to perform fault diagnosis on the fault node, to obtain a fault detection result of the fault node, wherein the fault detection result is used to represent a cause of the fault of the fault node.

[0010] According to another aspect of the embodiments of the present disclosure, a fault detection apparatus of a server cluster is also provided, which comprises: a first calling component configured to, in a case where a running interruption of the server cluster is monitored, acquire a running log before the running interruption of the server cluster by calling a first interface, wherein the first interface comprises a first parameter, and a parameter value of the first parameter comprises the running log; a third determining component configured to determine a fault node in the server cluster and a fault type of the server cluster based on the running log; a third diagnosing component configured to perform fault diagnosis on the fault node by calling a diagnosis module corresponding to the fault type, to obtain a fault detection result of the fault node, wherein the fault detection result is used to represent a cause of the fault of the fault node; and a second calling component configured to output the fault detection result by calling a second interface, wherein the second interface comprises a second parameter, and a parameter value of the second parameter comprises the fault detection result.

[0011] According to another aspect of the embodiments of the present disclosure, a fault detection system of a server cluster is also provided, which comprises: a plurality of diagnosis modules connected with the server cluster, different diagnosis modules corresponding to different fault types, and the plurality of diagnosis modules being used to perform fault diagnosis on nodes in the server cluster; and a detection device connected with the server cluster and the plurality of diagnosis modules, configured to, in a case where a running interruption of the server cluster is monitored, acquire a running log before the running interruption of the server cluster, and determine a fault node in the server cluster and a fault type of the server cluster based on the running log, call a diagnosis module corresponding to the fault type to perform fault diagnosis on the fault node, and obtain a fault detection result of the fault node, wherein the fault detection result is used to represent a cause of the fault of the fault node.

[0012] According to another aspect of the embodiments of the present disclosure, an electronic device is also provided, which comprises: a memory storing an executable program; and a processor configured to run the program, wherein the program performs the method in the embodiments of the present disclosure when running.

[0013] According to another aspect of the embodiments of the present disclosure, a computer readable storage medium is also provided, which comprises a stored executable program, wherein the computer readable storage medium controls a device where the computer readable storage medium is located to perform the method in the embodiments of the present disclosure when the executable program runs.

[0014] According to another aspect of the embodiments of the present disclosure, a computer program product is also provided, which comprises a computer program, and the computer program implements the method in the embodiments of the present disclosure when executed by a processor.

[0015] According to another aspect of the embodiments of the present disclosure, a computer program product is also provided, including a non-volatile computer readable storage medium storing a computer program, the computer program being executed by a processor to implement the method in various embodiments of the present disclosure.

[0016] According to another aspect of the embodiments of the present disclosure, a computer program is also provided, the computer program being executed by a processor to implement the method in various embodiments of the present disclosure.

[0017] In the embodiments of the present disclosure, in the case of monitoring the interruption of the server cluster running, the running log before the interruption of the server cluster running is acquired, then based on the running log, the fault node in the server cluster and the fault type of the server cluster are determined, and finally the fault node is diagnosed by calling the diagnosis module corresponding to the fault type to obtain the fault detection result of the fault node, thereby achieving the purpose of fault positioning of the server cluster. It is easy to note that the server cluster can be fault detected by the combination of the running log and the diagnosis module, and the cause of the fault is automatically located, not only achieving the technical effects of improving the fault detection accuracy, reducing the fault omission rate, and improving the fault detection efficiency, thereby providing a basis for subsequent node isolation and task recovery, effectively shortening the time of wasting computing resources caused by the fault, and further solving the technical problems of low fault detection efficiency and accuracy of the server cluster in the related art.

[0018] It is easy to note that the above general description and the following detailed description are only for exemplifying and explaining the present disclosure, and do not constitute a limitation on the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0019] The drawings described herein are used to provide further understanding of the present disclosure, and constitute a part of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure, and do not constitute an improper limitation on the present disclosure. In the drawings:

[0020] Fig. 1 is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a fault detection method of a server cluster according to an embodiment of the present disclosure;

[0021] Fig. 2 is a structure block diagram of a computing environment in which a computer terminal (or mobile device) is located according to an embodiment of the present disclosure;

[0022] Fig. 3 is a structure block diagram of a computer terminal (or mobile device) as a service mesh according to an embodiment of the present disclosure;

[0023] Fig. 4 is a flowchart of a fault detection method of a server cluster according to an embodiment 1 of the present disclosure;

[0024] FIG. 5 is a flowchart of an optional fault diagnosis and elimination of a server cluster according to an embodiment of the present disclosure;

[0025] FIG. 6 is a flowchart of an optional fault detection method of a server cluster according to an embodiment of the present disclosure;

[0026] FIG. 7 is a flowchart of a fault detection method of a server cluster according to Embodiment 2 of the present disclosure;

[0027] FIG. 8 is a flowchart of a fault detection method of a server cluster according to Embodiment 3 of the present disclosure;

[0028] FIG. 9 is a schematic diagram of a fault detection apparatus of a server cluster according to Embodiment 4 of the present disclosure;

[0029] FIG. 10 is a schematic diagram of a fault detection apparatus of a server cluster according to Embodiment 5 of the present disclosure;

[0030] FIG. 11 is a schematic diagram of a fault detection apparatus of a server cluster according to Embodiment 6 of the present disclosure;

[0031] FIG. 12 is a schematic diagram of a fault detection system of a server cluster according to an embodiment of the present disclosure;

[0032] FIG. 13 is a structural block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] In order to make the person skilled in the art better understand the present disclosure scheme, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, not all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the present disclosure.

[0034] It should be noted that the terms "first", "second" and the like in the specification and claims of the present disclosure and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0035] First, some of the nouns or terms that appear in the description of the embodiments of the present disclosure are applicable to the following explanations:

[0036] Distributed training: a method of efficiently training large models, especially when a single computing unit (such as a CPU or GPU) cannot handle a huge dataset or a complex model, in distributed training, the training task is divided into multiple parts, and then processed in parallel on multiple processing units of a computing cluster.

[0037] GPU: used to handle computer graphics and image processing tasks. Since these tasks usually involve a lot of parallel computing (for example, operating on each pixel point on the screen), GPUs are designed to perform hundreds of computing tasks at the same time, which is achieved by having a large number of computing cores. In machine learning and deep learning, training models usually involve a lot of matrix and vector operations, especially in neural networks, each forward propagation and back propagation requires large-scale matrix multiplication and other parallel computing. These computing tasks are similar to the original use cases of GPUs, so the parallel processing capabilities of GPUs can be very effectively used to accelerate the training process.

[0038] Checkpoint: the state saved during the model training process, including the parameters (weights and biases) of the model, the state of the optimizer, and other possible training state information, mainly to save the progress periodically during the training process, so that the training can be resumed from the previous state in case of unexpected failure.

[0039] NVLink: a high-speed, high-bandwidth interconnection technology used to connect GPUs and other computing devices, designed to improve internal data transmission speed and achieve higher performance, and can also be used to connect GPUs and CPUs to achieve more efficient data transmission and collaborative computing.

[0040] Through mechanisms such as automatic isolation of faulty nodes and automatic restart of failed tasks, the idle time of cluster computing resources caused by hardware failure can be shortened, but fault location technology is the premise for the above mechanisms to take effect, for example, if the non-faulty node is isolated, the failed task will still fail after automatic restart. Therefore, the present disclosure provides a fault detection method for a server cluster, which combines training logs with infrastructure diagnosis capabilities, and is log-oriented and infrastructure diagnosis-oriented.

[0041] According to the embodiments of the present disclosure, a method for detecting failure of a server cluster is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0042] The method embodiments provided by the embodiments of the present disclosure can be executed in a mobile terminal, a computer terminal or similar computing device. FIG. 1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing the method for detecting failure of a server cluster. As shown in FIG. 1, the computer terminal 10 (or mobile device) can include one or more processors 102 (the processor 102 can include but not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication function. In addition, it can also include a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which can be included as one of the ports in the BUS bus), a network interface, a power supply and / or a camera. Those skilled in the art can understand that the structure shown in FIG. 1 is only schematic, which does not limit the structure of the above-mentioned electronic device. For example, the computer terminal 10 can include more or less components than those shown in FIG. 1, or have a different configuration from that shown in FIG. 1.

[0043] It should be noted that the one or more processors 102 and / or other data processing circuits described above can be referred to as "data processing circuits" herein. The data processing circuit can be embodied in whole or in part as software, hardware, firmware or any other combination. In addition, the data processing circuit can be a single independent processing module, or any one of the other elements combined into the computer terminal 10 (or mobile device) in whole or in part. As referred to in the embodiments of the present disclosure, the data processing circuit serves as a processor to control (for example, selection of a variable resistance terminal path connected to an interface).

[0044] The memory 104 can be used to store software programs of application software and modules, such as program instructions / data storage means corresponding to the method in the embodiments of the present disclosure, and the processor 102 can execute various functional applications and data processing, i.e., implement the method in the above embodiments, by running the software programs and modules stored in the memory 104. The memory 104 can include a high-speed random access memory, and can also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory 104 can further include a memory remotely arranged with respect to the processor 102, which can be connected to the computer terminal 10 through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0045] The transmission device 106 is used to receive or send data via a network. The specific examples of the above network can include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to be able to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, RF) module, which is used to communicate with the Internet in a wireless manner.

[0046] The display can be, for example, a touch screen type liquid crystal display (Liquid Crystal Display, LCD), which can enable a user to interact with the user interface of the computer terminal 10 (or mobile device).

[0047] The hardware structure diagram shown in FIG. 1 can not only be used as an exemplary block diagram of the above computer terminal 10 (or mobile device), but also be used as an exemplary block diagram of the above server. In an optional embodiment, FIG. 2 shows an embodiment using the above computer terminal 10 (or mobile device) shown in FIG. 1 as a computing node in a computing environment 201. FIG. 2 shows a structural block diagram of a computing environment, as shown in FIG. 2, the computing environment 201 includes a plurality of (210-1, 210-2, …, are used to show in the figure) computing nodes (such as servers) running on a distributed network. The computing nodes all contain local processing and memory resources, and an end user 202 can remotely run an application program or store data in the computing environment 201. The application program can be provided as a plurality of services 220-1, 220-2, 220-3 and 220-4 in the computing environment 201, which represent services “A”, “D”, “E” and “H”, respectively.

[0048] End users 202 can provide and access services through a web browser or other software application on a client, in some embodiments, provisioning and / or requests of end users 202 can be provided to ingress gateway 230. Ingress gateway 230 can include a corresponding proxy to handle provisioning and / or requests for services (one or more services provided in computing environment 201).

[0049] Services are provided or deployed according to various virtualization technologies supported by computing environment 201. In some embodiments, services can be provided according to virtual machine (VM)-based virtualization, container-based virtualization, and / or the like. Virtual machine-based virtualization can be emulating a real computer by initializing a virtual machine to execute programs and applications without directly touching any actual hardware resources. While a virtual machine is virtualized, according to container-based virtualization, a container can be launched to virtualize an entire operating system (OS) so that multiple workloads can run on a single operating system instance.

[0050] In one embodiment of container-based virtualization, several containers of a service can be assembled into a Pod (e.g., a Kubernetes Pod). For example, as shown in FIG. 2, service 220-2 can be equipped with one or more Pods 240-1, 240-2, …, 240-N (collectively, Pods). A Pod can include a proxy 245 and one or more containers 242-1, 242-2, …, 242-M (collectively, containers). The one or more containers in a Pod handle requests related to one or more corresponding functions of a service, and proxy 245 generally controls network functions related to the service, such as routing, load balancing, and the like. Other services can also be equipped with similar Pods.

[0051] In operation, executing a user request from end user 202 can require invoking one or more services in computing environment 201, and executing one or more functions of a service can require invoking one or more functions of another service. As shown in FIG. 2, service “A” 220-1 receives a user request from end user 202 from ingress gateway 230, service “A” 220-1 can invoke service “D” 220-2, and service “D” 220-2 can request service “E” 220-3 to execute one or more functions.

[0052] The computing environment described above can be a cloud computing environment, where the allocation of resources is managed by a cloud service provider, allowing the development of functionality without the need to consider implementing, adjusting or extending servers. The computing environment allows developers to execute code that responds to events without building or maintaining complex infrastructure. Services can be split into a set of functions that can automatically scale independently, rather than scaling a single hardware device to handle potential loads.

[0053] In another alternative embodiment, FIG. 3 illustrates, in a block diagram, an embodiment of using the computer terminal 10 (or mobile device) shown in FIG. 1 described above as a service mesh. FIG. 3 illustrates a block diagram of a structure of a service mesh, as shown in FIG. 3, the service mesh 300 is mainly used to facilitate secure and reliable communication between a plurality of microservices, which refers to decomposing an application into a plurality of smaller services or instances and distributing them to run on different clusters / machines.

[0054] As shown in FIG. 3, the microservices can include an application service instance A and an application service instance B, which form a functional application layer of the service mesh 300. In an embodiment, the application service instance A is running in the form of a container / process 308 on a machine / workload container group 314 (Pod), and the application service instance B is running in the form of a container / process 310 on a machine / workload container group 316 (Pod).

[0055] In an embodiment, the application service instance A can be a commodity query service, and the application service instance B can be a commodity ordering service.

[0056] As shown in FIG. 3, the application service instance A and a mesh proxy 303 coexist in the machine workload container group 314, and the application service instance B and a mesh proxy 305 coexist in the machine workload container 316. The mesh proxy 303 and the mesh proxy 305 form a data plane layer of the service mesh 300. Among them, the mesh proxy 303 and the mesh proxy 305 run in the form of a container / process 304, a container / process 306, respectively, can receive a request 312 for commodity query service, and the mesh proxy 303 and the application service instance A can communicate bidirectionally, the mesh proxy 305 and the application service instance B can communicate bidirectionally. In addition, the mesh proxy 303 and the mesh proxy 305 can also communicate bidirectionally.

[0057] In one embodiment, the traffic of application service instance A is all routed through mesh proxy 303 to the appropriate destination, and the network traffic of application service instance B is all routed through mesh proxy 305 to the appropriate destination. It is noted that the network traffic referred to herein includes, but is not limited to, Hyper Text Transfer Protocol (HTTP), Representational State Transfer (REST), google Remote Procedure Call (gRPC), Redis, and the like.

[0058] In one embodiment, the functionality of the extended data plane layer can be implemented by writing custom filters for the proxies (Envoy) in the service mesh 300. The service mesh proxy configuration can be to make the service mesh correctly proxy service traffic, implement service interworking, and service governance. The mesh proxy 303 and the mesh proxy 305 can be configured to perform at least one of the following functions: service discovery, health checking, routing, load balancing, authentication and authorization, and observability.

[0059] As shown in FIG. 3, the service mesh 300 also includes a control plane layer. The control plane layer can be a set of services running in a dedicated namespace, hosted by the hosting control plane component 301 in machine / Pod 302. As shown in FIG. 3, the hosting control plane component 301 is in bidirectional communication with the mesh proxy 303 and the mesh proxy 305. The hosting control plane component 301 is configured to perform some control management functions. For example, the hosting control plane component 301 receives telemetry data transmitted by the mesh proxy 303 and the mesh proxy 305, which can further aggregate the telemetry data. The services, hosting control plane component 301 can also provide user-facing application programming interfaces (APIs) to more easily manipulate network behavior, provide configuration data to the mesh proxy 303 and the mesh proxy 305, and the like.

[0060] In the above operating environment, the disclosure provides a server cluster fault detection method as shown in FIG. 4. FIG. 4 is a flowchart of a server cluster fault detection method according to embodiment 1 of the disclosure. As shown in FIG. 4, the method can include the following steps:

[0061] Step S402, in the case of monitoring the interruption of the server cluster operation, obtaining the operation log before the interruption of the server cluster operation.

[0062] The above-mentioned server cluster can be a system composed of multiple servers connected to each other. The multiple servers can work together to improve availability, scalability and load balancing. The above-mentioned server can serve as a node in the server cluster. In different scenarios, the server cluster can perform different tasks. For example, in a large model training scenario, the server cluster can perform distributed training on a large model. For another example, in a data analysis scenario, the server cluster can perform data analysis tasks in parallel.

[0063] In order to achieve the purpose of fault detection of the server cluster based on the operation log, the operation information of the server cluster can be recorded in a timely manner during the execution of the task by the server cluster, and the corresponding operation log can be generated. If the server cluster performs a large model training task, the above-mentioned operation log can be a training log of the server cluster training a large model. Since different operation logs are generated at different times, in the case of interruption of the server cluster operation, the operation log recorded before the interruption of the server cluster operation can be collected.

[0064] In an optional embodiment, during the execution of the task by the server cluster, that is, during the operation of the server cluster, the operation state of the server cluster can be monitored by a monitoring tool. If the interruption of the server cluster operation is monitored, the fault detection process can be started. First, the operation log before the interruption of the server cluster operation can be collected.

[0065] Step S404, based on the operation log, determining the fault node in the server cluster and the fault type of the server cluster.

[0066] The above-mentioned fault node can be a node in the server cluster that has failed. The above-mentioned fault type can be a hardware failure, a software failure, or a network failure, but is not limited thereto.

[0067] In an optional embodiment, the running log before the server cluster running is interrupted can be analyzed to determine abnormal events contained in the running log, for example, error events, but not limited to. Since the abnormal events often contain network addresses of the communication opposite end (i.e., the node sending the abnormal event), the fault node that fails can be determined based on the network address. In another optional embodiment, a machine learning model for determining the fault node can be trained in advance by a machine learning method, and then the running log before the server cluster running is interrupted is input into the model to perform fault analysis by the model to determine the node that fails as the fault node.

[0068] After the fault node is determined, the fault type can be determined by analyzing the abnormal event, or the fault type can be determined based on the keywords in the running log. Optionally, a knowledge base can be formed based on operation and maintenance experience accumulation, and the knowledge base contains keywords corresponding to different fault types, so that the keywords in the running log can be extracted and matched with the knowledge base to determine the fault type. For example, if the running log contains keywords such as “TypeError” and “ValueError”, it can be determined that the fault type is software failure; if the running log contains keywords such as “ecc error” and “nvlink error”, it can be determined that the fault type is hardware failure; and if the running log contains keywords such as “error 12”, it can be determined that the fault type is software failure.

[0069] In step S406, the fault node is diagnosed by calling the diagnostic module corresponding to the fault type to obtain a fault detection result of the fault node, wherein the fault detection result is used to represent the cause of the fault of the fault node.

[0070] The diagnostic module described above can be a diagnostic tool for diagnosing different types of faults. For example, for hardware failure, the diagnostic module can be a tool for reading system logs, a tool for reading counters on hardware through a special driver, or a tool for detecting whether the fault node is damaged.

[0071] In an optional embodiment, to avoid the problem of easy mis-isolation based on the running log alone, after the fault node and the fault type are determined based on the running log, a more accurate fault cause can be determined by calling the diagnostic module, different countermeasures are further taken according to different fault causes, the misdiagnosis rate and the missed diagnosis rate are reduced, and the diagnosis time is shortened.

[0072] By the scheme provided by the above-mentioned embodiments of the present disclosure, in the case of monitoring the interruption of the server cluster operation, the running log before the interruption of the server cluster operation is obtained, and then based on the running log, the fault node in the server cluster and the fault type of the server cluster are determined, and finally the fault diagnosis module corresponding to the fault type is called to diagnose the fault node, and the fault detection result of the fault node is obtained, so as to realize the purpose of fault positioning of the server cluster. It is easy to note that the server cluster can be fault detected by the combination of the running log and the diagnosis module, and the cause of the fault is automatically located. Not only the technical effects of improving the fault detection accuracy, reducing the fault omission rate, and improving the fault detection efficiency are achieved, but also the basis for subsequent node isolation and task recovery is provided, the time of wasting computing resources caused by the fault is effectively shortened, and thus the technical problem of low fault detection efficiency and accuracy of the server cluster in the related art is solved.

[0073] In the above-mentioned embodiments of the present disclosure, based on the running log, the fault node in the server cluster is determined, including: analyzing the running log to obtain an abnormal event contained in the running log, wherein the abnormal event is used to represent an event corresponding to an abnormality of the server cluster; determining network address information contained in the abnormal event; and determining a node corresponding to the network address information in the server cluster as the fault node.

[0074] In an optional embodiment, the running log can be analyzed to determine the abnormal event contained in the running log, and the network address of the communication opposite end (i.e., the node sending the abnormal event) can be read from the abnormal event, so that the node corresponding to the network address information in the server cluster can be determined as the fault node.

[0075] In the above-mentioned embodiments of the present disclosure, based on the running log, the fault type of the server cluster is determined, including: extracting a target keyword in the running log; matching the target keyword with a plurality of preset keywords of a plurality of preset types to determine a keyword matched successfully with the target keyword, wherein the plurality of preset keywords of the plurality of preset types are used to represent keywords of different fault types; determining a preset type corresponding to the keyword matched successfully with the target keyword from the plurality of preset types as the fault type.

[0076] The plurality of preset keywords of the plurality of preset types described above can be keywords of different fault types formed based on operation and maintenance experience accumulation. One fault type often involves multiple keywords, and thus the correspondence between the fault type and the keyword can be stored, so as to form a knowledge base.

[0077] In an optional embodiment, after the target keywords are extracted from the operation log, each extracted keyword can be matched with each preset keyword, and if the two keywords are the same, the preset type corresponding to the preset keyword can be determined as the fault type.

[0078] In the above embodiments of the present disclosure, the fault node is diagnosed by calling the diagnostic module corresponding to the fault type to obtain the fault detection result of the fault node, including: in the case of software fault, the fault prompt information in the operation log is extracted by calling the software diagnostic module to obtain the fault detection result; in the case of hardware fault, the fault node is diagnosed by calling the hardware diagnostic module to obtain the fault detection result; in the case of network fault, the fault node and the switch connected to the fault node are diagnosed by calling the network diagnostic module to obtain the fault detection result.

[0079] In an optional embodiment, if the fault type is software fault, the software diagnostic module is called to extract the fault prompt information such as “TypeError” and “ValueError” in the log, and these fault prompt information are directly taken as the fault cause, i.e., the user needs to modify the code and submit the task again.

[0080] In another optional embodiment, if the fault type is hardware problem, a special hardware diagnostic tool can be called for fault diagnosis to determine the fault cause, and the hardware diagnostic tool can be various types of diagnostic tools, which are not described herein.

[0081] In yet another optional embodiment, if the fault type is network problem, the network diagnostic module can be called to perform software and hardware diagnosis on the fault node and diagnose the upper connection switch of the fault node to determine the fault cause.

[0082] In the above embodiments of the present disclosure, the method further includes: determining the fault elimination mode of the fault node based on the fault detection result; and performing fault elimination operation on the fault node based on the fault elimination mode.

[0083] It should be noted that for the same fault, some fault causes cause the fault to occur, which can be self-healing. For example, the fault of "flapping" of the optical module occurs, and the fault cause is determined by diagnosis to be construction or vibration on site in the machine room, which causes abnormal optical signal transmission, which can be restored to normal by restarting the optical module. However, some fault causes cause the fault to occur repeatedly. For example, the fault of "flapping" of the optical module occurs, and the fault cause is determined by diagnosis to be aging or removal of the optical module, which requires maintenance personnel to plug and repair. For the occasional and self-healing fault, if the fault node is directly isolated and repaired, the fault node will be unavailable, resulting in loss of computing resources.

[0084] In an optional embodiment, in order to avoid the above situation, it can be determined based on the fault cause whether the fault node needs to be isolated, and the specific repair means for repairing the fault node. If the fault node needs to be isolated, the corresponding operation is performed on the fault node, and then the repair is performed according to the repair means. If not, the fault node does not need to be isolated, and the repair is directly performed according to the repair means.

[0085] In the above embodiments of the present disclosure, the fault node is excluded based on the fault exclusion mode, including: in the case that the fault exclusion mode is not required to isolate the fault node, outputting first prompt information, wherein the first prompt information is used to prompt to update the code data in the fault node or control the fault node to re-execute the task; in the case that the fault exclusion mode is hot maintenance of the fault node, outputting second prompt information, wherein the second prompt information is used to prompt to perform hot maintenance on the fault node; in the case that the fault exclusion mode is isolation and maintenance of the fault node, isolating the fault node, and outputting third prompt information in the case that the fault node is successfully isolated, wherein the third prompt information is used to prompt to maintain the fault node.

[0086] In an optional embodiment, for the fault belonging to the fault node without isolation, if the fault type is a software fault, the first prompt information can be directly outputted to prompt the user to modify the code; if the fault type is a hardware fault, the first prompt information can be directly outputted to prompt the user to control the fault node to re-execute the task, but not limited to this.

[0087] In another optional embodiment, for the fault belonging to the hot maintenance of the fault node, if the fault type is a hardware fault, the second prompt information can be directly outputted to prompt the maintenance personnel to replace the fault equipment in the fault node during the running of the fault node; if the fault type is a software fault (system disk space is insufficient), the second prompt information can be directly outputted to prompt the maintenance personnel to clean up the files on the fault node during the running of the fault node, but not limited to this.

[0088] In yet another optional embodiment, for a fault belonging to the isolated and repaired fault node, a third prompt information can be directly outputted to prompt the operation and maintenance personnel to upgrade the hardware driver, patch and restart the fault node after the fault node is isolated, or the operation and maintenance personnel can be prompted to replace the network card, GPU, NVLink and other hardware after the fault node is isolated.

[0089] Next, a preferred embodiment of the present disclosure will be described in detail with reference to FIGS. 5 and 6, taking the server cluster for distributed training of a neural network model as an example. As shown in FIG. 5, during the distributed training, the distributed training can be monitored in real time, if the training is interrupted, the fault node, fault type and fault cause can be accurately detected by the technical solutions provided by the above method, the fault detection result is obtained, and whether the fault node needs to be isolated is determined based on the fault detection result, if needed, the fault node is isolated, and after the isolation is completed, the fault node is restarted, and the distributed training is continued.

[0090] As shown in FIG. 6, in the case of monitoring the training interruption, the training log before the training interruption (i.e. the historical training log) can be collected first, and then the fault point is circled, and optionally, the fault node can be determined based on the network address information carried in the training log, and the fault type corresponding to the fault node is further determined. After the fault node and the fault type are determined, the fault node can be further diagnosed by different types of diagnosis modules, such as hardware diagnosis module, software diagnosis module, network diagnosis module, to obtain the fault diagnosis result. Further, the diagnosis results diagnosed by the same diagnosis module can be summarized and output to the subsequent fault elimination module as the basis for isolating the fault node.

[0091] Through the above scheme, the training log can be combined with the infrastructure diagnosis capability, and the running log is oriented, and the diagnosis result output by the diagnosis module is the final conclusion. On the one hand, it avoids the problem that the infrastructure experience is cut off from the actual scene in the traditional operation and maintenance scheme, and achieves targeted attack, effectively reducing the misdiagnosis rate and the missed diagnosis rate; on the other hand, it avoids the problem that the pure log scheme is easy to misisolate, and obtains more accurate fault causes by using software and hardware and network diagnosis capability based on the log circled fault point, and different countermeasures are taken according to different reasons, and the misisolation rate is reduced. Compared with the scheme of frequently saving checkpoints, the present scheme can shorten the time required for diagnosis, and although the frequent saving of checkpoints can reduce the loss of restart, it is likely to fail in restarting under the condition that the root cause is not determined, and the computing resources are wasted.

[0092] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present disclosure are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0093] It should be noted that, for the foregoing method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the present disclosure is not limited by the action sequence described, because according to the present disclosure, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the present disclosure.

[0094] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software and necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the technical solutions of the present disclosure can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes a plurality of instructions for making a terminal device (which can be a mobile phone, computer, server, or network device, etc.) execute the method described in each embodiment of the present disclosure.

[0095] According to the embodiments of the present disclosure, a server cluster fault detection method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that here.

[0096] FIG. 7 is a flowchart of a server cluster fault detection method according to Embodiment 2 of the present disclosure, and the server cluster is used for distributed training of a neural network model. As shown in FIG. 7, the method can include the following steps:

[0097] Step S702, in the case where it is monitored that the distributed training of the neural network model is interrupted, obtaining the training log before the interruption of the neural network model.

[0098] Step S704, based on the training log, determining the fault node in the server cluster and the fault type of the distributed training.

[0099] In step S706, the fault diagnosis module corresponding to the fault type is called to perform fault diagnosis on the fault node, and a fault detection result of the fault node is obtained, wherein the fault detection result is used to represent the cause of the fault of the fault node.

[0100] In the above embodiments of the present disclosure, based on the training log, the fault node in the server cluster is determined, including: analyzing the training log to obtain an abnormal event contained in the training log, wherein the abnormal event is used to represent an event corresponding to an abnormality of the server cluster; determining network address information contained in the abnormal event; and determining a node corresponding to the network address information from the server cluster as the fault node.

[0101] In the above embodiments of the present disclosure, based on the training log, the fault type of the distributed training is determined, including: extracting a target keyword in the training log; matching the target keyword with preset keywords of multiple preset types to determine a keyword that is successfully matched with the target keyword, wherein the preset keywords of the multiple preset types are used to represent keywords of different fault types; and determining a preset type corresponding to the keyword that is successfully matched with the target keyword from the multiple preset types as the fault type.

[0102] In the above embodiments of the present disclosure, the fault diagnosis module corresponding to the fault type is called to perform fault diagnosis on the fault node, and a fault detection result of the fault node is obtained, including: in the case that the fault type is a software fault, the software diagnosis module is called to extract fault prompt information in the training log, and the fault detection result is obtained; in the case that the fault type is a hardware fault, the hardware diagnosis module is called to perform hardware fault diagnosis on the fault node, and the fault detection result is obtained; and in the case that the fault type is a network fault, the network diagnosis module is called to perform fault diagnosis on the fault node and a switch connected to the fault node, and the fault detection result is obtained.

[0103] It should be noted that the preferred embodiments involved in the above embodiments of the present disclosure are the same as the schemes, application scenarios and implementation processes provided by the above embodiments, but are not limited to the schemes provided by the above embodiments.

[0104] According to the embodiments of the present disclosure, a server cluster fault detection method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0105] FIG. 8 is a flowchart of a server cluster fault detection method according to Embodiment 3 of the present disclosure, as shown in FIG. 8, the method can include the following steps:

[0106] In step S802, when it is monitored that the server cluster is interrupted, the running log before the server cluster is interrupted is acquired by calling a first interface, wherein the first interface comprises a first parameter, and a parameter value of the first parameter comprises the running log.

[0107] In step S804, based on the running log, the fault node in the server cluster and the fault type of the server cluster are determined.

[0108] In step S806, the fault diagnosis of the fault node is performed by calling the diagnosis module corresponding to the fault type, and the fault detection result of the fault node is obtained, wherein the fault detection result is used to represent the cause of the fault of the fault node.

[0109] In step S808, the fault detection result is output by calling a second interface, wherein the second interface comprises a second parameter, and a parameter value of the second parameter comprises the fault detection result.

[0110] It should be noted that the preferred embodiments involved in the above embodiments of the present disclosure are the same as the schemes, application scenarios and implementation processes provided in the above embodiments, but are not limited to the schemes provided in the above embodiments.

[0111] According to the embodiments of the present disclosure, a fault detection device of a server cluster for implementing the fault detection method of the server cluster is also provided, as shown in FIG. 9, the device 900 comprises a first acquisition component 902, a first determination component 904 and a first calling component 906.

[0112] The first acquisition component 902 is configured to acquire the running log before the server cluster is interrupted when it is monitored that the server cluster is interrupted; the first determination component 904 is configured to determine the fault node in the server cluster and the fault type of the server cluster based on the running log; and the first calling component 906 is configured to call the diagnosis module corresponding to the fault type to perform the fault diagnosis of the fault node, and obtain the fault detection result of the fault node, wherein the fault detection result is configured to represent the cause of the fault of the fault node.

[0113] It should be noted that the first acquisition component 902, the first determination component 904 and the first calling component 906 correspond to steps S402 to S406 in the above embodiments, and the three components are the same as the instances and application scenarios realized by the corresponding steps, but are not limited to the contents disclosed in the above embodiments. It should be noted that the above components or units can be hardware components or software components stored in a memory (for example, the memory 104) and processed by one or more processors (for example, the processors 102a, 102b, …, 102n), and the above components can also be a part of the device and can run in the computer terminal 10 provided in the above embodiments.

[0114] It should be noted that the preferred embodiments involved in the above embodiments of the present disclosure are the same as the schemes, application scenarios and implementation processes provided in the above embodiments, but are not limited to the schemes provided in the above embodiments.

[0115] According to the embodiments of the present disclosure, a server cluster fault detection device for implementing the above-mentioned server cluster fault detection method is further provided. As shown in FIG. 10, the device 1000 includes a second acquisition component 1002, a second determination component 1004, and a second calling component 1006.

[0116] The second acquisition component 1002 is configured to acquire a training log before the interruption of the distributed training of the neural network model occurs in the case that the interruption of the distributed training of the neural network model is monitored; the second determination component 1004 is configured to determine a fault node in the server cluster and a fault type of the distributed training based on the training log; and the second calling component 1006 is configured to call a diagnostic module corresponding to the fault type to perform fault diagnosis on the fault node, and obtain a fault detection result of the fault node, wherein the fault detection result is used to represent a cause of the fault of the fault node.

[0117] It should be noted that the second acquisition component 1002, the second determination component 1004, and the second calling component 1006 correspond to steps S702 to S706 in the above embodiments, and the three components have the same instances and application scenarios as the corresponding steps, but are not limited to the contents disclosed in the above embodiments. It should be noted that the above components or units can be hardware components or software components stored in a memory (for example, the memory 104) and processed by one or more processors (for example, the processors 102a, 102b, …, 102n), and the above components can also be a part of the device and can run in the computer terminal 10 provided in the above embodiments.

[0118] It should be noted that the preferred embodiments involved in the above embodiments of the present disclosure are the same as the schemes, application scenarios and implementation processes provided in the above embodiments, but are not limited to the schemes provided in the above embodiments.

[0119] According to the embodiments of the present disclosure, a server cluster fault detection device for implementing the above-mentioned server cluster fault detection method is further provided. As shown in FIG. 11, the device 1100 includes a third calling component 1102, a third determination component 1104, a fourth diagnostic component 1106, and a fifth calling component 1108.

[0120] The third calling component 1102 is configured to, in the case where it is monitored that the server cluster operation is interrupted, acquire the operation log before the server cluster operation is interrupted by calling the first interface, where the first interface includes a first parameter, and a parameter value of the first parameter includes the operation log. The third determining component 1104 is configured to determine, based on the operation log, a fault node in the server cluster and a fault type of the server cluster. The fourth diagnosing component 1106 is configured to call a diagnosis module corresponding to the fault type to perform fault diagnosis on the fault node, to obtain a fault detection result of the fault node, where the fault detection result is used to represent a cause of the fault of the fault node. The fifth calling component 1108 is configured to output the fault detection result by calling a second interface, where the second interface includes a second parameter, and a parameter value of the second parameter includes the fault detection result.

[0121] It should be noted that the third calling component 1102, the third determining component 1104, the fourth diagnosing component 1106, and the fifth calling component 1108 correspond to steps S802 to S808 in the above embodiment, and the four components have the same instances and application scenarios as the corresponding steps, but are not limited to the solutions disclosed in the above embodiment. It should be noted that the above components or units can be hardware components or software components stored in the memory (for example, the memory 104) and processed by one or more processors (for example, the processors 102a, 102b,..., 102n), and the above components can also be a part of the apparatus and can run in the computer terminal 10 provided in the above embodiment.

[0122] It should be noted that the preferred embodiments involved in the above embodiment of the present disclosure have the same solutions, application scenarios, and implementation processes as the above embodiment, but are not limited to the solutions provided in the above embodiment.

[0123] The embodiments of the present disclosure can provide a fault detection system of a server cluster, as shown in FIG. 12, which includes:

[0124] A plurality of diagnosis modules 1202 (only one is shown in the figure) are connected with the server cluster 1204, different diagnosis modules correspond to different fault types, and the plurality of diagnosis modules are used to perform fault diagnosis on the nodes in the server cluster.

[0125] A detection device 1206 is connected with the server cluster and the plurality of diagnosis modules, and is used to, in the case where it is monitored that the server cluster operation is interrupted, acquire the operation log before the server cluster operation is interrupted, and determine, based on the operation log, a fault node in the server cluster and a fault type of the server cluster, call a diagnosis module corresponding to the fault type to perform fault diagnosis on the fault node, and obtain a fault detection result of the fault node, where the fault detection result is used to represent a cause of the fault of the fault node.

[0126] It should be noted that the preferred embodiments involved in the above embodiments of the present disclosure are the same as the schemes, application scenarios and implementation processes provided by the above embodiments, but are not limited to the schemes provided by the above embodiments.

[0127] Embodiments of the present disclosure can provide an electronic device, which can be any one of the electronic devices in the electronic device group. Alternatively, in the present embodiment, the above-mentioned electronic device can also be replaced by a terminal device such as a mobile terminal.

[0128] Alternatively, in the present embodiment, the above-mentioned electronic device can be located in at least one network device of a plurality of network devices of a computer network.

[0129] In the present embodiment, the above-mentioned electronic device can execute program codes in the method.

[0130] Alternatively, FIG. 13 is a structural block diagram of an electronic device according to an embodiment of the present disclosure. As shown in FIG. 13, the electronic device A can include one or more (only one is shown in the figure) processors 1302, a memory 1304, a storage controller, and a peripheral interface, wherein the peripheral interface is connected with a radio frequency component, an audio component, and a display.

[0131] The memory can be used to store software programs and modules, such as program instructions / modules corresponding to the method and device in the embodiments of the present disclosure. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, implements the method in the above embodiments. The memory can include a high-speed random access memory, and can also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory can further include a memory remotely arranged with respect to the processor, which can be connected to the terminal A through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0132] The processor can call information and application programs stored in the memory through the transmission device to perform the following steps: in the case that the running interruption of the server cluster is monitored, the running log before the running interruption of the server cluster is obtained; based on the running log, the fault node in the server cluster and the fault type of the server cluster are determined; the fault node is diagnosed by calling the diagnostic module corresponding to the fault type to obtain the fault detection result of the fault node, wherein the fault detection result is used to represent the reason for the fault of the fault node.

[0133] Optionally, the processor can further execute program codes of the following steps: parsing the running log to obtain an abnormal event contained in the running log, wherein the abnormal event is used to represent an event corresponding to an abnormality of the server cluster; determining network address information contained in the abnormal event; determining a node corresponding to the network address information as the fault node from the server cluster.

[0134] Optionally, the processor can further execute program codes of the following steps: extracting a target keyword in the running log; matching the target keyword with preset keywords of a plurality of preset types to determine a keyword matched successfully with the target keyword, wherein the preset keywords of the plurality of preset types are used to represent keywords of different fault types; determining a preset type corresponding to the keyword matched successfully with the target keyword from the plurality of preset types as the fault type.

[0135] Optionally, the processor can further execute program codes of the following steps: in a case where the fault type is a software fault, extracting fault prompt information in the running log by calling a software diagnosis module to obtain a fault detection result; in a case where the fault type is a hardware fault, performing hardware fault diagnosis on the fault node by calling a hardware diagnosis module to obtain the fault detection result; in a case where the fault type is a network fault, performing fault diagnosis on the fault node and a switch connected to the fault node by calling a network diagnosis module to obtain the fault detection result.

[0136] Optionally, the processor can further execute program codes of the following steps: determining a fault elimination manner of the fault node based on the fault detection result; and performing a fault elimination operation on the fault node based on the fault elimination manner.

[0137] Optionally, the processor can further execute program codes of the following steps: in a case where the fault elimination manner is not isolating the fault node, outputting first prompt information, wherein the first prompt information is used to prompt updating code data in the fault node or controlling the fault node to re-execute a task; in a case where the fault elimination manner is hot repairing the fault node, outputting second prompt information, wherein the second prompt information is used to prompt hot repairing the fault node; in a case where the fault elimination manner is isolating and repairing the fault node, isolating the fault node, and outputting third prompt information in a case where the fault node is successfully isolated, wherein the third prompt information is used to prompt repairing the fault node.

[0138] The processor can call information and application programs stored in the memory through the transmission device to perform the following steps: in the case that the distributed training of the neural network model is monitored to be interrupted, obtaining the training log before the neural network model is interrupted; based on the training log, determining the fault node in the server cluster and the fault type of the distributed training; by calling the diagnostic module corresponding to the fault type to diagnose the fault node, obtaining the fault detection result of the fault node, wherein the fault detection result is used to represent the reason for the fault of the fault node.

[0139] The processor can call information and application programs stored in the memory through the transmission device to perform the following steps: in the case that the distributed training of the neural network model is monitored to be interrupted, obtaining the training log before the neural network model is interrupted; based on the training log, determining the fault node in the server cluster and the fault type of the distributed training; by calling the diagnostic module corresponding to the fault type to diagnose the fault node, obtaining the fault detection result of the fault node, wherein the fault detection result is used to represent the reason for the fault of the fault node.

[0140] In the case that the server cluster is monitored to be interrupted, the running log before the server cluster is interrupted is obtained, and then based on the running log, the fault node in the server cluster and the fault type of the server cluster are determined, and finally the diagnostic module corresponding to the fault type is called to diagnose the fault node, and the fault detection result of the fault node is obtained, which realizes the purpose of fault positioning of the server cluster. It is easy to note that the server cluster can be fault detected by the combination of the running log and the diagnostic module, and the cause of the fault is automatically located, which not only achieves the technical effects of improving fault detection accuracy, reducing fault omission rate and improving fault detection efficiency, thereby providing a basis for subsequent node isolation and task recovery, effectively shortening the time of wasting computing resources caused by faults, and further solving the technical problems of low fault detection efficiency and accuracy of the server cluster in the related art.

[0141] Those skilled in the art can understand that the structure as shown in the figure is only schematic, and the electronic device can also be a terminal device such as a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a palm computer, a Mobile Internet Device (MID), a PAD, etc. The figure does not limit the structure of the above-mentioned electronic device. For example, the electronic device A can further include more or less components (such as a network interface, a display device, etc.) than those shown in the figure, or have a different configuration from that shown in the figure.

[0142] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiments can be completed by instructing the hardware related to the terminal device by a program, and the program can be stored in a computer readable storage medium, which can include a flash disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, etc.

[0143] The embodiments of the present disclosure further provide a computer readable storage medium. Optionally, in the present embodiment, the above-mentioned computer readable storage medium can be used to save the program code executed by the method provided by the above-mentioned embodiments.

[0144] Optionally, in the present embodiment, the above-mentioned storage medium can be located in any one of the electronic devices in the group of electronic devices in the computer network, or in any one of the mobile terminals in the group of mobile terminals.

[0145] Optionally, in the present embodiment, the computer readable storage medium is configured to store program code for performing the following steps: in the case that the server cluster running interruption is monitored, obtaining the running log before the server cluster running interruption; based on the running log, determining the fault node in the server cluster and the fault type of the server cluster; performing fault diagnosis on the fault node by calling the diagnosis module corresponding to the fault type to obtain the fault detection result of the fault node, wherein the fault detection result is used to represent the reason for the fault of the fault node.

[0146] Optionally, the computer readable storage medium is further configured to store program code for performing the following steps: analyzing the running log to obtain an abnormal event contained in the running log, wherein the abnormal event is used to represent an event corresponding to the abnormality of the server cluster; determining network address information contained in the abnormal event; determining the node corresponding to the network address information from the server cluster as the fault node.

[0147] Optionally, the computer readable storage medium is further configured to store program code for performing the following steps: extracting a target keyword in the running log; matching the target keyword with preset keywords of multiple preset types, determining keywords that are successfully matched with the target keyword, wherein the preset keywords of the multiple preset types are used to represent keywords of different fault types; determining a preset type corresponding to the keywords that are successfully matched with the target keyword from the multiple preset types, and taking the preset type as the fault type.

[0148] Optionally, the computer readable storage medium is further configured to store program code for performing the following steps: in a case where the fault type is a software fault, extracting fault prompt information in the running log by calling a software diagnosis module to obtain a fault detection result; in a case where the fault type is a hardware fault, performing hardware fault diagnosis on the fault node by calling a hardware diagnosis module to obtain the fault detection result; in a case where the fault type is a network fault, performing fault diagnosis on the fault node and a switch connected to the fault node by calling a network diagnosis module to obtain the fault detection result.

[0149] Optionally, the computer readable storage medium is further configured to store program code for performing the following steps: determining a fault elimination manner of the fault node based on the fault detection result; and performing a fault elimination operation on the fault node based on the fault elimination manner.

[0150] Optionally, the computer readable storage medium is further configured to store program code for performing the following steps: in a case where the fault elimination manner is not isolating the fault node, outputting first prompt information, wherein the first prompt information is used to prompt updating code data in the fault node or controlling the fault node to re-execute a task; in a case where the fault elimination manner is hot repairing the fault node, outputting second prompt information, wherein the second prompt information is used to prompt hot repairing the fault node; in a case where the fault elimination manner is isolating and repairing the fault node, isolating the fault node, and outputting third prompt information in a case where the fault node is successfully isolated, wherein the third prompt information is used to prompt repairing the fault node.

[0151] Optionally, in the embodiment, the computer readable storage medium is configured to store program code for performing the following steps: in a case where it is monitored that the distributed training of the neural network model is interrupted, obtaining a training log before the distributed training is interrupted; based on the training log, determining a fault node in a server cluster and a fault type of the server cluster; and performing fault diagnosis on the fault node by calling a diagnosis module corresponding to the fault type to obtain a fault detection result of the fault node, wherein the fault detection result is used to represent a cause of the fault of the fault node.

[0152] Optionally, in the embodiment, the computer readable storage medium is configured to store program code for performing the following steps: in the case of monitoring the server cluster running interruption, obtaining the running log before the server cluster running interruption by calling the first interface, wherein the first interface comprises a first parameter, and a parameter value of the first parameter comprises the running log; determining the fault node in the server cluster and the fault type of the server cluster based on the running log; performing fault diagnosis on the fault node by calling the diagnostic module corresponding to the fault type to obtain a fault detection result of the fault node, wherein the fault detection result is used to represent the cause of the fault of the fault node; and outputting the fault detection result by calling the second interface, wherein the second interface comprises a second parameter, and a parameter value of the second parameter comprises the fault detection result.

[0153] Embodiments of the present disclosure further provide a computer program product. Optionally, in the embodiment, the computer program product can include a computer program, and the computer program, when executed by a processor, implements the method provided by the above embodiments.

[0154] Embodiments of the present disclosure further provide a computer program product. Optionally, the computer program product can include a non-volatile computer readable storage medium, and the non-volatile computer readable storage medium can be used to store a computer program, and the computer program, when executed by a processor, implements the method provided by the above embodiments.

[0155] Embodiments of the present disclosure further provide a computer program. Optionally, in the embodiment, the computer program, when executed by a processor, implements the method provided by the above embodiments.

[0156] In the above embodiments of the present disclosure, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0157] In several embodiments provided by the present disclosure, it should be understood that the disclosed technology can be implemented in other ways. Of course, the embodiment described above is only schematic. For example, the division of units is only a logical function division, and there can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, units or modules, and can be electrical or other forms.

[0158] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e. may be located in one place, or may be distributed on multiple network units. Part or all of the units may be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0159] In addition, each functional unit in various embodiments of the present disclosure can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0160] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on this understanding, the technical scheme of the present disclosure essentially or the part that contributes to the prior art or the whole or part of the technical scheme can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in various embodiments of the present disclosure. The foregoing storage medium includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various program code storage media.

[0161] The above is only the preferred embodiment of the present disclosure, and it should be pointed out that for those skilled in the art, without departing from the principles of the present disclosure, a number of improvements and refinements can be made, and these improvements and refinements should be considered as the protection scope of the present disclosure. Industrial applicability

[0162] The scheme provided by the embodiments of the present disclosure can be applied in the fault detection process of a server cluster. In the case that the running of the server cluster is interrupted, the running log before the running of the server cluster is interrupted is obtained, then based on the running log, the fault node in the server cluster and the fault type of the server cluster are determined, finally the fault node is diagnosed by calling the diagnosis module corresponding to the fault type to obtain the fault detection result of the fault node, and the purpose of fault positioning of the server cluster is achieved. It is easy to note that the server cluster can be detected for fault by the combination of the running log and the diagnosis module, and the cause of the fault is automatically located. Not only the technical effects of improving the fault detection accuracy, reducing the fault omission rate and improving the fault detection efficiency are achieved, but also the basis for subsequent node isolation and task recovery is provided, the time of the idle consumption of the computing resources of the cluster caused by the fault is effectively shortened, and thus the technical problem of low fault detection efficiency and accuracy of the server cluster in the related art is solved.

Claims

1. A method for fault detection of a server cluster, comprising: in the case of monitoring that the server cluster is interrupted, obtaining a running log before the server cluster is interrupted; based on the running log, determining a fault node in the server cluster and a fault type of the server cluster; by calling a diagnosis module corresponding to the fault type to diagnose the fault node, obtaining a fault detection result of the fault node, wherein the fault detection result is used to represent the cause of the fault of the fault node.

2. The method of claim 1, wherein, The determination of the fault node in the server cluster based on the running log comprises: parsing the running log to obtain an abnormal event contained in the running log, wherein the abnormal event is used to represent an event corresponding to an abnormality of the server cluster; determining network address information contained in the abnormal event; determining the node corresponding to the network address information in the server cluster as the fault node.

3. The method of claim 1, wherein, The determination of the fault type of the server cluster based on the running log comprises: extracting a target keyword in the running log; matching the target keyword with a plurality of preset keywords of a plurality of preset types to determine a keyword matched successfully with the target keyword, wherein the plurality of preset keywords of the plurality of preset types are used to represent keywords of different fault types; determining a preset type corresponding to the keyword matched successfully with the target keyword from the plurality of preset types as the fault type.

4. The method of claim 1, wherein, The fault diagnosis of the fault node by calling the diagnosis module corresponding to the fault type comprises: in the case of the fault type being a software fault, extracting fault prompt information in the running log by calling a software diagnosis module to obtain the fault detection result; in the case of the fault type being a hardware fault, performing hardware fault diagnosis on the fault node by calling a hardware diagnosis module to obtain the fault detection result; in the case of the fault type being a network fault, performing fault diagnosis on the fault node and a switch connected to the fault node by calling a network diagnosis module to obtain the fault detection result.

5. The method according to any one of claims 1 to 4, wherein, The method further comprises: based on the fault detection result, determining a fault elimination method of the fault node; based on the fault elimination method, performing a fault elimination operation on the fault node.

6. The method of claim 5, wherein, The fault elimination operation on the fault node based on the fault elimination method comprises: in the case of the fault elimination method being not needing to isolate the fault node, outputting a first prompt information, wherein the first prompt information is used to prompt updating code data in the fault node or controlling the fault node to re-execute a task; in the case of the fault elimination method being hot maintenance of the fault node, outputting a second prompt information, wherein the second prompt information is used to prompt hot maintenance of the fault node. In a case where the troubleshooting mode is isolation and maintenance of the faulty node, the faulty node is isolated, and third prompt information is output in a case where the isolation of the faulty node is successful, where the third prompt information is used to prompt maintenance of the faulty node.

7. A fault detection method of a server cluster, the server cluster being used for distributed training of a neural network model, the method comprising: in a case where it is monitored that the distributed training of the neural network model is interrupted, obtaining a training log before the distributed training is interrupted; based on the training log, determining a faulty node in the server cluster and a fault type of the distributed training; by calling a diagnosis module corresponding to the fault type, performing fault diagnosis on the faulty node to obtain a fault detection result of the faulty node, where the fault detection result is used to represent a cause of the fault of the faulty node.

8. The method of claim 7, wherein, The determination of the faulty node in the server cluster based on the training log comprises: parsing the training log to obtain an abnormal event contained in the training log, where the abnormal event is used to represent an event corresponding to an abnormality of the server cluster; determining network address information contained in the abnormal event; determining, from the server cluster, a node corresponding to the network address information as the faulty node.

9. The method of claim 7, wherein, The determination of the fault type of the distributed training based on the training log comprises: extracting a target keyword in the training log; matching the target keyword with preset keywords of multiple preset types to determine a keyword matched successfully with the target keyword, where the preset keywords of the multiple preset types are used to represent keywords of different fault types; from the multiple preset types, determining a preset type corresponding to the keyword matched successfully with the target keyword as the fault type.

10. The method of claim 7, wherein, The fault diagnosis on the faulty node by calling the diagnosis module corresponding to the fault type to obtain the fault detection result of the faulty node comprises: in a case where the fault type is a software fault, extracting fault prompt information in the training log by calling a software diagnosis module to obtain the fault detection result; in a case where the fault type is a hardware fault, performing hardware fault diagnosis on the faulty node by calling a hardware diagnosis module to obtain the fault detection result; in a case where the fault type is a network fault, performing fault diagnosis on the faulty node and a switch connected with the faulty node by calling a network diagnosis module to obtain the fault detection result.

11. A fault detection method of a server cluster, comprising: in a case where it is monitored that the server cluster is interrupted, obtaining a running log before the server cluster is interrupted by calling a first interface, where the first interface comprises a first parameter, and a parameter value of the first parameter comprises the running log; based on the running log, determining a faulty node in the server cluster and a fault type of the server cluster; obtaining a fault detection result of the fault node by calling a diagnosis module corresponding to the fault type, wherein the fault detection result is used to represent a cause of the fault of the fault node; outputting the fault detection result by calling a second interface, wherein the second interface comprises a second parameter, and a parameter value of the second parameter comprises the fault detection result.

12. A fault detection system of a server cluster, comprising: a plurality of diagnosis modules connected to the server cluster, different diagnosis modules corresponding to different fault types, wherein the plurality of diagnosis modules are used to diagnose faults of nodes in the server cluster; a detection device connected to the server cluster and the plurality of diagnosis modules, configured to, in a case where a running interruption of the server cluster is monitored, obtain a running log before the running interruption of the server cluster, determine a fault node in the server cluster and a fault type of the server cluster based on the running log, call a diagnosis module corresponding to the fault type to diagnose faults of the fault node, and obtain a fault detection result of the fault node, wherein the fault detection result is used to represent a cause of the fault of the fault node.

13. The system of claim 12, wherein, The detection device is further configured to: analyze the running log to obtain an abnormal event contained in the running log, wherein the abnormal event is used to represent an event corresponding to an abnormality of the server cluster; determine network address information contained in the abnormal event; determine a node corresponding to the network address information in the server cluster as the fault node.

14. The system of claim 12, wherein, The detection device is further configured to: extract a target keyword in the running log; match the target keyword with preset keywords of a plurality of preset types to determine a keyword matched successfully with the target keyword, wherein the preset keywords of the plurality of preset types are used to represent keywords of different fault types; determine a preset type corresponding to the keyword matched successfully with the target keyword from the plurality of preset types as the fault type.

15. The system of claim 12, wherein, The detection device is further configured to: in a case where the fault type is a software fault, extract fault prompt information in the running log by calling a software diagnosis module to obtain the fault detection result; in a case where the fault type is a hardware fault, perform hardware fault diagnosis on the fault node by calling a hardware diagnosis module to obtain the fault detection result; in a case where the fault type is a network fault, perform fault diagnosis on the fault node and a switch connected to the fault node by calling a network diagnosis module to obtain the fault detection result.

16. The system of any one of claims 12 to 15, wherein, The system is further configured to: determine a fault elimination manner of the fault node based on the fault detection result; perform a fault elimination operation on the fault node based on the fault elimination manner.

17. The system of claim 16, wherein, The system is further configured to: In a case where the troubleshooting manner is that the fault node does not need to be isolated, first prompt information is output, wherein the first prompt information is used to prompt updating code data in the fault node or controlling the fault node to re-execute a task; In a case where the troubleshooting manner is that the fault node is hot repaired, second prompt information is output, wherein the second prompt information is used to prompt hot repairing the fault node; In a case where the troubleshooting manner is that the fault node is isolated and repaired, the fault node is isolated, and third prompt information is output in a case where the fault node is successfully isolated, wherein the third prompt information is used to prompt repairing the fault node. 18.An electronic device, comprising: a memory storing an executable program; a processor configured to run the program, wherein the program performs the method in any one of claims 1 to 11 when running.

19. A computer readable storage medium comprising a stored executable program, wherein, controlling a device where the storage medium is to perform the method in any one of claims 1 to 11 when the executable program runs. 20.A computer program product, comprising a computer program which, when executed by a processor, implements the method in any one of claims 1 to 11.

Images