Fault processing method and device, equipment, storage medium and computer program product

Abstract

The invention provides a fault processing method and device, equipment, a storage medium and a computer program product. The method comprises the steps of obtaining a running log of a host machine of at least one virtualization cluster; based on the operation logs of the host machines of the at least one virtualization cluster, determining fault host machines from the host machines of the at least one virtualization cluster and fault types corresponding to the fault host machines; classifying the fault host machines based on the fault types to obtain at least one group of fault host machines; determining a fault processing strategy corresponding to each group of fault host machines based on the fault type corresponding to each group of fault host machines; and respectively processing each group of fault host machines based on the corresponding fault processing strategy, and recovering the virtual machines running on each group of fault host machines. According to the method and the device, the automation degree of fault host machine processing can be improved, the situation of invalid fault processing is reduced, the processing efficiency of the fault host machine is improved, and the service environment provided by each virtualization cluster runs stably.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a fault handling method, apparatus, device, storage medium and computer program product. Background Art

[0002] With the widespread application of virtualization technology, the host failure handling of virtualization clusters in large-scale cloud computing data has become an important issue. Due to the host failures involving multiple virtualization clusters and multiple types of failures, the processing workload will increase when handling the faulty host, and the processing efficiency of the faulty host is low. Summary of the invention

[0003] The embodiments of the present application provide a fault handling method, apparatus, device, storage medium and computer program product, which can improve the automation level of fault host machine processing, reduce invalid fault processing, improve the processing efficiency of faulty host machines, and make the service environment provided by each virtualization cluster run stably.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] In a first aspect, an embodiment of the present application provides a fault handling method, the method comprising:

[0006] Obtaining an operation log of a host machine of at least one virtualization cluster; wherein each host machine runs at least one virtual machine;

[0007] Based on the operation log of the host machine of the at least one virtualization cluster, determining the faulty host machine and the fault type corresponding to each faulty host machine from the host machines of the at least one virtualization cluster;

[0008] Classifying the faulty host machines based on the fault type to obtain at least one group of faulty host machines; wherein the faulty host machines in the same group have the same fault type;

[0009] Based on the fault types corresponding to the faulty host machines in each group, determining the fault handling strategies corresponding to the faulty host machines in each group; wherein one fault handling strategy corresponds to one group of faulty host machines;

[0010] Each group of faulty host machines is processed separately based on the corresponding fault handling strategy, and the virtual machines running on each group of faulty host machines are recovered.

[0011] In a second aspect, an embodiment of the present application provides a fault handling device, including:

[0012] An acquisition module is configured to acquire an operation log of a host machine of at least one virtualization cluster; wherein each host machine runs at least one virtual machine;

[0013] A first determination module is configured to determine a faulty host machine and a fault type corresponding to each faulty host machine from the host machines of the at least one virtualization cluster based on the operation log of the host machine of the at least one virtualization cluster;

[0014] A classification module is configured to classify the faulty host machines based on the fault type to obtain at least one group of faulty host machines; wherein the faulty host machines in the same group have the same fault type;

[0015] The second determination module is configured to determine the fault handling strategy corresponding to each group of faulty host machines based on the fault type corresponding to each group of faulty host machines; wherein a group of faulty host machines corresponds to a fault handling strategy; the processing module is configured to process the corresponding groups of faulty host machines respectively based on the corresponding fault handling strategy, and to restore the virtual machines running on each group of faulty host machines. In a third aspect, an embodiment of the present application provides an electronic device, comprising:

[0016] Memory for storing computer programs or executable instructions;

[0017] The processor is used to implement the method provided in the first aspect of the embodiment of the present application when executing the computer program or executable instructions stored in the memory.

[0018] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having stored thereon a computer program or executable instructions, which, when executed by a processor, implements the method provided in the first aspect of the embodiment of the present application.

[0019] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program or executable instructions. When the computer program or executable instructions are executed by a processor, the method provided in the first aspect of the embodiment of the present application is implemented.

[0020] The embodiments of the present application have the following beneficial effects:

[0021] On the one hand, the operation logs of the hosts of each virtualization cluster can be used to automatically detect faults on faulty hosts without the need for manual fault detection, thereby improving the detection efficiency of faulty hosts while reducing labor and time costs; by grouping the faulty hosts of each virtualization cluster, faulty host groups corresponding to different fault types can be obtained, so that faulty hosts can be processed in batches, thereby reducing the accumulation of processing tasks on faulty hosts and improving the degree of automation in handling faulty hosts.

[0022] On the other hand, since the fault handling strategies corresponding to different fault types are stored in advance, after obtaining each group of faulty host machines, each group of faulty host machines can be handled according to the corresponding fault handling strategy, thereby reducing the situation of invalid fault handling and improving the processing efficiency of faulty host machines. At the same time, the virtual machines running on each group of faulty host machines can be restored, further optimizing the processing procedures of faulty host machines, so that the service environment provided by each virtualization cluster can run stably. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a schematic diagram of the architecture of a fault handling system provided in an embodiment of the present application. Figure 1 ;

[0024] Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0025] Figure 3 This is a flowchart of the fault handling method provided in the embodiment of the present application. Figure 1 ;

[0026] Figure 4 This is a flowchart of the fault handling method provided in the embodiment of the present application. Figure 2 ;

[0027] Figure 5 This is a schematic diagram of the architecture of a fault handling system provided in an embodiment of the present application. Figure 2 ;

[0028] Figure 6 This is a flowchart of the fault handling method provided in the embodiment of the present application. Figure 3 . DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0030] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0031] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0033] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0034] (1) A virtualization cluster is a computing resource pool composed of multiple host machines through virtualization technology. Each host machine runs virtualization software. Administrators can use these virtualization software to create and manage virtual machines on the host machines according to business needs, such as kernel-based virtual machines (KVM) and XEN virtual machines, and ensure the stability and performance of the system through the cluster's load balancing and fault tolerance mechanisms.

[0035] (2) Host refers to the physical host that runs virtualization software and hosts virtual machines. The host provides hardware resources and an operating environment so that virtual machines can be created, run, and managed on the operating environment. There is a master-slave relationship between the host and the virtual machines. As a physical host, the host creates, manages, and hosts multiple virtual machines through virtualization technology. Each virtual machine runs on the host and is isolated from each other. The host is responsible for allocating and managing the physical resources they share. Through virtualization technology, the host can dynamically allocate and adjust resources, thereby improving the flexibility, scalability, and resource utilization of the system.

[0036] (3) KVM runs the virtual machine as a Linux process of the operating system kernel, and can utilize the scheduler and other functions of the Linux kernel. At the same time, it uses the simulated processor software QEMU as a virtual machine monitor to provide a virtualized environment for each virtual machine. XEN introduces a hypervisor layer between the bare metal (i.e., computer hardware without an operating system or virtualization software installed) and the operating system to manage virtual machines and physical hardware resources.

[0037] When a virtualization cluster host fails, some problems may occur. For example, the virtual machines running on the failed host will be affected, causing related services to be unavailable, resulting in business interruption for enterprises that rely on related services. For another example, a host failure may cause the system to be unstable, exposing more security vulnerabilities, causing the key data of the host and its virtual machines to be stolen or tampered with, thus posing a serious security threat to the enterprise. Therefore, timely troubleshooting of the failed host is an urgent task.

[0038] In the related art, the fault handling method in which engineers perform fault detection on the host machine and perform on-site repair by the engineers is not conducive to improving the processing efficiency of the faulty host machine when there are multiple host machine failures in the virtualization cluster, because the steps of manual fault detection on the faulty host machine are cumbersome, and it is easy to make misjudgments or misoperations during manual operation. At the same time, since the fault handling of the faulty host machine depends on the response and operation of the engineer, when the number of faulty host machines is large, the engineer cannot meet the processing requirements of the faulty host machine quickly responding to the faulty host machine around the clock, resulting in the accumulation of processing tasks for the faulty host machine, and thus the timeliness of processing the faulty host machine cannot be met.

[0039] Based on the above problems, the embodiments of the present application provide a fault handling method, apparatus, device, computer-readable storage medium and computer program product, which can automatically detect faults on faulty host machines through the operation logs of the host machines of each virtualization cluster, thereby reducing the possibility of misjudgment or misoperation during manual operation; at the same time, fault handling strategies corresponding to different fault types are pre-stored, and the faulty host machine can be handled according to the corresponding fault handling strategies, thereby reducing invalid fault handling and improving the processing efficiency of the faulty host machine.

[0040] The electronic device provided in the embodiment of the present application for implementing the above-mentioned fault handling method is described below. The electronic device provided in the embodiment of the present application can be implemented as various types of terminal devices such as a laptop computer, a desktop computer, a mobile device, etc., and can also be implemented as a background server of a network platform (hereinafter referred to as a server). Below, an exemplary application when the electronic device is implemented as a server of a network platform will be described.

[0041] See also Figure 1 , Figure 1 This is a schematic diagram of the architecture of a fault handling system provided in an embodiment of the present application. Figure 1 To implement the fault handling method provided in the embodiment of the present application, the terminal device 101 (the number of terminal devices included in the fault handling system of the embodiment of the present disclosure is not limited) is connected to the server 102 via the network 103. The network 103 can be a wide area network or a local area network, or a combination of the two.

[0042] In some possible implementations, a user can log in to an application corresponding to the network platform through a terminal device 101, and the terminal device 101 interacts with the server 102 through a network 103; the server 102 can automatically detect faulty host machines based on the operation logs of the host machines of each virtualization cluster through the network 103, and group the faulty host machines. After the server 102 obtains each group of faulty host machines, it can process each group of faulty host machines according to a corresponding fault processing strategy, and restore the virtual machines running on each group of faulty host machines, and send the processing results to the terminal device 101.

[0043] In some embodiments, the server 102 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, and big data and artificial intelligence platforms. The terminal device 101 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart watch, etc., but is not limited thereto. The terminal device and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0044] See also Figure 2 , Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application; wherein, Figure 2 The electronic device 200 shown includes: at least one processor 210, a memory 250, at least one network interface 220 and a user interface 230. The various components in the electronic device 200 are coupled together via a bus system 240. It is understood that the bus system 240 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 240 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 240 is not described in detail. Figure 2 Various buses are labeled as bus system 240 .

[0045] The processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0046] The user interface 230 includes one or more output devices 231 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 230 also includes one or more input devices 232, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0047] The memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 250 may optionally include one or more storage devices that are physically remote from the processor 210.

[0048] The memory 250 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 250 described in the embodiments of the present application is intended to include any suitable type of memory.

[0049] In some embodiments, memory 250 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0050] Operating system 251, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0051] A network communication module 252, for reaching other computing devices via one or more (wired or wireless) network interfaces 220, exemplary network interfaces 220 include: Bluetooth, wireless compatibility certification (WiFi), and universal serial bus (USB), etc.;

[0052] a presentation module 253 for enabling presentation of information via one or more output devices 231 (e.g., display screen, speaker, etc.) associated with the user interface 230 (e.g., a user interface for operating peripherals and displaying content and information);

[0053] The input processing module 254 is used to detect one or more user inputs or interactions from one of the one or more input devices 232 and translate the detected inputs or interactions.

[0054] In some embodiments, the fault handling device provided in the embodiments of the present application can be implemented in a software manner. Figure 2 The fault handling device 255 stored in the memory 250 is shown, which can be software in the form of a program and a plug-in, etc., and includes the following software modules: an acquisition module 2551, a first determination module 2552, a classification module 2553, a second determination module 2554 and a processing module 2555. These modules are logical, and therefore can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0055] In other embodiments, the fault handling device provided in the embodiments of the present application can be implemented in hardware. As an example, the device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the fault handling method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD), field programmable gate array (FPGA) or other electronic components.

[0056] The fault handling method provided in the embodiment of the present application will be described below in conjunction with the exemplary application and implementation of the electronic device provided in the embodiment of the present application.

[0057] See also Figure 3 , Figure 3 This is a flowchart of the fault handling method provided in the embodiment of the present application. Figure 1 ,like Figure 3 As shown, the following steps may be included:

[0058] In step 301, the operation log of at least one host machine of the virtualization cluster is obtained; wherein each host machine runs at least one virtual machine;

[0059] In step 302, based on the operation log of the host machine of at least one virtualization cluster, a faulty host machine and a fault type corresponding to each faulty host machine are determined from the host machines of at least one virtualization cluster;

[0060] In step 303, the faulty host machines are classified based on the fault type to obtain at least one group of faulty host machines; wherein the faulty host machines in the same group have the same fault type.

[0061] In step 304, based on the fault types corresponding to the faulty host machines in each group, a fault handling strategy corresponding to each group of faulty host machines is determined; wherein one group of faulty host machines corresponds to one fault handling strategy;

[0062] In step 305, each group of failed host machines is processed separately based on the corresponding fault handling strategy, and the virtual machines running on each group of failed host machines are restored.

[0063] It should be noted that when there are host failures in multiple virtualization clusters, classifying and processing each faulty host machine can improve the processing efficiency of the faulty host machine. Therefore, the embodiment of the present disclosure can first classify the faulty host machines to obtain multiple groups of faulty host machines, and then perform fault processing based on the fault processing strategies corresponding to each group of faulty host machines.

[0064] Since the operation log of the host machine of the virtualization cluster records the startup and shutdown information, hardware status information, operation errors and warning information of the host machine, the operation log can reflect the operation status and operation performance of the host machine, so as to further determine whether the host machine is in a fault state.

[0065] Here, each virtualization cluster may correspond to an operation log, which records the operation information of each host machine in the virtualization cluster; or each host machine may correspond to an operation log, and the operation log of each host machine records the operation information of the corresponding host machine. The embodiments of the present disclosure are not limited to this.

[0066] In some embodiments, for Linux systems, operation logs are stored in the / var / log directory, such as / var / log / syslog, / var / log / messages, and commands such as cat, less, or tail can be used to view the operation logs. Here, a log processing tool (Remote System Logging, Rsyslog) is used to collect, process, and forward logs on a Linux system. Using multi-threading technology, log data of the operation log can be processed in parallel on multiple processors to improve the processing performance and response speed of the operation log. Therefore, the operation log of the host machine can be obtained through Rsyslog.

[0067] Here, the operation logs of the hosts of each virtualization cluster can be stored in the same / var / log directory or in different / var / log directories.

[0068] For example, in actual use, the operation log with the ID of the management controller (Baseboard Management Controller, BMC) log is obtained from the / var / log directory through Rsyslog to obtain the operation log of the host machine of the virtualization cluster.

[0069] It is understandable that after obtaining the operation logs of each host machine, since the operation logs contain hardware-related operation information and / or alarm information, the faulty host machines can be screened out from the various host machines based on the operation logs, and the fault types of the faulty host machines can be determined, thereby laying the foundation for subsequent batch processing of the faulty host machines and reducing human intervention.

[0070] Here, hardware includes but is not limited to a processor CPU, storage device, memory or network interface card, etc.

[0071] In some embodiments, in order to quickly screen the faulty host machines, after obtaining the operation log of the host machine, the key information in the operation log can be located by extracting keywords, such as "hardware name and alarm information" and other related keywords. After obtaining the above related keywords, it can be quickly determined that the host machine is in a fault state. Here, the alarm information includes but is not limited to the alarm level, the alarm duration and the cumulative number of alarms.

[0072] In some embodiments, the operation information related to the hardware in the operation log can reflect the operation status of the hardware, and the operation status of each hardware can be combined to determine not only whether the host machine is in a faulty state, but also the fault type of the faulty host machine. For example, if the operation log records information such as memory check error or insufficient memory, it can be determined that the fault type of the faulty host machine is a memory fault; if the operation log records information such as abnormal CPU occupancy or excessive CPU temperature, it can be determined that the fault type of the faulty host machine is a CPU fault.

[0073] In other embodiments, the alarm information in the operation log is caused by the failure or operation error of the host machine, for example, hard disk failure, memory error or CPU overheating will cause hardware alarm, kernel error, system file error or network anomaly will cause system abnormality alarm, high CPU usage and disk I / O performance degradation will cause performance bottleneck alarm, therefore, when the alarm information appears in the operation log, it can be determined that the host machine is in a faulty state. Since the alarm frequency and alarm level in the alarm information can reflect the fault degree of the faulty host machine, it is helpful to judge the fault type of the faulty host machine.

[0074] Here, since the fault handling of faulty hosts with the same fault type is similar, in order to improve the processing efficiency of the faulty hosts, after determining the fault type of each faulty host, the faulty hosts with the same fault type can be grouped together. In this way, groups of faulty hosts with different fault types can be obtained, so as to facilitate batch processing of each group of faulty hosts.

[0075] It can be understood that fault handling strategies corresponding to different fault types are pre-stored. After obtaining each group of faulty host machines, each group of faulty host machines can be processed according to the corresponding fault handling strategy, which can reduce the situation of invalid fault handling and help improve the reliability of the processing flow of the faulty host machines.

[0076] Here, different fault types correspond to different fault handling strategies. For example, (1) when the system is down, the fault handling strategy at least includes: checking hardware faults (i.e., the power supply, hard disk, memory or CPU of the host machine), checking software problems (i.e., checking whether there are software conflicts or configuration errors), restarting the faulty host machine, and restoring the faulty host machine; (2) when the memory fails, the fault handling strategy at least includes: checking the memory stick, updating the driver, checking the system settings, and performing a memory test; (3) when the CPU fails, the fault handling strategy at least includes: checking the cooling system, checking the installation and settings of the CPU, checking the device compatibility, and replacing the CPU; (4) when the power supply fails, the fault handling strategy at least includes: using a multimeter to test the input voltage and output voltage of the power supply, checking the power connection cable, reconnecting the power supply, and restarting the faulty host machine; (5) when the network card fails, the fault handling strategy at least includes: updating the driver, checking the system settings, restarting the network service, and checking the network card hardware; (6) when the data disk fails, the fault handling strategy at least includes: checking the hardware connection, checking the settings of the boot firmware on the motherboard, running a disk check, and replacing the data disk.

[0077] In some embodiments, in order to improve the automation level of the host processing flow, a host fault detection system may be set up, and the host fault detection system includes a configuration center, and the configuration center is used to configure the address information of the virtualization cluster, the host fault detection flow, the fault handling strategies corresponding to different fault types, and the handling personnel, etc. In this way, after the fault detection system performs fault detection on the faulty host, the corresponding fault handling strategy may be sent to the handling personnel.

[0078] In actual application, hardware failure of the host machine requires hardware replacement, so the hardware failure handling process is: send the IP address and failure type of the failed host machine to the processing personnel configured by the configuration center to wait for the processing personnel to handle it. During the waiting period, urging notifications can be sent to the processing personnel regularly to remind them to handle it as soon as possible to reduce the situation of missing processing and processing task accumulation. Here, the sending frequency of urging notifications can be set arbitrarily, for example, the sending frequency of urging notifications is 2 hours / time.

[0079] In some embodiments, at least one virtual machine runs on each host machine, and each virtual machine carries business applications and services. Failure to promptly restore the virtual machines running on the faulty host machine will result in business interruption. At the same time, a large amount of business data is stored on the virtual machines. Failure to promptly restore the virtual machines running on the faulty host machine will result in business data loss or damage.

[0080] Therefore, in addition to processing each group of faulty host machines based on the corresponding fault handling strategy, the virtual machines running on each group of faulty host machines can also be restored, so that the service environment provided by each virtualization cluster runs stably.

[0081] Here, the virtual machine running on the faulty host machine can be restored while the faulty host machine is being processed based on the fault handling strategy; or the virtual machine running on the faulty host machine can be restored after the faulty host machine is processed based on the fault handling strategy. This embodiment of the present disclosure is not limited to this.

[0082] Here, the virtual machine may be restored by backup recovery, snapshot recovery or recovery software, or may be restored by migration or reconstruction, which is not limited in the embodiments of the present disclosure.

[0083] In the disclosed embodiment, the faulty host machines and the fault types of the faulty host machines are first determined from the host machines of each virtualization cluster based on the operation logs of the host machines of each virtualization cluster; the faulty host machines are then classified based on the fault types to obtain at least one group of faulty host machines; and finally, the corresponding groups of faulty host machines are processed separately based on various fault handling strategies.

[0084] On the one hand, the operation logs of the hosts of each virtualization cluster can be used to automatically detect faults on faulty hosts without the need for manual fault detection, thereby improving the detection efficiency of faulty hosts while reducing labor and time costs; by grouping the faulty hosts of each virtualization cluster, faulty host groups corresponding to different fault types can be obtained, so that faulty hosts can be processed in batches, thereby reducing the accumulation of processing tasks on faulty hosts and improving the degree of automation in handling faulty hosts.

[0085] On the other hand, since the fault handling strategies corresponding to different fault types are stored in advance, after obtaining each group of faulty host machines, each group of faulty host machines can be handled according to the corresponding fault handling strategy, thereby reducing the situation of invalid fault handling and improving the processing efficiency of faulty host machines. At the same time, the virtual machines running on each group of faulty host machines can be restored, further optimizing the processing procedures of faulty host machines, so that the service environment provided by each virtualization cluster can run stably.

[0086] In some embodiments, based on the operation log of the host machine of at least one virtualization cluster, determining the faulty host machine and the fault type corresponding to each faulty host machine from the host machines of at least one virtualization cluster includes:

[0087] Based on the alarm information in the operation log of each host machine, determine the faulty host machine from the host machines of at least one virtualization cluster;

[0088] For any faulty host machine, determine the hardware-related operation information in the operation log of the faulty host machine as key text information;

[0089] Determine the similarity between the key text information and each preset text information respectively; wherein the preset text information is obtained from the historical fault log, and different preset text information indicates different preset fault types;

[0090] The preset fault type indicated by the preset text information having a similarity with the key text information greater than a preset threshold is determined as the fault type corresponding to the faulty host machine.

[0091] It is understandable that when a host machine fails to run or runs incorrectly, there will be alarm information in the host machine's running log. Therefore, in order to quickly identify the faulty host machine from each virtualization cluster, the running log of each host machine can be extracted, and the host machine corresponding to the running log with alarm information can be identified as the faulty host machine.

[0092] It should be noted that in order to improve the accuracy and efficiency of determining the fault type of the faulty host machine, preset text information indicating different fault types can be pre-set, and then the key text information in the operation log of the faulty host machine can be text-matched with each preset text information to obtain the fault type of the faulty host machine.

[0093] Since host machine failures are mainly caused by hardware failures of the host machine, the hardware-related operation information can be identified as key text information in the operation log of the faulty host machine, and information other than the hardware-related operation information can be identified as irrelevant text information in the operation log of the faulty host machine, thereby reducing the impact of irrelevant text information on fault detection and improving the accuracy of fault detection.

[0094] Here, hardware includes but is not limited to a processor CPU, storage device, memory or network interface card, etc.

[0095] It should be explained that the preset text information is obtained from the historical fault log, and the historical fault log includes the historical operation log of the historical fault host machine when the historical fault occurs, and different preset text information indicates different fault types, that is, different preset text information includes the operation information of the fault host machine with different fault types when the fault occurs. Therefore, by determining the similarity between the key text information and each preset text information, the fault type to which the fault host machine belongs can be determined.

[0096] Here, the similarity between the key text information and each preset text information may be determined based on a string method, a vector method, or a deep learning method, etc., which is not limited in this embodiment of the present disclosure.

[0097] In some embodiments, the key text information is converted into a single character required for the preset text information by means of Levenshtein Distance, and the similarity between the key text information and the preset text information can be determined based on the number of editing operations, wherein the editing operation at least includes inserting a character, deleting a character, or replacing a character. When the editing distance is smaller, it means that the difference between the two character strings is smaller, so fewer editing operations are required; and when the editing distance is larger, it means that the difference between the two character strings is larger, so more editing operations are required.

[0098] In other embodiments, the key text information and the preset text information are converted into TF-IDF vectors by means of term frequency-inverse document frequency (TF-IDF) vectors, and then the cosine value of the angle between the two TF-IDF vectors in the vector space is calculated to further determine the similarity between the key text information and the preset text information. For example, when the cosine value is closer to 1, it means that the directions of the two vectors are closer, that is, the key text information and the preset text information are more similar; and when the cosine value is closer to 0, it means that the directions of the two vectors are perpendicular, that is, the key text information and the preset text information are more different.

[0099] In some embodiments, after obtaining the similarities between the key text information and each preset text information, each similarity may be sorted, and the preset fault type indicated by the preset text information with the greatest similarity among the similarities may be determined as the fault type of the faulty host machine.

[0100] It can be understood that, considering that the differences between the various similarities are small, in order to reduce the misjudgment of the fault type of the faulty host machine and to fully handle the fault of the faulty host machine, a preset threshold can be set and each similarity can be compared with the preset threshold to determine the fault type of the faulty host machine.

[0101] Here, the preset threshold can be set arbitrarily according to the requirements, for example, 80%. In the embodiment of the present disclosure, the preset fault type indicated by the preset text information whose similarity with the key text information is greater than the preset threshold is determined as the fault type of the faulty host machine.

[0102] In some embodiments, when the similarity between the two key text information is greater than a preset threshold, the preset fault types indicated by the two preset text information with similarity between the two key text information are determined as the fault types of the faulty host machine. In subsequent processing, the faulty host machines are divided into different faulty host machine groups to process the faulty host machines according to different fault processing strategies, thereby ensuring the fault processing effect of the faulty host machines.

[0103] In some embodiments, by determining the associated information and unassociated information between the key text information and the preset text information, and then calculating the ratio between the associated information and the unassociated information, the similarity between the key text information and each preset text information is obtained. Here, the associated information refers to the overlapping information between the key text information and the preset text information, which can be understood as the intersection between the key text information and the preset text information; the unassociated information refers to the overlapping information between the key text information and the preset text information and the summary information of all the unassociated information, which can be understood as the union between the key text information and the preset text information.

[0104] The Jaccard similarity coefficient is a method for calculating set similarity, which is used to measure the similarity between two sets. The Jaccard similarity coefficient calculates the ratio of the intersection and union of two sets, indicating the proportion of the common parts of the two sets to the total parts, and is often used to calculate the similarity of text data. Therefore, the calculation formula for the similarity between the key text information and each preset text information can be as follows:

[0105] similarity(A,B)=|A∩B| / |A∪B| (1);

[0106] In formula (1), A is the key text information, and B is the preset sample information. Here, the fault type indicated by the preset sample information can be a known error in various data warehouses such as a CPU type error and a memory type error.

[0107] When the similarity between the key text information and the preset sample information Y is the highest, it can be determined that the faulty host machine is of the fault type indicated by the preset sample information Y.

[0108] In the disclosed embodiment, by determining the faulty host from the host of at least one virtualization cluster based on the alarm information in the operation log of each host, the accuracy and efficiency of determining the faulty host can be improved; and for any faulty host, by determining the hardware-related operation information in the operation log of the faulty host as key text information; then respectively determining the similarity between the key text information and each preset text information; finally, determining the preset fault type indicated by the preset text information whose similarity with the key text information is greater than a preset threshold as the fault type of the faulty host. In this way, the method of determining the fault type of the faulty host by calculating the text similarity can effectively improve the accuracy of fault detection of the faulty host, reduce human intervention, and thus help to improve the degree of automation of the processing flow of the faulty host.

[0109] In some embodiments, each group of failed hosts is processed separately based on a corresponding failure processing strategy, including:

[0110] Determine a fault level corresponding to each faulty host machine based on the operating parameters of each faulty host machine and / or the alarm information in the operating log of each faulty host machine;

[0111] Based on the fault level corresponding to each faulty host machine, determine the host machines to be processed from each group of faulty host machines;

[0112] Based on the corresponding fault handling strategy, the host machines to be processed in each group of faulty host machines are processed separately.

[0113] It should be noted that, considering that the faulty hosts in the same group have the same fault type but different fault levels, if all faulty hosts are handled, there will be ineffective processing and waste of processing resources, which is not conducive to improving the intelligence of the fault handling process. Therefore, the host to be processed can be determined from each group of faulty hosts according to the fault level of the faulty host, so that the host to be processed can be processed according to the corresponding fault handling strategy.

[0114] Here, the host machine to be processed is a faulty host machine with a higher fault level. In order to improve the accuracy of determining the host machine to be processed, a first level threshold can be set. After obtaining the fault level of each faulty host machine, each fault level is compared with the first level threshold, and the faulty host machine with a fault level greater than or equal to the first level threshold is determined as the host machine to be processed, and the faulty host machine with a fault level less than the first level threshold is determined as the host machine that does not need to be processed this time.

[0115] In some embodiments, in order to prevent a faulty host machine with a lower fault level from aggravating the fault severity, a faulty host machine with a lower fault level may be processed by observation or by performing status detection at a preset time.

[0116] It can be understood that the fault level of the faulty host machine is positively correlated with the degree of the fault, and the fault degree of the faulty host machine can be determined through the operating parameters of the faulty host machine or the alarm information in the operating log of the faulty host machine. Therefore, the method of determining the fault level through operating parameters or alarm information in the embodiment of the present disclosure can improve the accuracy of determining the fault level.

[0117] In some embodiments, in order to more accurately determine the fault level of the faulty host machine, the fault degree of the faulty host machine may be evaluated simultaneously through operating parameters and alarm information to obtain the fault level of the faulty host machine.

[0118] Here, after determining the host machines to be processed from each group of faulty host machines, the host machines to be processed determined from each group of faulty host machines can be processed separately based on the corresponding fault handling strategies to reduce the accumulation of fault handling tasks of the host machines to be processed, thereby improving the processing efficiency of the host machines to be processed.

[0119] In some embodiments, the fault levels of the host machines to be processed in each group of faulty host machines may be different, and a second level threshold may be set. By comparing the fault level of each host machine to be processed with the second level threshold, the processing priority of each host machine to be processed is obtained to reduce the situation where the host machine to be processed with a high degree of fault cannot be processed in time, resulting in serious damage to the host machine. Exemplarily, the processing priority of the host machine to be processed whose fault level is greater than or equal to the second level threshold is set higher than the processing priority of the host machine to be processed whose fault level is less than the second level threshold.

[0120] In the disclosed embodiment, considering that the faulty host machines in the same group have the same fault type but different fault levels, the fault level of each faulty host machine is determined, and the host machines to be processed in each group of faulty host machines are further determined to screen out the faulty host machines with low fault levels, thereby reducing invalid processing and waste of processing resources, and is conducive to improving the intelligence of the fault handling process.

[0121] In some embodiments, determining the fault level corresponding to each faulty host machine based on the operating parameters of each faulty host machine and / or the alarm information in the operating log of each faulty host machine includes:

[0122] For any faulty host machine, an adjustment coefficient is determined based on an operating parameter of the faulty host machine, and a preset standard value is adjusted based on the adjustment coefficient to obtain a first degree value; wherein the operating parameter is used to indicate the operating performance of the faulty host machine, and the adjustment coefficient is negatively correlated with the operating performance; and / or,

[0123] Determining a second degree value based on alarm information in the operation log of the faulty host machine; wherein the alarm information includes at least one of an alarm frequency, an alarm duration, or an alarm level;

[0124] Based on the first degree value and / or the second degree value, a fault level corresponding to the faulty host machine is determined.

[0125] It should be noted that in order to more accurately determine the fault level of the faulty host machine based on the operating parameters, a preset standard value can be set, and an adjustment coefficient can be first determined through the operating parameters, and then the preset standard value can be adjusted based on the adjustment coefficient to determine the first degree value of the faulty host machine to further determine the fault level. In this way, by quantifying the fault level of the faulty host machine, the fault level can be determined more clearly and accurately.

[0126] Here, the preset standard value represents the operating performance corresponding to the optimal operating state of the host machine. For example, the preset standard value may be 100.

[0127] It should be explained that since the operating parameters are used to indicate the current operating performance of the faulty host machine, and the preset standard value indicates the operating performance corresponding to the optimal operating state, in order to facilitate the calculation of the first fault value, an adjustment coefficient negatively correlated with the current operability can be determined.

[0128] Here, different operating parameters determine different adjustment coefficients, and the adjustment coefficient may be an integer or a fraction, which is not limited in the embodiments of the present disclosure.

[0129] Exemplarily, when the current operating performance is better, the determined adjustment coefficient is smaller, so that the adjustment coefficient is adjusted more to the preset standard value, and the value of the adjusted standard value is smaller, that is, the first degree value is smaller, for example, adjusted from 100 to 30, so that it can be determined that the current fault degree of the faulty host machine is small; and when the current operating performance is poor, the determined adjustment coefficient is larger, so that the adjustment coefficient is adjusted less to the preset standard value, and the value of the adjusted standard value is larger, that is, the first degree value is larger, for example, adjusted from 100 to 70, so that it can be determined that the current fault degree of the faulty host machine is large.

[0130] It is understandable that the alarm information includes at least one of the alarm frequency, alarm duration or alarm level. By obtaining the numerical alarm information, the second degree value of the faulty host machine can be obtained, and the fault level can also be clearly and accurately determined.

[0131] In the embodiment of the present disclosure, after obtaining the first degree value and the second degree value, the sum of the first degree value and the second degree value can be calculated to further determine the fault level of the faulty host machine. In this way, the method of jointly determining the fault level by the first degree value and the second degree value can effectively improve the accuracy of determining the fault level.

[0132] In some embodiments, various fault levels are set in advance, each fault level corresponds to a fault degree range, and each fault degree range has multiple preset degree values. After obtaining the degree value of the faulty host machine (the first degree value and / or the second degree value), the degree value can be compared with the various preset degree values ​​in each degree range to determine the fault degree range in which the faulty host machine is located, thereby further determining the fault level of the faulty host machine.

[0133] Exemplarily, the preset standard value can be adjusted based on the adjustment coefficient determined by the operating parameters to obtain a first degree value, and based on the alarm frequency, alarm duration or alarm level, a second degree value can be obtained; finally, based on the first degree value and the second degree value, the fault degree of the faulty host machine can be obtained. The specific implementation process can be as follows:

[0134]

[0135] In formula (2), T is the alarm duration in the alarm information, representing the time from the occurrence of this type of alarm to the present; N is the alarm frequency in the alarm information, L is the alarm level in the alarm information, and D is the adjustment coefficient determined based on the operating parameters.

[0136] In some embodiments, to facilitate calculation of the first degree value, the adjustment coefficient can be understood as an indicator of whether the faulty host machine is down, and the adjustment coefficient takes a value of 0 or 1, where 0 indicates that the faulty host machine is not down, and 1 indicates that the faulty host machine is down.

[0137] In some embodiments, for a hardware failure of a faulty host machine, after the processing personnel replace the hardware, the operation log will reset N=0. When the recalculated fault degree score=0, it indicates that the hardware failure processing is complete and the faulty host machine hardware is restored. Finally, the restored host machine is marked as online.

[0138] In the disclosed embodiment, for any faulty host machine, an adjustment coefficient is determined based on the operating parameters of the faulty host machine, and a preset standard value is adjusted based on the adjustment coefficient to obtain a first degree value. At the same time, a second degree value is determined based on the alarm information in the operating log of the faulty host machine. Finally, based on the first degree value and / or the second degree value, the fault level of the faulty host machine is determined, thereby improving the accuracy of determining the fault level, laying a foundation for subsequent fault handling, and reducing invalid fault handling.

[0139] In some embodiments, recovering virtual machines running on each group of failed host machines includes:

[0140] After processing any group of faulty host machines based on the corresponding fault processing strategy, detecting the operating status of the faulty host machines in the current group after processing;

[0141] Based on the running status of the faulty host machine after processing, if it is determined that the faulty host machine is successfully repaired, the virtual machine running on the repaired faulty host machine is restored.

[0142] It needs to be explained that in order to improve the processing flow of the faulty host machine, after processing the faulty host machine based on the fault handling strategy, the operating status of the faulty host machine in the current group can be checked again to determine whether the faulty host machine has been successfully repaired, so as to facilitate subsequent processing of the faulty host machine that has not been successfully repaired. In this way, the situation of continued failure of the faulty host machine that has not been successfully repaired can be reduced, and the processing flow of the faulty host machine can be optimized.

[0143] Here, there are many ways to detect the running status of the faulty host after processing, such as remote login to the system response, viewing the running log, using command tools to perform response detection or detect the service status, etc.

[0144] Exemplarily, the CPU and memory usage are checked through the task manager to determine whether the CPU and memory are in a stable state, and the disk I / O and network status are checked through the resource monitor to determine whether there are any abnormalities; you can also check whether there are warning messages in the operation log to determine the operating status of the faulty host machine after processing.

[0145] It is understandable that, when it is determined that the faulty host machine is successfully repaired, the virtual machine running on the repaired faulty host machine is restored, thereby reducing the situation of ineffective recovery of the virtual machine.

[0146] In some embodiments, when a host machine fails, the data of the virtual machine can be backed up in time from a remote server, a cloud storage server, or a backup system of a virtualization platform. When it is determined that the failed host machine has been successfully repaired, the backup data is imported into the virtualization platform of the failed host machine through the recovery function of the virtualization platform or a third-party recovery tool, and the virtual machine running on the repaired failed host machine is restored.

[0147] In the disclosed embodiment, after the faulty host machine is processed based on the fault handling strategy, the running status of the faulty host machine in the current group can be detected again to determine whether the faulty host machine is successfully repaired, and if the faulty host machine is successfully repaired, the virtual machine running on the repaired faulty host machine can be restored. In this way, the situation of invalid recovery of virtual machines can be reduced, and the intelligence and automation of fault handling can be improved.

[0148] In some embodiments, the method further comprises:

[0149] If it is determined based on the running state of the faulty host machine after processing that the faulty host machine has not been successfully repaired, a prompt message that the faulty host machine is to be repaired is generated based on the identification information and the fault type corresponding to the faulty host machine;

[0150] After the faulty host machine is successfully repaired based on the prompt information, the virtual machine running on the repaired faulty host machine is restored.

[0151] Here, after detecting the processed running state, if it is determined that the processed running state is a normal state, it is determined that the faulty host machine has been successfully repaired; if it is determined that the processed running state is a faulty state, it is determined that the faulty host machine has not been successfully repaired.

[0152] It is understandable that when it is determined that the faulty host machine has not been successfully repaired, a prompt message can be generated based on the identification and fault type of the faulty host machine to prompt the faulty host machine to be processed, thereby reducing the situation of missing faulty hosts and improving the efficiency of processing faulty hosts.

[0153] Here, the prompt information is mainly used to send to the administrator to inform the administrator to repair the unrepaired faulty host machine on site.

[0154] In the disclosed embodiment, when a faulty host machine is not successfully repaired, prompt information that the faulty host machine is to be repaired is generated based on the identification information and fault type corresponding to the faulty host machine, so as to facilitate subsequent processing of the faulty host machine that has not been successfully repaired, thereby optimizing the processing flow of the faulty host machine, reducing the situation where the faulty host machine that has not been successfully repaired continues to fail, and improving the intelligence and automation of fault processing.

[0155] In some embodiments, restoring a virtual machine running on a repaired failed host machine includes:

[0156] Based on the data disk of the repaired faulty host machine, determine whether the virtual machine running on the repaired faulty host machine is in a recoverable state;

[0157] Perform data recovery or migration on the virtual machine in a recoverable state;

[0158] Rebuild the virtual machine that is in an unrecoverable state.

[0159] In some embodiments, if the host machine failure is caused by a hardware problem of the host machine, such as a damaged hard disk, a power failure, or a motherboard failure, handling the failed host machine involves replacing hardware components, which may result in data loss on the failed host machine. If the host machine failure is caused by a software failure or configuration error, such as an operating system crash, a file system corruption, or a virtualization platform configuration error, improper operation or failure to properly back up data when handling the failed host machine may also result in data loss on the failed host machine.

[0160] Therefore, in the embodiment of the present disclosure, after the faulty host machine is repaired, it is first determined whether there is data loss or damage on the data disk of the repaired faulty host machine, and then it is further determined whether the virtual machine running on the repaired faulty host machine is in a recoverable state, thereby improving the accuracy and reliability of recovering the virtual machine.

[0161] Here, when it is determined that there is no data loss or damage on the data disk of the repaired faulty host machine, it is determined that the virtual machine running on the repaired faulty host machine is in a recoverable state; and when it is determined that there is data loss or damage on the data disk of the repaired faulty host machine, it is determined that the virtual machine running on the repaired faulty host machine is in an unrecoverable state.

[0162] It is understandable that when a virtual machine is in a recoverable state, by determining whether the data of the virtual machine is backed up and / or the extent of damage to the virtual machine data caused by the host machine failure, it is determined whether to perform data recovery processing or data migration processing on the virtual machine in a recoverable state, thereby improving the efficiency of recovering the virtual machine.

[0163] In some embodiments, if the virtual machine data has been backed up in advance, the virtual machine in a recoverable state will be directly restored; if the virtual machine data has not been backed up in advance, the virtual machine in a recoverable state will be directly migrated and the virtual machine will be restored from the new environment after migration.

[0164] In other embodiments, if the damage to the virtual machine data is minor, the virtual machine in a recoverable state will be directly recovered; if the damage to the virtual machine data is severe, the virtual machine in a recoverable state will be directly migrated to avoid further loss of virtual machine data.

[0165] When the virtual machine is in an unrecoverable state, it is determined that the data disk of the repaired faulty host machine has lost the data of the original virtual machine, and the virtual machine is directly rebuilt on the repaired faulty host machine.

[0166] In the embodiment of the present disclosure, based on the data disk of the repaired faulty host machine, the recovery steps of the virtual machine running on the repaired faulty host machine are determined, which can improve the accuracy and reliability of recovering the virtual machine.

[0167] In some embodiments, the method further comprises:

[0168] After the faulty host machine is successfully repaired based on the prompt information, a target file is generated based on a processing strategy corresponding to the repair and a fault type corresponding to the faulty host machine;

[0169] Based on the target file, update the fault handling strategy corresponding to the fault type.

[0170] It is understandable that since the faulty host machine to be processed is debugged and repaired by the administrator, in order to optimize the processing flow of the faulty host machine, after the faulty host machine is repaired based on the prompt information, the processing strategy of the administrator's on-site repair can be updated to the pre-stored fault processing strategy.

[0171] Here, a target file may be generated according to the fault type of the faulty host machine and the processing strategy corresponding to the repair, so as to update the fault processing strategy corresponding to the fault type.

[0172] In some embodiments, a task report can be generated based on the processing of the fault handling, and the task report can be stored in the data warehouse. Through the task report, relevant personnel can understand the situation and results of the fault handling, and make relevant records and analysis, so as to effectively optimize the host machine fault handling process. Exemplarily, the entries of the task report can be: {virtualization cluster: cluster name, fault host: IP, type: fault type, level: fault level, function: fault handling method, time: fault handling time}.

[0173] In the disclosed embodiment, after the faulty host machine is repaired based on the prompt information, the fault handling strategy corresponding to the fault type can be updated based on the processing strategy and fault type corresponding to the repair, thereby optimizing the processing flow of the faulty host machine and improving the degree of automation of fault handling.

[0174] Based on this, the implementation process of the fault handling method described in the above embodiment in an actual application scenario is specifically described below. Figure 4 This is a flowchart of the fault handling method provided in the embodiment of the present application. Figure 2 ,like Figure 4 As shown, the fault handling method provided in the embodiment of the present application may include the following steps:

[0175] In step 401, the operation log of each faulty host is obtained.

[0176] In some embodiments, based on the alarm information in the operation log of each host machine, a faulty host machine is determined from the host machines of at least one virtualization cluster, and then the operation log of each faulty host machine is further obtained.

[0177] In step 402, for any faulty host machine, operation information related to hardware in the operation log of the faulty host machine is determined as key text information.

[0178] In some embodiments, when there are multiple faulty host machines in at least one virtualization cluster, hardware-related operation information is extracted from operation logs of the faulty host machines respectively, and the hardware-related operation information is determined as key text information.

[0179] In step 403, the similarity between the key text information and each preset text information is determined respectively.

[0180] In step 404, the preset fault type indicated by the preset text information having a similarity with the key text information greater than a preset threshold is determined as the fault type of the faulty host machine.

[0181] In step 405, after the fault type of each faulty host machine is obtained, the faulty host machines are classified based on the fault type to obtain at least one group of faulty host machines.

[0182] In step 406, based on the fault type of each group of faulty host machines, a fault handling strategy corresponding to each group of faulty host machines is determined.

[0183] In step 407, each group of corresponding faulty host machines is processed separately based on various fault processing strategies.

[0184] In step 408, after the faulty host machine is successfully processed based on the fault processing strategy, it is determined whether the faulty host machine is successfully repaired.

[0185] In step 409, in response to the successful repair of the faulty host machine, the virtual machines running on the repaired faulty host machine are restored.

[0186] In step 410, in response to the failure of the host machine to be repaired, prompt information is generated based on the identification of the host machine and the failure type.

[0187] In some embodiments, after the faulty host machine is successfully repaired based on the prompt information, the virtual machine running on the repaired faulty host machine is restored.

[0188] The fault handling method proposed in the embodiment of the present application, on the one hand, can automatically perform fault detection on the faulty host machine through the operation log of the host machine of each virtualization cluster, without the need for manual fault detection, thereby improving the detection efficiency of the faulty host machine while reducing labor costs and time costs; by grouping the faulty host machines of each virtualization cluster, faulty host machine groups corresponding to different fault types can be obtained, so that the faulty host machines can be processed in batches, thereby reducing the accumulation of processing tasks of the faulty host machines and improving the degree of automation of faulty host machine processing.

[0189] On the other hand, since the fault handling strategies corresponding to different fault types are stored in advance, after obtaining each group of faulty host machines, each group of faulty host machines can be handled according to the corresponding fault handling strategy, thereby reducing the situation of invalid fault handling and improving the processing efficiency of faulty host machines. At the same time, the virtual machines running on each group of faulty host machines can be restored, further optimizing the processing procedures of faulty host machines, so that the service environment provided by each virtualization cluster can run stably.

[0190] Figure 5 This is a schematic diagram of the architecture of a fault handling system provided in an embodiment of the present application. Figure 2 ,like Figure 5 As shown, the fault handling system provided in the embodiment of the present application may include the following modules:

[0191] The configuration module 501 is used to configure the address information of the virtualization cluster, the host machine fault detection process, the fault handling strategies and handling personnel corresponding to different fault types, etc.

[0192] The log collection module 502 is used to obtain the operation log of each host machine in at least one virtualization cluster.

[0193] The storage module 503 is used to store the operation log of each host machine.

[0194] Here, the storage module 503 may be a database.

[0195] The classifier 504 is used to determine the operation information related to the hardware in the operation log as key text information; and respectively determine the similarity between the key text information and each preset text information; determine the preset fault type indicated by the preset text information whose similarity with the key text information is greater than a preset threshold as the fault type of the faulty host machine; classify the faulty host machines based on the fault type to obtain at least one group of faulty host machines; wherein the fault types of the faulty host machines in the same group are the same.

[0196] The first diagnostic module 505 is used to determine an adjustment coefficient based on an operating parameter for any faulty host machine, and adjust a preset standard value based on the adjustment coefficient to obtain a first degree value; wherein the operating parameter is used to indicate the operating performance of the faulty host machine, and the adjustment coefficient is negatively correlated with the operating performance; based on the alarm information, a second degree value is determined; wherein the alarm information includes at least one of an alarm frequency, an alarm duration or an alarm level; based on the first degree value and the second degree value, the fault level of the faulty host machine is determined.

[0197] The second diagnosis module 506 is used to determine host machines to be processed from each group of faulty host machines based on the fault level corresponding to each faulty host machine.

[0198] In some embodiments, after the second diagnosis module determines the host machines to be processed from each group of faulty host machines based on the fault level of the faulty host machine, the subsequent process is performed by the fault processing module.

[0199] In other embodiments, after the second diagnostic module fails to determine a host machine to be processed from each group of faulty host machines based on the fault level of the faulty host machine, the classifier performs fault detection on the host machine again and detects the fault type of the faulty host machine.

[0200] The fault processing module 507 is used to process the host machines to be processed determined from each group of faulty host machines respectively based on the corresponding fault processing strategy.

[0201] The confirmation and repair module 508 is used to detect the operating status of the faulty host machines after processing any group of faulty host machines based on the corresponding fault processing strategy.

[0202] In some embodiments, after the confirmation repair module determines that the faulty host machine is successfully repaired based on the processed running status, the report generation module executes a subsequent process, that is, generates a report of the processing process.

[0203] Here, the report generation module is also used to generate a task report based on the processing of the fault handling, and store the task report in the data warehouse. Through the task report, relevant personnel can understand the situation and results of the fault handling, and make relevant records and analysis, so as to effectively optimize the host machine fault handling process. Exemplarily, the entries of the task report can be: {virtualization cluster: cluster name, fault host: IP, type: fault type, level: fault level, function: fault handling method, time: fault handling time}.

[0204] In some embodiments, after the confirmation repair module determines that the faulty host machine has not been successfully repaired based on the processed running status, the prompt module executes the subsequent process.

[0205] The prompt module 509 is used to generate prompt information based on the identifier and the fault type of the faulty host machine in response to the faulty host machine not being repaired successfully; wherein the prompt information is used to prompt that the faulty host machine needs to be repaired.

[0206] A report generation module 510 is used to generate a target file based on a processing strategy corresponding to the repair and a fault type of the faulty host machine after the faulty host machine is repaired based on the prompt information;

[0207] Based on the target file, update the fault handling strategy corresponding to the fault type.

[0208] Figure 6 This is a flowchart of the fault handling method provided in the embodiment of the present application. Figure 3 ,like Figure 6 As shown, the fault handling method provided in the embodiment of the present application may include the following steps:

[0209] In step 601, it is determined that the faulty host machine is successfully repaired.

[0210] In step 602, it is determined whether the virtual machine running on the repaired faulty host machine is in a recoverable state.

[0211] Here, after the faulty host machine is repaired, it is first determined whether there is data loss or damage on the data disk of the repaired faulty host machine, and then it is further determined whether the virtual machine running on the repaired faulty host machine is in a recoverable state.

[0212] In some embodiments, when it is determined that the virtual machine running on the failed host machine is in a recoverable state, step 603 is performed.

[0213] In other embodiments, when it is determined that the virtual machine running on the failed host machine is in an unrecoverable state, step 606 is executed.

[0214] In step 603, it is determined whether to restore locally.

[0215] Here, by determining whether the data of the virtual machine is backed up and / or the extent of damage to the virtual machine data caused by the failure of the host machine, it is determined whether the virtual machine in a recoverable state can be locally repaired.

[0216] In some embodiments, if it is determined that the virtual machine in the recoverable state can be recovered locally, step 605 is performed.

[0217] In other embodiments, when it is determined that the virtual machine in the recoverable state cannot be recovered locally, step 604 is performed.

[0218] In step 604, the virtual machine is migrated.

[0219] In step 603, data recovery is performed on the virtual machine.

[0220] In step 606, the virtual machine is rebuilt.

[0221] In some embodiments, when the virtual machine is in an unrecoverable state, it is determined that the data disk of the repaired faulty host machine has lost the data of the original virtual machine, and the virtual machine is directly rebuilt on the repaired faulty host machine.

[0222] The following is a description of an exemplary structure of the fault handling device 255 provided in the application embodiment implemented as a software module. In some embodiments, Figure 2 As shown, the software modules stored in the fault processing device 255 of the memory 250 may include:

[0223] The acquisition module 2551 is configured to acquire the operation log of at least one host machine of the virtualization cluster; wherein each host machine runs at least one virtual machine;

[0224] A first determination module 2552 is configured to determine a faulty host machine and a fault type corresponding to each faulty host machine from the host machines of the at least one virtualization cluster based on the operation log of the host machine of the at least one virtualization cluster;

[0225] The classification module 2553 is configured to classify the faulty host machines based on the fault type to obtain at least one group of faulty host machines; wherein the faulty host machines in the same group have the same fault type.

[0226] The second determination module 2554 is configured to determine the fault handling strategy corresponding to each group of faulty host machines based on the fault type corresponding to each group of faulty host machines; wherein one group of faulty host machines corresponds to one fault handling strategy;

[0227] The processing module 2555 is configured to process the corresponding groups of faulty host machines respectively based on the corresponding fault processing strategy, and to recover the virtual machines running on the groups of faulty host machines.

[0228] In some embodiments, the first determination module 2552 is specifically configured as follows:

[0229] Determine a faulty host from the host machines of the at least one virtualization cluster based on the alarm information in the operation log of each host machine;

[0230] For any faulty host machine, determining hardware-related operation information in the operation log of the faulty host machine as key text information;

[0231] Determine the similarity between the key text information and each preset text information respectively; wherein the preset text information is obtained from a historical fault log, and different preset text information indicates different preset fault types;

[0232] The preset fault type indicated by the preset text information having a similarity with the key text information greater than a preset threshold is determined as the fault type corresponding to the faulty host machine.

[0233] In some embodiments, the processing module 2555 is specifically configured as follows:

[0234] Determine a fault level corresponding to each faulty host machine based on the operating parameters of each faulty host machine and / or the alarm information in the operating log of each faulty host machine;

[0235] Based on the fault level corresponding to each faulty host machine, determine the host machines to be processed from each group of faulty host machines;

[0236] Based on the corresponding fault handling strategy, the host machines to be processed in each group of faulty host machines are processed separately.

[0237] In some embodiments, the processing module 2555 is further configured to:

[0238] For any faulty host machine, an adjustment coefficient is determined based on an operating parameter of the faulty host machine, and a preset standard value is adjusted based on the adjustment coefficient to obtain a first degree value; wherein the operating parameter is used to indicate an operating performance of the faulty host machine, and the adjustment coefficient is negatively correlated with the operating performance; and / or,

[0239] Determining a second degree value based on alarm information in the operation log of the faulty host machine; wherein the alarm information includes at least one of an alarm frequency, an alarm duration, or an alarm level;

[0240] Based on the first degree value and / or the second degree value, a fault level corresponding to the faulty host machine is determined.

[0241] In some embodiments, the processing module 2555 is further configured to:

[0242] After processing any group of faulty host machines based on the corresponding fault processing strategy, detecting the operating status of the faulty host machines in the current group after processing;

[0243] If it is determined based on the running state of the faulty host machine after processing that the faulty host machine is successfully repaired, the virtual machine running on the repaired faulty host machine is restored.

[0244] In some embodiments, the software module in the fault handling device 255 further includes:

[0245] A repair module, configured to generate prompt information that the faulty host machine is to be repaired based on identification information and a fault type corresponding to the faulty host machine when it is determined that the faulty host machine has not been successfully repaired based on the running state of the faulty host machine after processing;

[0246] After the faulty host machine is successfully repaired based on the prompt information, the virtual machine running on the repaired faulty host machine is restored.

[0247] In some embodiments, the processing module 2555 is further configured to:

[0248] Based on the data disk of the repaired faulty host machine, determining whether the virtual machine running on the repaired faulty host machine is in a recoverable state;

[0249] Perform data recovery or migration on the virtual machine in a recoverable state;

[0250] Rebuild the virtual machine that is in an unrecoverable state.

[0251] In some embodiments, the software module in the fault handling device 255 further includes:

[0252] A generating module configured to generate a target file based on a processing strategy corresponding to the repair and a fault type corresponding to the faulty host machine after the faulty host machine is successfully repaired based on the prompt information;

[0253] Based on the target file, a fault handling strategy corresponding to the fault type is updated.

[0254] It should be noted that the description of the fault handling device in the embodiment of the present application is similar to the description of the fault handling method embodiment described above, and has similar beneficial effects as the method embodiment, so it will not be repeated.

[0255] The embodiment of the present application provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the fault handling method described in the embodiment of the present application.

[0256] The embodiment of the present application provides a computer-readable storage medium storing executable instructions, wherein a computer program or executable instructions are stored. When the computer program or executable instructions are executed by a processor, the processor will be caused to execute the fault handling method provided in the embodiment of the present application, for example, Figure 3 The troubleshooting method shown.

[0257] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0258] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0259] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0260] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0261] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A fault handling method, characterized in that: The method comprises: Obtaining an operation log of a host machine of at least one virtualization cluster; wherein each host machine runs at least one virtual machine; Based on the operation log of the host machine of the at least one virtualization cluster, determining the faulty host machine and the fault type corresponding to each faulty host machine from the host machines of the at least one virtualization cluster; Classifying the faulty host machines based on the fault type to obtain at least one group of faulty host machines; wherein the faulty host machines in the same group have the same fault type; Based on the fault types corresponding to the faulty host machines in each group, determining the fault handling strategies corresponding to the faulty host machines in each group; wherein one fault handling strategy corresponds to one group of faulty host machines; Each group of faulty host machines is processed separately based on the corresponding fault handling strategy, and the virtual machines running on each group of faulty host machines are recovered.

2. The method according to claim 1, characterized in that: The determining, based on the operation log of the host machine of the at least one virtualization cluster, a faulty host machine and a fault type corresponding to each faulty host machine from the host machines of the at least one virtualization cluster includes: Determine a faulty host from the host machines of the at least one virtualization cluster based on the alarm information in the operation log of each host machine; For any faulty host machine, determining hardware-related operation information in the operation log of the faulty host machine as key text information; Determine the similarity between the key text information and each preset text information respectively; wherein the preset text information is obtained from a historical fault log, and different preset text information indicates different preset fault types; The preset fault type indicated by the preset text information having a similarity with the key text information greater than a preset threshold is determined as the fault type corresponding to the faulty host machine.

3. The method according to claim 1, characterized in that The processing of each group of faulty host machines based on the corresponding fault handling strategy includes: Determine a fault level corresponding to each faulty host machine based on the operating parameters of each faulty host machine and / or the alarm information in the operating log of each faulty host machine; Based on the fault level corresponding to each faulty host machine, determine the host machines to be processed from each group of faulty host machines; Based on the corresponding fault handling strategy, the host machines to be processed in each group of faulty host machines are processed separately.

4. The method according to claim 3, characterized in that The determining, based on the operating parameters of each faulty host machine and / or the alarm information in the operating log of each faulty host machine, the fault level corresponding to each faulty host machine includes: For any faulty host machine, an adjustment coefficient is determined based on an operating parameter of the faulty host machine, and a preset standard value is adjusted based on the adjustment coefficient to obtain a first degree value; wherein the operating parameter is used to indicate an operating performance of the faulty host machine, and the adjustment coefficient is negatively correlated with the operating performance; and / or, Determining a second degree value based on alarm information in the operation log of the faulty host machine; wherein the alarm information includes at least one of an alarm frequency, an alarm duration, or an alarm level; Based on the first degree value and / or the second degree value, a fault level corresponding to the faulty host machine is determined.

5. The method according to claim 1, characterized in that The method of restoring the virtual machines running on each group of failed host machines includes: After processing any group of faulty host machines based on the corresponding fault processing strategy, detecting the operating status of the faulty host machines in the current group after processing; If it is determined based on the running state of the faulty host machine after processing that the faulty host machine is successfully repaired, the virtual machine running on the repaired faulty host machine is restored.

6. The method according to claim 5, characterized in that The method further comprises: If it is determined based on the running state of the faulty host machine after processing that the faulty host machine has not been successfully repaired, based on the identification information and the fault type corresponding to the faulty host machine, a prompt message indicating that the faulty host machine is to be repaired is generated; After the faulty host machine is successfully repaired based on the prompt information, the virtual machine running on the repaired faulty host machine is restored.

7. The method according to claim 5 or 6, characterized in that: The method of restoring the virtual machine running on the repaired faulty host machine includes: Based on the data disk of the repaired faulty host machine, determining whether the virtual machine running on the repaired faulty host machine is in a recoverable state; Perform data recovery or migration on the virtual machine in a recoverable state; Rebuild the virtual machine that is in an unrecoverable state.

8. The method according to claim 6, characterized in that The method further comprises: After the faulty host machine is successfully repaired based on the prompt information, a target file is generated based on a processing strategy corresponding to the repair and a fault type corresponding to the faulty host machine; Based on the target file, a fault handling strategy corresponding to the fault type is updated.

9. A fault handling device, characterized in that: The device comprises: An acquisition module is configured to acquire an operation log of a host machine of at least one virtualization cluster; wherein each host machine runs at least one virtual machine; A first determination module is configured to determine a faulty host machine and a fault type corresponding to each faulty host machine from the host machines of the at least one virtualization cluster based on the operation log of the host machine of the at least one virtualization cluster; A classification module is configured to classify the faulty host machines based on the fault type to obtain at least one group of faulty host machines; wherein the faulty host machines in the same group have the same fault type; A second determination module is configured to determine a fault handling strategy corresponding to each group of faulty host machines based on the fault type corresponding to each group of faulty host machines; wherein a group of faulty host machines corresponds to one fault handling strategy; The processing module is configured to process the corresponding groups of faulty host machines respectively based on the corresponding fault processing strategy, and to restore the virtual machines running on the groups of faulty host machines.

10. An electronic device, characterized in that: include: Memory for storing computer programs or executable instructions; A processor, configured to implement the method according to any one of claims 1 to 8 when executing the computer program or the executable instructions stored in the memory.

Images