Fault root cause positioning method and device, equipment and medium

Abstract

The invention discloses a fault root cause positioning method and device, equipment and a medium, and relates to the technical field of cloud computing and virtualization. According to the scheme, the related logs of the fault virtual machine and the performance indexes of the equipment where the fault virtual machine is located are collected, multi-source data fusion is achieved, comprehensive and reliable data input is provided for fault positioning of the virtual machine, and key clues are prevented from being omitted; according to the method, the causal experience library containing the fault root cause information of various virtual machine fault types is pre-constructed, the target fault root cause information of the fault virtual machine is retrieved in the causal experience library by using the alarm information and output, a rapid analysis means is provided for the virtual machine fault, and the fault positioning efficiency is higher; meanwhile, if the search based on the causal experience library fails, the target fault root cause information of the faulted virtual machine is reasoned in real time according to the alarm information, the related log and the multi-source data of the performance index, so that the real-time performance and the accuracy of fault root cause positioning of the virtual machine are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of cloud computing and virtualization, and particularly to a method, device, equipment and medium for fault root cause location. Background Art

[0002] Cloud computing and virtualization technologies have been deeply integrated into enterprise operations and personal life, significantly improving the efficiency of data storage and processing. However, with the continuous growth of demand, system failures and performance issues have become increasingly frequent. Quickly troubleshooting and locating problems to improve system stability directly affects customer satisfaction with products.

[0003] Currently, fault location mainly relies on manual log analysis and monitoring tools, but these traditional methods have obvious drawbacks: they usually analyze logs, metrics, or topology data in isolation, lacking the ability to correlate and analyze multi-source data, resulting in difficulties in root cause location. In addition, these methods highly rely on the experience of operation and maintenance experts for manual troubleshooting, which is not only inefficient but also prone to missing key clues.

[0004] In view of the above, how to solve the problem that the current virtual machine fault location method lacks the ability to correlate and analyze multi-source data, relies on manual troubleshooting, is inefficient and prone to missing key clues is an urgent problem for those skilled in the art. Summary of the Invention

[0005] The present invention provides a method, device, equipment and medium for fault root cause location to at least solve the problem that the current virtual machine fault location method lacks the ability to correlate and analyze multi-source data, relies on manual troubleshooting, is inefficient and prone to missing key clues.

[0006] The present invention provides a method for fault root cause location, including:

[0007] Obtaining the alarm information of the faulty virtual machine;

[0008] Collecting the relevant logs of the faulty virtual machine and the performance metrics of the device where the faulty virtual machine is located according to the alarm information;

[0009] Retrieving in a pre-constructed causal experience library according to the alarm information to determine whether the target fault root cause information corresponding to the faulty virtual machine is retrieved; wherein, the causal experience library contains the fault root cause information of various virtual machine fault types generated according to the historical alarm information of multiple virtual machines;

[0010] If not, inferring the target fault root cause information corresponding to the faulty virtual machine according to the alarm information, relevant logs and performance metrics;

[0011] If so, outputting the target fault root cause information corresponding to the faulty virtual machine.

[0012] The present invention also provides a fault root cause location device, including:

[0013] An acquisition module, configured to acquire alarm information of a faulty virtual machine;

[0014] A collection module, configured to collect relevant logs of the faulty virtual machine and performance metrics of the device where the faulty virtual machine is located according to the alarm information;

[0015] A retrieval and judgment module, configured to retrieve in a pre-constructed causal experience library according to the alarm information and judge whether target fault root cause information corresponding to the faulty virtual machine is retrieved; wherein, the causal experience library contains fault root cause information of various virtual machine fault types generated according to historical alarm information of multiple virtual machines; if not, the inference module is triggered; if so, the output module is triggered;

[0016] An inference module, configured to infer target fault root cause information corresponding to the faulty virtual machine according to the alarm information, relevant logs and performance metrics;

[0017] An output module, configured to output target fault root cause information corresponding to the faulty virtual machine.

[0018] The present invention also provides an electronic device, including: a memory, configured to store a computer program; a processor, configured to implement the steps of any one of the above-mentioned fault root cause location methods when executing the computer program.

[0019] The present invention also provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program implements the steps of any one of the above-mentioned fault root cause location methods when executed by a processor.

[0020] The beneficial effects of the present invention are as follows: by collecting relevant logs of the faulty virtual machine and performance metrics of the device where the faulty virtual machine is located, multi-source data fusion is achieved, providing comprehensive and reliable data input for virtual machine fault location and avoiding missing key clues; a causal experience library containing fault root cause information of various virtual machine fault types is pre-constructed, and the alarm information is used to retrieve and output the target fault root cause information of the faulty virtual machine in the causal experience library, providing a fast analysis means for virtual machine faults and having higher fault location efficiency; at the same time, if the retrieval based on the causal experience library fails, the target fault root cause information of the faulty virtual machine is inferred in real time according to multi-source data of the alarm information, relevant logs and performance metrics, improving the real-time performance and accuracy of virtual machine fault root cause location.

[0021] In addition, the present invention also provides a fault root cause location device, equipment and medium, and the effects are the same as above. Description of the Drawings

[0022] To more clearly illustrate the embodiments of the present invention, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0023] Figure 1 It is a flowchart of a fault root cause location method provided by an embodiment of the present invention;

[0024] Figure 2 It is an architecture diagram of a fault root cause location system provided by an embodiment of the present invention;

[0025] Figure 3 It is a schematic diagram of multi-source data samples provided by an embodiment of the present invention;

[0026] Figure 4 It is a schematic diagram of a fault root cause location device provided by an embodiment of the present invention. Detailed implementation manners

[0027] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the protection scope of the present invention.

[0028] It should be noted that in the description of the present invention, the terms "include", "comprise" or any other variant thereof are intended to cover a non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in the present invention are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0029] To enable those skilled in the art of the present technology to better understand the solution of the present invention, the following will further elaborate on the present invention in conjunction with the drawings and specific implementation manners.

[0030] Currently, virtual machine fault localization mainly relies on manual log analysis and monitoring tools. Usually, logs, metrics, or topology data are analyzed in isolation, lacking the ability to correlate multi-source data, resulting in difficulties in root cause localization. At the same time, relying on the experience of operation and maintenance experts for manual troubleshooting is not only inefficient but also prone to missing key clues. Therefore, to solve the above problems, the present invention provides a method for fault root cause localization. In combination with the specific application environment architecture or specific hardware architecture on which the execution of the fault root cause localization method depends, the specific application environment architecture or specific hardware architecture is described herein.

[0031] The fault root cause localization method provided by the present invention is applied to the fault root cause localization system in the control host. Through this system, multiple virtual machines in the virtual machine platform can be monitored, and analysis and localization can be automatically triggered when a virtual machine anomaly is detected. In this embodiment, the specific type of the control host is not limited. For example, it can be a server or a personal host. The method will be described in detail below in combination with the execution process of the fault root cause localization method.

[0032] Figure 1 It is a flowchart of a fault root cause localization method provided by an embodiment of the present invention. As Figure 1 shown, the method includes:

[0033] S10: Obtain the alarm information of the faulty virtual machine.

[0034] Figure 2 It is an architecture diagram of the fault root cause localization system provided by an embodiment of the present invention. When the system runs, the fault root cause localization method is executed. As Figure 2 shown, the fault alarm detection unit monitors the alarm information of the virtual machines in the virtual machine platform in real time. After receiving the real-time alarm information, it will parse and obtain the identifier (Identity, ID), alarm time, and alarm type of the faulty virtual machine for the current alarm. Among them, the alarm type can include cloud host downtime, cloud host input / output interruption, cloud host disconnection, etc. The alarm information may also include the alarm level.

[0035] S11: Collect relevant logs of the faulty virtual machine and performance metrics of the device where the faulty virtual machine is located according to the alarm information.

[0036] Further, after receiving the trigger signal from the fault alarm detection unit, the multi-source data fusion unit in the system collects the relevant logs of the faulty virtual machine and the performance metrics of the device where the faulty virtual machine is located according to the alarm information, so as to obtain multi-source data samples. In this embodiment, there is no limitation on the specific type of the relevant logs. For example, it may include virtualization platform logs, virtual machine kernel logs, and application logs, etc.; there is also no limitation on the type of the performance metrics of the device where the faulty virtual machine is located. For example, it may include the performance metrics of the processor, memory, disk, and network. In addition, in this embodiment, there is no limitation on the specific collection process of the relevant logs of the faulty virtual machine and the performance metrics of the device where the faulty virtual machine is located, which depends on the specific implementation situation.

[0037] S12: Retrieve in the pre-constructed causal experience library according to the alarm information to determine whether the target fault root cause information corresponding to the faulty virtual machine is retrieved. If not, go to step S13; if so, go to step S14.

[0038] Among them, the causal experience library contains the fault root cause information of various virtual machine fault types generated according to the historical alarm information of multiple virtual machines.

[0039] S13: Infer the target fault root cause information corresponding to the faulty virtual machine according to the alarm information, relevant logs, and performance metrics.

[0040] S14: Output the target fault root cause information corresponding to the faulty virtual machine.

[0041] Subsequently, the multi-source data fusion unit inputs the collected multi-source data samples into the causal inference engine. The causal inference engine retrieves in the pre-constructed causal experience library according to the alarm information to determine whether the target fault root cause information corresponding to the faulty virtual machine is retrieved.

[0042] It should be noted that the causal experience library contains the fault root cause information corresponding to various virtual machine fault types, and these fault root cause information are generated according to the historical alarm information of multiple virtual machines. The fault root cause information actually manifests as a directed acyclic graph of the causal relationship between multi-source data variables. For example, a decrease in the number of input / output operations per second (IOPS) of the disk (variable 1) and an increase in the memory usage rate (variable 3) lead to an increase in network latency (variable 2), which in turn causes the virtual machine to freeze. It should be noted that in this embodiment, there is no limitation on the construction process of the causal experience library.

[0043] If the target root cause information corresponding to the faulty virtual machine is retrieved from the cause-and-effect experience library according to the alarm type in the alarm information, the target root cause information corresponding to the faulty virtual machine is directly output, so as to quickly locate the fault problem. If the target root cause information corresponding to the faulty virtual machine is not retrieved from the cause-and-effect experience library according to the alarm type in the alarm information, it is considered that there is no corresponding root cause information for the current fault type in the cause-and-effect experience library, and the cause-and-effect inference engine needs to perform cause-and-effect inference in real time based on the above multi-source data samples, that is, infer the target root cause information corresponding to the faulty virtual machine according to the alarm information, relevant logs, and performance metrics, so as to determine the cause of the fault. In this embodiment, the specific process of performing cause-and-effect inference in real time based on multi-source data samples is not limited and depends on the specific implementation situation.

[0044] In this embodiment, by collecting the relevant logs of the faulty virtual machine and the performance metrics of the device where the faulty virtual machine is located, multi-source data fusion is achieved, providing comprehensive and reliable data input for virtual machine fault location and avoiding missing key clues; a cause-and-effect experience library containing root cause information for various virtual machine fault types is pre-constructed, and the target root cause information of the faulty virtual machine is retrieved and output in the cause-and-effect experience library using the alarm information, providing a fast analysis method for virtual machine faults and higher fault location efficiency; at the same time, if the retrieval based on the cause-and-effect experience library fails, the target root cause information of the faulty virtual machine is inferred in real time based on the multi-source data of the alarm information, relevant logs, and performance metrics, improving the real-time performance and accuracy of virtual machine root cause location.

[0045] Based on the above embodiment, in some embodiments, collecting the relevant logs of the faulty virtual machine and the performance metrics of the device where the faulty virtual machine is located according to the alarm information includes:

[0046] S111: Determine the alarm time of the faulty virtual machine according to the alarm information, and construct a preset time window according to the alarm time.

[0047] Among them, the alarm time is within the preset time window.

[0048] S112: Collect performance metrics within the preset time window through the monitoring alarm tool.

[0049] Among them, the performance metrics at least include processor performance metrics, memory performance metrics, disk performance metrics, and network performance metrics.

[0050] S113: Obtain the topology information of the faulty virtual machine.

[0051] Among them, the topology information represents the connection and relationship structure between the faulty virtual machine and its related resources.

[0052] S114: Retrieve relevant logs within the preset time window according to the topology information.

[0053] Among them, the relevant logs at least include virtualization platform logs, virtual machine kernel logs, and application logs.

[0054] It can be understood that data collection is the basis for fault root cause location, and it is necessary to ensure the accuracy, timeliness, and comprehensiveness of the data. Therefore, in this embodiment, in order to ensure the complete acquisition of relevant data, the alarm time of the faulty virtual machine is specifically determined according to the alarm information, and a preset time window is constructed according to the alarm time, and the alarm time is within the preset time window. For example, when the alarm time is t, the preset time window is set to [t - n, t + n]. In this embodiment, the size of the preset time window is not limited. For example, let n be 5 minutes, and the size of the preset time window is 10 minutes.

[0055] After setting the preset time window, performance metrics are collected within the preset time window through a monitoring and alarm tool. It should be noted that the monitoring and alarm tool is mainly used to collect, store, query, and visualize real-time metric data of systems, applications, and hardware. In this embodiment, the type of the specifically selected monitoring and alarm tool is not limited. In addition, the performance metrics at least include processor performance metrics, memory performance metrics, disk performance metrics, and network performance metrics. The processor performance metrics may specifically include processor utilization, steal time, and the number of context switches, etc. The memory performance metrics may specifically include memory usage, ballooning, and swap usage. The disk performance metrics may specifically include disk IOPS, throughput, latency, and read / write error rate, etc. The network performance metrics may specifically include network latency, packet loss rate, bandwidth, and the number of connections, etc.

[0056] Furthermore, the topology information of the faulty virtual machine is obtained through the topology awareness module in the system. It should be noted that the topology information represents the connection and relationship structure between the faulty virtual machine and its related resources, such as the network of the virtual machine, the storage of the virtual machine, the physical host where the virtual machine is located, the switch associated with the virtual machine, and so on. The topology information may specifically include the mapping relationship between the virtual machine and the host, that is, the host to which the virtual machine belongs and the resource allocation situation; include the storage topology, that is, the storage volumes mounted by the virtual machine and the load status of the storage nodes; and also include the network topology, that is, the network connection of the virtual machine and the configuration of the switch and router.

[0057] Finally, the multi-source data fusion unit parses the topology information of the faulty virtual machine, and traverses all associated logs in the topology relationship of the faulty virtual machine within the preset time window, at least including virtualization platform logs, virtual machine kernel logs, and application logs.

[0058] In this way, by collecting multi-source data for subsequent root cause location, the accurate identification of the root cause of the failure is ensured.

[0059] Based on the above embodiments, in some embodiments, before retrieving in the pre-constructed causal experience library according to the alarm information, after collecting the relevant logs of the faulty virtual machine and the performance metrics of the device where the faulty virtual machine is located according to the alarm information, it further includes:

[0060] S15: Perform data preprocessing on the performance metrics, topology information, and relevant logs;

[0061] S16: Generate a multi-source data sample corresponding to the faulty virtual machine according to the performance metrics, topology information, and relevant logs after data preprocessing.

[0062] In order to better perform root cause retrieval and reasoning for faults, after obtaining the relevant logs of the faulty virtual machine and the performance metrics of the device where the faulty virtual machine is located, in some embodiments, it is also necessary to preprocess the collected raw data and convert it into a data format that is readable and easy to analyze.

[0063] Specifically, the multi-source data fusion unit performs data preprocessing on the collected performance metrics, topology information, and relevant logs, including but not limited to removing abnormal data, filling null values, and unit standardization. Further, the multi-source data fusion unit performs data fusion on the performance metrics, topology information, and relevant logs after data preprocessing, and finally generates a multi-source data sample corresponding to the faulty virtual machine, so as to facilitate subsequent root cause retrieval or causal reasoning using the multi-source data sample. The data preprocessing process and the generation process of the multi-source data sample will be described in detail below with specific embodiments:

[0064] In some embodiments, performing data preprocessing on the performance metrics, topology information, and relevant logs includes:

[0065] S151: Remove the outliers in the performance metrics and replace the null values in the performance metrics with the average value of the corresponding performance metrics within a preset time window.

[0066] S152: Unify the timestamp format of the performance metrics, standardize the performance metrics of the same type, and generate a performance metric form according to the standardized performance metrics.

[0067] S153: Construct a topology graph according to the topology information and generate a topology form according to the topology graph.

[0068] S154: Extract the key log information of the relevant logs according to the regular expression and standardize the key log information.

[0069] S155: Generate a log form according to the standardized key log information.

[0070] To implement the preprocessing of multi-source data, in this embodiment, outliers in performance metrics such as processor performance metrics and memory performance metrics are first removed, and null values in the performance metrics are replaced with the average values of the corresponding performance metrics within a preset time window, thereby completing the preliminary processing of the data. Further, the timestamp formats of the performance metrics are unified, for example, the timestamp format of the performance metrics is corrected to a unified UTC time. Finally, the performance metrics of the same type are standardized, that is, converted to a unified data unit to eliminate the dimension difference, so as to generate a performance metric form based on the standardized performance metrics. It can be understood that the performance metric form contains the standard performance metrics after data preprocessing.

[0071] Furthermore, a topology map is constructed according to the topology information, for example, virtual machines-hosts-storage-network devices and their dependencies are marked, and a topology form is generated according to the topology map. The key log information of the relevant logs is extracted according to the regular expression, and the key log information is standardized to unify the format. The key log information includes but is not limited to log timestamps, error codes, and error log details. Finally, a log form is generated according to the standardized key log information.

[0072] In summary, by generating a performance metric form, a topology form, and a log form, the preprocessing and standardization of multi-source data are completed, so as to better perform fault root cause retrieval and reasoning based on the standardized processing.

[0073] Correspondingly, according to the performance metrics, topology information, and relevant logs after data preprocessing, multi-source data samples corresponding to the faulty virtual machines are generated, including:

[0074] S161: Determine the average value of each performance metric in the performance metric form.

[0075] S162: Determine the faulty performance metrics at the alarm time as the average values of each performance metric.

[0076] S163: Determine the faulty key log information at the alarm time according to each key log information in the log form.

[0077] S164: Taking the alarm time as the time reference, integrate the faulty performance metrics in the performance metric form, the faulty topology map in the topology form, and the faulty key log information in the log form at the alarm time to generate multi-source data samples.

[0078] To achieve data fusion of multi-source data, in this embodiment, it is first necessary to determine the fault data at the alarm time. Specifically, since the performance metrics collect the performance fluctuation data within a preset time window rather than the performance metrics corresponding to the alarm time, it is necessary to calculate an average value for each performance metric and unify the timestamps to the performance fault time point, that is, to determine the average value of each performance metric as the fault performance metric at the alarm time.

[0079] Furthermore, since the log form also counts the data within a preset time window, it is also necessary to unify the timestamps to the performance fault time point, that is, to determine the fault key log information at the alarm time according to each key log information in the log form. Finally, taking the alarm time as the time reference, integrate the fault performance metrics in the performance metric form, the fault topology map in the topology form, and the fault key log information in the log form at the alarm time to generate a multi-source data sample.

[0080] Figure 3 It is a schematic diagram of the multi-source data sample provided by the embodiment of the present invention. As Figure 3 shown, a group of samples corresponds to one timestamp, including the performance metrics, topology information, and key log information at this fault time point. In summary, by fusing the data in the performance metric form, log form, and topology form at the alarm time into a group of multi-source data samples, multi-source data fusion is achieved, providing comprehensive and reliable data input for virtual machine fault location.

[0081] Based on the above embodiments, in some embodiments, inferring the target fault root cause information corresponding to the faulty virtual machine according to the alarm information, relevant logs, and performance metrics includes:

[0082] S131: Determine the alarm type of the faulty virtual machine according to the alarm information.

[0083] S132: Obtain the fault performance metrics, fault topology map, and fault key log information in the multi-source data sample.

[0084] S133: Generate the target fault root cause information according to the alarm type, fault performance metrics, fault topology map, fault key log information, and causal discovery algorithm.

[0085] To infer the target root cause information corresponding to the faulty virtual machine, in this embodiment, the alarm type of the faulty virtual machine is specifically determined according to the alarm information. The fault performance metrics, fault topology maps, and fault key log information in the multi-source data samples are obtained, that is, the variables related to the virtual machine fault are retrieved from the samples. For example, the variables are CPU StealTime, disk IOPS, network packet loss, and memory utilization rate, and the fault type is virtual machine lag. Finally, according to the alarm type, fault performance metrics, fault topology maps, fault key log information, and causal discovery algorithm, the target root cause information is generated.

[0086] It should be noted that causal discovery algorithms are a class of computational methods for inferring causal relationships between variables from observational data. These algorithms aim to construct a directed acyclic graph (DAG), where nodes represent variables and directed edges represent the causal influence of one variable on another. By analyzing the conditional independence between variables, causal discovery algorithms can identify potential causal relationships, rather than just discovering statistical correlations. Common methods include constraint-based algorithms (such as the PC algorithm), score-based algorithms (such as Bayesian networks), and function-based algorithms (such as LiNGAM). In this embodiment, the type of causal discovery algorithm specifically selected is not limited and depends on the specific implementation situation.

[0087] In summary, generating the target root cause information according to the alarm type, fault performance metrics, fault topology maps, fault key log information, and causal discovery algorithm can accurately identify the root cause of the fault and improve the real-time performance and accuracy of root cause localization.

[0088] In addition, after obtaining the target root cause information, in order to more accurately locate and troubleshoot the fault, in some embodiments, it is also necessary to evaluate the impact scope of the fault on other systems or services. This includes identifying all dependencies and interaction points related to the faulty virtual machine and determining whether the fault has caused service interruption or performance degradation. According to the evaluation results, it is determined whether emergency measures need to be taken to restore the service to minimize the impact on business operations.

[0089] Furthermore, according to the information provided by the target root cause information, corresponding repair measures are taken. This may involve replacing faulty hardware, such as hard disks or memory modules; updating software versions to address known bugs or vulnerabilities; or modifying configuration files to correct incorrect settings or parameters. When implementing repair measures, best practices and vendor guidelines should be followed to ensure the safety and effectiveness of the repair process.

[0090] Finally, after implementing the repair measures, test the virtual machine to verify the repair effect. This includes checking system logs and monitoring metrics to confirm that the fault has been resolved and the system is running normally. In addition, functional tests and performance tests should also be performed to ensure that the virtual machine can meet the expected requirements and verify that the repair measures have not introduced new errors or problems. Through comprehensive verification, it can be ensured that the fault has been completely resolved and the stability and reliability of the system have been restored.

[0091] Based on the above embodiments, in some embodiments, after inferring the target root cause information of the faulty virtual machine according to the alarm information, relevant logs and performance metrics, it further includes:

[0092] S17: Output the fault cause diagram corresponding to the target root cause information through the display interface.

[0093] S18: Store the target root cause information in the causal experience database.

[0094] In specific implementation, after inferring the target root cause information of the faulty virtual machine, in order to facilitate the staff to more intuitively determine the cause of the virtual machine fault, the fault cause diagram corresponding to the target root cause information can also be output through the display interface, so that the staff can directly analyze the fault cause according to the fault cause diagram.

[0095] At the same time, since the target root cause information of the inferred faulty virtual machine does not exist in the causal experience database, the target root cause information can also be stored in the causal experience database, so as to update the causal experience database and improve the efficiency of fault root cause location during the next retrieval.

[0096] In addition, the construction process of the causal experience database is actually similar to the inference process of the root cause information. The following is a specific description:

[0097] Different from monitoring the alarm information of virtual machines on the virtualization platform during fault root cause detection, when constructing the causal experience database, the time range can be manually input, and the fault alarm detection unit can automatically retrieve the alarm information within this time range by calling the alarm history record, and parse and obtain all alarm types, faulty virtual machine IDs and alarm times within this time period from the alarm information to obtain a large amount of historical alarm data.

[0098] Similarly, when constructing the causal experience library, data collection, data preprocessing, and multi-source data fusion also need to be performed. Specifically, according to the alarm time in the historical alarm data, performance metrics, topology information, and relevant logs within the corresponding time window are collected. Subsequently, the performance metrics, topology information, and relevant logs are preprocessed, and based on the preprocessed performance metrics, topology information, and relevant logs, multi-source data samples corresponding to the faulty virtual machines are generated. The specific process is the same as the data collection process, data preprocessing process, and data fusion process in the above embodiments, and will not be elaborated here. It should be noted that different from only obtaining a set of multi-source data samples corresponding to the faulty virtual machines in the above embodiments, when constructing the causal experience library, multiple sets of multi-source data samples will be generated based on the historical alarm data (such as Figure 3 as shown).

[0099] Finally, using the causal inference engine, with multiple sets of multi-source data samples output by the multi-source data fusion unit as input samples, an array A = [list 1, list 2, list 3, list i,..., list n] is constructed, and a set of lists is a set of multi-source data samples. Taking list i as an example, list i is the variable value of the i-th group of samples, and list i = [CPU Steal Time, Disk IOPS, Memory Usage, Network Packet Loss]. Using the causal discovery algorithm to construct a causal graph, with array A as the input of the causal discovery algorithm, the causal analysis result will be output, including the root cause information of faults under various alarm types, thus completing the construction of the causal experience library.

[0100] Based on the above embodiments, in some embodiments, it further includes:

[0101] S19: Detect the topology information of each virtual machine according to a preset period.

[0102] S20: When it is detected that the topology information of the virtual machine has changed, update the topology information of the corresponding virtual machine.

[0103] In specific implementation, the topology awareness module inputs the initial topology information of the virtual machine, monitors the basic information of the virtual machines in the target cloud platform through a preset period, and can obtain the topology changes of the virtual machines in real time, such as the mounting and unmounting of storage volumes and network cards of the virtual machines; when it is detected that the topology information of the virtual machine has changed, update the topology information of the corresponding virtual machine, ensuring the accuracy of log data collection.

[0104] Through the description of the above implementation manners, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware, but in many cases, the former is a better implementation manner.

[0105] Figure 4The schematic diagram of a fault root cause location device provided by an embodiment of the present invention. As Figure 4 shown, the device includes:

[0106] An acquisition module 10, configured to acquire the alarm information of a faulty virtual machine.

[0107] A collection module 11, configured to collect the relevant logs of the faulty virtual machine and the performance metrics of the device where the faulty virtual machine is located according to the alarm information.

[0108] A retrieval and judgment module 12, configured to retrieve in a pre-constructed causal experience library according to the alarm information, and judge whether the target fault root cause information corresponding to the faulty virtual machine is retrieved; wherein, the causal experience library contains the fault root cause information of various virtual machine fault types generated according to the historical alarm information of multiple virtual machines; if not, the inference module 13 is triggered; if so, the output module 14 is triggered.

[0109] An inference module 13, configured to infer the target fault root cause information corresponding to the faulty virtual machine according to the alarm information, relevant logs and performance metrics.

[0110] An output module 14, configured to output the target fault root cause information corresponding to the faulty virtual machine.

[0111] In some embodiments, the collection module 11 includes:

[0112] A first determination sub-module, configured to determine the alarm time of the faulty virtual machine according to the alarm information, and construct a preset time window according to the alarm time; wherein, the alarm time is within the preset time window;

[0113] A first collection sub-module, configured to collect performance metrics within the preset time window through a monitoring alarm tool; wherein, the performance metrics at least include processor performance metrics, memory performance metrics, disk performance metrics and network performance metrics;

[0114] A first acquisition sub-module, configured to acquire the topology information of the faulty virtual machine; wherein, the topology information characterizes the connection and relationship structure between the faulty virtual machine and its related resources;

[0115] A first retrieval sub-module, configured to retrieve relevant logs within the preset time window according to the topology information; wherein, the relevant logs at least include virtualization platform logs, virtual machine kernel logs and application logs.

[0116] In some embodiments, it further includes:

[0117] A data preprocessing module, configured to perform data preprocessing on the performance metrics, topology information and relevant logs;

[0118] A data fusion module, configured to generate multi-source data samples corresponding to faulty virtual machines based on the performance metrics, topology information, and relevant logs after data preprocessing.

[0119] In some embodiments, the data preprocessing module includes:

[0120] A numerical processing module, configured to remove outliers from the performance metrics and replace null values in the performance metrics with the average value of the corresponding performance metrics within a preset time window;

[0121] A performance metric form construction module, configured to unify the timestamp format of the performance metrics, standardize the performance metrics of the same type, and generate a performance metric form based on the standardized performance metrics;

[0122] A topology form construction module, configured to construct a topology graph based on the topology information and generate a topology form according to the topology graph;

[0123] A standardization module, configured to extract key log information from the relevant logs according to a regular expression and standardize the key log information;

[0124] A log form construction module, configured to generate a log form based on the standardized key log information;

[0125] Correspondingly, the data fusion module includes:

[0126] An average value determination module, configured to determine the average value of each performance metric in the performance metric form;

[0127] A second determination sub-module, configured to determine the faulty performance metric at the alarm time as the average value of each performance metric;

[0128] A third determination sub-module, configured to determine the faulty key log information at the alarm time according to each key log information in the log form;

[0129] A data integration module, configured to integrate the faulty performance metrics in the performance metric form, the faulty topology graph in the topology form, and the faulty key log information in the log form at the alarm time with the alarm time as the time reference to generate multi-source data samples.

[0130] In some embodiments, the inference module 13 includes:

[0131] A fourth determination sub-module, configured to determine the alarm type of the faulty virtual machine according to the alarm information;

[0132] A second acquisition sub-module, configured to acquire the faulty performance metrics, the faulty topology graph, and the faulty key log information in the multi-source data samples;

[0133] A generating sub-module, configured to generate target fault root cause information according to alarm types, fault performance metrics, fault topology maps, fault critical log information, and causal discovery algorithms.

[0134] In some embodiments, it further includes:

[0135] An output sub-module, configured to output a fault root cause diagram corresponding to the target fault root cause information through a display interface;

[0136] A storage sub-module, configured to store the target fault root cause information in a causal experience database.

[0137] In some embodiments, it further includes:

[0138] A topology detection sub-module, configured to detect topology information of each virtual machine according to a preset period;

[0139] A topology update sub-module, configured to update the topology information of the corresponding virtual machine when it is detected that the topology information of the virtual machine has changed.

[0140] For the description of the features in the corresponding embodiments of the fault root cause location device, reference can be made to the relevant descriptions in the corresponding embodiments of the fault root cause location method, which will not be elaborated here one by one.

[0141] An embodiment of the present invention further provides an electronic device, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any of the above embodiments of the fault root cause location method.

[0142] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps in any of the above embodiments of the fault root cause location method when running.

[0143] In an exemplary embodiment, the above computer-readable storage medium may include, but is not limited to: USB flash drives, read-only memories (ROM for short), random access memories (RAM for short), mobile hard disks, magnetic disks, or optical discs, etc., various media that can store computer programs.

[0144] An embodiment of the present invention further provides a computer program product. The above computer program product includes a computer program, and when the computer program is executed by a processor, it implements the steps in any of the above embodiments of the fault root cause location method.

[0145] An embodiment of the present invention further provides another computer program product, including a non-volatile computer-readable storage medium storing a computer program, where the computer program, when executed by a processor, implements the steps in any of the above-described embodiments of the root cause localization method.

[0146] Those skilled in the art can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0147] The above has introduced in detail a root cause localization method, device, equipment, and medium provided by the present invention. Specific examples are used herein to elaborate on the principle and implementation manner of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention. It should be noted that for those of ordinary skill in the art in the technical field, without departing from the principle of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the protection scope of the present invention.

Claims

1. A method for locating the root cause of a fault, characterized in that: include: Get the alarm information of the faulty virtual machine; Collect relevant logs of the faulty virtual machine and performance indicators of the device where the faulty virtual machine is located according to the alarm information; Searching a pre-built causal experience database according to the alarm information to determine whether target fault root cause information corresponding to the faulty virtual machine is retrieved; wherein the causal experience database contains fault root cause information of multiple virtual machine fault types generated according to historical alarm information of multiple virtual machines; If not, inferring the target fault root cause information corresponding to the faulty virtual machine according to the alarm information, the relevant logs and the performance indicators; If so, the target fault root cause information corresponding to the faulty virtual machine is output.

2. The method for locating the root cause of a fault according to claim 1, characterized in that: Collecting the relevant logs of the faulty virtual machine and the performance indicators of the device where the faulty virtual machine is located according to the alarm information includes: Determine the alarm time of the faulty virtual machine according to the alarm information, and construct a preset time window according to the alarm time; wherein the alarm time is within the preset time window; The performance indicators are collected within the preset time window by a monitoring alarm tool; wherein the performance indicators include at least processor performance indicators, memory performance indicators, disk performance indicators and network performance indicators; Acquire topology information of the faulty virtual machine; wherein the topology information represents the connection and relationship structure between the faulty virtual machine and its related resources; The relevant logs are retrieved within the preset time window according to the topology information; wherein the relevant logs at least include virtualization platform logs, virtual machine kernel logs and application logs.

3. The method for locating the root cause of a fault according to claim 2, characterized in that: Before searching the pre-built causal experience database according to the alarm information, and after collecting the relevant logs of the faulty virtual machine and the performance indicators of the device where the faulty virtual machine is located according to the alarm information, the method further includes: Performing data preprocessing on the performance indicator, the topology information and the related logs; A multi-source data sample corresponding to the faulty virtual machine is generated according to the performance indicator, the topology information and the related logs after data preprocessing.

4. The method for locating the root cause of a fault according to claim 3, characterized in that: The performance indicator, the topology information and the related logs are subjected to data preprocessing, including: Remove abnormal values ​​in the performance indicator, and replace null values ​​in the performance indicator with an average value corresponding to the performance indicator within the preset time window; Unifying the timestamp format of the performance indicators, standardizing the performance indicators of the same type, and generating a performance indicator form according to the standardized performance indicators; Construct a topology map according to the topology information, and generate the topology form according to the topology map; Extract key log information of the relevant logs according to regular expressions, and standardize the key log information; Generate a log form according to the standardized key log information; Correspondingly, generating a multi-source data sample corresponding to the faulty virtual machine according to the performance indicator, the topology information and the related log after data preprocessing, including: Determine the average value of each performance indicator in the performance indicator table; Determine the average value of each of the performance indicators as the fault performance indicator of the alarm time; Determine the critical log information of the fault at the alarm time according to each critical log information in the log form; Taking the alarm time as the time reference, the fault performance indicators in the performance indicator form at the alarm time, the fault topology map in the topology form, and the fault critical log information in the log form are integrated to generate the multi-source data sample.

5. The method for locating the root cause of a fault according to claim 4, characterized in that: Inferring the target fault root cause information corresponding to the faulty virtual machine according to the alarm information, the relevant logs and the performance indicators includes: Determine the alarm type of the faulty virtual machine according to the alarm information; Acquire the fault performance indicator, the fault topology map and the key fault log information in the multi-source data sample; The target fault root cause information is generated according to the alarm type, the fault performance index, the fault topology map, the key fault log information and the causal discovery algorithm.

6. The fault root cause location method according to any one of claims 2 to 5, characterized in that: After inferring the target fault root cause information corresponding to the faulty virtual machine according to the alarm information, the relevant logs and the performance indicators, the method further includes: Outputting a fault root cause graph corresponding to the target fault root cause information through a display interface; The target fault root cause information is stored in the causal experience library.

7. The method for locating the root cause of a fault according to claim 6, characterized in that: Also includes: Detecting topology information of each virtual machine according to a preset period; When a change in the topology information of a virtual machine is detected, the topology information of the corresponding virtual machine is updated.

8. A fault root cause location device, characterized in that: include: An acquisition module is used to obtain alarm information of a faulty virtual machine; A collection module, used to collect relevant logs of the faulty virtual machine and performance indicators of the device where the faulty virtual machine is located according to the alarm information; A retrieval and judgment module is used to search the pre-built causal experience library according to the alarm information to determine whether the target fault root cause information corresponding to the faulty virtual machine is retrieved; wherein the causal experience library contains fault root cause information of multiple virtual machine fault types generated according to the historical alarm information of multiple virtual machines; if not, the reasoning module is triggered; if yes, the output module is triggered; The inference module is used to infer the target fault root cause information corresponding to the faulty virtual machine according to the alarm information, the relevant logs and the performance indicators; The output module is used to output the target fault root cause information corresponding to the faulty virtual machine.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor is used to implement the steps of the fault root cause locating method as claimed in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the fault root cause locating method according to any one of claims 1 to 7.

Images