A network fault root cause analysis method, terminal device and storage medium

Abstract

The present application relates to a network fault root cause analysis method, terminal equipment and storage medium, and the method includes the following steps: collecting initial data, the initial data includes system logs, operation logs and business data; obtaining multiple alarm events from the system logs, and determining the fault time period based on the alarm events; performing feature analysis on the multiple alarm events in combination with the business data to determine the initial fault root cause; performing operational analysis on the initial fault root cause and the operation log to determine the final fault root cause; and saving the final fault cause to a preset fault data warehouse. The present application has the effect of reducing the troubleshooting time and scope of monitoring personnel and quickly locating the root cause of the fault.

Description

Technical Field

[0001] The present application relates to the technical field of network fault analysis, and in particular to a network fault root cause analysis method, terminal equipment and storage medium. Background Art

[0002] Enterprises often encounter production environment failures, which will cause many monitoring indicators to be abnormal. Currently, network failures in network monitoring systems are mostly based on two solutions: network logs and traffic collection. Failure alarms are triggered based on keywords in the logs and traffic exceeding the limit.

[0003] Regarding the above-mentioned related technologies, the inventor believes that the fault alarms obtained through keywords in the log or through traffic monitoring cannot accurately reflect network failures. Since the network connection relationship is relatively complex, when a core switch fails, multiple devices interconnected with the switch will also trigger related alarms of the network link, resulting in a large number of alarm email bombardment. The monitoring personnel cannot accurately know the cause of the failure based on the alarm, and need to spend a lot of time on manual investigation. Summary of the invention

[0004] In order to reduce the troubleshooting time and scope of monitoring personnel and quickly locate the root cause of the fault, the present application provides a network fault root cause analysis method, terminal device and storage medium.

[0005] In a first aspect, the present application provides a network fault root cause analysis method, which adopts the following technical solution:

[0006] A network fault root cause analysis method comprises the following steps:

[0007] Collecting initial data, including system logs, operation logs and business data;

[0008] Acquire multiple alarm events from the system log, and determine a fault time period based on the alarm events;

[0009] Performing feature analysis on the plurality of alarm events in combination with the business data to determine the initial fault root cause;

[0010] Performing an operation analysis on the initial fault root cause and the operation log to determine a final fault root cause;

[0011] The final fault cause is saved in a preset fault data warehouse.

[0012] By adopting the above technical solution, the alarm events in the system log are obtained, and by collecting the business data within the fault time period, the initial root cause of the equipment failure, that is, the hardware cause, is determined and accurately located by analyzing the operation logs collected within the fault time period and the initial root cause of the failure. It can be determined that the final root cause of the failure is caused by improper operation of the operation and maintenance personnel or by hardware failure itself. When multiple alarm events occur, this application can reduce the monitoring personnel's troubleshooting time and scope, improve the efficiency of fault handling, and reduce the losses caused by equipment failure.

[0013] Optionally, the obtaining of multiple alarm events from the system log comprises the following steps:

[0014] Building a fault template based on the system log;

[0015] Alarm events are extracted from the system log based on the fault template.

[0016] By adopting the above technical solution and building a fault template according to the newly received system log, alarm events in massive system logs can be quickly located.

[0017] Optionally, the constructing a fault template based on the system log includes the following steps:

[0018] Obtaining a sample log from the system log;

[0019] Extracting feature words from the sample log;

[0020] A fault template is constructed based on the feature words.

[0021] Optionally, the performing feature analysis on the alarm event in combination with the business data to determine the initial fault cause comprises the following steps:

[0022] determining the fault levels of a plurality of the alarm events;

[0023] Acquire the real value and the predicted value of the business data within the fault time period, and calculate the deviation of the business data based on the real value and the predicted value;

[0024] Obtaining anomaly similarities of sub-dimensions constituting the alarm event;

[0025] An initial fault root cause is determined based on the fault level, the deviation, and the abnormality similarity.

[0026] By adopting the above technical solution, business data within the fault time period is collected, and analyzed from multiple aspects such as time correlation, dependency correlation and level correlation. Alarm events with high influence, large business data deviation and high anomaly similarity are selected as the initial root cause of the fault, and the root cause of the fault is predicted with accuracy and comprehensiveness.

[0027] Optionally, the performing an operational analysis on the initial fault root cause and the alarm event to determine the final fault root cause comprises the following steps:

[0028] Determining whether to perform operation and maintenance operations within the fault time period based on the operation log;

[0029] If the operation and maintenance operation is not performed, the initial fault root cause is used as the final fault root cause;

[0030] If the operation and maintenance operation has been performed, the operation and maintenance operation is analyzed to obtain an analysis result, and the final root cause of the fault is determined based on the analysis result.

[0031] By adopting the above technical solution, it is possible to determine whether the final root cause of the failure is a hardware failure or improper operation by the operation and maintenance personnel, which facilitates processing.

[0032] Optionally, analyzing the operation and maintenance operation to obtain an analysis result, and determining the final root cause of the fault based on the analysis result comprises the following steps:

[0033] Compare the operation and maintenance operation with the historical operation data in the preset operation warehouse to determine whether the operation and maintenance operation is a high-risk operation;

[0034] If the operation and maintenance operation is a high-risk operation, determining the high-risk operation as the final fault root cause;

[0035] If the operation and maintenance operation is a normal operation, determining the initial fault root cause as the final fault root cause;

[0036] If the operation and maintenance operation is a new category operation, the operation and maintenance operation is rated to determine whether the operation and maintenance operation is the high-risk operation or the normal operation, and the operation and maintenance operation is saved in a preset operation warehouse as a sample.

[0037] Optionally, the step of saving the final fault cause to a preset fault data warehouse comprises the following steps:

[0038] Correct the final cause of the failure;

[0039] The final fault cause is saved in a fault data warehouse.

[0040] By adopting the above technical solution, the corrected final root cause of the fault is stored in the fault data warehouse. As time accumulates, the fault data warehouse will become richer and richer, and the fault location time can be further shortened, thereby reducing the troubleshooting time.

[0041] In a second aspect, the present application provides a terminal device, which adopts the following technical solution:

[0042] A terminal device comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor. When the processor loads and executes the computer program, the above-mentioned network fault root cause analysis method is adopted.

[0043] By adopting the above technical solution, the above network fault root cause analysis method is generated into a computer program and stored in a memory so as to be loaded and executed by a processor, thereby manufacturing a terminal device based on the memory and the processor for easy use.

[0044] In a third aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution:

[0045] A computer-readable storage medium stores a computer program, and when the computer program is loaded and executed by a processor, the above-mentioned network fault root cause analysis method is adopted.

[0046] By adopting the above technical solution, the above network fault root cause analysis method is generated into a computer program and stored in a computer-readable storage medium so as to be loaded and executed by a processor. The computer-readable storage medium facilitates the reading and storage of the computer program. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a schematic diagram of the overall process of a network fault root cause analysis method according to an embodiment of the present application.

[0048] Figure 2 It is a flowchart of step S21-step S22 in a network fault root cause analysis method according to an embodiment of the present application.

[0049] Figure 3 It is a flowchart of steps S31 to S33 in a network fault root cause analysis method according to an embodiment of the present application.

[0050] Figure 4 It is a flowchart of steps S41 to S44 in a network fault root cause analysis method according to an embodiment of the present application.

[0051] Figure 5 It is a model diagram of an association tree in a network fault root cause analysis method in an embodiment of the present application.

[0052] Figure 6 It is a flowchart of steps S51 to S53 in a network fault root cause analysis method according to an embodiment of the present application.

[0053] Figure 7 It is a flowchart of steps S61 to S64 in a network fault root cause analysis method according to an embodiment of the present application.

[0054] Figure 8 It is a flowchart of step S71-step S72 in a network fault root cause analysis method according to an embodiment of the present application. DETAILED DESCRIPTION

[0055] The following is combined with Figure 1-8 This application is described in further detail.

[0056] The present application discloses a network fault root cause analysis method. Figure 1 , including the following steps:

[0057] S11. Collect initial data, including system logs, operation logs and business data;

[0058] S12, obtaining multiple alarm events from the system log, and determining the fault time period based on the alarm events;

[0059] S13, analyzing the characteristics of multiple alarm events in combination with business data to determine the initial fault root cause;

[0060] S14, performing operation analysis on the initial fault root cause and operation log to determine the final fault root cause;

[0061] S15. Save the final fault cause to a preset fault data warehouse.

[0062] Among them, in step S11, in the Internet operation and maintenance service, the initial data of the device is collected and analyzed. In this embodiment, the initial data includes system logs, operation logs and business data. The system logs of the device are collected by the log collector; the operation logs are collected by Logstash; the business data includes but is not limited to business traffic, business request times, business success times, etc.

[0063] In step S12, after receiving the fault alarm, the alarm event is extracted from the system log for analysis. Figure 2 , specifically including the following steps:

[0064] S21. Build a fault template based on the system log;

[0065] S22. Extract alarm events from the system log based on the fault template.

[0066] Specifically, for the case of fault alarms in the system log, a fault template can be constructed for each fault alarm, that is, one fault template corresponds to one fault alarm event, so that when the same system log content is encountered again, the cause of the fault can be quickly found. In this embodiment, when a new system log fault occurs, a fault template can be constructed by extracting the characteristic words of the fault. Figure 3 , specifically including the following steps:

[0067] S31, classifying system logs and obtaining sample logs;

[0068] S32, extracting feature words from the sample log;

[0069] S33. Construct a fault template based on the feature words.

[0070] More specifically, in this embodiment, when the system log of a new device is obtained, the log is first classified, the feature words in the system log are extracted, and a log template is constructed according to the frequency of the feature words. When a device fails, the log template extracted from the system log is used as a fault template. The judgment process can be performed manually. After obtaining the fault template, the same system log is obtained again, and the fault template can be quickly matched, and the log is determined to be an alarm event. At the same time, each time a new fault template is determined, the fault template can be saved. More and more fault templates are saved, which can cover the system faults more comprehensively, so that the alarm event can be quickly located from the massive system logs, greatly reducing the workload of manual troubleshooting.

[0071] More specifically, multiple alarm events may be generated at the same time. For example, if the host fan of the device fails, a series of chain reactions will occur, generating multiple alarm events, such as fan stoppage, high host CPU temperature, slow disk reading speed, etc. These alarm events will be analyzed later to find the ultimate root cause of the failure. At the same time, the fault time period is set according to the time when multiple alarms appear. In this embodiment, the preset fault time period is 5 minutes.

[0072] In step S13, the business data within the fault time period is collected, and the characteristics of multiple alarm events are analyzed in combination with the changes in the business data to determine the initial fault root cause. Figure 4 , specifically including the following steps:

[0073] S41, determining the fault levels of multiple alarm events;

[0074] S42, obtaining the actual value and predicted value of the business data during the fault period, and calculating the deviation of the business data based on the actual value and the predicted value;

[0075] S43, obtaining the abnormal similarity of the sub-dimensions constituting the alarm event;

[0076] S44. Determine the initial fault root cause based on the fault level, deviation and abnormal similarity.

[0077] Specifically, in step S41, the fault level of the alarm event is set, and the influence of the alarm event is judged according to the set fault level. In this embodiment, the fault level of the alarm event is set to 5 levels, namely: emergency, serious, normal, warning and notification. An association tree for each type of fault is constructed based on historical alarm event data. The root node in the association tree corresponds to the alarm event with the highest fault level, and a series of alarm events generated by the alarm event with the highest fault level are child nodes of the association tree. The first construction of the association tree requires manual confirmation of the fault level of each alarm event to construct the association tree. For example, referring to Figure 5 , if the fan of the device fails, the alarm event priority of the fan failure is set to severe, and the fan failure is set as the root node of the association tree. The high CPU temperature, high power temperature, and high disk temperature caused by the fan failure are used as child nodes of the fan failure, and the fault levels are defined as severe, serious, and ordinary, respectively, to build an association tree. When the number of association trees is large enough, each type of alarm event can correspond to a type of association tree, so that multiple alarm events correspond to one type of association tree.

[0078] Specifically, in step S42, the real value and predicted value of the business data are obtained during the fault time period. Taking the business growth as an example, the business growth under normal circumstances is 10%, that is, 10% is the predicted value of the business growth, and the business growth during the fault time period is 1%, and 1% is the real value of the business growth. In this embodiment, the obtained business data deviation is the difference between the real value and the predicted value.

[0079] Specifically, in step S43, after obtaining the alarm event, the abnormal similarity of the sub-dimensions constituting the alarm event is determined according to the association tree. In this embodiment, the abnormal degree between each sub-dimension is measured by the coefficient of variation of the sub-dimension, and the smaller the coefficient of variation, the more consistent the abnormal degree. For example, if the fan fails, the CPU temperature of the sub-dimension is abnormal, and the power supply temperature is abnormal, then the abnormal similarity of determining that the alarm event is a fan failure is high.

[0080] Specifically, in step S44, the alarm data is evaluated according to the influence, deviation and abnormal similarity of each alarm event, and the alarm event with high influence, large business data deviation and high abnormal similarity is determined as the initial fault root cause.

[0081] More specifically, in order to optimize the process of predicting the root cause of the initial fault and improve the accuracy of fault prediction, this embodiment performs optimization through four steps: data prediction, pruning search, root cause decision, and algorithm evaluation.

[0082] Data prediction: using Monte Carlo book search algorithm and second-order exponential smoothing algorithm;

[0083] Pruning search: Search the root cause layer by layer, and filter according to the influence value and PS value of the ripple effect;

[0084] Root cause decision: sort the root causes and correct the root cause combination through Occam’s razor;

[0085] Algorithm evaluation: Score evaluation of root cause output results.

[0086] Through these four processes, the above fault prediction process can be optimized, the impact of extreme values ​​can be reduced, and timeliness and accuracy can be improved.

[0087] Among them, in step S14, the initial fault root cause is obtained, and the faulty hardware can be located. However, in fact, among common production failures, there are not many failures caused by hardware aging. Most of the real failures are caused by improper operation of the operation and maintenance personnel. Therefore, after obtaining the initial fault root cause, it is necessary to determine whether the operation and maintenance personnel have performed high-risk operations to obtain the final fault root cause. Figure 6 , specifically including the following steps:

[0088] S51. Determine whether to perform operation and maintenance operations within the fault time period based on the operation log;

[0089] S52: If the operation and maintenance operation is not performed, the initial fault root cause is used as the final fault root cause;

[0090] S53: If the operation and maintenance operation has been performed, the operation and maintenance operation is analyzed to obtain an analysis result, and the final root cause of the fault is determined based on the analysis result.

[0091] Specifically, after the alarm event is acquired, the operation logs within the fault time period before the alarm event occurs are collected. In this embodiment, the operation logs within 5 minutes before the alarm event are collected. The operation logs are analyzed. If the operation and maintenance personnel have not performed any operation and maintenance operations within the fault time period, it is determined that the fault is not caused by improper operation. At this time, the initial root cause of the fault is used as the final root cause and output; if the operation and maintenance personnel have performed any operation and maintenance operations within the fault time period, it is necessary to analyze the operation and maintenance operations, obtain the analysis results, and determine the final root cause of the fault based on the analysis results. Figure 7 , specifically including the following steps:

[0092] S61, comparing the operation and maintenance operation with the historical operation data in the preset operation warehouse to determine whether the operation and maintenance operation is a high-risk operation;

[0093] S62. If the operation and maintenance operation is a high-risk operation, determine the high-risk operation as the final root cause of the fault;

[0094] S63. If the operation and maintenance operation is normal, determining the initial fault root cause as the final fault root cause;

[0095] S64. If the operation and maintenance operation is a new category operation, the operation and maintenance operation is rated to determine whether the operation and maintenance operation is a high-risk operation or a normal operation, and the operation and maintenance operation is saved in a preset operation warehouse as a sample.

[0096] Among them, in step S61, the operation and maintenance operations of the operation and maintenance personnel will be manually graded and stored in a preset operation warehouse, including high-risk operations and normal operations. After the operation and maintenance personnel perform the operation and maintenance operation within the fault time period, the operation and maintenance operation is compared with the historical operation data in the operation warehouse. If the operation and maintenance operation can be matched to a high-risk operation, it is determined that the equipment failure is caused by improper operation of the operation and maintenance personnel, and the final root cause of the failure is determined to be the operation and maintenance operation of the operation and maintenance personnel; if the operation and maintenance operation can be matched to a normal operation, it is determined that the operation and maintenance personnel operate correctly, and the equipment failure is caused by a hardware problem. At this time, the initial root cause of the failure is determined to be the final root cause of the failure; if the corresponding historical operation data is not matched in the operation warehouse, the operation and maintenance operation is determined to be a new type of operation. At this time, it is necessary to manually grade the operation and maintenance operation to determine whether it is a high-risk operation or a normal operation. If it is a high-risk operation, it is determined that the equipment failure is caused by improper operation of the operation and maintenance personnel, and the final root cause of the failure is determined to be the operation and maintenance operation of the operation and maintenance personnel; if it is a normal operation, it is determined that the operation and maintenance personnel operate correctly, and the equipment failure is caused by a hardware problem. At this time, it is determined that the initial root cause of the failure is the final root cause of the failure, and the new type of operation and maintenance operation is saved in the operation warehouse as a new sample for subsequent matching of subsequent operation and maintenance operations.

[0097] Among them, refer to Figure 8 , step S15 comprises the following steps:

[0098] S71. Correct the final cause of the failure;

[0099] S72. Save the final fault cause to the fault data warehouse.

[0100] Specifically, after determining the final root cause of the fault, in order to further ensure the accuracy of the final root cause of the fault, the final root cause of the fault is manually analyzed and corrected, and the corrected final root cause of the fault is stored in the fault data warehouse. At the same time, the fault data warehouse and the operation warehouse can be the same. When a fault occurs, the fault data warehouse can be queried first to match the sample, thereby improving the efficiency of fault location. Of course, as time accumulates, the fault data warehouse will become more and more abundant, and the fault location time can be further shortened, thereby reducing the troubleshooting time and reducing the adverse effects of equipment failures.

[0101] The implementation principle of a network fault root cause analysis method in an embodiment of the present application is: obtain alarm events in the system log, collect business data within the fault time period, analyze from multiple aspects such as time correlation, operation correlation, dependency correlation and level correlation, select alarm events with high influence, large business data deviation and high abnormal similarity as the initial fault root cause, and accurately locate it; by collecting operation logs within the fault time period, it can be determined whether the final root cause of the fault is caused by improper operation of the operation and maintenance personnel or by hardware failure itself. This application is able to narrow the scope of the fault when multiple alarm events occur, reduce the time of manual troubleshooting, thereby improving the efficiency of fault handling and reducing the losses caused by equipment failure.

[0102] An embodiment of the present application further discloses a terminal device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein when the processor executes the computer program, the network fault root cause analysis method in the above embodiment is adopted.

[0103] The terminal device may be a computer device such as a desktop computer, a laptop computer or a cloud server, and the terminal device includes but is not limited to a processor and a memory. For example, the terminal device may also include input and output devices, a network access device and a bus.

[0104] Among them, the processor can adopt a central processing unit (CPU). Of course, according to actual usage, other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. can also be adopted. The general-purpose processor can adopt a microprocessor or any conventional processor, etc., and this application does not impose any restrictions on this.

[0105] Among them, the memory can be an internal storage unit of the terminal device, such as a hard disk or memory of the terminal device, or it can be an external storage device of the terminal device, such as a plug-in hard disk, a smart memory card (SMC), a secure digital card (SD) or a flash memory card (FC) equipped on the terminal device, and the memory can also be a combination of an internal storage unit and an external storage device of the terminal device. The memory is used to store computer programs and other programs and data required by the terminal device. The memory can also be used to temporarily store data that has been output or is to be output, and this application does not impose any restrictions on this.

[0106] Among them, through this terminal device, the network fault root cause analysis method in the above embodiment is stored in the memory of the terminal device, and is loaded and executed on the processor of the terminal device, which is convenient for use.

[0107] The embodiment of the present application further discloses a computer-readable storage medium, and the computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the network fault root cause analysis method in the above embodiment is adopted.

[0108] Among them, the computer program can be stored in a computer-readable medium, the computer program includes computer program code, the computer program code can be in the form of source code, object code, executable file or certain middleware, etc. The computer-readable medium includes any entity or device that can carry the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the computer-readable medium includes but is not limited to the above-mentioned components.

[0109] Among them, through this computer-readable storage medium, the network fault root cause analysis method in the above embodiment is stored in a computer-readable storage medium, and is loaded and executed on a processor to facilitate the storage and application of the above method.

[0110] The above are all preferred embodiments of the present application, and the protection scope of the present application is not limited thereto. Therefore, any equivalent changes made according to the structure, shape, and principle of the present application should be included in the protection scope of the present application.

Claims

1. A network fault root cause analysis method, characterized in that: The following steps are involved: Collecting initial data, including system logs, operation logs and business data; Acquire multiple alarm events from the system log, and determine a fault time period based on the alarm events; Performing feature analysis on the plurality of alarm events in combination with the business data to determine the initial fault root cause; Performing an operation analysis on the initial fault root cause and the operation log to determine a final fault root cause; Saving the final root cause of the fault to a preset fault data warehouse; The feature analysis of the alarm event in combination with the business data to determine the initial fault root cause comprises the following steps: determining the fault levels of a plurality of the alarm events; Acquire the real value and the predicted value of the business data within the fault time period, and calculate the deviation of the business data based on the real value and the predicted value; Acquire the abnormal similarity of the sub-dimensions constituting the alarm event, wherein the abnormal similarity of the sub-dimensions is the degree of abnormal similarity between the sub-dimensions; determining an initial fault root cause based on the fault level, the deviation, and the abnormal similarity; The alarm data is evaluated according to the fault level, deviation and anomaly similarity of each alarm event, and the alarm events with high fault level, large business data deviation and high anomaly similarity of sub-dimensions are determined as the initial fault root causes.

2. A network fault root cause analysis method according to claim 1, characterized in that: The step of obtaining multiple alarm events from the system log comprises the following steps: Building a fault template based on the system log; Alarm events are extracted from the system log based on the fault template.

3. A network fault root cause analysis method according to claim 2, characterized in that: The constructing of the fault template based on the system log comprises the following steps: Obtaining a sample log from the system log; Extracting feature words from the sample log; A fault template is constructed based on the characteristic words.

4. A network fault root cause analysis method according to claim 1, characterized in that: The performing of operation analysis on the initial fault root cause and the operation log to determine the final fault root cause comprises the following steps: Determining whether to perform operation and maintenance operations within the fault time period based on the operation log; If the operation and maintenance operation is not performed, the initial fault root cause is used as the final fault root cause; If the operation and maintenance operation has been performed, the operation and maintenance operation is analyzed to obtain an analysis result, and the final root cause of the fault is determined based on the analysis result.

5. A network fault root cause analysis method according to claim 4, characterized in that: Analyzing the operation and maintenance operation to obtain an analysis result, and determining the final root cause of the fault based on the analysis result comprises the following steps: Compare the operation and maintenance operation with the historical operation data in the preset operation warehouse to determine whether the operation and maintenance operation is a high-risk operation; If the operation and maintenance operation is a high-risk operation, determining the high-risk operation as the final fault root cause; If the operation and maintenance operation is a normal operation, determining the initial fault root cause as the final fault root cause; If the operation and maintenance operation is a new category operation, the operation and maintenance operation is rated to determine whether the operation and maintenance operation is the high-risk operation or the normal operation, and the operation and maintenance operation is saved in a preset operation warehouse as a sample.

6. A network fault root cause analysis method according to claim 1, characterized in that: The step of saving the final root cause of the fault to a preset fault data warehouse comprises the following steps: Correct the final root cause of the failure; The final root cause of the fault is saved in a fault data warehouse.

7. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: When the processor loads and executes the computer program, the method according to any one of claims 1 to 6 is adopted.

8. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, characterized in that: When the computer program is loaded and executed by a processor, the method according to any one of claims 1 to 6 is adopted.

Images