Cloud system fault diagnosis research method and system based on log data

Abstract

The invention relates to the technical field of cloud computing system management and maintenance, in particular to a cloud system fault diagnosis research method and system based on log data, and the method comprises the following steps: collecting and preprocessing log data; carrying out workflow mining and model construction; a log filtering mechanism; online fault diagnosis is carried out; designing and realizing a fault diagnosis system; the cloud system fault diagnosis research method and system based on log data provided by the invention have the beneficial effects that the problem of incomplete workflow mining is effectively solved through a proposed fault diagnosis algorithm, so that fault information can be more comprehensively mined and labeled, and the working efficiency is improved. And a more complete and accurate workflow state automaton model is constructed. Therefore, the accuracy and timeliness of online fault diagnosis are improved, and the false alarm rate is greatly reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of cloud computing system management and maintenance, and in particular to a cloud system fault diagnosis research method and system based on log data. Background Art

[0002] With the widespread application of cloud computing technology, the stability and reliability of cloud systems have become the focus of industry and academia. As an important data source for system monitoring, log data contains rich system runtime information and is of great significance for fault diagnosis of cloud systems.

[0003] However, existing cloud system fault diagnosis methods face many challenges when applied to specific cloud systems such as OpenStack, such as incomplete workflow mining and redundant log data affecting online processing efficiency. These problems limit the accuracy and efficiency of fault diagnosis and make it difficult to meet the needs of practical applications. Summary of the invention

[0004] The purpose of the present invention is to provide a cloud system fault diagnosis research method and system based on log data to solve the problems raised in the above background technology.

[0005] To achieve the above object, the present invention provides the following technical solution: a cloud system fault diagnosis research method based on log data, the method comprising the following steps:

[0006] Log data collection and preprocessing;

[0007] Workflow mining and model building;

[0008] Log filtering mechanism;

[0009] Online fault diagnosis;

[0010] Design and implementation of fault diagnosis system.

[0011] Preferably, the specific operations of log data collection and preprocessing include:

[0012] Use the LogStash tool to collect log files of various components of the OpenStack cloud system and store them centrally in databases such as ElasticSearch for subsequent processing and analysis;

[0013] The collected raw logs are preprocessed, including coarse-grained processing and fine-grained processing. Coarse-grained processing divides the logs into timestamps, request IDs, and key parts of the log content through regular expression matching; fine-grained processing further parses the log content and extracts useful information.

[0014] Preferably, the specific operations of workflow mining and model building include:

[0015] Workflow mining technology is used to mine task workflows from historical log data and assign fault information labels to these workflows;

[0016] Aiming at the problem of incomplete workflow mining, a log identifier expansion method is proposed to supplement the missing log identifiers and ensure the integrity of the workflow.

[0017] A finite state automaton (FSA) model is constructed for each workflow for fault detection and location in the online fault diagnosis phase.

[0018] Preferably, the specific operations of the log filtering mechanism include:

[0019] The IDF log filtering method is introduced to evaluate the importance of log templates and select log templates with lower weight values ​​to build a log filtering knowledge base.

[0020] In the online fault diagnosis stage, by matching the current log with the log template in the filtering knowledge base, redundant logs are identified and filtered out, thereby improving the processing efficiency and accuracy of the fault diagnosis model.

[0021] Preferably, the specific operations of online fault diagnosis include: designing and implementing an online fault diagnosis process, parsing the current log to be tested in real time, matching it with the constructed FSA model, detecting and locating faults in the cloud system; if the FSA model reaches a termination state, outputting the fault diagnosis result according to the model label; if no matching FSA model is found, querying the fault knowledge base to obtain a new FSA model to be tested, and continuing the diagnosis process.

[0022] A cloud system fault diagnosis research system based on log data is applied to a cloud system fault diagnosis research method based on log data, the method comprising the following steps:

[0023] The log data collection module is used to collect log files of various components of the OpenStack cloud system using the LogStash tool and store these logs in the ElasticSearch database;

[0024] The log preprocessing module is used to preprocess the collected raw logs, including coarse-grained processing through regular expression matching, dividing the logs into timestamps, request IDs, and key parts of the log content, and further parsing the log content for fine-grained processing to extract useful information.

[0025] Preferably, it also includes: a workflow mining and model building module, which is used to use workflow mining technology to mine task workflows from historical log data and assign fault information labels to these workflows; at the same time, a log identifier expansion method is proposed to supplement missing log identifiers to ensure the integrity of the workflow; a finite state automaton FSA model is constructed for each workflow for fault detection and location in the online fault diagnosis stage.

[0026] Preferably, it also includes: a log filtering module, which is used to introduce the IDF log filtering method to evaluate the importance of log templates, and select log templates with lower weight values ​​to build a log filtering knowledge base; in the online fault diagnosis stage, by matching the current log with the log template in the filtering knowledge base, redundant logs are identified and filtered out to improve the processing efficiency and accuracy of fault diagnosis.

[0027] Preferably, it also includes: an online fault diagnosis module, which is used to design and implement an online fault diagnosis process, and detect and locate faults in the cloud system by real-time parsing the current log to be tested, matching it with the constructed FSA model; if the FSA model reaches a termination state, the fault diagnosis result is output according to the model label; if no matching FSA model is found, the fault knowledge base is queried to obtain a new FSA model to be tested, and the diagnosis process continues.

[0028] Preferably, the system integrates a log data collection module, a log preprocessing module, a workflow mining and model building module, a log filtering module and an online fault diagnosis module, collects log data of the OpenStack cloud system in real time, and realizes automatic detection and location of cloud system faults through preprocessing, workflow mining, log filtering and online fault diagnosis steps; the system is also scalable and configurable, and can be customized and optimized according to the characteristics and requirements of different cloud systems.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] The cloud system fault diagnosis research method and system based on log data proposed in the present invention, through the proposed fault diagnosis algorithm, the present invention effectively solves the problem of incomplete workflow mining, so that fault information can be more comprehensively mined and labeled, and then a more complete and accurate workflow state automaton model is constructed. This not only improves the accuracy and timeliness of online fault diagnosis, but also greatly reduces the false alarm rate. In addition, the introduced IDF log filtering method effectively reduces the impact of redundant logs on online processing efficiency, and significantly improves the system's data processing capability and response speed by intelligently identifying and filtering unimportant log templates. In general, the implementation of the present invention will greatly improve the fault diagnosis capability of the OpenStack cloud system, enhance the stability and reliability of the system, and provide strong support for the efficient operation and maintenance of the cloud computing platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose and technical solution of the present invention clearly and completely described, and the advantages more clearly understood, the embodiments of the present invention are further described in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are part of the embodiments of the present invention, rather than all of the embodiments, and are only used to explain the embodiments of the present invention, and are not used to limit the embodiments of the present invention. All other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] For example, see Figure 1 The present invention provides a technical solution: a cloud system fault diagnosis research method based on log data, the method comprising the following steps:

[0034] Log data collection and preprocessing:

[0035] Use tools such as LogStash to collect log files of various components of the OpenStack cloud system and store them centrally in databases such as ElasticSearch for subsequent processing and analysis.

[0036] The collected raw logs are preprocessed, including coarse-grained processing and fine-grained processing. Coarse-grained processing divides the logs into key parts such as timestamp, request ID and log content through regular expression matching; fine-grained processing further parses the log content and extracts useful information.

[0037] Workflow Mining and Model Building:

[0038] Workflow mining technology is used to mine task workflows from historical log data, and fault information labels are assigned to these workflows.

[0039] Aiming at the problem of incomplete workflow mining, a log identifier expansion method is proposed to supplement the missing log identifiers and ensure the integrity of the workflow.

[0040] A finite state automaton (FSA) model is constructed for each workflow for fault detection and location in the online fault diagnosis phase.

[0041] Log filtering mechanism:

[0042] The IDF (Inverse Document Frequency) log filtering method is introduced to evaluate the importance of log templates and select log templates with lower weight values ​​to build a log filtering knowledge base.

[0043] In the online fault diagnosis stage, by matching the current log with the log template in the filtering knowledge base, redundant logs are identified and filtered out, thereby improving the processing efficiency and accuracy of the fault diagnosis model.

[0044] Online fault diagnosis:

[0045] Design and implement an online fault diagnosis process to detect and locate faults in the cloud system by real-time parsing of the current log to be tested and matching it with the constructed FSA model.

[0046] If the FSA model reaches the termination state, the fault diagnosis result is output according to the model label; if no matching FSA model is found, the fault knowledge base is queried to obtain a new FSA model to be tested, and the diagnosis process continues.

[0047] Fault diagnosis system design and implementation:

[0048] Based on the above technical solution, a cloud system fault diagnosis system based on log data is designed and implemented. The system can collect log data of the OpenStack cloud system in real time, and realize automatic detection and location of cloud system faults through preprocessing, workflow mining, log filtering and online fault diagnosis.

[0049] Embodiment 2, based on embodiment 1, proposes a cloud system fault diagnosis research system based on log data, which is applied to a cloud system fault diagnosis research method based on log data. The method comprises the following steps:

[0050] The log data collection module is used to collect log files of various components of the OpenStack cloud system using the LogStash tool and store these logs in the ElasticSearch database;

[0051] The log preprocessing module is used to preprocess the collected raw logs, including coarse-grained processing through regular expression matching, dividing the logs into timestamps, request IDs, and key parts of the log content, and further parsing the log content for fine-grained processing to extract useful information.

[0052] It also includes: a workflow mining and model building module, which uses workflow mining technology to mine task workflows from historical log data and assign fault information labels to these workflows; at the same time, a log identifier expansion method is proposed to supplement missing log identifiers to ensure the integrity of the workflow; a finite state automaton FSA model is built for each workflow for fault detection and location in the online fault diagnosis stage.

[0053] It also includes: a log filtering module, which is used to introduce the IDF log filtering method to evaluate the importance of log templates and select log templates with lower weight values ​​to build a log filtering knowledge base; in the online fault diagnosis stage, by matching the current log with the log template in the filtering knowledge base, redundant logs are identified and filtered out to improve the processing efficiency and accuracy of fault diagnosis.

[0054] It also includes: an online fault diagnosis module, which is used to design and implement an online fault diagnosis process, detect and locate faults in the cloud system by real-time parsing of the current log to be tested, matching it with the constructed FSA model; if the FSA model reaches the termination state, the fault diagnosis result is output according to the model label; if no matching FSA model is found, the fault knowledge base is queried to obtain a new FSA model to be tested, and the diagnosis process continues.

[0055] The system integrates log data collection module, log preprocessing module, workflow mining and model building module, log filtering module and online fault diagnosis module to collect log data of OpenStack cloud system in real time. Through preprocessing, workflow mining, log filtering and online fault diagnosis steps, it can realize automatic detection and location of cloud system faults. The system is also scalable and configurable, and can be customized and optimized according to the characteristics and requirements of different cloud systems.

[0056] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A cloud system fault diagnosis research method based on log data, characterized by: The method comprises the following steps: Log data collection and preprocessing; Workflow mining and model building; Log filtering mechanism; Online fault diagnosis; Design and implementation of fault diagnosis system.

2. According to a cloud system fault diagnosis research method based on log data according to claim 1, it is characterized by: The specific operations of log data collection and preprocessing include: Use the LogStash tool to collect log files of various components of the OpenStack cloud system and store them centrally in databases such as ElasticSearch for subsequent processing and analysis; The collected raw logs are preprocessed, including coarse-grained processing and fine-grained processing. Coarse-grained processing divides the logs into timestamps, request IDs, and key parts of the log content through regular expression matching; fine-grained processing further parses the log content and extracts useful information.

3. The cloud system fault diagnosis research method based on log data according to claim 1 is characterized by: The specific operations of workflow mining and model building include: Workflow mining technology is used to mine task workflows from historical log data and assign fault information labels to these workflows; Aiming at the problem of incomplete workflow mining, a log identifier expansion method is proposed to supplement the missing log identifiers and ensure the integrity of the workflow. A finite state automaton (FSA) model is constructed for each workflow for fault detection and location in the online fault diagnosis phase.

4. The cloud system fault diagnosis research method based on log data according to claim 1 is characterized by: The specific operations of the log filtering mechanism include: The IDF log filtering method is introduced to evaluate the importance of log templates and select log templates with lower weight values ​​to build a log filtering knowledge base. In the online fault diagnosis stage, by matching the current log with the log template in the filtering knowledge base, redundant logs are identified and filtered out, thereby improving the processing efficiency and accuracy of the fault diagnosis model.

5. The cloud system fault diagnosis research method based on log data according to claim 1 is characterized by: The specific operations of online fault diagnosis include: designing and implementing the online fault diagnosis process, parsing the current log to be tested in real time, matching it with the constructed FSA model, detecting and locating faults in the cloud system; if the FSA model reaches the termination state, outputting the fault diagnosis result according to the model label; if no matching FSA model is found, querying the fault knowledge base to obtain a new FSA model to be tested, and continuing the diagnosis process.

6. A cloud system fault diagnosis research system based on log data, applied to a cloud system fault diagnosis research method based on log data according to any one of claims 1 to 5, characterized in that: The method comprises the following steps: The log data collection module is used to collect log files of various components of the OpenStack cloud system using the LogStash tool and store these logs in the ElasticSearch database; The log preprocessing module is used to preprocess the collected raw logs, including coarse-grained processing through regular expression matching, dividing the logs into timestamps, request IDs, and key parts of the log content, and further parsing the log content for fine-grained processing to extract useful information.

7. The cloud system fault diagnosis research method based on log data according to claim 6 is characterized by: Also includes: The workflow mining and model building module is used to mine task workflows from historical log data using workflow mining technology and assign fault information labels to these workflows. At the same time, a log identifier expansion method is proposed to supplement missing log identifiers to ensure the integrity of the workflow. A finite state automaton (FSA) model is built for each workflow for fault detection and location in the online fault diagnosis stage.

8. The cloud system fault diagnosis research method based on log data according to claim 7 is characterized by: Also includes: The log filtering module is used to introduce the IDF log filtering method to evaluate the importance of log templates and select log templates with lower weight values ​​to build a log filtering knowledge base; In the online fault diagnosis stage, redundant logs are identified and filtered out by matching the current logs with the log templates in the filtering knowledge base to improve the processing efficiency and accuracy of fault diagnosis.

9. The cloud system fault diagnosis research method based on log data according to claim 8 is characterized by: Also includes: The online fault diagnosis module is used to design and implement the online fault diagnosis process. It analyzes the current log to be tested in real time, matches it with the constructed FSA model, detects and locates faults in the cloud system; if the FSA model reaches the termination state, the fault diagnosis result is output according to the model label; if no matching FSA model is found, the fault knowledge base is queried to obtain a new FSA model to be tested, and the diagnosis process continues.

10. The cloud system fault diagnosis research method based on log data according to claim 9 is characterized in that: The system integrates log data collection module, log preprocessing module, workflow mining and model building module, log filtering module and online fault diagnosis module to collect log data of OpenStack cloud system in real time. Through preprocessing, workflow mining, log filtering and online fault diagnosis steps, it can realize automatic detection and location of cloud system faults. The system is also scalable and configurable, and can be customized and optimized according to the characteristics and requirements of different cloud systems.

Images