A cloud system fault diagnosis method based on log data

Abstract

This invention discloses a cloud system fault diagnosis method based on log data. The method includes the following steps: preprocessing the input log data to extract log templates, timestamps, and fingerprint sets; caching the preprocessed data and sorting the cached data by timestamp and extracting additional fingerprint sets; dividing discrete logs into workflows by task; generating references with task tags and determining the task category of the workflow based on the similarity between the generated workflow and the references; constructing a diagnostic automaton for each task based on an automaton model; calculating the term frequency-inverse document frequency value of each log template in the training dataset to obtain a term frequency-inverse document frequency dictionary; filtering the real-time log data using the term frequency-inverse document frequency dictionary, and matching the filtered log sequences with the diagnostic automaton to complete fault diagnosis and location.

Description

Technical Field

[0001] This invention belongs to the field of computer application technology, and in particular relates to a cloud system fault diagnosis method based on log data. Background Technology

[0002] With the continuous development of new cloud computing technologies, including SDN (Software Defined Network), high availability deployment, and monitoring and auditing technologies, cloud computing is playing a significant positive role in the daily operations of enterprises and individuals. Due to the widespread use of cloud systems, industry and academia have placed certain demands on their stability and reliability. Therefore, how to accurately diagnose and locate faults in cloud systems has become an important research topic. This not only helps maintenance personnel quickly locate the modules causing errors in the cloud system, but also, combined with automated fault recovery tools, ensures rapid recovery of the cloud system in the event of a failure, improving user experience.

[0003] Cloud systems generate massive amounts of monitoring data during operation, including monitoring metrics, logs, and service call traces. This makes it difficult for operations and maintenance personnel to detect system faults in real time and identify the modules or nodes causing the failures. To address these issues, many researchers have proposed fault diagnosis methods based on system monitoring data to quickly identify faults or find their root causes, thereby ensuring the reliability and stability of cloud systems.

[0004] Numerous methods have been proposed for cloud system fault diagnosis. The paper "CloudSeer: Using Logs to Detect Errors in the Cloud Infrastructure (US2016179600A1)" aims to address runtime errors in tasks within cloud systems, achieving rapid and accurate fault location diagnosis when errors occur. CloudSeer first extracts the workflow for each task in the cloud system from logs. A workflow is a sequence of logs printed by various service processes during the task's execution, arranged chronologically. Using the task's workflow as input, CloudSeer constructs a corresponding automaton model for each task. Fault diagnosis is achieved through the matching process of the task automaton. However, CloudSeer's workflow extraction is limited to a serial environment, which makes its workflow lack universality in parallel environments. The paper "A Workflow-Based Fault Diagnosis Method and System for Cloud System Task Execution (CN110489317A)" proposes LogChain, which improves the workflow extraction method based on Cloudseer, making it applicable to parallel environments. It also proposes a workflow labeling algorithm based on similarity calculation to address the problems of excessive manual intervention or stringent environmental requirements in existing methods during reference construction. All of the above works extract workflows based on the premise that "logs belonging to the same workflow contain common variables." However, in actual cloud system operation, due to program design or system configuration issues, some logs belonging to the same workflow may not contain common variables, or the current log may not have common variables with previous workflow log sequences but may have common variables with subsequent log sequences, leading to incomplete workflow extraction. Furthermore, during online diagnosis, a large amount of irrelevant background log data can impose additional overhead on the diagnostic process. Therefore, constructing a suitable online fault diagnosis method for cloud systems remains a challenge for intelligent cloud computing operations and maintenance. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a cloud system fault diagnosis method based on log data. This method utilizes log data in the cloud system to extract the workflow in the cloud system and matches it with a reference workflow automaton to achieve fault diagnosis, promptly identify the modules or causes that lead to cloud system malfunctions, and thus improve the reliability and stability of the cloud system.

[0006] The objective of this invention is achieved by at least one of the following technical solutions.

[0007] A cloud system fault diagnosis method based on log data includes the following steps:

[0008] S1. The distributed cloud system includes multiple nodes. The logs of each module in the nodes are collected to obtain a log set. The log data in the log set is preprocessed to obtain preprocessed log data. Each log in the log set is preprocessed and transformed into a corresponding tuple consisting of a log template, timestamp, and fingerprint set.

[0009] S2. Add the preprocessed log data to the cache queue, sort it according to the timestamp in the tuple, get the first log in the queue as the current log, and extract the additional fingerprint set of the current log from the remaining logs in the queue.

[0010] S3. Based on the current log and its additional fingerprint set obtained in step S2, determine whether there is a test workflow that matches the current log. If there is, add the current log to the end of the matching test workflow. Otherwise, create a new test workflow and use the current log as the first log entry of the new test workflow.

[0011] S4. By running the specified task type multiple times, multiple workflows belonging to the same task type are obtained. The common parts in the workflows are retained to obtain a reference workflow with a task type identifier. Based on the similarity between the workflow to be tested and the reference workflow, the task type to which the workflow to be tested belongs is determined.

[0012] S5. Build a corresponding finite state automaton for each workflow to be tested, and merge finite state automata with the same starting state.

[0013] S6. Based on the preprocessed log data obtained in step S1, extract all the log templates that have appeared in it, calculate the term frequency-inverse file frequency value for each log template, and obtain the log template term frequency-inverse file frequency dictionary;

[0014] S7. Based on the log template term frequency-inverse document frequency dictionary obtained in step S6, and the finite state automaton obtained in step S5, online fault diagnosis is performed. The real-time input log data is processed into log tuples consisting of log templates, timestamps, and fingerprint sets. The term frequency-inverse document frequency dictionary is used to determine whether to discard or retain the log tuples. The retained log tuples are input into the subsequent finite state automaton for matching and recognition, thereby completing the fault diagnosis and location.

[0015] Further, in step S1, the preprocessing refers to obtaining log components that conform to the corresponding matching rules from log entries using regular expressions, including timestamps, log templates, and fingerprint sets;

[0016] The timestamp refers to the system time when the log was generated; the log template refers to the non-variable part of the log content; the fingerprint set refers to a set of one or more variables in the log entry.

[0017] Further, in step S2, for each log entry of the preprocessed log data, a tuple consisting of a log template, a fingerprint set, and a timestamp is generated; each generated tuple is added to the cache queue and sorted according to the timestamp of the tuple to form an ordered queue, until the timestamp interval of the tuple at the head and tail of the queue is greater than a given threshold.

[0018] Get the head log tuple as the current log tuple, and search the remaining cache queue for log tuples that have common elements with the fingerprint set in the current log tuple. Get the fingerprint set of the searched log tuple as the additional fingerprint set of the current log tuple, and merge the additional fingerprint set into the fingerprint set of the current log tuple.

[0019] Furthermore, in step S3, the matching process between the current log and the workflow under test is as follows:

[0020] Based on the log tuple of the current log obtained in step S2, if the fingerprint set of the current log tuple has a common element with the fingerprint set of the workflow under test, and the update time of the common element in the fingerprint set of the workflow under test has the minimum time interval with the timestamp of the current log tuple, then the current log matches the workflow under test.

[0021] Further, step S4 includes the following steps:

[0022] S4.1 Inject a fault of a specified type and run a specified task type. Collect the log sequence within the time window between the task start time and end time as the workflow of that type of task, and assign fault and task type labels to the workflow.

[0023] S4.2. The workflows that have been assigned fault and task type labels are preprocessed, cached and sorted, and the workflow is mined to filter out redundant background logs. The fingerprint set of the background logs does not have common variables with the fingerprint set of other logs in the workflow. A workflow log template sequence with task type identifier is obtained. The common part of multiple workflow log template sequences belonging to the same task type is retained to obtain a reference workflow log template sequence.

[0024] S4.3. Based on the log sequence of the workflow to be tested obtained in step S3, extract the log template of each log to obtain the log template sequence of the workflow to be tested. Based on the similarity between the log template sequence of the workflow to be tested and the reference workflow log template sequence, determine the task type of the workflow to be tested.

[0025] Furthermore, the similarity calculation process between the workflow log template sequence to be tested and the reference workflow log template sequence in step S4.3 is as follows:

[0026] Calculate the proportion (CommRate) of common elements in the workflow log template sequence to be tested and the reference workflow log template sequence, as follows:

[0027]

[0028] Where ls_l represents the number of elements in the workflow log template sequence to be tested, rs_l represents the number of elements in the reference workflow log template sequence, and l represents the number of common elements in the workflow log template sequence to be tested and the reference workflow log template sequence;

[0029] Calculate the proportion of common elements (CommRate) between the workflow log template sequence under test and each reference workflow log template sequence, obtain the reference workflow log template sequences whose proportion of common elements (CommRate) is greater than a given threshold, and obtain multiple similar reference workflow log template sequences.

[0030] The task type of the workflow under test is determined by calculating the longest common subsequence ratio (SubseqRate) between the log template sequence of the workflow under test and the similar reference workflow log template sequences, as follows:

[0031]

[0032] Where ls_l represents the number of elements in the workflow log template sequence to be tested, rs_l represents the number of elements in the reference workflow log template sequence, and sub_l represents the number of elements in the longest common subsequence of the workflow log template sequence to be tested and the reference workflow log template sequence;

[0033] Calculate the longest common subsequence ratio (SubseqRate) between the workflow log template sequence under test and each similar reference workflow log template sequence. Obtain the similar reference workflow log template sequence with the largest longest common subsequence ratio (SubseqRate) value that is greater than a given threshold. Determine the task type of the workflow under test based on the task type identifier of this reference workflow.

[0034] Further, in step S5, based on the obtained sequence of labeled workflow log templates to be tested, a corresponding finite state automaton is constructed for each workflow to be tested. The construction process is as follows:

[0035] Create a new state for the i-th log in the workflow under test. Add a unique connection edge between the new state and the last state of the automaton. The value of this connection edge is the log template of the i-th log.

[0036] Merge automata that have the same starting state.

[0037] Furthermore, the merger process is as follows:

[0038] Select two automata with the same starting state, and traverse both automata simultaneously until a branch point with unequal states is encountered. Then, append the state sequence after the branch point of one automaton to the branch point of the other automaton.

[0039] Further, in step S6, the calculation process of the term frequency-inverse document frequency value is as follows:

[0040] Based on the obtained log template set including all log templates, the term frequency value of log template j is calculated as follows:

[0041]

[0042] Among them, TF j The term frequency is the word frequency value of log template j. j Let ∑frequency be the number of times log template j appears in the training dataset. j This represents the number of times each log template appears in the training dataset.

[0043] For log template j, the inverse file frequency value is calculated as follows:

[0044]

[0045] Among them, IDF j wfnum is the inverse file frequency value of log template j, wflogt is the number of workflows to be tested, and wflogt is the inverse file frequency value of log template j. j The number of workflows to be tested, including log template j;

[0046] For log template j, its term frequency-inverse document frequency (TFIDF) value j The calculation is as follows:

[0047] TFIDF j =TF j ·IDF j (5)

[0049] Save the correspondence between each log template and its term frequency-inverse file frequency value to obtain a dictionary of log template term frequency-inverse file frequency values.

[0050] Furthermore, step S7 specifically includes the following steps:

[0051] S7.1 For the input real-time log data, preprocessing is first performed to convert the raw log into log tuples that include timestamps, fingerprint sets, and log templates;

[0052] S7.2. Based on the log template term frequency-inverse file frequency value dictionary obtained in step S6, find the term frequency-inverse file frequency value of the log template of the current log entry through the correspondence between the log template and the term frequency-inverse file frequency value. If the term frequency-inverse file frequency value of the log template of the current log entry is less than the given threshold, then input the log tuple of the current log into the subsequent diagnostic steps; otherwise, discard the current log entry.

[0053] S7.3. Based on the log tuple of the current log obtained in step S7.2, add the log tuple to the cache queue and sort it by timestamp. When the cache queue grows to the point where the difference between the timestamps of the log tuples at the tail and the head of the queue is greater than a given threshold, obtain the log tuple at the head of the queue as the current log tuple, and obtain the additional fingerprint set of the current log tuple from the remaining log tuples in the cache queue and add it to the fingerprint set of the current log tuple.

[0054] S7.4. Based on the current log tuple obtained in step S7.3, determine whether there exists an automaton in the diagnostic automaton set where the value of the connection edge from the current state to the next state of the automaton is the same as the log template of the current log tuple, and the fingerprint set of the automaton and the fingerprint set of the current log tuple have common elements, and the update time of the common element in the fingerprint set of the automaton has the minimum time interval with the timestamp of the common element in the fingerprint set of the current log tuple; if it exists, the automaton jumps to the next state and the fingerprint set of the automaton is updated; otherwise, obtain the automaton whose initial state matches the log template of the current log tuple and add it to the diagnostic state machine set.

[0055] Check whether the automaton after the state transition has reached the last state. If the automaton has reached the last state, output the workflow fault and task type label of the automaton as the diagnostic result. At this time, a cloud system fault diagnosis is completed.

[0056] Compared with the prior art, the present invention has the following advantages and technical achievements:

[0057] 1. This paper presents a cloud system fault diagnosis method based on log data, which extracts workflows through additional log variables and filters out irrelevant background log data. Previous fault diagnosis methods relied on variables within the current log itself for workflow extraction, and irrelevant background logs were input into the fault diagnosis algorithm during the online diagnosis process.

[0058] 2. The proposed method for workflow extraction using additional common variables in logs is based on the principle of locality. It caches log data within a given time window after the current log generation time, retrieves log sequences belonging to the same workflow as the current log based on the existence of common variables, and extracts their variables as additional variables for the current log for workflow extraction. This method can solve the problem of incomplete workflow extraction caused by the lack of common variables between the current log and previous workflows in actual cloud system logs.

[0059] 3. The proposed irrelevant background log filtering method uses the correspondence between log templates obtained from the training set and their TF-IDF values ​​to query the TF-IDF values ​​of the input real-time logs during the online fault diagnosis stage, and filters log data that does not meet the threshold requirements, thereby achieving the filtering of irrelevant background logs and improving the efficiency of online fault diagnosis. Attached Figure Description

[0060] Figure 1 This is a flowchart illustrating the cloud system fault diagnosis method based on log data in a specific embodiment of the method of the present invention.

[0061] Figure 2 This is a schematic diagram of log preprocessing in a specific embodiment of the method of the present invention.

[0062] Figure 3 This is a diagram of the additional fingerprint set extraction algorithm in a specific embodiment of the method of the present invention.

[0063] Figure 4 This is a diagram of the matching algorithm between the current log and the workflow to be tested in a specific embodiment of the method of the present invention. Detailed Implementation

[0064] To make the technical solutions and advantages of the present invention clearer, the following detailed description is provided in conjunction with the accompanying drawings, but the implementation and protection of the present invention are not limited thereto.

[0065] Example:

[0066] A cloud system fault diagnosis method based on log data, such as Figure 1 As shown, it includes the following steps:

[0067] S1. The distributed cloud system includes multiple nodes. The logs of each module in the collection node are collected and summarized by log collection tools such as Logstash. The logs generated by each service module of the cloud system distributed on various physical machines are collected and summarized to obtain a log set. The log data in the log set is preprocessed to obtain preprocessed log data. Each log in the log set is preprocessed and transformed into a corresponding tuple consisting of a log template, timestamp, and fingerprint set.

[0068] The preprocessing refers to obtaining log components that conform to the corresponding matching rules from log entries using regular expressions, including timestamps, log templates, and fingerprint sets;

[0069] like Figure 2 As shown, the timestamp refers to the system time at the time the log was generated, consisting of the date (Date) and the specific time (hour, minute, second); the fingerprint set refers to a set of one or more variables in the log entry, such as... Figure 2 The network address IP1, the universally unique identifier UUIDi, and various numeric variables NUMi shown are all variables in the log entries. The fingerprint set is generated by extracting them from the log data using a regular expression algorithm. The log template refers to the non-variable part of the log content, which is the part remaining after the variables have been extracted from the log entries.

[0070] S2. Add the preprocessed log data to the cache queue, sort it according to the timestamp in the tuple, get the first log in the queue as the current log, and extract the additional fingerprint set of the current log from the remaining logs in the queue.

[0071] For each log entry in the preprocessed log data, a tuple consisting of a log template, fingerprint set, and timestamp is generated. Each generated tuple is added to a cache queue and sorted by timestamp to form an ordered queue until the timestamp interval between the tuples at the head and tail of the queue is greater than a given threshold.

[0072] Get the head log tuple as the current log tuple, and search the remaining cache queue for log tuples that have common elements with the fingerprint set in the current log tuple. Get the fingerprint set of the searched log tuple as the additional fingerprint set of the current log tuple, and merge the additional fingerprint set into the fingerprint set of the current log tuple.

[0073] During workflow extraction, the latest log entries are first added to the cache queue and sorted according to the timestamps in the log entries to form an ordered queue. This continues until the difference between the timestamps of the log entries at the head and tail of the queue is greater than a given threshold. At this point, the log entry at the head of the queue is taken as the current log entry for matching the workflow under test.

[0074] Before performing workflow matching, an additional fingerprint set for the current log entry is first extracted, the specific process of which is as follows: Figure 3 As shown: The extra fingerprint set extraFins is initially an empty set, for each log entry. curr Extract its fingerprint set fins curr For each log entry in the cache queue logbuffer, extract its fingerprint set fins. If fins matches fins... curr There is an intersection, i.e., CommonSize(fins) currIf fins > 0, then fins will be merged into extraFins until every element in the cache queue logbuffer has been traversed.

[0075] S3. Based on the current log and its additional fingerprint set obtained in step S2, determine whether there is a test workflow that matches the current log. If there is, add the current log to the end of the matching test workflow. Otherwise, create a new test workflow and use the current log as the first log entry of the new test workflow.

[0076] The matching process between the current logs and the workflow under test is as follows:

[0077] Based on the log tuple of the current log obtained in step S2, if the fingerprint set of the current log tuple has a common element with the fingerprint set of the workflow under test, and the update time of the common element in the fingerprint set of the workflow under test has the minimum time interval with the timestamp of the current log tuple, then the current log matches the workflow under test.

[0078] like Figure 4 As shown, after extracting the additional fingerprint set of the current log entry and adding it to its fingerprint set, the current log entry is matched with the workflow under test based on the updated fingerprint set and timestamp. The specific process is as follows: For each workflow under test wf in the workflow under test set WFS, its fingerprint set wfingp is obtained. It is determined whether wfingp and the fingerprint set fingerp of the current log entry have an intersection. If not, the next element of the workflow under test set is traversed. If it exists, i.e., CommonSize(wfingp, fingerp) > 0, the minimum time interval min_tmp between wfingp and the elements in fingerp is obtained. If min_tmp is less than the current minimum time interval interval, the process is as follows: min Then the interval min Set to min_tmp, and set the target workflow to the current workflow wf.

[0079] If there is a best-matching workflow under test, i.e. target is not NULL, then add the current log entry and its log template to the tail of the log queue of the best-matching workflow under test, and merge the fingerprint set of the current log entry into the fingerprint set of the workflow under test, and update the timestamp of the corresponding variable in the fingerprint set to the timestamp of the current log entry.

[0080] If there is no best matching workflow to be tested, i.e., target equals NULL, then a new workflow to be tested is constructed. The first log in the log queue of the workflow to be tested is the log template of the current log entry, the fingerprint set of the workflow to be tested is the fingerprint set of the current log entry, the timestamp of the corresponding variable in the fingerprint set is the timestamp of the current log entry, and the newly constructed workflow to be tested is added to the set of workflows to be tested.

[0081] S4. By running the specified task type multiple times, multiple workflows belonging to the same task type are obtained. The common parts of the workflows are retained to obtain a reference workflow with a task type identifier. Based on the similarity between the workflow to be tested and the reference workflow, the task type to which the workflow to be tested belongs is determined, including the following steps:

[0082] S4.1 Inject a fault of a specified type and run a specified task type. Collect the log sequence within the time window between the task start time and end time as the workflow of that type of task, and assign fault and task type labels to the workflow.

[0083] S4.2. The workflows that have been assigned fault and task type labels are preprocessed, cached and sorted, and the workflow is mined to filter out redundant background logs. The fingerprint set of the background logs does not have common variables with the fingerprint set of other logs in the workflow. A workflow log template sequence with task type identifier is obtained. The common part of multiple workflow log template sequences belonging to the same task type is retained to obtain a reference workflow log template sequence.

[0084] S4.3. Based on the log sequence of the workflow to be tested obtained in step S3, extract the log template of each log to obtain the log template sequence of the workflow to be tested. Based on the similarity between the log template sequence of the workflow to be tested and the reference workflow log template sequence, determine the task type of the workflow to be tested, as follows:

[0085] Calculate the proportion (CommRate) of common elements in the workflow log template sequence to be tested and the reference workflow log template sequence, as follows:

[0086]

[0087] Where ls_l represents the number of elements in the workflow log template sequence to be tested, rs_l represents the number of elements in the reference workflow log template sequence, and l represents the number of common elements in the workflow log template sequence to be tested and the reference workflow log template sequence;

[0088] Calculate the proportion of common elements (CommRate) between the workflow log template sequence under test and each reference workflow log template sequence, obtain the reference workflow log template sequences whose proportion of common elements (CommRate) is greater than a given threshold, and obtain multiple similar reference workflow log template sequences.

[0089] The task type of the workflow under test is determined by calculating the longest common subsequence ratio (SubseqRate) between the log template sequence of the workflow under test and the similar reference workflow log template sequences, as follows:

[0090]

[0091] Where ls_l represents the number of elements in the workflow log template sequence to be tested, rs_l represents the number of elements in the reference workflow log template sequence, and sub_l represents the number of elements in the longest common subsequence of the workflow log template sequence to be tested and the reference workflow log template sequence;

[0092] Calculate the longest common subsequence ratio (SubseqRate) between the workflow log template sequence under test and each similar reference workflow log template sequence. Obtain the similar reference workflow log template sequence with the largest longest common subsequence ratio (SubseqRate) value that is greater than a given threshold. Determine the task type of the workflow under test based on the task type identifier of this reference workflow.

[0093] S5. Build a corresponding finite state automaton for each workflow to be tested, and merge finite state automata with the same starting state.

[0094] Based on the obtained sequence of labeled workflow log templates to be tested, a corresponding finite state automaton is constructed for each workflow to be tested. The construction process is as follows:

[0095] Create a new state for the i-th log in the workflow under test. Add a unique connection edge between the new state and the last state of the automaton. The value of this connection edge is the log template of the i-th log.

[0096] Merging automata with the same initial state involves the following process:

[0097] Select two automata with the same starting state, and traverse both automata simultaneously until a branch point with unequal states is encountered. Then, append the state sequence after the branch point of one automaton to the branch point of the other automaton.

[0098] S6. Based on the preprocessed log data obtained in step S1, extract all the log templates that have appeared in it, calculate the term frequency-inverse file frequency value for each log template, and obtain a dictionary of log template term frequency-inverse file frequency values.

[0099] The calculation process for the term frequency-inverse document frequency value is as follows:

[0100] Based on the obtained log template set containing all log templates, the term frequency value of log template j is calculated as follows:

[0101]

[0102] Among them, TF j The term frequency is the word frequency value of log template j. j Let ∑frequency be the number of times log template j appears in the training dataset. j This represents the number of times each log template appears in the training dataset.

[0103] For log template j, the inverse file frequency value is calculated as follows:

[0104]

[0105] Among them, IDF j wfnum is the inverse file frequency value of log template j, wflogt is the number of workflows to be tested, and wflogt is the inverse file frequency value of log template j. j The number of workflows to be tested, including log template j;

[0106] For log template j, its term frequency-inverse document frequency (TFIDF) value j The calculation is as follows:

[0107] TFIDF j =TF j ·IDF j (5)

[0109] Save the correspondence between each log template and its term frequency-inverse file frequency value to obtain a dictionary of log template term frequency-inverse file frequency values.

[0110] S7. Based on the log template term frequency-inverse document frequency dictionary obtained in step S6, and the finite state automaton obtained in step S5, online fault diagnosis is performed. The real-time input log data is processed into log tuples composed of log templates, timestamps, and fingerprint sets. The term frequency-inverse document frequency dictionary is used to determine whether to discard or retain log tuples. The retained log tuples are input into the subsequent finite state automaton for matching and recognition, thereby completing the fault diagnosis and location. Figure 1 As shown, the specific steps include:

[0111] S7.1 For the input real-time log data, preprocessing is first performed to convert the raw log into log tuples that include timestamps, fingerprint sets, and log templates;

[0112] S7.2. Based on the log template term frequency-inverse file frequency value dictionary obtained in step S6, find the term frequency-inverse file frequency value of the log template of the current log entry through the correspondence between the log template and the term frequency-inverse file frequency value. If the term frequency-inverse file frequency value of the log template of the current log entry is less than the given threshold, then input the log tuple of the current log into the subsequent diagnostic steps; otherwise, discard the current log entry.

[0113] S7.3. Based on the log tuple of the current log obtained in step S7.2, add the log tuple to the cache queue and sort it by timestamp. When the cache queue grows to the point where the difference between the timestamps of the log tuples at the tail and the head of the queue is greater than a given threshold, obtain the log tuple at the head of the queue as the current log tuple, and obtain the additional fingerprint set of the current log tuple from the remaining log tuples in the cache queue and add it to the fingerprint set of the current log tuple.

[0114] S7.4. Based on the current log tuple obtained in step S7.3, determine whether there exists an automaton in the diagnostic automaton set where the value of the connection edge from the current state to the next state of the automaton is the same as the log template of the current log tuple, and the fingerprint set of the automaton and the fingerprint set of the current log tuple have common elements, and the update time of the common element in the fingerprint set of the automaton has the minimum time interval with the timestamp of the common element in the fingerprint set of the current log tuple; if it exists, the automaton jumps to the next state and the fingerprint set of the automaton is updated; otherwise, obtain the automaton whose initial state matches the log template of the current log tuple and add it to the diagnostic state machine set.

[0115] In this embodiment, the diagnostic process is as follows:

[0116] Traverse the current set of automata to be tested, find the automata to be tested that can accept the current log entry’s log template and perform state transitions, and add it to the set of automata to be matched. An automaton can accept the current log entry’s log template and jump from state A to state B if and only if the log template represented by the transition edge from state A to state B is the same as the log template of the current log entry.

[0117] Traverse the set of automata to be matched to obtain the best matching automaton. The fingerprint set of the best matching automaton and the fingerprint set of the current log entry have common elements, and the difference between the update time of the common element in the fingerprint set of the automaton and the timestamp of the common element in the fingerprint set of the current log tuple is the smallest. If the best matching automaton exists, the automaton jumps to the next state and updates the fingerprint set and the timestamp of the corresponding variable of the automaton. Otherwise, obtain a new automaton to be tested from the workflow automaton set. The initial state of the automaton can accept the log template of the current log entry and perform state transition. Add the fingerprint set of the current log entry to the fingerprint set of the new automaton to be tested. Update the corresponding variable in the fingerprint set of the new automaton to be tested to the timestamp of the current log entry. Add the new automaton to be tested to the set of automata to be tested.

[0118] Check whether the automaton after the state transition has reached the last state. If the automaton has reached the last state, output the workflow fault and task type label of the automaton as the diagnostic result. At this time, a cloud system fault diagnosis is completed.

[0119] In this embodiment, the invention was tested on the open-source cloud operating system OpenStack. The experiment involved deploying the OpenStack cloud system on a cluster of five servers. User requests were simulated to generate load using scripts, and fault injection was controlled by scripts to simulate cloud system failure scenarios. Log data generated by various system service modules distributed across the servers was collected using the LogStash log collection tool. Regular expressions were used to preprocess the log data, extracting information such as timestamps, variable sets, and log templates. The preprocessed log data underwent workflow extraction, workflow tagging, workflow automaton generation, and filter dictionary extraction to generate a filter dictionary and workflow diagnostic automaton set suitable for online fault diagnosis. The workflow diagnostic automaton was used to match workflows present in the current real-time log data. The identifier of the matched workflow automaton indicated whether a fault existed in the cloud system and the type of fault.

[0120] Example 2:

[0121] Compared to Example 1, this example conducts experiments on the open-source dataset HDFS. The HDFS dataset is generated by running standard workloads on a private cloud system, with abnormal data manually labeled. The experiment divides the HDFS dataset into a training set and a validation set. The training set is used to train the diagnostic method model, and the validation set is used to verify the accuracy of the diagnostic method model. This example achieves good fault diagnosis accuracy on the open-source dataset HDFS.

[0122] Example 3:

[0123] Compared to Example 1, this example conducts experiments on the open-source Hadoop dataset. The Hadoop dataset is generated from a Hadoop cluster distributed across 5 server hosts. Task load is simulated by running WordCount and PageRank, and service failures in a production environment are simulated through fault injection. Similar to Example 2, this example divides the Hadoop dataset into training and validation sets, and achieves better fault diagnosis accuracy on the Hadoop dataset.

Claims

1. A cloud system fault diagnosis method based on log data, characterized in that, Includes the following steps: S1. The distributed cloud system includes multiple nodes. Logs from each module within these nodes are aggregated to form a log set. The log data in this set is preprocessed to obtain preprocessed log data. Each log entry in the log set is preprocessed into a tuple consisting of a log template, a timestamp, and a fingerprint set. The timestamp refers to the system time at the time the log was generated. The log template refers to the non-variable portion of the log content. The fingerprint set refers to a set of one or more variables in the log entry. S2. Add the preprocessed log data to the cache queue, sort it according to the timestamp in the tuple, get the first log in the queue as the current log, and extract the additional fingerprint set of the current log from the remaining logs in the queue. S3. Based on the current log and its additional fingerprint set obtained in step S2, determine whether there is a test workflow that matches the current log. If there is, add the current log to the end of the matching test workflow. Otherwise, create a new test workflow and use the current log as the first log entry of the new test workflow. S4. By running the specified task type multiple times, multiple workflows belonging to the same task type are obtained. The common parts in the workflows are retained to obtain a reference workflow with a task type identifier. Based on the similarity between the workflow to be tested and the reference workflow, the task type to which the workflow to be tested belongs is determined. S5. Build a corresponding finite state automaton for each workflow to be tested, and merge finite state automata with the same starting state. S6. Based on the preprocessed log data obtained in step S1, extract all the log templates that have appeared in it, calculate the term frequency-inverse file frequency value for each log template, and obtain the log template term frequency-inverse file frequency dictionary; S7. Based on the log template term frequency-inverse document frequency dictionary obtained in step S6, and the finite state automaton obtained in step S5, online fault diagnosis is performed. The real-time input log data is processed into log tuples consisting of log templates, timestamps, and fingerprint sets. The term frequency-inverse document frequency dictionary is used to determine whether to discard or retain the log tuples. The retained log tuples are input into the subsequent finite state automaton for matching and recognition, thereby completing the fault diagnosis and location.

2. The cloud system fault diagnosis method based on log data according to claim 1, characterized in that, In step S1, the preprocessing refers to obtaining log components that conform to the corresponding matching rules from log entries using regular expressions, including timestamps, log templates, and fingerprint sets.

3. The cloud system fault diagnosis method based on log data according to claim 1, characterized in that, In step S2, for each log entry of the preprocessed log data, a tuple consisting of a log template, fingerprint set, and timestamp is generated; each generated tuple is added to the cache queue and sorted by timestamp to form an ordered queue until the timestamp interval of the tuples at the head and tail of the queue is greater than a given threshold. Get the head log tuple as the current log tuple, and search the remaining cache queue for log tuples that have common elements with the fingerprint set in the current log tuple. Get the fingerprint set of the searched log tuple as the additional fingerprint set of the current log tuple, and merge the additional fingerprint set into the fingerprint set of the current log tuple.

4. The cloud system fault diagnosis method based on log data according to claim 3, characterized in that, In step S3, the matching process between the current log and the workflow under test is as follows: Based on the log tuple of the current log obtained in step S2, if the fingerprint set of the current log tuple has a common element with the fingerprint set of the workflow under test, and the update time of the common element in the fingerprint set of the workflow under test has the minimum time interval with the timestamp of the current log tuple, then the current log matches the workflow under test.

5. The cloud system fault diagnosis method based on log data according to claim 4, characterized in that, Step S4 includes the following steps: S4.1 Inject a fault of a specified type and run a specified task type. Collect the log sequence within the time window between the task start time and end time as the workflow of that type of task, and assign fault and task type labels to the workflow. S4.

2. The workflows that have been assigned fault and task type labels are preprocessed, cached and sorted, and the workflow is mined to filter out redundant background logs. The fingerprint set of the background logs does not have common variables with the fingerprint set of other logs in the workflow. A workflow log template sequence with task type identifier is obtained. The common part of multiple workflow log template sequences belonging to the same task type is retained to obtain a reference workflow log template sequence. S4.

3. Based on the log sequence of the workflow to be tested, extract the log template of each log to obtain the log template sequence of the workflow to be tested. Based on the similarity between the log template sequence of the workflow to be tested and the reference workflow log template sequence, determine the task type of the workflow to be tested.

6. The cloud system fault diagnosis method based on log data according to claim 5, characterized in that, The similarity calculation process between the workflow log template sequence to be tested and the reference workflow log template sequence in step S4.3 is as follows: Calculate the percentage of common elements in the workflow log template sequence to be tested and the reference workflow log template sequence. The details are as follows: in, This indicates the number of elements included in the workflow log template sequence to be tested. This indicates the number of elements included in the referenced workflow log template sequence. This indicates the number of common elements in the workflow log template sequence to be tested and the reference workflow log template sequence; Calculate the percentage of common elements between the workflow log template sequence under test and each reference workflow log template sequence. Get the proportion of common elements A sequence of reference workflow log templates exceeding a given threshold is used to obtain multiple similar sequence of reference workflow log templates. The proportion of the longest common subsequence between the workflow log template sequence under test and similar reference workflow log template sequences is calculated. Determine the task type of the workflow to be tested, as follows: in, This indicates the number of elements included in the workflow log template sequence to be tested. This indicates the number of elements included in the referenced workflow log template sequence. This represents the number of elements in the longest common subsequence of the workflow log template sequence to be tested and the reference workflow log template sequence; Calculate the percentage of the longest common subsequence among the workflow log template sequence under test and each similar reference workflow log template sequence. Obtain the percentage of the longest common subsequence The task type of the workflow to be tested is determined based on the task type identifier of the reference workflow, which is the sequence of similar reference workflow log templates with the largest value that is greater than a given threshold.

7. The cloud system fault diagnosis method based on log data according to claim 6, characterized in that, In step S5, based on the obtained sequence of labeled workflow log templates to be tested, a corresponding finite state automaton is constructed for each workflow to be tested. The construction process is as follows: For the first in the workflow to be tested Each log entry creates a new state, and a unique edge is added between the new state and the last state of the automaton. The value of this edge is the first... Log template for each log entry; Merge automata that have the same starting state.

8. The cloud system fault diagnosis method based on log data according to claim 7, characterized in that, The merger process is as follows: Select two automata with the same starting state, and traverse both automata simultaneously until a branch point with unequal states is encountered. Then, append the state sequence after the branch point of one automaton to the branch point of the other automaton.

9. The cloud system fault diagnosis method based on log data according to claim 1, characterized in that, In step S6, the calculation process of the term frequency-inverse document frequency value is as follows: Based on the obtained log template set including all log templates, the term frequency value of log template j is calculated as follows: in, For log template j, the word frequency value. Let j be the number of times log template j appears in the training dataset. This represents the number of times each log template appears in the training dataset. For log template j, the inverse file frequency value is calculated as follows: in, The inverse file frequency value of log template j. The number of workflows to be tested. The number of workflows to be tested, including log template j; For log template j, its term frequency-inverse document frequency value The calculation is as follows: Save the correspondence between each log template and its term frequency-inverse file frequency value to obtain a dictionary of log template term frequency-inverse file frequency values.

10. A cloud system fault diagnosis method based on log data according to any one of claims 1 to 9, characterized in that, Step S7 specifically includes the following steps: S7.1 For the input real-time log data, preprocessing is first performed to convert the raw log into log tuples that include timestamps, fingerprint sets, and log templates; S7.

2. Based on the log template term frequency-inverse file frequency value dictionary obtained in step S6, find the term frequency-inverse file frequency value of the log template of the current log entry through the correspondence between the log template and the term frequency-inverse file frequency value. If the term frequency-inverse file frequency value of the log template of the current log entry is less than the given threshold, then input the log tuple of the current log into the subsequent diagnostic steps; otherwise, discard the current log entry. S7.

3. Based on the log tuple of the current log obtained in step S7.2, add the log tuple to the cache queue and sort it by timestamp. When the cache queue grows to the point where the difference between the timestamps of the log tuples at the tail and the head of the queue is greater than a given threshold, obtain the log tuple at the head of the queue as the current log tuple, and obtain the additional fingerprint set of the current log tuple from the remaining log tuples in the cache queue and add it to the fingerprint set of the current log tuple. S7.

4. Based on the current log tuple obtained in step S7.3, determine whether there exists an automaton in the diagnostic automaton set where the value of the connection edge from the current state to the next state of the automaton is the same as the log template of the current log tuple, and the fingerprint set of the automaton and the fingerprint set of the current log tuple have common elements, and the update time of the common element in the fingerprint set of the automaton has the minimum time interval with the timestamp of the common element in the fingerprint set of the current log tuple; if it exists, the automaton jumps to the next state and the fingerprint set of the automaton is updated; otherwise, obtain the automaton whose initial state matches the log template of the current log tuple and add it to the diagnostic state machine set. Check whether the automaton after the state transition has reached the last state. If the automaton has reached the last state, output the workflow fault and task type label of the automaton as the diagnostic result. At this time, a cloud system fault diagnosis is completed.

Images