Log Exception Detection System Based on Process Invariants

Abstract

The present invention discloses a log anomaly detection system based on process invariants. The system is divided into two main parts: a process invariant mining subsystem and a log anomaly detection subsystem. The process invariant mining subsystem is the core component of the system. It is responsible for extracting process invariants from logs and mainly works in the training stage. Its main process is divided into three steps: mining event correlation relationships, restoring software business processes, and analyzing event variable associations. The log anomaly detection subsystem works in the detection stage. According to the process invariants obtained in the training stage, it detects possible abnormal records in the logs and discovers software anomalies. The present invention adopts a window mechanism to discover event correlations based on the frequency principle, reconstructs the branch structure of the software business process, and at the same time introduces event variables into the process invariants, making the content and form of the process invariants richer and more accurate.

Description

Technical Field

[0001] The present invention belongs to the technical field of log anomaly detection, and particularly relates to a log anomaly detection system based on process invariants. Background Art

[0002] Works similar to the present invention are Document [1] [Lou J G, Qiang F, Yang S, et al. Mining invariants from console logs for system problem detection [J]. Proc of Usenix Atc, 2010.] and Document [3] [Du M, Li F, Zheng G, et al. DeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning [C] / / Computer and Communications Security. ACM, 2017.]. It follows the principal component analysis method proposed in Document [2] [Xu W, Huang L, Fox A, et al. Detecting large-scale system problems by mining console logs [C] / / Symposium on Operating Systems Principles. ACM, 2009.] to perform daily anomaly detection, and proposes to mine invariants to perform anomaly detection. Its core idea is to count event information and search for the relationships between various events and between event count values. For example, if there is an event of opening a file, there should be a corresponding event of closing the file. However, in our opinion, log events should not only be related to count values, but should be examined for anomalies from the aspects of the software business workflow. Continuing with the previous example: the event of opening a file should obviously occur before the event of closing the same file. Here, there is not only a counting problem, but also an order problem.

[0003] The solution proposed in Document [3] is named DeepLog, which is a representative method for log analysis using a deep neural network. Its core idea is to abstract logs into a vector form that can be processed by a deep learning model, use the deep learning model for training and learning, discover the workflow model of the software, and thus perform anomaly detection. However, using a deep learning model for learning is costly, especially when dealing with massive data, and its computational cost is difficult for general systems to bear. In addition, in our opinion, logs not only contain log event information, but also variable information, which should also be important information for determination.

[0004] The existing work mainly faces the problems of high computational cost and ignoring event variables. First, the computational cost is high. In order to support invariant mining, reference [1] adopted the MapReduce architecture to implement distributed computing, while the deep learning model used in reference [3] itself requires a large amount of computation. Due to the incomplete logs, some low-probability events may be ignored, which makes it necessary to frequently extract invariants from the logs. The method with a large amount of computation is not really practical. Second, they all ignore the variables related to events and only focus on event sequences or event counts. In our view, event variables are as important as event counts, and some event variables themselves imply the information of subsequent event sequences. Summary of the Invention

[0005] In view of the problems of high computational cost and ignoring event variables existing in the existing log anomaly detection methods, the present invention proposes a log anomaly detection system based on process invariants. First, the present invention proposes a software process invariant mining subsystem that can achieve a relatively low computational cost and a more in-depth analysis, and constructs a log anomaly detection system based on process invariants on this basis. In order to reduce the computational cost, instead of using algorithms with high training costs such as deep neural networks, we adopt a window mechanism based on the frequency principle to discover event associations and reconstruct the branch structure of the software business process. In addition, in order to make full use of the information recorded in the logs, we also introduce event variables into the process invariants, making the content and form of the process invariants richer and more accurate.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] A log anomaly detection system based on process invariants, the system includes a process invariant mining subsystem and a log anomaly detection subsystem;

[0008] The process invariant mining subsystem includes an event association relationship mining module, a software workflow restoration module, and an event variable association analysis module; the event association relationship mining module is used to extract valuable event sequence fragments from log events; the software workflow restoration module is used to partially restore the software workflow from the event sequence fragments; the event variable association analysis module is used to analyze the possible association relationships between event variables and the software workflow based on the restored software workflow, so as to form the final process invariant;

[0009] The log anomaly detection subsystem includes a to-be-detected event sequence maintenance module, a current detection event sequence maintenance module, and an anomaly detection module; the to-be-detected event sequence maintenance module is used to maintain the to-be-detected event sequence; the current detection event sequence maintenance module is used to maintain the current detection event sequence; the anomaly detection module is used to discover log anomalies by reading process invariants with the support of the to-be-detected event sequence and the current detection event sequence.

[0010] Further, the event association relationship mining module is specifically used for:

[0011] Sliding window processing: Use a sliding window to count the frequency of events within the window, mine the association degree between events, and thus discover regular event sequence segments; for the design of the window size and window overlap size, use time as the measurement unit;

[0012] Single window processing: Start from an event sequence of length 1, count the event association support degree and event association credibility of all event sequence segments of length 1 within a certain time window, and remove the event sequences with event association support degree and event association credibility less than the thresholds; then, based on the event sequence of length 1, add subsequent events after it, gradually increase the length of the event sequence, and mine the event association relationship within a certain time window in the above manner;

[0013] Multi-window processing: Combine the event sequences generated by different windows according to the actual situation to form a unified result: when the event sequences generated by different windows do not have intersections and overlaps, directly incorporate the event sequences generated by the same window into the final result set; when the event sequences generated by different windows have overlaps, without changing the content of the event sequences, but accumulate the event association support degree and event association credibility corresponding to the two overlapping sequences; when the event sequences generated by different windows have intersections, retain the original sequences, extract the common intersection part to form a new event sequence, and accumulate the event association support degree and event association credibility of the two original sequences with intersections as the event association support degree and event association credibility of the new event sequence respectively.

[0014] Further, the software workflow restoration module is specifically used for:

[0015] Branch structure mining: Split all sequence segments in the sequential event sequence event_A into several binary relationships, that is, the length of all event sequence segments is 2. Count the leading and subsequent relationships of all events in event_A, and generate selection, convergence, and loop branch structures according to the common leading and common subsequent situations. Try to extend the length of the event sequence, and further discover loop structures during this process; for overly long multi-hop loop structures, regard them as sequential structures; for complex loops, focus on the local part of the loop structure, mainly single loops and a small number of multi-step loops.

[0016] Optimize the obtained selection branch structure: Lower the minimum frequency requirement, and conduct another search on the log-related event sequence. Starting from the event where the selection condition of the selection branch structure is located, construct a time window with a size of 1 minute; lower the event association support threshold and the event association confidence threshold, and conduct another event association relationship mining according to a similar single-window event association relationship mining method.

[0017] Optimize the obtained loop structure: Lower the minimum frequency requirement, and conduct another search on the log-related event sequence. Based on the loop structure obtained through branch structure mining, construct a time window with a size of 2 minutes starting from the earliest event; lower the event association support threshold and the event association confidence threshold, and conduct another event association relationship mining according to a similar single-window event association relationship mining method.

[0018] Furthermore, the event variable association analysis module is specifically used for:

[0019] Event variable type analysis: If the change situation of the event variables in the log does not exceed 10 types, it is considered an enumerated variable, and only the frequently occurring enumerated values are taken as the event variables that should be concerned about.

[0020] Event variable analysis of branch structure:

[0021] For each optimized loop structure, traverse the set of enumerated variables, and check whether all events in the current structure contain enumerated variables by comparison: If none of the events contain enumerated variables, process the next loop structure; temporarily mark the events containing enumerated variables as enumerated events; starting from the earliest time of the enumerated events in the current structure, open a window with a size of 1 minute; count the variable association support and variable association confidence between the variables of each enumerated event and the variables of its subsequent events; if the requirements of variable association support and variable association confidence are met, take the variable of the current enumerated event as part of the invariant, mark it as an event condition variable, and supplement it to the loop structure set.

[0022] For each optimized selection structure, traverse the set of enumerated variables, and check whether all events in the current structure contain the enumerated variables by comparison: If none of the events contain the enumerated variables, process the next selection branch structure; temporarily mark the events that contain the enumerated variables as enumerated events; starting from the earliest time of the enumerated events in the current structure, open a window with a size of 10 seconds; count the variable association support and variable association credibility between the variables of each enumerated event and its subsequent events; if the requirements of variable association support and variable association credibility are met, use the variables of the current enumerated event as part of the invariant, mark it as an event conditional variable, and supplement it to the set of selection structures.

[0023] Analysis of sequential structure event status variables: Set different variable association support thresholds for event status variables and event conditional variables. Based on the set variable association support thresholds, divide event status variables and event conditional variables. For event status variables, if it is part of the process invariant, it means it should occur frequently and there is no need to consider branch judgment issues during detection. For event conditional variables, they do not occur frequently and branch judgment issues need to be considered during detection;

[0024] Form the final process invariant through the above process.

[0025] Furthermore, the module for maintaining the event sequence to be detected is specifically used for:

[0026] After the process invariant changes, generate or optimize the event sequence to be detected, sort the events according to their frequency, place the frequent events in the front part of the sequence, and use the sum of the event association support between the event and all its next adjacent events to describe the frequency of the event's occurrence. Moreover, it is required to merge the event association support when merging event sequences.

[0027] Furthermore, the module for maintaining the current detected event sequence is specifically used for:

[0028] During anomaly detection, if the detected event matches the first event in the process invariant, it will trigger the action of adding a node, add a new node at the head of the current detected event sequence, the new node records the information of the matched first event, and the record of the new node points to the possible subsequent events; during anomaly detection, if the matching of a certain structure in the process invariant is completed, remove the corresponding matching node of the structure; during anomaly detection, if the matching of a certain structure in the process invariant is not completed, it means that an anomaly may occur.

[0029] Furthermore, the anomaly detection module is specifically used for:

[0030] Receive the event sequence, determine the structure of the event sequence to be detected according to the event sequence to be detected first and then the current detected event sequence, and determine whether there is an anomaly within the time window. If the current event does not hit any of the first events in the current detected event sequence, it means that there is no partially detected event sequence, then go to match the current detected event sequence. If the current event still does not hit the event sequence to be detected, it means that the event does not belong to any process invariant, and it is directly skipped. Repeat the above process until all events in the log are processed.

[0031] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0032] Aiming at the problems of high computational cost and ignoring event variables existing in the existing log anomaly detection methods, the present invention first proposes a software process invariant mining subsystem that can achieve a relatively small computational cost and a more in-depth analysis, and constructs a log anomaly detection system based on process invariants on this basis. To reduce the computational cost, the present invention does not adopt algorithms such as deep neural networks with high training costs, but uses a window mechanism based on the frequency principle to discover event associations and reconstruct the branch structure of the software business process. In addition, to make full use of the information recorded in the log, the present invention also introduces event variables into the process invariants, making the content and form of the process invariants richer and more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a schematic diagram of the log anomaly detection principle of the embodiment of the present invention;

[0034] Figure 2 It is a schematic diagram of mining process invariants of the embodiment of the present invention;

[0035] Figure 3 It is an architecture diagram of a log anomaly detection system based on process invariants of the embodiment of the present invention;

[0036] Figure 4 It is a schematic diagram of the structure of the process invariant mining subsystem of the embodiment of the present invention;

[0037] Figure 5 It is a schematic diagram of the sliding window processing method of the embodiment of the present invention;

[0038] Figure 6 It is a schematic diagram of multi-window result fusion of the embodiment of the present invention;

[0039] Figure 7 It is a schematic diagram of software workflow mining of the embodiment of the present invention;

[0040] Figure 8 It is a schematic diagram of the algorithm idea of business process mining of the embodiment of the present invention;

[0041] Figure 9 Schematic diagram of loop simplification according to an embodiment of the present invention;

[0042] Figure 10 Schematic diagram of the necessity for optimizing the branch structure according to an embodiment of the present invention;

[0043] Figure 11 Schematic diagram of introducing an event variable into a process invariant according to an embodiment of the present invention;

[0044] Figure 12 Schematic diagram of two types of event variables according to an embodiment of the present invention;

[0045] Figure 13 Schematic diagram of a method for mining event variable invariants with a branch structure according to an embodiment of the present invention;

[0046] Figure 14 Schematic diagram of a method for mining event state variable invariants of a sequential structure according to an embodiment of the present invention;

[0047] Figure 15 Schematic diagram of a daily anomaly detection subsystem according to an embodiment of the present invention;

[0048] Figure 16 Two key data structures according to an embodiment of the present invention;

[0049] Figure 17 Process invariant and event sequence to be detected according to an embodiment of the present invention;

[0050] Figure 18 Process invariant and current detected event sequence according to an embodiment of the present invention. Detailed implementation manners

[0051] For ease of understanding, the following explanations are given for some terms that appear in the detailed implementation manners of the present invention:

[0052] Process invariant: The software works according to a pre-set working process, and this working process will also be partially reflected in the log. The process invariant is the relatively fixed part among them. In the present invention, the process invariant includes: event sequence and variables. For example, the process invariant A(p)→B means that if event A occurs and its variable is p, then event B occurs.

[0053] Sliding event window: We adopt the method of a sliding window, with time as the measurement unit and a window width of 10 minutes, to analyze the event sequence within the window. This can improve the analysis efficiency and also respect the event correlation.

[0054] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments:

[0055] When software runs, it usually writes some key information into logs to facilitate subsequent fault detection and other tasks. Naturally, some of the process information reflected by the software will also be recorded in the logs. Once an exception occurs in the software, it may cause changes in the process information recorded in the logs. For example, a buffer overflow attack may cause the software process to interrupt. Therefore, the present invention hopes to extract the relatively invariant features in the business process during the software operation from the software logs, and use these features as a benchmark to discover the software exception situations reflected in the logs.

[0056] Software logs record event information in software activities. Software designers will insert several pieces of code that generate logs in the software source code to report the software running status, and these log information become an important information source for software fault analysis. And these log information also become the data source for anomaly detection. Combining Figure 1 Description. In the figure, t i represents time, e i represents an event. In the figure, the circles represent the abstracted software business processes. The circles with shaded parts represent those that will generate log information, and the circles without shaded parts will not generate log information. This is because software designers do not always generate logs as we expect.

[0057] When the software is normal or abnormal, the information recorded in the logs is different. Normally, the running trajectory of the software is 1→2→3→6 or 1→4→5→6, then the log information they generate should correspond to e1→e3→e6 and e1→e4→e6. As Figure 1 shown. If the event sequence extracted from the log information is e1→e4→e7, this means that event 6 did not occur. This may be the consequence of software exception and is reflected in the logs.

[0058] The main challenge faced by log anomaly detection based on such an idea is: how to find these process invariants that can be used as a benchmark from the logs. As Figure 2 shown, our solution is: based on the principle of event correlation, discover important event sequences from the logs, construct multi-type branch structures (such as: selection, loop, etc.) that partially reflect the business workflow by statistically analyzing these sequential event sequences, extract possible enumeration variables from the logs, combine the enumeration variables to mine the potential relationships between the variables and these event sequences, and finally restore the process invariants as the detection benchmark.

[0059] 1.1 Overview

[0060] A process invariant is a relatively fixed part recorded in the log that reflects the operation of the software. Generally, software operates according to a pre-set workflow, and this workflow will also be partially reflected in the log. This forms a relatively fixed sequence of events in the log. For example, given existing events A and B, and there is a relationship A → B, indicating that event B always occurs after event A, this is a process invariant. The existing large-scale logs are more rich and complex. To discover such process invariants, people often use algorithms with high computational costs.

[0061] We adopt the frequency-based principle and combine event variables to mine process invariants. The frequent occurrence of events means that something important may happen or represents a certain pattern. Therefore, from the perspective of frequency, we count the frequent events in the log events and use this as an anchor to repeatedly scan local fragments of the log, thereby restoring the working process that reflects the software business flow. According to the characteristics of the log variables, some log variables are introduced into the process invariant, improving both the content and form of the process invariant.

[0062] Based on the above considerations, a log anomaly detection system based on process invariants is given. The system architecture is as Figure 3 shown. The system is divided into two main parts: a process invariant mining subsystem and a log anomaly detection subsystem. The process invariant mining subsystem is the core component of the system. It is responsible for extracting process invariants from the log. Its main work is in the training stage, and its main process is divided into three steps: mining event correlation relationships, restoring the software business process, and analyzing event variable correlations. The log anomaly detection subsystem works in the detection stage. Based on the process invariants obtained in the training stage, it detects possible abnormal records recorded in the log and discovers software anomalies.

[0063] From the log to the process invariant, there is actually also the problem of how to extract the event sequence from the original log data. There are some existing algorithms to solve this problem, such as the Drain algorithm. The present invention does not make requirements for the extraction of the event sequence. We assume that there are already mature algorithms that can accurately extract the event sequence from the log. The present invention directly uses the event sequence as the input (both in the training stage and the detection stage).

[0064] 1.2 Process Invariant Mining Subsystem

[0065] The main components of the process invariant mining subsystem are as Figure 4As shown in the figure, it mainly includes: an event correlation relationship mining module, a software workflow restoration module, and an event variable correlation analysis module. The event correlation relationship mining module is responsible for extracting valuable event sequence fragments from log events. The software workflow restoration module partially restores the software workflow from the event sequence fragments. Since the software workflow not only has a sequential structure but also has branch structures such as selection and loop, it is necessary to restore the branch structure as much as possible from the sequential structure to be closer to the real situation of the software workflow. However, complete restoration is neither possible nor realistic. The event variable correlation analysis module analyzes the possible correlation relationships between event variables and the software workflow based on the restored software workflow, thereby forming the final process invariant.

[0066] 1.2.1 Event correlation relationship mining module

[0067] The event correlation relationship mining module mines the associated event sequence fragments from the log event sequence. To improve the mining efficiency, we adopt a sliding window processing method to detect the most relevant and limited number of event sequences.

[0068] 1.2.1.1 Sliding window processing method

[0069] As Figure 5 shown, we use a sliding window to count the frequency of events within the window and mine the correlation degree between events, thereby discovering regular event sequence fragments. The starting point for designing the sliding window is that, from the perspective of correlation, most associated events are adjacent in time, which means that associated events are often adjacent in the position in the log. In other words, we don't need to search in the entire log event sequence, but only in a local part. To avoid the situation where boundary events of the window are missed, we require partial overlap between windows. As Figure 5 shown by the shaded events E and F, they may be associated with events G and H. If there is no overlap between window 1 and window 2 and window 2 starts directly from event G, this association relationship may not be captured by us.

[0070] For the design of window size and overlap size, we use time as the measurement unit. The reason for using time as the measurement unit is that there are strong time characteristics in the occurrence of log events. Moreover, within a certain time period, a large number of events occur, which itself reflects the possibility of a certain system event occurring. Given that we are mainly used for the massive log analysis generated by large-scale distributed software such as Hadoop, and the operations of these software generally complete within a few seconds, and the operations completed collaboratively generally complete within a few minutes, so we set the time unit to minutes. More specifically, we default the window size to 10 minutes and the overlap size to 1 minute. This ensures the accuracy of the analysis and also ensures the efficiency of the analysis.

[0071] 1.2.1.2 Single Window Processing Flow

[0072] For the convenience of explanation, we give the following concepts: event association, event association support, event association credibility, event association support threshold, and event association credibility threshold.

[0073] Event association should be an implicative formula, which can be expressed as A→B, indicating that if event A occurs, event B will definitely occur. However, due to the incompleteness of the log data generated by the software, or due to the incompleteness of the extraction of log events, etc., it is often not possible to be completely sure that if event A occurs, event B will definitely occur. Therefore, other concepts need to be combined to describe the reliability of this relationship.

[0074] Event association: It indicates the situation that if event A occurs, event B will occur, and can be expressed as A→B s,c , where s and c represent event association support and event association credibility. For the convenience of discussion, without special explanation, A→B s,c is simplified to A→B.

[0075] Event association support: If there is an association relationship between event A and event B, then the probability that they appear simultaneously within a single window is the event association support.

[0076] Event association credibility: If event A occurs, then the probability that event B occurs within the same window is the event association credibility.

[0077] For example: In a certain window, there is an event sequence {ABAABC}, then the association support of the association relationship A→B is 2 / 6, and the association credibility of the association relationship A→B is 2 / 3.

[0078] Event association support threshold: If there is a strong association relationship between events, then the minimum value that the event association support should reach.

[0079] Event association confidence threshold: The minimum value that the event association confidence should reach if there is a strong association relationship between events.

[0080] To find as long an event sequence as possible as the result, we start with an event sequence of length 1 and gradually increase the length of the event sequence by adding subsequent events after it. For example, in a certain window, there is an event sequence {ABAABC}. First, all event sequences of length 1 are formed {A, B, A, A, B, C}; to increase the sequence length, we refer to the original sequence and try to increase its length, converting it into a set of event sequences of length 2 {AB, BA, AA, AB, BC}; and so on. For distinction, we call the set of event sequences of length i the i-set.

[0081] Regarding the event association support threshold and the event association confidence threshold, we default to set them to 10% and 20% respectively. For the event association support threshold, our consideration is that since large-scale distributed software often has multiple business flows intersecting, in a window, a certain type of event sequence may not seem very frequent. If the threshold is set too high, most events may be filtered out, so it is set to 10%. For the event association confidence threshold, our consideration is that the current sequential event sequence often also includes branch selection situations, and not all branches are so frequent, so it is set to 20%.

[0082] Based on the above description, the single-window event association relationship mining process is given.

[0083]

[0084] Among them, the candidate (i + 1)-set refers to the candidate set of event sequences with a length of i + 1. Subsequently, based on it, the relationship support and confidence will be statistically calculated, and only the event sequences that meet the conditions will be included in the (i + 1)-set.

[0085] 1.2.1.3 Multi-window processing flow

[0086] The purpose of multi-window processing is to reasonably integrate the single-window processing results. After the single-window processing is completed, several event sequences will be output to events_a. events_a is a temporary structure used to accommodate the output results of all windows. Due to the repetition of software operations, different windows may have different results for the event sequences generated by the same operation. At this time, the event sequences generated by different windows need to be merged according to the actual situation to form a unified result.

[0087] There are mainly three situations that need to be processed.

[0088] (1) No intersection and overlap: Such asFigure 6 For the event sequences {HI} and {JK} shown, if there is no intersection between the two windows, they will be directly included in the final result set.

[0089] (2) Overlap: As Figure 6 For the event sequence {EF} shown, if the event sequences of the two windows are exactly the same, then they will be directly merged. The merging method is as follows: without changing the content of the event sequence, but adding up the event correlation support degree and the event correlation credibility corresponding to the two sequences.

[0090] Intersection: As Figure 6 For the event sequences {ABCD} and {BCDG} shown, if there is an intersection between them, then keep the original sequences, extract the common intersection part to form a new event sequence {BCD}, and assign the event correlation support degree and the event correlation credibility (the sum of the event correlation support degrees and the event correlation credibilities of the two original sequences).

[0091] Note: The current event correlation support degree and the event correlation credibility have lost their original meanings, and are only used as the basis for sorting the event sequences to be detected in subsequent processing. For event sequences with higher event correlation support degrees and event correlation credibilities, we think they should be processed first.

[0092] Based on the above discussion, the mining process of multi-window event correlation relationship is given.

[0093]

[0094] 1.2.3 Software Workflow Restoration Module

[0095] The main purpose of the software workflow restoration module is as Figure 7 shown, to restore the software workflow as much as possible from the sequential events_A. events_A is a sequential event sequence, but the software workflow not only presents a sequential state, but also has multiple types of branch structures. An ideal flow invariant should not only be a sequential event sequence, but an event sequence that is closer to the software workflow and contains branch structures. To make the software workflow more clear, we mainly focus on three branch structures: selection, aggregation (parallel), and loop. Since events_A itself is only a partial restoration, the restoration result is also only a part of the software workflow.

[0096] 1.2.3.1 Branch Structure Mining Process

[0097] The basic idea of branch structure mining is as follows: all sequence segments in event_A are split into several binary relationships, that is, the length of all event sequence segments is 2. The leading and subsequent relationships of all events in event_A are counted, and branch structures such as selection, convergence, and loop are generated according to the situations of common leading and common subsequent events, and the length of the event sequence is tried to be extended. During this process, loop structures are further discovered. Combined with Figure 8 Explanation. Figure 8 For the 2 event sequence segments shown, we first split them into 5 binary relationships. By counting the leading events and subsequent events of these events, a selection structure {A,B / C} can be obtained; based on this structure, the length of the event sequence is tried to be extended, and a loop structure such as {A,C,A} is discovered.

[0098] The discovery of loop structures is more complex and mainly faces two challenges. One is the complexity of loop structures. Since there may be various situations such as nested loops, cross loops, and nested cross loops in the loop itself. And after the mining and filtering of associated events, these events themselves are incomplete and it is difficult to reconstruct a complete loop structure. The other is the multi-hop feature of loop structures. For multi-hop loop structures, since the event sequences participating in the loop are too long, it is difficult to discover the loop.

[0099] For multi-hop loops, for overly long multi-hop loop structures, we regard them as sequential structures and no longer pursue comprehensive and accurate discovery. For complex loops, our solution is: focus on the local part of the loop structure, mainly single loops and a small number of multi-step loops. We do not aim to accurately restore the loop structure, so complex loop structures such as nested loops and cross loops are replaced with single loop structures as much as possible. As Figure 9 shown, there is a nested loop here, and there is a jump between the inner and outer loops, but we split it into two single loop event sequences and a sequential event sequence (that is, Figure 9 CF in it. In fact, due to the mining of association relationships, such sequential event sequences may have a low occurrence frequency and have been eliminated and will not appear in the current link). Although this destroys the integrity of the loop and cannot accurately reflect the software workflow, for anomaly detection, this simplification will not overly affect the detection efficiency.

[0100] The method for discovering a cyclic structure is as follows: Check whether there are reverse events in the event sequence. For example, in {A,...,C},... represents that there can be zero or more events between event A and event C. Then the reverse event of the event sequence {A,...,C} is {C,...,A}. Once a reverse event sequence is found in the event sequence, it is considered that there is a cyclic structure. More specifically: First, we count the number of occurrences of all events in the event sequence. If an event appears more than twice, then there will be a reverse event. An extreme case is that there are multiple cycles intersecting in an event sequence. For the sake of simplifying the processing: We extract all cycles as alternatives. Another case is that an event appears more than twice. For the sake of simplifying the processing: We take the two events with the longest distance as the boundaries of the cyclic event sequence. That is: If there is {C,..,B...,C,...,A,.....,C}, we regard {C,..,B...,C,...,A,.....,C} as a cycle, and {C,..,B...,C} and {C,...,A,.....,C} will no longer be processed separately.

[0101] Based on the above considerations, the discovery process of the branch structure is given.

[0102]

[0103] 1.2.3.2 Select branch structure optimization

[0104] It is necessary to optimize the selected branch structure obtained in the previous stage. Due to the incompleteness of the log information, as well as the elimination and filtering in the previous analysis, the selected branch structure we obtained is also incomplete. As Figure 10 shown, the software workflow has three branches, among which the hit probabilities of two branches are 48%, and the hit probability of one branch is 4%. According to our method, the first two branches can be easily discovered, while for the last branch, due to its low frequency of occurrence in the log, it is very likely to be missed.

[0105] Our solution is to lower the minimum frequency requirement and conduct another search on the log-related event sequence. Starting from the event where the selection condition of the selected branch structure is located, construct a time window with a size of 1 minute; according to a similar method for mining event correlation relationships in a single window, conduct another mining of event correlation relationships. The difference is that at this time, the event correlation support threshold and the event correlation confidence threshold are set to 2% and 4% respectively. Such settings are to mine relevant low-frequency events. The reason for the setting comes from the evaluation: The number of events generated by the log per minute ranges from several hundred to several thousand.

[0106] To avoid including irrelevant events, such mining should be carried out closely following the existing event sequences of the selection branch structure. Still taking Figure 10 as an example, when a new event X is found to meet the event association support threshold and the event association confidence threshold, but event A or event B appears on its subsequent path, then we consider that an interfering event has occurred. Such an event is just noise generated because we have lowered the threshold. The conditions for in-depth mining of branch events in the selection branch structure are: the new branch event cannot appear on the existing branch path and must meet the event association support threshold and the event association confidence threshold.

[0107]

[0108] 1.2.3.3 Loop Structure Optimization

[0109] The loop structure also needs to be optimized. In the loop structure, since the events within the loop body occur repeatedly, they are more likely to be discovered. However, the events after the loop body ends are difficult to discover due to frequency issues. On the other hand, the events that occur after the loop often imply some important software state changes and are more representative instead. Therefore, it is very necessary to specifically discover loop-ending events.

[0110] Our solution is to lower the minimum requirement for frequency and conduct another search on the log-related event sequences. Based on the loop structure within event_Cir, taking the earliest-occurring event as the starting time, construct a time window with a size of 2 minutes (the reason for making it larger than the time window of the selection branch structure is that the loop itself takes more time); conduct another event association relationship mining according to a similar mining method for single-window event association relationships; for the event association support threshold and the event association confidence threshold, we set them to 2% and 4%.

[0111]

[0112] 1.2.4 Event Variable Association Analysis Module

[0113] Event variables may also become part of the process invariant. For the events in the selection branch structure, the variables of the event where the selection condition of the selection structure is located may indirectly reflect the selection branch condition or the result of the selection branch. Such as Figure 11As shown in the figure, event A has a selection branch structure with events B and C. After analyzing the statistical log events, it is found that the variables of event A have only two values, 1 and 2. When the variable is 1, event B follows; when the variable is 2, event C follows. Thus, we can consider the variable of event A as part of the process invariant. In contrast, the variables of events B and C have no pattern and cannot be regarded as invariants. Based on this discovery, we will further analyze the event variables and use them as part of the process invariant.

[0114] 1.2.4.1 Analysis of Event Variable Types

[0115] Event variables are often components with fixed meanings output by programmers from the internal state of the software. According to our statistics and analysis, programmers often output a status message to the log for later analysis of the software state. These variables representing status are usually fixed enumeration values. For example, Figure 12 as shown in the figure, in such logs, words like "down" and "up" are used to represent status information. We call them enumeration variables. In contrast, another common type of identification variable is used to identify the subject recorded in the log. However, due to its excessive variety of changes, we only focus on enumeration variables.

[0116] The characteristic of enumeration variables is relative fixity, which provides a basis for us to automatically discover enumeration variables from the log. For the logs of the same event, we believe that if it contains enumeration variables, usually there will be no more than 10 enumeration values, such as the status values of the business: down, up, suspend, etc. Therefore, we think that if the change situation of the event variables in the log does not exceed 10 kinds, they are enumeration variables. We only take the frequently occurring enumeration values as the event variables we should focus on. Of course, other types of variables may still be part of the process invariant, but this invention does not make such a requirement.

[0117] The method for mining enumeration variables is given.

[0118]

[0119]

[0120] 1.2.4.2 Analysis of Event Variables in Branch Structures

[0121] To more objectively describe the relationship between event variables and process invariants, we introduce the following concepts.

[0122] Variable association: Indicates that a variable is associated with a certain event sequence and can be expressed as p→B sp,cp, where sp and cp represent the variable association support and the variable association confidence. For the convenience of elaborating variable association, without special explanation, p→B sp,cp , which is simplified as p→B.

[0123] Variable association support: If there is an association relationship between variable p and event sequence B, then the probability that they appear simultaneously within a single window.

[0124] Variable association confidence: If variable p appears, then within the same window, the probability that event sequence B also occurs.

[0125] Variable association support threshold: If there is a strong association relationship between a variable and an event sequence, then the minimum value that the variable association support should reach.

[0126] Variable association confidence threshold: If there is a strong association relationship between a variable and an event sequence, then the minimum value that the variable association confidence should reach.

[0127] Event variables are divided into event condition variables and event state variables. Event condition variables refer to event variables that are closely related to branch events. For example, when event variable x is 1, the subsequent event is A, and when x is 2, the subsequent event is B. Event state variables refer to variables that reflect the current state of an event. For example, event B always appears with event variable y = 3. Event state variables are the reinforcement of the event itself.

[0128] For event condition variables, the method of incorporating them into invariants is as follows. For all existing branch structures, count the events and their event variables before the branch, as well as the mapping of subsequent branch events. Figure 13 Selection and loop are two branch structures. The shaded events are the events and variables to be counted. Figure 13 The left part in shows the existing mapping situations {A(1), B} and {A(2), C}. Then we can consider that there is a certain association relationship between the variable of event A and the branch events. Figure 13 The right part in shows the existing mapping relationships {A(1), B} and {A(2), C)}, then we can consider that when the event variable of A is 2, the loop is exited.

[0129] For the event variable association of the branch structure, we set both the variable association support threshold and the variable association credibility threshold to 0.1 and 0.8. Our consideration is that there is often a strong association between the variables in the branch structure and the branch judgment. In theory, the variable association credibility threshold should be set to 1. However, due to the limitations and ambiguity of log output, it may lead to a functional relationship between the log variables and the actual branch judgment variables, that is, the log variables only have a certain connection with the judgment variables in the branch structure, but are not necessarily equivalent. Therefore, we can only approximately assume it, rather than fully determine it. The variable association support threshold is relatively low because the branch itself may occur less frequently, so its frequency of occurrence cannot be too high.

[0130]

[0131]

[0132] 1.2.4.3 Analysis of Event State Variables in Sequential Structure

[0133] The event variables in the event sequence of the sequential structure may essentially be the event variables in the event sequence of the branch structure. In the previous section, we focused more on the events before and after the branch, and for the long event sequences before or after, we regarded them as sequential structures. In fact, it is also difficult to distinguish them from the event sequences of the sequential structure in the strict sense. For example, there is a sequential structure as shown Figure 14 which may actually be a selection branch structure. The reason for this situation may be that after event A, there should originally be an event related to the branch condition, but due to program design reasons, it is not output to the log, or due to the filtering of the previous algorithm, the event is missing.

[0134] It is necessary to distinguish the event condition variables and event state variables found in the sequential structure. For event state variables, if they are part of an invariant, it means they should occur frequently, and we do not need to consider branch judgment issues during detection. For event condition variables, they do not occur frequently, and we also need to consider branch judgment issues during detection. The main difference between the two is that the variable association support thresholds are different. We set the variable association support threshold for event state variables to 0.9, while the corresponding value for event condition variables is 0.1.

[0135] Give the process of analyzing event state variables in the sequential structure

[0136]

[0137]

[0138] 1.2.5 Summary

[0139] Any structure in event_O, event_S, event_Cir, and event_Con is called a process invariant, which reflects the relatively fixed part in the software workflow. We call event_O, event_S, event_Cir, and event_Con the process invariant set. A process invariant may be just an event sequence or may also include event variables. The purpose of the process invariant mining subsystem is to discover the process invariants existing in the software workflow from the trained log set as the benchmark for subsequent anomaly detection. This part is the core part of the present invention.

[0140] 1.3 Log Anomaly Detection Subsystem

[0141] 1.3.1 Overview

[0142] As Figure 15 shown, the daily anomaly detection subsystem uses process invariants as the detection benchmark, takes the log event sequence as the input, and outputs the possible anomalies in the log. The daily anomaly detection subsystem mainly includes: a to-be-detected event sequence maintenance module, a current detected event sequence maintenance module, and an anomaly detection module. The to-be-detected event sequence and the current detected event sequence are two core data structures in the log anomaly detection subsystem, which are maintained by the to-be-detected event sequence maintenance module and the current detected event sequence maintenance module respectively. The anomaly detection module, with the support of the two data structures, reads the process invariants to discover log anomalies.

[0143] The descriptions of the to-be-detected event sequence and the current detected event sequence are as follows. The to-be-detected event sequence is a sequence formed by extracting the first items of all structures in the process invariant and arranging them in descending order according to the frequency of each first event. The current detected event sequence is a sequence that records the intermediate state of the event sequence being detected by comparing with the process invariant. For example, if the event sequence AB has been detected, then the current detected event sequence, according to the process invariant, records the possible next event C (of course, there may be multiple next events for the event sequence AB).

[0144] To Figure 16 further explain, assume there is a Figure 16 process invariant as shown. The to-be-detected event sequence is a combination composed of the first events of all possible event sequences. For example, if there is an event sequence A→B→C in event_O, then event A should be included in the to-be-detected event sequence; if there is an event sequence D / G→H in event_Con, then events D and G should be included in the to-be-detected event sequence. The current detected event sequence plays a crucial role during detection. As Figure 16As shown, when the event (I, x) arrives at the current detected event sequence maintenance module, it checks whether event I is an event in the process invariant. As shown in the legend, it first checks whether it is A→B→C or A→B→E; if not, it further checks whether it is D→H; if not, it further checks whether it is H→I or H→L. Obviously, at this time, event I hits the event sequence H→I.

[0145] 1.3.2 To-be-detected event sequence maintenance module

[0146] The to-be-detected event sequence contains the first events of all structures in the process invariant and points to the corresponding structures in the process invariant, as Figure 17 shown. When implementing anomaly detection, the to-be-detected event sequence is the basis for checking events, and according to the pointer of the to-be-detected event sequence, it can turn to the corresponding event sequence in the process invariant. There may be multiple structures with the same first event, such as Figure 17 event A in event_O and event A in event_Cir shown in, so at the event A node in the to-be-detected event sequence, it will point to two structures at the same time. Note: Non-first events do not need to be included in the to-be-detected event sequence, such as Figure 17 event B in event_O shown in.

[0147] After the process invariant changes, the to-be-detected event sequence is generated or optimized by the to-be-detected event sequence maintenance module. To improve the detection efficiency, we sort the events according to their frequency, and place the frequent events in the front part of the sequence, so that events can be detected at a relatively small time cost. The frequency of an event can be described by the sum of the event association supports between the event and all its next adjacent events (there may be multiple next adjacent events for a certain event), and we require that the event association supports need to be merged when the event sequences are merged, so that the frequency can be more objectively reflected (the merged event association support no longer conforms to its definition, but can still reflect its frequency).

[0148]

[0149]

[0150] 1.3.3 Current detected event sequence maintenance module

[0151] The current detected event sequence is used to record the matching process for the process invariant. As Figure 18As shown, events (AB), event (D), and event (H) have been matched in the current detection event sequence. The first node records the event sequence AB and points to the possible subsequent events of the event sequence AB in the process invariant, namely event C in event_O and events A and E in event_Cir. In other words, this means that events A and B have been discovered, so the currently detected event needs to match event C, event A, or event E. As Figure 18 shown, when matching the current detection event I and the attached variables, the subsequent events of node (AB), node (D), and node (H) will be checked in sequence, and finally it is found that it matches event I in event_S.

[0152] The current detection event sequence maintenance module adds or deletes nodes in the current detection event sequence during anomaly detection. During anomaly detection, if the detection event matches the first event in the process invariant, the add node action will be triggered. The specific added content is: adding a new node at the head of the current detection event sequence; the new node records the information of the matched first event; the record of the new node points to the possible subsequent events. As Figure 18 shown, after the detection event matches event A, a node will be added to record the matched event A, and the node record points to the possible subsequent events, including event B in event_O and event B in event_Cir. During anomaly detection, if the matching of a certain structure in the process invariant is completed, the corresponding matching node of this structure will be removed. As Figure 18 shown, after event C appears, the matching of the event sequence A→B→C in the process invariant is completed, then the node (AB) recording this structure can be removed. During anomaly detection, if the matching of a certain structure in the process invariant is not completed, it means that an anomaly may occur.

[0153]

[0154]

[0155] 1.3.4 Anomaly Detection Module

[0156] The anomaly detection module is responsible for receiving the event sequence, determining the structure of the event sequence to be detected according to the event sequence to be detected first and then the current detected event sequence, and determining whether there is an anomaly within the time window. Since the event sequence to be detected records the first event information of the event sequence that has been partially detected, its quantity is relatively less than that of the current detected event sequence and it is most likely to be queried, so the event sequence to be detected is matched first. If the current event does not hit any of the first events of the current detected event sequence, it means that there is no event sequence that has been partially detected, so the anomaly detection module turns to match the current detected event sequence. If the current event still does not hit the event sequence to be detected, it means that the event does not belong to any process invariant and is directly skipped.

[0157] The criterion for detecting an anomaly is that within a certain time window, the expected events in the process invariant do not occur. For example, within the time window, the process invariant is: A→B→C. However, the event sequence A→B has occurred, but event C has not occurred within the time window, which is regarded as an anomaly. The reason for this anomaly may be that the host has stopped, resulting in the inability to continue executing event C, or it may be due to an attack causing the software to stop working. Since the expected event may occur at the window boundary, if no event is found in the current window, the window crossover designed by us can be used to detect again in the adjacent window.

[0158]

[0159] In summary, in view of the problems of high computational cost and ignoring event variables existing in the existing log anomaly detection method, the present invention first proposes a software process invariant mining subsystem that can achieve a relatively low computational cost and a more in-depth analysis, and constructs a log anomaly detection system based on process invariants on this basis. To reduce the computational cost, the present invention does not adopt algorithms such as deep neural networks with high training costs, but adopts a window mechanism based on the frequency principle to discover event associations and reconstruct the branch structure of the software business process. In addition, to make full use of the information recorded in the logs, the present invention also introduces event variables into the process invariants, making the content and form of the process invariants richer and more accurate.

[0160] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A log anomaly detection system based on process invariants, characterized in that, The system includes a process invariant mining subsystem and a log anomaly detection subsystem; The process invariant mining subsystem includes an event correlation relationship mining module, a software workflow restoration module, and an event variable correlation analysis module; The event correlation relationship mining module is used to extract event sequence fragments from log events; The software workflow restoration module is used to partially restore the software workflow from the event sequence fragments, including branch structure mining, optimizing the obtained selection branch structure, and optimizing the obtained loop structure; The event variable correlation analysis module is used to analyze the possible correlation relationships between event variables and the software workflow based on the restored software workflow, so as to form the final process invariant; The log anomaly detection subsystem includes a to-be-detected event sequence maintenance module, a current detection event sequence maintenance module, and an anomaly detection module; The to-be-detected event sequence maintenance module is used to maintain the to-be-detected event sequence; the current detection event sequence maintenance module is used to maintain the current detection event sequence; the anomaly detection module is used to discover log anomalies by reading the process invariant with the support of the to-be-detected event sequence and the current detection event sequence; The event variable correlation analysis module is specifically used for: Event variable type analysis: If the change situation of the event variables in the log does not exceed 10 types, the event variables are considered enumeration variables; Branch structure event variable analysis: For each optimized loop structure, traverse the set of enumeration variables, and check whether all events in the current structure contain enumeration variables by comparison: If none of the events contain enumeration variables, process the next loop structure; temporarily mark the events containing enumeration variables as enumeration events; start a window with a size of 1 minute starting from the earliest time of the enumeration events in the current structure; count the variable correlation support degree and variable correlation credibility between the variables of each enumeration event and the variables of its subsequent events; if the requirements of the variable correlation support degree and variable correlation credibility are met, take the variables of the current enumeration event as part of the invariant, mark it as an event condition variable, and supplement it to the loop structure set; For each optimized selection structure, traverse the set of enumeration variables, and check whether all events in the current structure contain enumeration variables by comparison: If none of the events contain enumeration variables, process the next selection branch structure; temporarily mark the events containing enumeration variables as enumeration events; start a window with a size of 10 seconds starting from the earliest time of the enumeration events in the current structure; count the variable correlation support degree and variable correlation credibility between the variables of each enumeration event and the variables of its subsequent events; if the requirements of the variable correlation support degree and variable correlation credibility are met, take the variables of the current enumeration event as part of the invariant, mark it as an event condition variable, and supplement it to the selection structure set; Sequential structure event status variable analysis: Different variable association support thresholds are set for event status variables and event condition variables. Based on the set variable association support thresholds, event status variables and event condition variables are divided. For event status variables, if an event status variable is part of a process invariant, the branch judgment problem does not need to be considered during detection. For event condition variables, the branch judgment problem also needs to be considered during detection; The final process invariant is formed through the above process.

2. The log anomaly detection system based on process invariants according to claim 1, characterized in that, The event association relationship mining module is specifically used for: Sliding window processing: A sliding window is adopted to count the frequency of events within the window and mine the association degree between events, so as to discover regular event sequence segments; for the design of the window size and window overlap size, time is used as the measurement unit; Single window processing: Starting from an event sequence of length 1, count the event association support and event association credibility of all event sequence segments of length 1 within a certain time window, and remove the event sequences with event association support and event association credibility less than the set values; then, based on the event sequence of length 1, add subsequent events behind it, gradually increasing the length of the event sequence, and mine the event association relationship within a certain time window in the above manner; Multi-window processing: The event sequences generated by different windows are merged according to the actual situation to form a unified result: when the event sequences generated by different windows do not intersect or overlap, the event sequences generated by the same window are directly included in the final result set; When the event sequences generated by different windows overlap, the content of the event sequences is not changed, but the event association support and event association credibility corresponding to the two overlapping sequences are accumulated; When the event sequences generated by different windows intersect, the original sequences are retained, the common intersection part is extracted to form a new event sequence, and the event association support and event association credibility of the two original sequences with intersection are accumulated and used as the event association support and event association credibility of the new event sequence respectively.

3. The log anomaly detection system based on process invariants according to claim 2, wherein The software work process restoration module is specifically used for: Branch structure mining: All sequence segments in the sequential event sequence event_A are split into several binary relationships, that is, the length of all event sequence segments is 2. The leading and subsequent relationships of all events in event_A are counted, and selection, convergence, and loop branch structures are generated according to the situation of common leading and common subsequent, and the length of the event sequence is tried to be extended. During this process, loop structures are further discovered; Optimize the obtained selection branch structure: Lower the minimum requirement of frequency, and conduct another search on the event sequence related to the log. Starting from the event where the selection condition of the selection branch structure is located, construct a time window with a size of 1 minute; Lower the event association support threshold and event association credibility threshold, and conduct another event association relationship mining according to a method similar to the single window event association relationship mining method; Optimize the obtained loop structure: lower the minimum frequency requirement, search the log-related event sequence again, based on the loop structure mined through the branch structure, construct a time window with the first-occurring event as the starting time, and the size of the time window is 2 minutes; Lower the event association support threshold and the event association confidence threshold, and perform event association relationship mining again according to the mining method similar to that of the single-window event association relationship.

4. The log anomaly detection system based on process invariants according to claim 1, wherein The module for maintaining the event sequence to be detected is specifically used for: After the process invariant changes, generate or optimize the event sequence to be detected, sort the events according to their frequencies, place the frequent events in the front part of the sequence, describe the frequency of occurrence of an event by the sum of the event association supports between the event and all its next adjacent events, and require that the event association support be merged when the event sequences are merged.

5. The log anomaly detection system based on process invariants according to claim 1, characterized in that The module for maintaining the current detected event sequence is specifically used for: During anomaly detection, if the detected event matches the first event in the process invariant, it will trigger the action of adding a node, add a new node at the head of the current detected event sequence, the new node records the information of the first event that has been matched, and the record of the new node points to the possible subsequent events; during anomaly detection, if the matching of a certain structure in the process invariant is completed, the corresponding matching node of the structure will be removed; during anomaly detection, if the matching of a certain structure in the process invariant is not completed, it means that an anomaly may occur.

6. The log anomaly detection system based on process invariants according to claim 1, characterized in that The anomaly detection module is specifically used for: Receive the event sequence, determine the structure of the event sequence to be detected according to the event sequence to be detected first and then the current detected event sequence, and determine whether there is an anomaly within the time window. If the current event does not hit any of the first events in the current detected event sequence, it means that there is no partially detected event sequence, then turn to match the current detected event sequence. If the current event still does not hit the event sequence to be detected, it means that the event does not belong to any process invariant, and it is directly skipped. Repeat the above process until all events in the log are processed.

Images