Log association analysis method and device based on FP-Growth algorithm and storage medium

Abstract

The invention relates to a log association analysis method based on an FP-Growth algorithm, and the method is characterized in that the method comprises the steps: S1, obtaining an original log, and generating an event; s2, based on a given division standard, dividing log events, and constructing a transaction data set; s3, for the transaction data set, mining a frequent item set by adopting an FP-Growth algorithm in a frequent event mode of the FP-Growth algorithm; and S4, generating a strong association event rule. According to the method, the FP-Growth algorithm is adopted to mine the association relationship of the log events, and the FP-tree is constructed to store the frequent item set information, so that the memory occupation can be reduced, and the mining process of the frequent pattern is simplified. The invention further provides a device for implementing the log association analysis method based on the FP-Growth algorithm and a storage medium.

Description

Technical Field

[0001] The present invention belongs to the field of data mining, and in particular relates to a log association analysis method, device and storage medium based on FP-Growth algorithm. Background Art

[0002] Log correlation analysis refers to the process of analyzing massive amounts of log data to identify the correlation between log events, aiming to connect seemingly discrete or isolated log events through some logic or rules, thereby revealing the larger-scale events or behavior patterns hidden behind them. Log correlation analysis technology is of great significance in understanding the overall operation of the system, detecting potential security threats, and optimizing system performance.

[0003] Faced with massive amounts of log data, traditional methods of manually mining association rules are time-consuming and laborious. First, the scale of log data makes it infeasible to manually formulate rules. Even experienced operation and maintenance or security analysts find it difficult to effectively process hundreds of millions or even more log entries in a short period of time. Second, the diversity and complexity of log data further exacerbate the difficulty of analysis. Different systems, applications, and even different modules within the same system often produce log records with different formats and complex contents. Analysts need to have a deep field background and knowledge reserves to discover the associations among them.

[0004] At present, log association analysis methods mainly include cluster association analysis and rule association analysis. Cluster association analysis refers to the classification of similar log events into the same category through clustering methods, so that it is considered that there is a certain correlation between these events. However, there are some disadvantages in clustering-based association analysis methods. First, the association results obtained are affected by the initial conditions and parameter selection. Different initial conditions and parameter settings may lead to different clustering results, thus affecting the accuracy and reliability of association analysis. Second, clustering algorithms are sensitive to noise and anomalies, which may lead to the neglect of important data points or the wrong association of abnormal data in association analysis. Third, clustering results are not very interpretable. Clustering is based on statistical data for classification, so it may not be able to directly explain why some data points are classified into one category, which requires further manual analysis. Summary of the invention

[0005] The purpose of the present invention is to provide a log association analysis method, device and storage medium based on FP-Growth algorithm to solve the problems existing in the prior art.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A log association analysis method based on FP-Growth algorithm, the method comprising:

[0008] S1: Get raw logs and generate events;

[0009] S2: Based on the given partitioning criteria, divide the log events and construct the transaction data set;

[0010] S3: For transaction data sets, frequent event patterns are obtained using the FP-Growth algorithm, and frequent itemsets are mined using the FP-Growth algorithm;

[0011] S4: Generate strong correlation event rules.

[0012] Furthermore, obtaining the original log and generating events is to obtain the log of a single device, collect the original logs generated by the required analysis device by Syslog forwarding or client reporting, parse the original logs using rule-based, template-based or machine learning parsing methods, and generate events.

[0013] Furthermore, the division standard described in S2 is a fixed time window, which means that log events are divided sequentially based on the occurrence time of the original log.

[0014] Furthermore, the transaction data set described in S2 is generated based on the log event division result.

[0015] Furthermore, the FP-Growth algorithm described in S3 is used to mine frequent itemsets, including:

[0016] Constructing an FP-tree based on the acquired transaction data set;

[0017] Mining frequent item sets from FP-tree is to obtain frequent patterns of events.

[0018] Furthermore, the construction of the FP-tree includes:

[0019] S71: Scan the transaction data set, and generate a set of frequent 1-item sets according to the preset minimum support: Scan the transaction data set, calculate the support of each event; set the minimum support, and items with a support less than the minimum support are non-frequent 1-item sets, delete non-frequent 1-item sets according to the minimum support, and the remaining events with support greater than the minimum support are frequent 1-item sets; arrange the frequent 1-item sets in descending order according to their support, and generate a set of frequent 1-item sets;

[0020] S72: sorting the transaction data set according to the frequent 1-item set set: for the items of the transaction data set, comparing the frequent 1-item set set, deleting the events that are not in the frequent 1-item set set; sorting the items of the transaction data set according to the events in the frequent 1-item set set; generating a sorted transaction data set;

[0021] S73: constructing an FP-tree, that is, constructing an FP-tree according to the generated sorted transaction data set;

[0022] S74: Generate a node linked list, that is, connect the corresponding nodes according to the correspondence between the frequent 1-item set set and the FP-tree, so as to facilitate tracing in the subsequent mining process.

[0023] Furthermore, the mining of frequent itemsets includes:

[0024] S81: Generate a conditional pattern base for each node of the FP-tree, where the conditional pattern base is a set of paths ending with the element item to be searched, and each path is a prefix path;

[0025] S82: constructing a corresponding conditional FP-tree according to the conditional pattern base, specifically, taking the conditional pattern base as a sub-transaction data set, and the FP-tree generated according to the sub-transaction data set is the conditional FP-tree;

[0026] S83: recursively construct a conditional FP-tree and simultaneously increase the frequent item sets contained therein. Recursive construction means repeatedly executing S81 and S82 for the conditional FP-tree until the conditional pattern base of the search node is "null".

[0027] S84: If the conditional FP-tree has only one path, the frequent item sets contained therein are directly generated; if the FP-tree constructed in step S82 has only one path and no redundant branches, the frequent item sets are directly generated.

[0028] Furthermore, the support refers to the number of transactions containing a specific item set, which is a measure of the importance of the association rule, that is, the greater the support of the association rule, the more important the association rule is.

[0029] Furthermore, the generating of the strong correlation event rule is to select and discard the generated frequent item sets based on the preset minimum confidence, thereby generating the strong correlation rule;

[0030] The confidence level is defined as follows:

[0031]

[0032] Among them, c(X→Y) represents the confidence of the association rule X→Y, s(X,Y) represents the support of the item set {X,Y}, and s(X) represents the support of the item set {X};

[0033] Confidence is a measure of the reliability of association rules. For a given rule X→Y, the higher the confidence, the greater the possibility that Y is included in the transaction of X.

[0034] A rule that satisfies both the minimum support and the minimum confidence is called a strong association rule.

[0035] The present invention provides a log association analysis device based on the FP-Growth algorithm, the analysis device comprising:

[0036] Data collection and event generation module, used to obtain raw logs and generate events;

[0037] The event partitioning module is used to partition log events based on given partitioning criteria and construct transaction data sets;

[0038] Data mining module, used to mine frequent itemsets using FP-Growth algorithm for frequent event patterns in transaction data sets;

[0039] The event correlation module is used to generate strongly correlated event rules.

[0040] The present invention also provides a computer-readable storage medium, which stores computer program instructions. When the computer program instructions are executed by a computer, the computer executes the steps of any of the above-mentioned log association analysis methods based on the FP-Growth algorithm.

[0041] By adopting the above technical solution, the present invention has the following beneficial effects:

[0042] 1. The FP-Growth algorithm is highly efficient and can complete the mining of frequent itemsets by scanning the data set twice. It also reduces memory usage and improves search speed by building a compact FP-tree.

[0043] 2. The FP-Growth algorithm does not need to generate a large number of candidate sets, but directly uses the structural information of the FP-tree to mine frequent item sets, which reduces unnecessary calculations and improves the execution efficiency of the algorithm.

[0044] 3. The FP-Growth algorithm is easy to parallelize. The data set can be divided into multiple subsets, each of which can independently build an FP-tree and mine frequent item sets.

[0045] 4. The FP-Growth algorithm can adapt to stream data processing scenarios. It can dynamically update the FP-tree without rebuilding the entire tree. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 A flow chart of the log association analysis method provided by the present invention;

[0048] Figure 2 A flowchart for constructing an FP-tree provided by the present invention;

[0049] Figure 3 A flowchart of mining frequent item sets provided by the present invention;

[0050] Figure 4 A schematic diagram of the FP-tree construction process provided by the present invention;

[0051] Figure 5 A schematic diagram of the node chain provided by the present invention;

[0052] Figure 6 A schematic diagram of the FP-tree mining frequent item set process provided by the present invention;

[0053] Figure 7 A schematic diagram of a log association analysis device provided by the present invention;

[0054] Figure 8 A schematic diagram of a computer-readable storage medium according to the present invention;

[0055] Fig. 9 This is a schematic diagram of the time axis of the present invention. DETAILED DESCRIPTION

[0056] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] Combination Figure 1 As shown, the present invention provides a log association analysis method based on the FP-Growth algorithm, the method comprising:

[0058] S1: Get raw logs and generate events;

[0059] S2: Based on the given partitioning criteria, divide the log events and construct the transaction data set;

[0060] S3: For transaction data sets, frequent event patterns are obtained using the FP-Growth algorithm, and frequent itemsets are mined using the FP-Growth algorithm;

[0061] S4: Generate strong correlation event rules.

[0062] Specifically, see Figure 1 , Figure 2 The obtaining of original logs and generating events as shown is to obtain the logs of a single device, collect the original logs generated by the required analysis device by Syslog forwarding or client reporting, parse the original logs by rule-based, template-based or machine learning parsing methods, and generate events.

[0063] The specific parsing example is as follows:

[0064] The original log sample collected from the switch is " <190> Apr 16 2024 02:05:25c1-FW%%01HTTPD / 6 / PASS(l)

[24279] :User lily(IP:10.100.1.80ID:246) login succeeded".

[0065] Use the regular expression matching method to parse. The given regular expression samples are shown in the following table:

[0066] Table 1

[0067] Event Name Regular Expressions User login / User.*login succeeded /

[0068] After the original log sample is successfully matched with the regular expression sample, a "user login" event can be generated.

[0069] After the original log is parsed using the log parsing method, multiple events may be generated within a period of time, which are defined as E1, E2, E3, ...En.

[0070] Specifically, the division standard described in S2 is a fixed time window, which means that log events are divided sequentially based on the occurrence time of the original log.

[0071] The specific parsing example is as follows:

[0072] The division criteria are fixed time windows, such as Fig. 9 As shown in the figure, Δt is a fixed value, and each Δt represents a time window. i (i=0,1,2,3,...n) represents different time periods.

[0073] Assume that for time t 0 To time t 9The event division of the event sequence is as follows:

[0074] Table 2

[0075] Time Window Sequence of events <![CDATA[t 0 -t 1 ]]> E2,E1,E4 <![CDATA[t 1 -t 2 ]]> E2,E4,E3 <![CDATA[t 2 -t 3 ]]> E2,E1,E3 <![CDATA[t 3 -t 4 ]]> E2,E1,E4,E6 <![CDATA[t 4 -t 5 ]]> E1,E3 <![CDATA[t 5 -t 6 ]]> E2,E3,E7 <![CDATA[t 6 -t 7 ]]> E2,E1,E3 <![CDATA[t 7 -t 8 ]]> E2,E1,E3,E5

[0076] In the above table, each row represents the event sequence under the corresponding time window, where the time window is the event division standard. 0 -t 1 Indicates t 0 Time to t 1 The time window of time. In the time window, the corresponding event sequence is E2, E1, E4, which means t 0 Time to t 1 At this moment, three events are generated, first event E2, then event E1, and finally event E4.

[0077] Specifically, the transaction data set described in S2 is generated based on the log event division result.

[0078] The specific parsing example is as follows:

[0079] Based on the event segmentation result, a transaction data set is generated.

[0080] Based on the event sequence divided in step S201, after sorting, the transaction data set W is generated as follows:

[0081] Table 3

[0082] Transaction ID item 1 {E2,E1,E4} 2 {E2,E4,E3} 3 {E2,E1,E3} 4 {E2,E1,E4,E6} 5 {E1,E3} 6 {E2,E3,E7} 7 {E2,E1,E3} 8 {E2,E1,E3,E5}

[0083] Specifically, the FP-Growth algorithm described in S3 is used to mine frequent itemsets, including:

[0084] Constructing an FP-tree based on the acquired transaction data set;

[0085] Mining frequent item sets from FP-tree is to obtain frequent patterns of events.

[0086] Specifically, the construction of FP-tree includes:

[0087] S71: Scan the transaction data set, and generate a set of frequent 1-item sets according to the preset minimum support: Scan the transaction data set, calculate the support of each event; set the minimum support, and items with a support less than the minimum support are non-frequent 1-item sets, delete non-frequent 1-item sets according to the minimum support, and the remaining events with support greater than the minimum support are frequent 1-item sets; arrange the frequent 1-item sets in descending order according to their support, and generate a set of frequent 1-item sets;

[0088] The support refers to the number of transactions containing a specific item set, which is a measure of the importance of the association rule. That is, the greater the support of the association rule, the more important the association rule is.

[0089] First, scan the transaction data set and calculate the support of each event, that is, the number of times each event occurs, to generate the event count table as follows:

[0090] Table 4

[0091] E1 E2 E3 E4 E5 E6 E7 6 7 6 3 1 1 1

[0092] Then, the minimum support is set, and items with a support less than the minimum support are non-frequent 1-itemsets, and non-frequent 1-itemsets are deleted according to the minimum support. Assuming the minimum support is 2, in Table 4, the minimum support of events E5, E6, and E7 is less than 2, so they are deleted. The remaining events have a support greater than the minimum support and are frequent 1-itemsets.

[0093] Finally, the frequent 1-itemset is sorted in descending order according to its support, generating a set of frequent 1-itemset, which is defined as F1 = {{E2:7},{E1:6},{E3:6},{E4:3}}.

[0094] S72: sorting the transaction data set according to the frequent 1-item set set: for the items of the transaction data set, comparing the frequent 1-item set set, deleting the events that are not in the frequent 1-item set set; sorting the items of the transaction data set according to the events in the frequent 1-item set set; generating a sorted transaction data set;

[0095] Specific examples:

[0096] For the transaction data set, infrequent 1-item sets are deleted, and items of the transaction data set are sorted according to events in a frequent 1-item set set.

[0097] First, for the items in the transaction data set, compare the frequent 1-item set set and delete the events that are not in the frequent 1-item set set. For example, when transaction ID = 8, the corresponding items are {E2, E1, E3, E5}, where That is, event E5 is not in the frequent 1-item set, so E5 is deleted and the remaining events are retained, that is, the items corresponding to transaction ID = 8 are {E2, E1, E3}.

[0098] Then, for the transaction data set, the items are sorted according to the events in the frequent 1-item set. For example, when transaction ID = 2, the corresponding item is {E2, E4, E3}, and the order of the events in the frequent 1-item set is E2, E3, E4, then the item is adjusted to {E2, E3, E4}.

[0099] The final generated sorted transaction data set Ws is as follows:

[0100] Table 5

[0101] Transaction ID item 1 {E2,E1,E4} 2 {E2,E3,E4} 3 {E2,E1,E3} 4 {E2,E1,E4} 5 {E1,E3} 6 {E2,E3} 7 {E2,E1,E3} 8 {E2,E1,E3}

[0102] S73: constructing an FP-tree, that is, constructing an FP-tree according to the generated sorted transaction data set;

[0103] Specific examples:

[0104] Taking the transaction data set in Table 5 as an example, the construction process of FP-tree is explained as follows:

[0105] (1) Create the root node of the tree, represented by "null";

[0106] (2) Scan the first transaction, i.e., transaction ID = 1, and build a tree in the order of items, i.e., build a tree section by section in the order of E2-E1-E4, and mark the support of the event on the node. The FP-tree constructed in this step is shown in Figure 4 Middle T1;

[0107] (3) Scan the second transaction, i.e., transaction ID = 2, and build the tree in the order of the items, i.e., build the tree section by section in the order of E2-E3-E4, and mark the support of the event on the node. The nodes that are repeated in (2) do not need to be rebuilt, and only the support of the node needs to be increased by 1. For example, node E2 has been created in the construction process in (2), so in this step, only the support of the node needs to be increased by 1. The FP-tree constructed after this step is shown in Figure 4 Middle T2;

[0108] (4) Scan the third transaction, i.e., transaction ID = 3, and build the tree section by section in the order of E2-E1-E3. The FP-tree constructed in this step is shown in Figure 4 Middle T3;

[0109] (5) Scan the fourth transaction, i.e., transaction ID = 4, and construct the tree section by section in the order of E2-E1-E4. The FP-tree constructed in this step is shown in Figure 4 Middle T4;

[0110] (6) Scan the fifth transaction, i.e., transaction ID = 5. The first event in this item is E1, and the first event in the previous items is E2. Therefore, we need to start another branch and build the tree section by section in the order of E1-E3, and mark the support of the event on the node. The FP-tree constructed after this step is shown in Figure 4 Middle T5;

[0111] (7) Scan the sixth transaction, i.e., transaction ID = 6, and construct the tree section by section according to the order of E2-E3. The FP-tree constructed in this step is shown in Figure 4 Middle T6;

[0112] (8) Scan the seventh transaction, i.e., transaction ID = 7, and construct the tree section by section according to the order of E2-E1-E3. The FP-tree constructed in this step is shown in Figure 4 Middle T7;

[0113] (9) Scan the eighth transaction, i.e., transaction ID = 8. Same as (8), build the tree section by section in the order of E2-E1-E3. The FP-tree constructed in this step is shown in Figure 4 Medium T8.

[0114] After the above steps, the complete FP-tree generated is T8.

[0115] S74: Generate a node linked list, that is, connect the corresponding nodes according to the correspondence between the frequent 1-item set set and the FP-tree, so as to facilitate tracing in the subsequent mining process.

[0116] Based on the frequent 1-item set F1 and FP-tree T8, the generated node chain representation example is shown in Figure 5 .

[0117] Specifically, the mining of frequent itemsets includes:

[0118] S81: Generate a conditional pattern base for each node of the FP-tree, where the conditional pattern base is a set of paths ending with the element item to be searched, and each path is a prefix path;

[0119] S82: constructing a corresponding conditional FP-tree according to the conditional pattern base, specifically, taking the conditional pattern base as a sub-transaction data set, and the FP-tree generated according to the sub-transaction data set is the conditional FP-tree;

[0120] S83: recursively construct a conditional FP-tree and simultaneously increase the frequent item sets contained therein. Recursive construction means repeatedly executing S81 and S82 for the conditional FP-tree until the conditional pattern base of the search node is "null".

[0121] S84: If the conditional FP-tree has only one path, the frequent item sets contained therein are directly generated; if the FP-tree constructed in step S82 has only one path and no redundant branches, the frequent item sets are directly generated.

[0122] Taking T8 as an example, the above mining process is explained in detail. Figure 6Schematic diagram of the mining process for each node.

[0123] like Figure 6 In P1, consider the node E3 in T8, whose conditional pattern base is {(E2 E1:3), (E2:2), (E1:1)}, that is, the node E3 has three prefix paths: (E2 E1:3), (E2:2) and (E1:1). The conditional pattern base of E3 is used as the sub-transaction data set to construct the FP-tree, see the conditional FP-tree of E3 for details. Since the FP-tree of E3 is not a single path, recursive mining is required.

[0124] First, consider the node E1 in the E3 conditional FP-tree, connect it with E3, and get the frequent pattern {E1 E3:4}. Then generate the conditional FP-tree of {E1 E3}, see the conditional FP-tree of {E1 E3}. Consider the node E2 in the {E1 E3} conditional FP-tree, connect it with the generated frequent pattern {E1 E3:4}, and get the frequent pattern {E2 E1 E3:4}. Then generate the conditional FP-tree of {E2 E1 E3}, and its conditional FP-tree is empty.

[0125] Then, consider the node E2 in the E3 conditional FP-tree, connect it with E3, and get the frequent pattern {E2 E3:4}. Then generate the conditional FP-tree of {E2 E3}, and its conditional FP-tree is empty.

[0126] In summary, the frequent itemsets mined from node E3 include: {E1 E3:4}, {E2 E3:4}, {E2 E1E3:4}.

[0127] like Figure 6 In P2, consider node E4 in T8, whose conditional pattern base is {(E2 E1:2), (E2E3:1)}, that is, node E4 has two prefix paths, (E2 E1:2) and (E2 E3:1). Take the conditional pattern base of E4 as the sub-transaction data set. According to the set minimum support 2, the support of E3 is less than 2, so the event is deleted and the FP-tree is constructed. For details, see the conditional FP-tree of E4. The FP-tree of E4 has only one path and no extra branches, so the frequent item sets are generated directly. The frequent item sets generated by this path include: {E2 E4:3}, {E1 E4:2}, {E2 E1 E4:2}.

[0128] like Figure 6In P3, consider the node E1 in T8, whose conditional pattern base is {(E2:5)}, that is, the node E4 has only a prefix path (E2:5). The conditional pattern base of E1 is used as the sub-transaction data set to construct the FP-tree, see the conditional FP-tree of E1 for details. The FP-tree of E1 has only one path, so a frequent item set is directly generated. The frequent item set generated by this path is {E2 E1:5}.

[0129] Considering the node E2 in T8, it can be seen that the conditional pattern base of the node E2 is empty, so there is no need to continue mining.

[0130] According to the FP-Growth algorithm, the frequent item sets with support greater than 2 are finally obtained as follows:

[0131] Table 6

[0132] The association rules corresponding to the frequent itemsets are as follows:

[0133] Table 7

[0134] Frequent Itemsets Association Rules {E1 E3:4} E1→E3 {E2 E3:4} E2 → E3 {E2 E1 E3:4} {E2,E1}→E3 {E2 E4:3} E2 → E4 {E1 E4:2} E1→E4 {E2 E1 E4:2} {E2,E1}→E4 {E2 E1:5} E2 → E1

[0135] According to the definition of support, it can be seen that among the above association rules, E2→E1 has the largest support, which is 5, indicating that this association rule is the most important.

[0136] Specifically, the support refers to the number of transactions containing a specific item set, which is a measure of the importance of the association rule. That is, the greater the support of the association rule, the more important the association rule is.

[0137] Specifically, the generating of the strong correlation event rule is to select and discard the generated frequent item sets based on the preset minimum confidence, and then generate the strong correlation rule;

[0138] The confidence level is defined as follows:

[0139]

[0140] Among them, c(X→Y) represents the confidence of the association rule X→Y, s(X,Y) represents the support of the item set {X,Y}, and s(X) represents the support of the item set {X};

[0141] Confidence is a measure of the reliability of association rules. For a given rule X→Y, the higher the confidence, the greater the possibility that Y is included in the transaction of X.

[0142] A rule that satisfies both the minimum support and the minimum confidence is called a strong association rule.

[0143] Taking the frequent item set mined in step S3 as an example, the generation process of strong association rules is explained in detail: the association rules generated in step S3 have satisfied the minimum support, that is, the support of the generated rules is greater than the minimum support. Next, the confidence of the association rules is calculated. Taking the association rule E1→E3 as an example, the calculation process is as follows:

[0144]

[0145] Among them, s(E1,E3) represents the support of {E1,E3}, that is, the number of times the item set {E1,E3} appears in the transaction data set. It can be seen that the item set appears in the items with transaction IDs 3, 5, 7, and 8, so the support of the item set is 4. Similarly, it can be concluded that the support of s(E1) is 6, so the confidence calculation result is 66.7%.

[0146] Similarly, the confidence of the remaining association rules is calculated, and the results are shown in the following table:

[0147] Table 8

[0148] Association Rules Confidence level (keep 1 decimal place) E1→E3 66.7％ E2 → E3 71.4％ {E2,E1}→E3 60.0％ E2 → E4 14.3％ E1→E4 33.3％ {E2,E1}→E4 40.0％ E2 → E1 71.4％

[0149] Assuming that the minimum confidence is 60%, the generated strong association rules are E1→E3, E2→E3, {E2,E1}→E3, E2→E1.

[0150] In the confidence calculation, if the rule X→Y does not satisfy the minimum confidence, then the rule of the form X'→Y must also not satisfy the minimum confidence, where X' is a subset of X.

[0151] In actual programs, when calculating confidence, in order to save resources, the frequent item sets corresponding to the calculated association rules can be calculated in descending order. If multiple association rules do not meet the minimum confidence, then their subsets do not meet the confidence, and there is no need to calculate the subset confidence. For example, the confidence of the association rule {E2, E1}→E4 is 40%, which is less than the preset minimum confidence. Therefore, the confidence of the association rules E2→E4 and E1→E4 must not meet the minimum confidence, so there is no need to calculate and the rule is skipped directly.

[0152] The present invention provides a log association analysis device based on the FP-Growth algorithm, see Figure 7 , the analyzing device comprises:

[0153] Data collection and event generation module, used to obtain raw logs and generate events;

[0154] The event partitioning module is used to partition log events based on given partitioning criteria and construct transaction data sets;

[0155] Data mining module, used to mine frequent itemsets using FP-Growth algorithm for frequent event patterns in transaction data sets;

[0156] The event correlation module is used to generate strongly correlated event rules.

[0157] The data collection and event generation module obtains the logs of a single device, collects the original logs generated by the required analysis device by Syslog forwarding or client reporting, parses the original logs using rule-based, template-based or machine learning parsing methods, and generates events.

[0158] The present invention also provides a computer readable storage medium, see Figure 8 The computer-readable storage medium stores computer program instructions. When the computer program instructions are executed by a computer, the computer performs any of the steps of the log association analysis method based on the FP-Growth algorithm as described above.

[0159] It should be noted that the storage medium shown in the embodiment of the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any storage medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the storage medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0160] The accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present invention. Each box in the figure can represent a module, program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that each box in the block diagram or flow chart, and the combination of boxes in the block diagram or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0161] The units involved in the embodiments of the present invention may be implemented by software or hardware, and the units described may also be arranged in a processor. The names of these units do not, in some cases, limit the units themselves.

[0162] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A log association analysis method based on FP-Growth algorithm, characterized in that: The method comprises: S1: Get raw logs and generate events; S2: Based on the given partitioning criteria, divide the log events and construct the transaction data set; S3: For transaction data sets, frequent event patterns are obtained using the FP-Growth algorithm, and frequent itemsets are mined using the FP-Growth algorithm; S4: Generate strong correlation event rules.

2. The log association analysis method based on the FP-Growth algorithm according to claim 1, characterized in that: The acquisition of original logs and generation of events described in S1 is to acquire the logs of a single device, collect the original logs generated by the required analysis device by Syslog forwarding or client reporting, parse the original logs by rule-based, template-based or machine learning parsing methods, and generate events.

3. The log association analysis method based on the FP-Growth algorithm according to claim 1, characterized in that: The division standard described in S2 is a fixed time window, which means that log events are divided sequentially based on the occurrence time of the original log.

4. The log association analysis method based on the FP-Growth algorithm according to claim 1, characterized in that: The transaction data set described in S2 is generated based on the log event division result.

5. The log association analysis method based on the FP-Growth algorithm according to claim 4, characterized in that: The FP-Growth algorithm described in S3 is used to mine frequent itemsets, including: Constructing an FP-tree based on the acquired transaction data set; Mining frequent item sets from FP-tree is to obtain frequent patterns of events.

6. The log association analysis method based on the FP-Growth algorithm according to claim 5, characterized in that: The construction of FP-tree includes: S71: Scan the transaction data set, and generate a set of frequent 1-item sets according to the preset minimum support: Scan the transaction data set, calculate the support of each event; set the minimum support, and items with a support less than the minimum support are non-frequent 1-item sets, delete non-frequent 1-item sets according to the minimum support, and the remaining events with support greater than the minimum support are frequent 1-item sets; arrange the frequent 1-item sets in descending order according to their support, and generate a set of frequent 1-item sets; S72: sorting the transaction data set according to the frequent 1-item set set: for the items of the transaction data set, comparing the frequent 1-item set set, deleting the events that are not in the frequent 1-item set set; sorting the items of the transaction data set according to the events in the frequent 1-item set set; generating a sorted transaction data set; S73: constructing an FP-tree, that is, constructing an FP-tree according to the generated sorted transaction data set; S74: Generate a node linked list, that is, connect the corresponding nodes according to the correspondence between the frequent 1-item set set and the FP-tree, so as to facilitate tracing in the subsequent mining process.

7. The log association analysis method based on the FP-Growth algorithm according to claim 5, characterized in that: The mining of frequent itemsets includes: S81: Generate a conditional pattern base for each node of the FP-tree, where the conditional pattern base is a set of paths ending with the element item to be searched, and each path is a prefix path; S82: constructing a corresponding conditional FP-tree according to the conditional pattern base, specifically, taking the conditional pattern base as a sub-transaction data set, and the FP-tree generated according to the sub-transaction data set is the conditional FP-tree; S83: recursively construct a conditional FP-tree and simultaneously increase the frequent item sets contained therein. Recursive construction means repeatedly executing S81 and S82 for the conditional FP-tree until the conditional pattern base of the search node is "null". S84: If the conditional FP-tree has only one path, the frequent item sets contained therein are directly generated; if the FP-tree constructed in step S82 has only one path and no redundant branches, the frequent item sets are directly generated.

8. The log association analysis method based on the FP-Growth algorithm according to claim 5, characterized in that: The support refers to the number of transactions containing a specific item set, and is a measure of the importance of the association rule. That is, the greater the support of the association rule, the more important the association rule is.

9. The log association analysis method based on the FP-Growth algorithm according to claim 8, characterized in that: The generating of the strong correlation event rule is to select and discard the generated frequent item sets based on the preset minimum confidence, and then generate the strong correlation rule; The confidence level is defined as follows: Among them, c(X→Y) represents the confidence of the association rule X→Y, s(X,Y) represents the support of the item set {X,Y}, and s(X) represents the support of the item set {X}; Confidence is a measure of the reliability of association rules. For a given rule X→Y, the higher the confidence, the greater the possibility that Y is included in the transaction of X. A rule that satisfies both the minimum support and the minimum confidence is called a strong association rule.

10. A log association analysis device based on FP-Growth algorithm, characterized in that: The analysis device comprises: Data collection and event generation module, used to obtain raw logs and generate events; The event partitioning module is used to partition log events based on given partitioning criteria and construct transaction data sets; Data mining module, used to mine frequent itemsets using FP-Growth algorithm for frequent event patterns in transaction data sets; The event correlation module is used to generate strongly correlated event rules.

11. A computer-readable storage medium storing computer program instructions, wherein when the computer program instructions are executed by a computer, the computer executes the steps of the log association analysis method based on the FP-Growth algorithm according to any one of claims 1 to 9.

Images