Log detection method, device and storage medium

Abstract

The present invention provides a log detection method, device, and storage medium. Based on multiple current log data, multiple current coded information groups are obtained; a first matrix is ​​constructed based on multiple log information included in the multiple current coded information groups; multiple current log information groups corresponding to the multiple current log data are obtained according to the number of times the multiple log information appear in the first matrix; the multiple current log information groups are matched with multiple pre-stored log information groups in a pattern library to obtain matching results of the multiple current log information groups, and detection results for the multiple current log data are determined based on the matching results. Since the path relationship between the log information of each current coded information group in the first matrix is ​​clearer, it is convenient for a server to quickly construct a frequent pattern tree to mine the multiple current log information groups, and then the detection results of the multiple current log data can be quickly determined, thereby improving the efficiency of the server in detecting log data.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of data processing technology, and in particular to a log detection method, device, and storage medium. Background Art

[0002] Logs are key information that records server status. By examining logs, you can explore server operating conditions and identify potential security issues in a timely manner.

[0003] In the prior art, when detecting logs, it is necessary to detect the behavioral characteristics of the server within a period of time. There are two commonly used algorithms for detecting logs, namely: the Apriori algorithm and the FP-growth algorithm. Among them, the Apriori algorithm needs to scan the database multiple times before generating a complete set of frequent patterns, and at the same time generate a large number of candidate frequent sets. This makes the time and space complexity of the Apriori algorithm larger, resulting in low efficiency of the Apriori algorithm in detecting logs. The FP-growth algorithm needs to scan the transaction database twice. When the database capacity is large, the scanning time is longer, making the efficiency of detecting logs lower. Therefore, the prior art has the problem of low efficiency in detecting logs. Summary of the Invention

[0004] The embodiments of the present invention provide a log detection method, device, and storage medium, which can improve the efficiency of log detection.

[0005] The technical solution of the present invention is achieved as follows:

[0006] An embodiment of the present invention provides a log detection method, including:

[0007] Acquire multiple current log data, and obtain multiple current coded information groups based on the multiple current log data; the multiple current coded information groups include: multiple log information in the multiple current log data; the log information represents the event information in the corresponding log data;

[0008] Constructing a first matrix based on a plurality of log information included in a plurality of current encoding information groups;

[0009] According to the number of times the multiple log information appears in the first matrix, multiple current log information groups corresponding to the multiple current log data are obtained; the multiple current log information groups are overlapping log information groups in the frequent pattern tree; the frequent pattern tree is constructed according to the number of times the multiple log information appears;

[0010] The multiple current log information groups are matched with the multiple pre-stored log information groups in the pattern library to obtain matching results of the multiple current log information groups, and the detection results for the multiple current log data are determined based on the matching results.

[0011] An embodiment of the present invention further provides a log detection device, comprising:

[0012] A data acquisition unit is configured to acquire a plurality of current log data and obtain a plurality of current coded information groups based on the plurality of current log data; the plurality of current coded information groups include: a plurality of log information in the plurality of current log data; the log information represents event information in the corresponding log data;

[0013] a matrix construction unit, configured to construct a first matrix based on the plurality of log information included in the plurality of current encoding information groups;

[0014] an association extraction unit, configured to obtain, based on the number of occurrences of the plurality of log information in the first matrix, a plurality of current log information groups corresponding to the plurality of current log data; the plurality of current log information groups being overlapping log information groups in a frequent pattern tree; the frequent pattern tree being constructed based on the number of occurrences of the plurality of log information;

[0015] The matching unit is used to match the multiple current log information groups with multiple pre-stored log information groups in the pattern library to obtain matching results of the multiple current log information groups, and determine detection results for the multiple current log data based on the matching results.

[0016] An embodiment of the present invention further provides a log detection device, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor executes the steps in the above method.

[0017] An embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps in the above method are implemented.

[0018] The embodiment of the present invention obtains multiple current log data based on the multiple current log data, wherein the multiple current coded information groups include: multiple log information in the multiple current log data; the log information represents the event information in the corresponding log data; based on the multiple log information included in the multiple current coded information groups, a first matrix is ​​constructed; according to the number of times the multiple log information appears in the first matrix, multiple current log information groups corresponding to the multiple current log data are obtained; the multiple current log information groups are overlapping log information groups in a frequent pattern tree; the frequent pattern tree is constructed according to the number of times the multiple log information appears; the multiple current log information groups are matched with multiple pre-stored log information groups in a pattern library to obtain matching results of the multiple current log information groups, and the detection results for the multiple current log data are determined based on the matching results. Since the server stores each current coded information group through the first matrix, the path relationship between the log information of each current coded information group in the first matrix is ​​clearer, which facilitates the server to quickly construct a frequent pattern tree to mine the multiple current log information groups, and then can quickly determine the detection results of the multiple current log data, thereby improving the efficiency of the server in detecting log data. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 An optional flow chart of a log detection method provided in an embodiment of the present invention;

[0020] Figure 2 An optional flow chart of a log detection method provided in an embodiment of the present invention;

[0021] Figure 3 An optional flow chart of a log detection method provided in an embodiment of the present invention;

[0022] Figure 4 A schematic diagram of an optional effect of the log detection method provided by an embodiment of the present invention;

[0023] Figure 5 A schematic diagram of an optional effect of the log detection method provided by an embodiment of the present invention;

[0024] Figure 6 An optional flow chart of a log detection method provided in an embodiment of the present invention;

[0025] Figure 7 A schematic diagram of the structure of a log detection device provided by an embodiment of the present invention;

[0026] Figure 8 A schematic diagram of a hardware entity of a log detection device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0027] Figure 1 An optional flow chart of the log detection method provided by the embodiment of the present invention is combined with Figure 1 The steps shown are explained.

[0028] S101. Acquire multiple current log data, and obtain multiple current coded information groups based on the multiple current log data; the multiple current coded information groups include: multiple log information in the multiple current log data; the log information represents the event information in the corresponding log data.

[0029] In an embodiment of the present invention, a server retrieves multiple current log data from a cache or database. Based on the multiple current log data, the server obtains multiple current coded information groups. The multiple current coded information groups include multiple log information from the multiple current log data. The log information represents event information from the corresponding log data.

[0030] In an embodiment of the present invention, a server can obtain multiple current log data within a predetermined time period before the current moment in the server's cache or database through a log acquisition tool. The server deletes duplicate log data from the multiple current log data. The server extracts the multiple current log data after deleting the duplicate log data, and the server determines the encoding information of multiple preset log information for each current log data according to a predetermined rule, and then can determine the current encoding information group corresponding to each current log data, thereby obtaining multiple current encoding information groups. The encoding information can be digital information or character information.

[0031] In an embodiment of the present invention, the multiple current log data may be log data of interactions between multiple clients and a server within a preset time period. The log information in each current log data reflects information such as the interaction time between the corresponding client and the server, the client Internet Protocol (IP) address information, the network target IP address information, and the Hypertext Transfer Protocol (HTTP) request method information.

[0032] In an embodiment of the present invention, the server may obtain multiple current log data in a cache or database of the server through a log collection system, a logstash log collection tool, or a syslog tool. The multiple current log data may be in text format.

[0033] For example, in combination with Table 1, Table 1 is a log information table corresponding to multiple current log data

[0034]

[0035]

[0036] Table 1

[0037] Among them, the user name represents the name information of the client, the time represents the interaction time information between the client and the server, the host IP represents the IP address information of the client, the application name represents the network target IP address information of the client, and the operation represents the HTTP request method information of the client.

[0038] In an embodiment of the present invention, the server can set different codes for different clients, different times, different host IP addresses, different application names, and different operations. The server determines the coding information corresponding to each log message in Table 1, and then obtains the current coding information group corresponding to each current log data, thereby obtaining multiple current coding information groups.

[0039] For example, in combination with Table 2, Table 2 is a table of details of multiple coded information groups.

[0040]

[0041]

[0042] Table 2

[0043] The transaction database represents the current encoding information group of the current log data corresponding to different clients. For example, in the current encoding information group "U1 T10 P3 E1 O1", U1 corresponds to Charles, T10 corresponds to 09:10:18, P3 corresponds to 192.168.0.33, E1 corresponds to regedit.exe, and O1 corresponds to PATH.REG_EXPAND_SZ. The current encoding information group "U1 T10P3 E1 O1" indicates that user Charles ran regedit.exe on the host with IP address 192.168.0.33 between 9:00 and 10:00 and performed operations on the PATH.REG_EXPAND_SZ file.

[0044] S102: Construct a first matrix based on multiple log information included in multiple current encoding information groups.

[0045] In the embodiment of the present invention, the server may construct a first matrix based on multiple log information and log information categories included in multiple current encoded information groups.

[0046] In the embodiment of the present invention, the server may construct a first matrix with the log information category as the first column and the encoding information of each log information in each current encoding information group and the tag value of the corresponding log information category as subsequent columns.

[0047] The tag value may be a predetermined value for the encoding information of each log message and the corresponding log message category. The tag value for each log message and the corresponding log message category may be a unified value of 1 or other predetermined values.

[0048] In the embodiment of the present invention, since the server stores the log information corresponding to each current coded information group in the form of a two-dimensional matrix and groups the log information corresponding to each current coded information group, the server can efficiently determine the parent-child path relationship between the log information, so the server can efficiently construct a frequent pattern tree, which facilitates the server to quickly determine multiple current log information groups of multiple current log data.

[0049] S103. Obtain multiple current log information groups corresponding to multiple current log data according to the number of times the multiple log information appear in the first matrix; the multiple current log information groups are overlapping log information groups in the frequent pattern tree; the frequent pattern tree is constructed according to the number of times the multiple log information appear.

[0050] In an embodiment of the present invention, the server obtains multiple current log information groups corresponding to multiple current log data based on the number of times the multiple log information appear in the first matrix. The multiple current log information groups are overlapping log information groups in a frequent pattern tree. The frequent pattern tree is constructed based on the number of times the multiple log information appear.

[0051] In the embodiment of the present invention, since the relationship between the log information groups corresponding to each current coded information group in the first matrix is ​​determined, the server can quickly determine the relationship between the log information groups of each current coded information group in the first matrix, and then construct a frequent pattern tree. The server extracts the frequent patterns of multiple current coded information groups from the frequent pattern tree as multiple current log information groups through an association extraction algorithm.

[0052] The log information group is a combination of at least one adjacent log information of the corresponding current coding information group in the first matrix.

[0053] In the embodiment of the present invention, the server may extract multiple current log information groups from the frequent pattern tree using the FP-growth algorithm.

[0054] In an embodiment of the present invention, the server can calculate the support of the log information category by the tag value of a row corresponding to each log information category in the first matrix. The server then selects a target log information category that is greater than a support threshold. The server sorts the support corresponding to the target log information category in descending order and establishes a second matrix again. The server extracts the first log information group corresponding to each current encoded information group in the second matrix. The server constructs a frequent pattern tree based on the parent-child path relationship between each first log information group. The server extracts multiple current log information groups in the frequent pattern tree through an association extraction algorithm.

[0055] S104 : Match the multiple current log information groups with the multiple pre-stored log information groups in the pattern library to obtain matching results of the multiple current log information groups, and determine detection results for the multiple current log data based on the matching results.

[0056] In an embodiment of the present invention, the server matches multiple current log information groups with multiple pre-stored log information groups in a pattern library to obtain matching results of the multiple current log information groups, and determines detection results for the multiple current log data based on the matching results.

[0057] In an embodiment of the present invention, before the server detects multiple current log data, the server needs to build a pattern library. The server obtains multiple sample log data before the current moment. The server builds a sample matrix of multiple sample log data and sample log information categories, and associates and extracts multiple pre-stored log information groups based on the number of times the corresponding log information appears in the multiple sample log data in the sample matrix. The server forms a pattern library based on multiple pre-stored log information groups. The server then matches each current log information group in the multiple current log information groups with the multiple pre-stored log information groups one by one to obtain a matching result for each current log information group. If they can match, it means that the current log data corresponding to the corresponding current log information group is normal. If they cannot match, it means that the current log data corresponding to the corresponding current log information group is abnormal.

[0058] Among them, multiple pre-stored log information groups in the pattern library are obtained by obtaining multiple sample log data before the current moment, constructing a sample matrix of multiple sample log data and sample log information categories, and performing association extraction in the sample matrix according to the number of occurrences of the corresponding log information in the multiple sample log data.

[0059] The embodiment of the present invention obtains multiple current log data based on the multiple current log data, wherein the multiple current coded information groups include: multiple log information in the multiple current log data; the log information represents the event information in the corresponding log data; based on the multiple log information included in the multiple current coded information groups, a first matrix is ​​constructed; according to the number of times the multiple log information appears in the first matrix, multiple current log information groups corresponding to the multiple current log data are obtained; the multiple current log information groups are overlapping log information groups in a frequent pattern tree; the frequent pattern tree is constructed according to the number of times the multiple log information appears; the multiple current log information groups are matched with multiple pre-stored log information groups in a pattern library to obtain matching results of the multiple current log information groups, and the detection results for the multiple current log data are determined based on the matching results. Since the server stores each current coded information group through the first matrix, the path relationship between the log information of each current coded information group in the first matrix is ​​clearer, which facilitates the server to quickly construct a frequent pattern tree to mine the multiple current log information groups, and then can quickly determine the detection results of the multiple current log data, thereby improving the efficiency of the server in detecting log data.

[0060] In some embodiments, Figure 1 S105 is also included after S104 shown, and will be explained in combination with each step.

[0061] S105. If the matching result indicates that at least one current log information group matches at least one pre-stored log information group, the current log data corresponding to the at least one current log information group is added to the multiple sample log data extracted when determining the pattern library to obtain multiple updated pre-stored log information groups and a pattern library updated based on the updated multiple pre-stored log information groups.

[0062] In an embodiment of the present invention, if the server detects that at least one current log information group corresponds to at least one pre-stored log information group, the current log data corresponding to the at least one current log information group is added to multiple sample log data to form a new sample set. The server processes the multiple sample log data in the sample set again using the processing method for the multiple sample log data in S104 to obtain multiple pre-stored log information groups, forming a new pattern library. At least one current log information group belongs to multiple current log information groups; at least one pre-stored log information group belongs to multiple pre-stored log information groups.

[0063] In an embodiment of the present invention, if the server matches a current log information group among multiple current log information groups and finds that the current log information group does not match any pre-stored log information group, the server will issue an alarm to inform the staff that the current log data corresponding to the current log information group is abnormal.

[0064] In the embodiment of the present invention, after each time the server detects multiple current log data, it adds the normal current log data to the multiple sample log data and performs the process of obtaining multiple pre-stored log information groups again to update the pattern library.

[0065] Among them, multiple sample log data are log data extracted when calculating the pattern library.

[0066] In an embodiment of the present invention, since the server adds the current log data corresponding to the current log information group that can match the pre-stored log information group to multiple sample log data to calculate a new pattern library, the data in the pattern library is more complete, allowing the server to efficiently detect log data while improving the accuracy of detection.

[0067] In some embodiments, Figure 1 S102 - S103 shown can be implemented through S106 to S112 , which will be described in conjunction with each step.

[0068] S106: Form a first column of a first matrix based on multiple log information categories in multiple current encoding information groups.

[0069] In the embodiment of the present invention, the server forms a first column of a first matrix using multiple log information categories in multiple current encoded information groups.

[0070] S107 : Form an nth column of a first matrix based on each log information in the (n−1)th current coded information group and a preset label value of the corresponding log information category.

[0071] In an embodiment of the present invention, the server forms the nth column of the first matrix using each log message in the n-1th current coded information group and a preset tag value of the corresponding log message category. n is a positive integer greater than 1. The number of columns in the first matrix is ​​n, where n is a positive integer greater than 1.

[0072] Exemplarily, the server may form the second column of the first matrix by combining each log information in the first currently encoded information group with the label value of the corresponding log information category.

[0073] In an embodiment of the present invention, the server constructs a first matrix with the log information category as the first column and the tag value of the log information in each current coded information group and the corresponding log information category as the subsequent columns. The first column of the first matrix is ​​the log information category, and the nth column of the first matrix is ​​the tag value of each log information in the n-1th current coded information group and the corresponding log information category. The tag value can be uniformly set to 1. n is a positive integer greater than 1 and less than the number of current coded information groups.

[0074] That is, the second column of the first matrix is ​​the label value of each log information in the first current coded information group and the corresponding log information category.

[0075] Exemplarily, the multiple current coded information groups may include T01 and T02. The log information categories may include four categories: A, B, C, and D. The tag value is set to 1. Furthermore, the server may construct a first matrix based on the log information in T01 and T02 and the tag values ​​of the corresponding log information categories.

[0076] Please refer to Table 3 for the schematic diagram of the structure of the first matrix.

[0077] Log information category T01 T02 A 1 0 B 1 1 C 0 0 D 1 1

[0078] S108 . Calculate the support of each log information category based on the number of occurrences of each log information in the multiple current coded information groups in the first matrix.

[0079] In the embodiment of the present invention, the server calculates the support degree of each log information category based on the number of occurrences of each log information in the multiple current coded information groups in the first matrix.

[0080] In an embodiment of the present invention, when the log information of each current log coding information in the first matrix appears once at the position corresponding to the log information category in the first matrix, the server will record the tag value at that position. The tag value corresponding to each log information is 1. The server can add the tag values ​​of a row corresponding to each log information category in the first matrix to obtain an intermediate value. The server divides the intermediate value corresponding to each log information category by the number of multiple current coding information groups to obtain the support degree corresponding to each log information category.

[0081] For example, referring to Table 3, the server calculates the support of log information category A by adding the tag values ​​of a row corresponding to log information category A to obtain an intermediate value of 1. The server divides the intermediate value 1 by the number of current coded information groups, 2, and obtains a support of log information category A of 0.5.

[0082] In some embodiments, see Figure 2 , Figure 2 An optional flow chart of the log detection method provided by an embodiment of the present invention is provided. Figure 2 S106 to S108 shown can be implemented through S201 to S202 , and will be described in conjunction with each step.

[0083] S201. Input the transaction database, minimum support, number of log information categories, and number of current coded information groups.

[0084] In an embodiment of the present invention, a server inputs a transaction database into an association extraction algorithm model. The transaction database includes multiple current coded information groups. Simultaneously, backend system personnel on the server can preset parameter information for the association extraction algorithm. The parameter information includes minimum support, the number of log information categories, and the number of current coded information groups. The server can construct a first matrix using the parameter information and the transaction database.

[0085] S202 : Scan the transaction database, store the current coded information group into the first matrix, and count the support of each log information category.

[0086] In an embodiment of the present invention, a server scans a transaction database. The server stores the log information category in the first column of a first matrix, and stores the log information in each current coded information group and the tag value of the corresponding log information category in subsequent columns of the first matrix. The server can add the tag values ​​of a row corresponding to each log information category in the first matrix to obtain an intermediate value. The server divides the intermediate value corresponding to each log information category by the number of multiple current coded information groups to obtain the support level corresponding to each log information category.

[0087] S109: Determine the target log information category corresponding to the target support greater than the support threshold.

[0088] In the embodiment of the present invention, the server determines a target log information category corresponding to a target support greater than a support threshold.

[0089] In this embodiment of the present invention, the support threshold for the association extraction algorithm is pre-set by the backend system staff. Therefore, after calculating the support corresponding to each log information category, the server determines a target support greater than the support threshold. The server then determines the target log information category corresponding to the target support.

[0090] In the embodiment of the present invention, the server may delete the log information categories corresponding to the support levels that are not greater than the support level threshold in the first matrix, thereby obtaining the target log information category.

[0091] In the embodiment of the present invention, the server may extract the log information category corresponding to the target support degree from the first matrix, and thus obtain the target log information category.

[0092] S110 , determining the first log information groups corresponding to the multiple current coded information groups in a second matrix constructed based on the target log information category.

[0093] In the embodiment of the present invention, the server determines the first log information groups corresponding to the multiple current coded information groups in the second matrix constructed based on the target log information category.

[0094] In an embodiment of the present invention, the server needs to construct a second matrix of target log information categories and label values ​​corresponding to each log information in multiple current coded information groups. In the second matrix, the server determines the label values ​​of adjacent log information corresponding to each current coded information group as the corresponding log information group. The server then determines the first log information group corresponding to each current coded information in the second matrix as the first log information group.

[0095] Each current encoding information group may correspond to one log information group or multiple log information groups.

[0096] When there is only one log information in each current encoding information group, the log information forms a log information group corresponding to the current encoding information group.

[0097] In an embodiment of the present invention, the server can determine the position information (M, N) of each log information group. Wherein, M is the number of rows of the corresponding log information group in the second matrix, and N is the number of columns of the corresponding log information group in the second matrix. The server determines that the log information group with the smallest number of rows corresponding to each coded information group is the corresponding first log information group.

[0098] S111 . Construct a frequent pattern tree based on attribute information corresponding to each of the plurality of first log information groups.

[0099] In the embodiment of the present invention, the server may construct a frequent pattern tree based on the attribute information corresponding to the plurality of first log information groups.

[0100] In an embodiment of the present invention, the server can determine the number of rows and columns corresponding to each leading log information, as well as the corresponding category name information. Furthermore, the server can determine the parent path information of each leading log information group. Based on the parent path information of each leading log information group, the server can calculate the parent-child relationship between multiple leading log information groups, thereby constructing a frequent pattern tree.

[0101] The server may determine the first log information group corresponding to the first current coded information group as the root node, determine the parent-child relationship between the multiple first log information groups based on the attribute information between the multiple first log information groups, and construct a frequent pattern tree.

[0102] In some embodiments, see Figure 2 , Figure 2 This is an optional flow chart of the log detection method provided by an embodiment of the present invention. S109-S111 can be implemented through S203 to S206, which will be described in conjunction with each step.

[0103] S203 : Sort the log information categories in the first matrix according to the support, to obtain a second matrix that meets the support threshold.

[0104] In this embodiment of the present invention, the server first deletes a row of data corresponding to a log information category whose support is less than a support threshold from the first matrix. Then, the server reorders the rows of data corresponding to the remaining log information categories in descending order of their support, thereby forming a new second matrix.

[0105] S204 , dividing the second matrix into log information groups by column, and establishing relationships such as parent-child paths between each log information group.

[0106] In the embodiment of the present invention, the server determines that adjacent log information in a column corresponding to each current coded information group is a log information group. Since the position information of each log information group in the second matrix is ​​determined, the server can quickly determine the parent path relationship of each log information group. As well as the parent-child relationship between multiple log information groups.

[0107] S205: Optimize the log information group and create a tree relationship set.

[0108] In the embodiment of the present invention, the server determines the first log information group corresponding to each current coded information group in the second matrix, and then determines the parent-child relationship between the plurality of first log information groups based on the position information of the plurality of first log information groups, that is, creates a tree relationship set.

[0109] S206: Construct a frequent pattern tree.

[0110] In the embodiment of the present invention, the server determines the first log information group corresponding to the first current coded information group as the root node, and then constructs a frequent pattern tree based on the parent-child relationship between multiple first log information groups.

[0111] S112 : In the frequent pattern tree, determine a first frequent pattern of a left node in the frequent pattern tree and a second frequent pattern of a right node in the frequent pattern tree, and determine overlapping log information groups of the first frequent pattern and the second frequent pattern as multiple current log information groups.

[0112] In an embodiment of the present invention, the server determines the first frequent pattern of the left node in the frequent pattern tree and the second frequent pattern of the right node in the frequent pattern tree, and determines the overlapping log information groups of the first frequent pattern and the second frequent pattern as multiple current log information groups.

[0113] In an embodiment of the present invention, a server can determine a first frequent pattern of the left node in the frequent pattern tree by traversing the father node and the left node in the frequent pattern tree. The server can determine a second frequent pattern of the right node in the frequent pattern tree by traversing the father node and the right node in the frequent pattern tree. The server determines overlapping log information groups of the first frequent pattern and the second frequent pattern as multiple current log information groups.

[0114] The first frequent pattern may include at least one leading log information group corresponding to a left node. The second frequent pattern may include at least one leading log information group corresponding to a right node. The first frequent pattern may be a left frequent pattern. The second frequent pattern may be a right frequent pattern.

[0115] In an embodiment of the present invention, the server can perform a post-order traversal of the frequent pattern tree, adding the total support corresponding to each left node to the total support corresponding to the corresponding parent node to obtain first counting information. If the first counting information is greater than a support threshold, the first log information group corresponding to the parent node is classified as the first frequent pattern. If the first counting information is not greater than the support threshold, the server continues to traverse the next left node until the frequent pattern tree traversal is completed.

[0116] In an embodiment of the present invention, the server can perform a pre-order traversal of the frequent pattern tree, adding the total support corresponding to each right node to the log count information corresponding to the corresponding parent node to obtain second count information. If the second count information is greater than a support threshold, the first log information group corresponding to the parent node and the right node is classified as a second frequent pattern. If the second count information is not greater than the support threshold, the server continues to traverse the next right node until the frequent pattern tree traversal is completed.

[0117] The log count information reflects the number of log information included in the first log information group of the corresponding node.

[0118] In some embodiments, see Figure 2 , Figure 2 This is an optional flow chart of the log detection method provided by an embodiment of the present invention. S112 can be implemented through S207 to S211, which will be described in conjunction with each step.

[0119] S207: Mining the first frequent pattern of the left node of the frequent pattern tree.

[0120] In the embodiment of the present invention, the server mines out the first frequent pattern of the left node in the frequent pattern tree through post-order traversal.

[0121] S208. The left node mining is completed, and the set L is obtained.

[0122] In the embodiment of the present invention, the server forms a set L with the first frequent pattern set formed by mining.

[0123] S209: Mining the second frequent pattern of the right node of the frequent pattern tree.

[0124] In the embodiment of the present invention, the server mines out the second frequent pattern of the right node in the frequent pattern tree through pre-order traversal.

[0125] S210. The right node mining is completed, and the set R is obtained.

[0126] In the embodiment of the present invention, the server forms a set R with the second frequent pattern set formed by mining.

[0127] S211. Calculate the union of set L and set R to obtain multiple current log information groups.

[0128] In the embodiment of the present invention, the server determines the overlapping portion of the set L and the set R, and uses the first log information group of the overlapping portion as the multiple current log information groups.

[0129] In some embodiments, Figure 2 The illustrated S108 can be implemented through S113 to S114 , which will be described in conjunction with each step.

[0130] S113 . In the first matrix, add the mark values ​​corresponding to a row of each log information category to obtain the median value of each log information category.

[0131] In the embodiment of the present invention, the server adds the tag values ​​of a row corresponding to each log information category in the first matrix to obtain an intermediate value corresponding to each log information category.

[0132] The tag value can be 1.

[0133] S114: Compare the intermediate value with the number of multiple current coded information groups to obtain the support degree of each log information category.

[0134] In the embodiment of the present invention, the server divides the median value corresponding to each log information category by the number of multiple current coded information groups to obtain the support degree corresponding to each log information category.

[0135] The support reflects the number of times the log information of the corresponding log information category appears in multiple current coded information groups.

[0136] In some embodiments, S110 can be implemented through S115 to S117 , which will be described in conjunction with each step.

[0137] S115 , sorting a row of data corresponding to the target log information category in the first matrix according to the order of the target support corresponding to the target log information category, to construct a second matrix.

[0138] In the embodiment of the present invention, the server sorts a row of data corresponding to the target log information category in the first matrix according to the order of the target support corresponding to the target log information category, and constructs a second matrix.

[0139] In the embodiment of the present invention, the server determines the target log information category. The server also previously obtains the target support corresponding to the target log information category. The server reorders the row of data (i.e., the tag value) corresponding to the target log information category in the first matrix in descending order of target support, thereby obtaining the second matrix.

[0140] S116. Determine in the second matrix that the log information categories to which the adjacent tag values ​​corresponding to the multiple current coded information groups belong are log information groups.

[0141] In the embodiment of the present invention, since a column corresponding to each current coded information group may contain multiple tag values ​​corresponding to log information, the server determines the log information categories corresponding to adjacent tag values ​​in a column corresponding to each current coded information group as a log information group.

[0142] For example, in combination with Table 3, the server may determine that the log information groups corresponding to the current coded information group T01 are (A, B) and (D).

[0143] In an embodiment of the present invention, the server may have three grouping rules for the log information corresponding to each current coded information group. The first is that mat[j][i] is 0 in the second matrix and mat[j-1][i] is 1, indicating that there is at least one group in the column, and the server creates a log information group including log information categories whose adjacent tag values ​​are all 1. Among them, mat[j][i] represents the tag value of the j-th row and i-th column in the second matrix. Among them, mat[j][i] represents the tag value of the j-th row and i-th column in the second matrix.

[0144] The second type is that mat[j][i] in the second matrix is ​​1 and mat[j-1][i] is 0, indicating that the first j rows of the column are 0. The server continues to traverse downward until the traversal ends. The server creates a log information group that includes log information categories whose adjacent tag values ​​are all 1.

[0145] The third type is that mat[j][i] in the second matrix is ​​1, and mat[j-1][i] is 1, which means that the jth column is all 1 starting from a certain row of non-zero values. Then the server creates a log information group including log information categories whose adjacent tag values ​​are all 1.

[0146] S117 . In the second matrix, number the log information groups corresponding to each current coding information group according to a predetermined rule, and determine the first log information group with a preset number in a column of log information groups corresponding to the multiple current coding information groups.

[0147] In an embodiment of the present invention, the server may encode a column of log information groups corresponding to each current encoded information group in ascending order, and determine the log information group with the smallest code as the first log information group of the corresponding current encoded information group.

[0148] For example, the server may number a column of log information groups corresponding to each current encoding information group in the order of 1, 2, and 3. The server determines the log information group numbered 1 corresponding to each current encoding information group as the first log information group.

[0149] In some embodiments, S111 can be implemented through S118 to S120, which will be described in conjunction with each step.

[0150] S119: Obtain attribute information corresponding to each of the plurality of first-place log information groups in the second matrix.

[0151] In the embodiment of the present invention, since the position information of the first log information group corresponding to each current coded information group in the second matrix is ​​determined, the server can obtain the category name information, parent node information, total support, parent path information, log set information, and log count information corresponding to multiple first log information groups in the second matrix.

[0152] S120 . In the second matrix, determine the first log information group with the minimum number of rows as the root node.

[0153] In the embodiment of the present invention, since multiple first log information groups can have different numbers of rows, the smaller the number of rows corresponding to the first log information group, the greater the support of the log information category corresponding to the first log information group. The server determines the first log information group with the smallest number of rows as the root node.

[0154] In the embodiment of the present invention, if there are multiple first log information groups with the minimum number of rows, the server determines the first log information group with the minimum number of columns as the root node.

[0155] S120 : Construct a frequent pattern tree based on the root node and the known attribute information corresponding to the plurality of first log information groups.

[0156] In the embodiment of the present invention, the server constructs a frequent pattern tree according to the construction rules of the frequent pattern tree, based on the root node and based on the attribute information corresponding to the plurality of first log information groups.

[0157] The attribute information includes: category name information, parent node information, total support, parent path information, log set information, and log count information corresponding to multiple first-place log information groups. The total support information represents the sum of the support of the log information category corresponding to the first-place log information group.

[0158] In an embodiment of the present invention, if the category name information of the root node is the parent node of the node to be inserted, the node to be inserted is determined to be in a parent-child relationship with the root node. The server inserts the node to be inserted into the frequent pattern tree at the left node position of the root node. If the next node to be inserted is still in a parent-child relationship with the root node, the server inserts the next node to be inserted into the right node position of the root node. If the parent path information of a node is the parent path information of the node to be inserted, it indicates that the two nodes are sibling nodes. The server then determines whether the category name information of the two nodes contains a relationship. If so, the parent-child relationship between the two nodes is established and the two nodes are inserted into the frequent pattern tree.

[0159] In some embodiments, S112 can be implemented through S121 to S123, which will be described in conjunction with each step.

[0160] S121 . Perform post-order traversal in the frequent pattern tree, and determine a first frequent pattern based on the total support of the father node and the left node in the frequent pattern tree.

[0161] In an embodiment of the present invention, the server performs post-order traversal in the frequent pattern tree and determines a first frequent pattern based on the total support of the father node and the left node in the frequent pattern tree, wherein the first frequent pattern includes the first log information group of the corresponding father node.

[0162] In an embodiment of the present invention, the server can perform a post-order traversal of the frequent pattern tree, adding the total support corresponding to each left node to the total support corresponding to the corresponding parent node to obtain first counting information. If the first counting information is greater than a support threshold, the first log information group corresponding to the parent node is classified as the first frequent pattern. If the first counting information is not greater than the support threshold, the server continues to traverse the next left node until the frequent pattern tree is traversed. Ultimately, the first frequent pattern is obtained.

[0163] S122 : Perform a pre-order traversal in the frequent pattern tree, and determine a second frequent pattern based on the log count information of the father node and the total support of the right node in the frequent pattern tree.

[0164] In an embodiment of the present invention, the server performs a pre-order traversal in the frequent pattern tree and determines a second frequent pattern based on the log count information of the parent node and the total support of the right node in the frequent pattern tree. The second frequent pattern includes the first log information group of the corresponding parent node and the right node.

[0165] In an embodiment of the present invention, the server can perform a pre-order traversal of the frequent pattern tree, adding the total support corresponding to each right node to the log count information corresponding to the corresponding parent node to obtain second count information. If the second count information is greater than a support threshold, the first log information group corresponding to the parent node and the right node is classified as a second frequent pattern. If the second count information is not greater than the support threshold, the server continues to traverse the next right node until the frequent pattern tree traversal is completed.

[0166] The log count information reflects the number of log information included in the first log information group of the corresponding node.

[0167] S123: Determine overlapping log information groups of the first frequent pattern and the second frequent pattern as multiple current log information groups.

[0168] In the embodiment of the present invention, the server determines the overlapping first log information groups of the first frequent pattern and the second frequent pattern as multiple current log information groups.

[0169] In some embodiments, S121 can be implemented through S124 to S125, which will be explained in conjunction with each step.

[0170] S124 , performing post-order traversal in the frequent pattern tree, and sequentially adding the total support of the left node and the total support of the corresponding parent node to obtain first counting information.

[0171] In the embodiment of the present invention, the server performs a post-order traversal in the frequent pattern tree and sequentially adds the total support of each left node to the total support of the parent node corresponding to the left node to obtain corresponding first counting information.

[0172] The first counting information reflects the occurrence frequency of the left node and the corresponding parent node.

[0173] S125 : If the first counting information is greater than the support threshold, the first log information group corresponding to the father node is classified into the first frequent pattern, and the next traversal is continued until the frequent pattern tree traversal is completed to obtain the first frequent pattern.

[0174] In the embodiment of the present invention, if the first count information is greater than the support threshold, the server classifies the first log information group corresponding to the father node into the first frequent pattern, and continues the next traversal until the frequent pattern tree traversal is completed, thereby obtaining the first frequent pattern.

[0175] For example, in combination with Table 3, the first log information group corresponding to the father node may be (A, B).

[0176] In some embodiments, S125 can be implemented through S126, which will be explained in conjunction with each step.

[0177] S126. If the first counting information is not greater than the support threshold, add the total support of the next left node and the total support of the corresponding father node to obtain the first counting information, and continue the process of comparing with the support threshold.

[0178] In this embodiment of the present invention, if the server obtains the first count information from the previous calculation that is not greater than the support threshold, the server traverses the next left node in the frequent pattern tree. The server adds the total support of the next left node to the total support of its corresponding parent node to obtain the next first count information. The server then continues to compare the first count information with the support threshold.

[0179] In some embodiments, Figure 7 The illustrated S122 can be implemented through S127 to S128, which will be described in conjunction with each step.

[0180] S127 , performing a pre-order traversal in the frequent pattern tree, and sequentially adding the total support of the right node and the log count information of the corresponding father node to obtain second counting information.

[0181] In the embodiment of the present invention, the server performs a sequential traversal in the frequent pattern tree and sequentially adds the total support of each right node to the log count information of the parent node corresponding to the right node to obtain corresponding second counting information.

[0182] The log count information reflects the number of log information categories in the corresponding father node.

[0183] S128: If the second count information is greater than the support threshold, the first log information group corresponding to the father node and the right node is classified into the second frequent pattern, and the next traversal is continued until the frequent pattern tree traversal is completed to obtain the second frequent pattern.

[0184] In this embodiment of the present invention, if the second counting information is greater than the support threshold, the server classifies the first log information group corresponding to the father node and the right node into the second frequent pattern and continues the next traversal until the frequent pattern tree traversal is completed to obtain the second frequent pattern.

[0185] In some embodiments, S128 can be implemented through S129, which will be explained in conjunction with each step.

[0186] S129. If the second counting information is not greater than the support threshold, the total support of the next right node and the log count information of the corresponding father node are added to obtain the second counting information, and the comparison process with the support threshold is continued.

[0187] In this embodiment of the present invention, if the server obtains the second count information from the previous calculation that is not greater than the support threshold, the server traverses the next right node in the frequent pattern tree. The server adds the total support of the next right node to the log count information of the corresponding parent node to obtain the next second count information. The server then continues to compare the second count information with the support threshold.

[0188] In some embodiments, Figure 1 The illustrated S101 may also include S130 to S132 , which will be described in conjunction with each step.

[0189] S130: Acquire multiple sample log data before the current moment, and obtain multiple sample coding information groups based on the multiple sample log data.

[0190] In an embodiment of the present invention, the server can obtain multiple sample log data before the current moment in the server's cache or database through a log acquisition tool. The server deletes duplicate log data from the multiple sample log data. The server extracts multiple preset sample log information from the multiple sample log data after deleting the duplicate log data. The server determines the encoding information of the multiple preset sample log information for each sample log data according to a predetermined rule, and then can determine the sample encoding information group corresponding to each sample log data, thereby obtaining multiple sample encoding information groups.

[0191] S131. Construct a sample matrix based on the sample log information and sample log information categories in multiple sample coding information groups.

[0192] In an embodiment of the present invention, the server may construct a sample matrix with the log information category as the first column and the label value of each sample log information and the corresponding log information category in each sample coding information group as subsequent columns.

[0193] S132. According to the number of times the sample log information appears in the sample matrix, a plurality of pre-stored log information groups are obtained to form a pattern library.

[0194] In the embodiment of the present invention, since the relationship between the sample log information groups corresponding to each sample coding information group in the sample matrix is ​​determined, the server can quickly determine the relationship between the sample log information groups of each sample coding information group in the sample matrix, and then construct a frequent pattern tree. The server uses an association extraction algorithm to extract frequent patterns of multiple sample coding information groups from the frequent pattern tree as multiple pre-stored coding information groups.

[0195] The sample log information group is the log information of the corresponding sample coding information group that is adjacent in the sample matrix.

[0196] In some embodiments, see Figure 3 , Figure 3 This is an optional flow chart of the log detection method provided by an embodiment of the present invention. S130 to S132 can also be implemented through S212 to S216, which will be described in conjunction with each step.

[0197] S212: The server obtains multiple sample log data.

[0198] S213: The server pre-processes the sample log data.

[0199] S214. The server converts the sample log data format.

[0200] S215, algorithm mining.

[0201] S216, pattern library.

[0202] In some embodiments, Figure 1 The illustrated S101 can also be implemented through S133 to S136 , which will be described in conjunction with each step.

[0203] S133. Mining multiple current log data using a log acquisition tool.

[0204] In an embodiment of the present invention, the server may obtain multiple current log data in a cache or database of the server through a log collection system, a logstash log collection tool, or a syslog tool. The multiple current log data may be in text format.

[0205] S134 , performing deduplication processing on the multiple current log data to obtain multiple first-screened log data.

[0206] In an embodiment of the present invention, if k pieces of current log data have duplicate content among the multiple current log data, the server deletes k-1 pieces of current log data from the k pieces of current log data, thereby obtaining multiple pieces of primary filtered log data. k is a positive integer greater than or equal to 1.

[0207] S135 , deleting missing log data with missing log information from the plurality of first-screened log data to obtain a plurality of second-screened log data.

[0208] In an embodiment of the present invention, if there is missing log data in the log information among the multiple first-screened log data, the server deletes the log data with missing log information from the multiple first-screened log data, thereby obtaining multiple second-screened log data.

[0209] S136 , extracting a plurality of preset log information corresponding to the plurality of secondary screening log data respectively from the plurality of secondary screening log data, and encoding the plurality of preset log information to obtain a plurality of current encoding information groups corresponding to the plurality of secondary screening log data respectively.

[0210] In an embodiment of the present invention, the server extracts multiple preset log information from each secondary screening log data. The server determines the encoding information of the multiple preset log information according to a predetermined rule, and further determines the current encoding information group corresponding to each secondary screening log data, thereby obtaining multiple current encoding information groups.

[0211] In some embodiments, see Figure 3 , Figure 3 This is an optional flowchart of the log detection method provided by an embodiment of the present invention. S133 to S136 can also be implemented through S217 to S219, which will be described in conjunction with each step.

[0212] S217: The server obtains multiple current log data.

[0213] S218. The server pre-processes the current log data.

[0214] S219. The server converts the current log data format.

[0215] In some embodiments, Figure 1 The illustrated S104 can also be implemented through S137 to S138, which will be described in conjunction with each step.

[0216] S137 , matching each current log information group with a plurality of pre-stored log information groups in the pattern library one by one to obtain a matching result for each current log information group.

[0217] In the embodiment of the present invention, the server matches each current log information group with a plurality of pre-stored log information groups in the pattern library one by one to obtain a matching result for each current log information group.

[0218] S138: If the matching result of at least one current log information group indicates that at least one current log information group matches at least one pre-stored log information group, then the detection result of at least one current log information group is normal.

[0219] In some embodiments, see Figure 3 , Figure 3 This is an optional flow chart of the log detection method provided by an embodiment of the present invention. S137 to S138 can also be implemented through S220 to S221, which will be described in conjunction with each step.

[0220] S220, pattern matching.

[0221] S221: Is there any abnormality?

[0222] It should be noted that if the server detects that at least one current log data is normal, the server will add the at least one log data to the Figure 3 The plurality of sample log data in S212 are collected, and the process of determining the pattern library is re-executed.

[0223] In some embodiments, S138 can also be implemented through S139, which will be explained in conjunction with each step.

[0224] S139: If the matching result of at least one current log information group indicates that at least one current log information group does not match any pre-stored log information group, then the detection result of at least one current log information group is abnormal.

[0225] In some embodiments, see Figure 3 , Figure 3 This is an optional flow chart of the log detection method provided by an embodiment of the present invention. S139 can also be implemented through S222, which will be described in conjunction with each step.

[0226] S222: Detect abnormality.

[0227] To verify the effectiveness of the improved MGFP-growth algorithm in mining massive data sets, two different experimental comparisons were conducted using the Apriori algorithm, the FP-growth algorithm, and the MGFP-growth algorithm under the same experimental environment. One set was based on the dense mushroom dataset, and the other on the sparse T10I4D100K dataset. The experimental data characteristics are shown in Table 4.

[0228] Table 4 Dataset characteristics

[0229]

[0230]

[0231] The two sets of experimental preconditions do not change the parameters, only the minimum support threshold is changed, and the execution time corresponding to different support levels of the three algorithms is counted. The shorter the running time, the higher the execution efficiency of the MGFP-growth algorithm. The experimental results are as follows Figure 4 、 5 As shown in the figure, the improved MGFP-growth algorithm has obvious advantages in execution efficiency compared with traditional classical algorithms on dense and sparse data sets, so it can be used to mine massive log data.

[0232] In some embodiments, see Figure 6 , Figure 6 An optional flow chart of the log detection method provided by an embodiment of the present invention will be described in conjunction with each step.

[0233] The log detection method provided in the embodiment of the present invention is divided into two parts. The first part can be the determination part of the pattern library, which can be implemented through S224 to S226.

[0234] S224: Log preprocessing and format conversion.

[0235] S225, algorithm mining.

[0236] S226, pattern library.

[0237] The second part may be the detection part of the current log, which may be implemented through S227 to S230.

[0238] S228, algorithm mining.

[0239] S229, multiple current log information groups.

[0240] S229, match.

[0241] S229. Test results.

[0242] See also Figure 7 , Figure 7 A schematic diagram of the structure of a log detection device provided by an embodiment of the present invention.

[0243] The embodiment of the present invention further provides a log detection device 800 , comprising: a data acquisition unit 803 , a matrix construction unit 804 , a first correlation extraction unit 805 , and a matching unit 806 .

[0244] The data acquisition unit 803 is configured to acquire a plurality of current log data and obtain a plurality of current coded information groups based on the plurality of current log data; the plurality of current coded information groups include: a plurality of log information in the plurality of current log data; the log information represents event information in the corresponding log data;

[0245] A matrix construction unit 804 is configured to construct a first matrix based on the plurality of log information included in the plurality of current encoding information groups;

[0246] The association extraction unit 805 is configured to obtain, based on the number of occurrences of the multiple log information in the first matrix, multiple current log information groups corresponding to the multiple current log data; the multiple current log information groups are overlapping log information groups in a frequent pattern tree; the frequent pattern tree is constructed based on the number of occurrences of the multiple log information;

[0247] Matching unit 806 is used to match the multiple current log information groups with multiple pre-stored log information groups in the pattern library to obtain matching results of the multiple current log information groups, and determine detection results for the multiple current log data based on the matching results.

[0248] The association extraction unit 805 is used to add the current log data corresponding to the at least one current log information group to the multiple sample log data extracted when determining the pattern library if the matching result indicates that at least one current log information group matches at least one pre-stored log information group, so as to obtain the updated multiple pre-stored log information groups and the updated pattern library based on the updated multiple pre-stored log information groups.

[0249] In an embodiment of the present invention, the matrix construction unit 804 in the log detection device 800 is used to form the first column of the first matrix based on the categories of multiple log information in the multiple current encoded information groups; based on each log information in the n-1th current encoded information group and the preset mark value of the corresponding log information category, form the nth column of the first matrix; the number of columns of the first matrix is ​​n; n is a positive integer greater than 1.

[0250] In an embodiment of the present invention, the association extraction unit 805 in the log detection device 800 is used to calculate the support of each log information category based on the number of occurrences of each log information in the multiple current encoded information groups in the first matrix; determine the target log information category corresponding to the target support greater than the support threshold; determine the first log information group corresponding to the multiple current encoded information groups in the second matrix constructed based on the target log information category; construct a frequent pattern tree based on the attribute information corresponding to the multiple first log information groups; the attribute information includes: category name information, parent node information, total support, parent path information, log set information and log count information corresponding to the multiple first log information groups; the total support represents the sum of the support of the log information category corresponding to the first log information group; in the frequent pattern tree, determine the first frequent pattern of the left node in the frequent pattern tree and the second frequent pattern of the right node in the frequent pattern tree, and determine the overlapping log information groups of the first frequent pattern and the second frequent pattern as the multiple current log information groups.

[0251] In an embodiment of the present invention, the association extraction unit 805 in the log detection device 800 is used to add the label values ​​corresponding to a row of each log information category in the first matrix to obtain the intermediate value of each log information category; and compare the intermediate value with the number of the multiple current encoded information groups to obtain the support of each log information category.

[0252] In the embodiment of the present invention, the matrix construction unit 804 in the log detection device 800 is used to sort a row of data corresponding to the target log information category in the first matrix according to the order of the target support corresponding to the target log information category to construct a second matrix.

[0253] In an embodiment of the present invention, the association extraction unit 805 in the log detection device 800 is used to determine in the second matrix that the log information category to which the adjacent tag values ​​corresponding to the multiple current coding information groups belong is a log information group; in the second matrix, the log information group corresponding to each current coding information group is numbered according to a predetermined rule, and the first log information group with a preset number in a column of log information groups corresponding to the multiple current coding information groups is determined.

[0254] In an embodiment of the present invention, the association extraction unit 805 in the log detection device 800 is used to obtain attribute information corresponding to multiple first-place log information groups in the second matrix; in an embodiment of the present invention, the association extraction unit 805 in the log detection device 800 is used to determine the first-place log information group with the minimum number of rows as the root node in the second matrix; based on the known attribute information corresponding to the root node and the multiple first-place log information groups, the frequent pattern tree is constructed.

[0255] In an embodiment of the present invention, the association extraction unit 805 in the log detection device 800 is used to perform a post-order traversal in the frequent pattern tree, and determine the first frequent pattern based on the total support of the father node and the left node in the frequent pattern tree; the first frequent pattern includes the first log information group of the corresponding father node; perform a pre-order traversal in the frequent pattern tree, and determine the second frequent pattern based on the log count information of the father node and the total support of the right node in the frequent pattern tree; the second frequent pattern includes the first log information group of the corresponding father node and the right node; and determine the overlapping log information groups of the first frequent pattern and the second frequent pattern as the multiple current log information groups.

[0256] In an embodiment of the present invention, the association extraction unit 805 in the log detection device 800 is used to perform a post-order traversal in the frequent pattern tree, and sequentially add the total support of the left node and the total support of the corresponding parent node to obtain first counting information; if the first counting information is greater than the support threshold, the first log information group corresponding to the father node is classified into the first frequent pattern, and the next traversal is continued until the traversal of the frequent pattern tree is completed to obtain the first frequent pattern.

[0257] In an embodiment of the present invention, the association extraction unit 805 in the log detection device 800 is used to add the total support of the next left node and the total support of the corresponding father node if the first counting information is not greater than the support threshold, to obtain the first counting information, and continue the process of comparing with the support threshold.

[0258] In an embodiment of the present invention, the association extraction unit 805 in the log detection device 800 is used to perform a pre-order traversal in the frequent pattern tree, sequentially adding the total support of the right node and the log count information of the corresponding father node to obtain second counting information; if the second counting information is greater than a support threshold, the first log information group corresponding to the father node and the right node is classified as the second frequent pattern, and the next traversal is continued until the traversal of the frequent pattern tree is completed to obtain the second frequent pattern.

[0259] In an embodiment of the present invention, the association extraction unit 805 in the log detection device 800 is used to add the total support of the next right node and the log count information of the corresponding father node if the second counting information is not greater than the support threshold, to obtain the second counting information, and continue the process of comparing with the support threshold.

[0260] In an embodiment of the present invention, the association extraction unit 807 in the log detection device 800 is used to obtain multiple sample log data before the current moment, and obtain multiple sample coding information groups based on the multiple sample log data; construct a sample matrix based on the sample log information and sample log information categories in the multiple sample coding information groups; and obtain the multiple pre-stored log information groups based on the number of times the sample log information appears in the sample matrix to form the pattern library.

[0261] In an embodiment of the present invention, the data acquisition unit 803 in the log detection device 800 is used to mine the multiple current log data through a log acquisition tool; deduplicate the multiple current log data to obtain multiple first-screened log data; delete the missing log data in which log information is missing from the multiple first-screened log data to obtain multiple second-screened log data; extract multiple preset log information corresponding to the multiple second-screened log data from the multiple second-screened log data, and encode the multiple preset log information to obtain the multiple current encoded information groups corresponding to the multiple second-screened log data.

[0262] In an embodiment of the present invention, the matching unit 806 in the log detection device 800 is used to match each current log information group with the multiple pre-stored log information groups in the pattern library one by one to obtain a matching result for each current log information group; if the matching result of the at least one current log information group indicates that the at least one current log information group matches the at least one pre-stored log information group, then the detection result of the at least one current log information group is normal; or, if the matching result of the at least one current log information group indicates that the at least one current log information group does not match any pre-stored log information group, then the detection result of the at least one current log information group is abnormal.

[0263] In an embodiment of the present invention, a data acquisition unit 803 acquires a plurality of current log data, and obtains a plurality of current coded information groups based on the plurality of current log data; the plurality of current coded information groups include: a plurality of log information in the plurality of current log data; the log information represents event information in the corresponding log data; a matrix construction unit 804 constructs a first matrix based on the plurality of log information included in the plurality of current coded information groups; an association extraction unit 805 obtains a plurality of current log information groups corresponding to the plurality of current log data based on the number of occurrences of the plurality of log information in the first matrix; the plurality of current log information groups are overlapping log information groups in a frequent pattern tree; the frequent pattern tree is constructed based on the number of occurrences of the plurality of log information; a matching unit 806 matches the plurality of current log information groups with a plurality of pre-stored log information groups in a pattern library to obtain matching results of the plurality of current log information groups, and determines detection results for the plurality of current log data based on the matching results. Since the server stores each current coded information group through the first matrix, the path relationship between the log information of each current coded information group in the first matrix is ​​clearer, which facilitates the server to quickly build a frequent pattern tree to mine multiple current log information groups, and then quickly determine the detection results of multiple current log data, thereby improving the efficiency of the server in detecting log data.

[0264] It should be noted that, in the embodiment of the present invention, if the above-mentioned cluster construction and subscription information processing method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention, or the part that contributes to the relevant technology, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a log detection device (which can be a personal computer, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiment of the present invention is not limited to any specific combination of hardware and software.

[0265] Correspondingly, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the steps in the above method when executed by a processor.

[0266] Correspondingly, an embodiment of the present invention provides a log detection device 800, including a memory 802 and a processor 801. The memory 802 stores a computer program that can be run on the processor 801. When the processor 801 executes the program, the steps in the above method are implemented.

[0267] It should be noted that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of the present invention, please refer to the description of the method embodiments of the present invention for understanding.

[0268] It should be noted that Figure 8 A hardware entity diagram of the log detection device 800 provided in an embodiment of the present invention is shown as follows: Figure 8 As shown, the hardware entity of the log detection device 800 includes: a processor 801 and a memory 802, wherein;

[0269] The processor 801 generally controls the overall operation of the log detection apparatus 800 .

[0270] The memory 802 is configured to store instructions and applications executable by the processor 801, and can also cache data to be processed or processed by the processor 801 and each module in the log detection device 800 (for example, image data, audio data, voice communication data and video communication data), which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0271] The above description is merely an embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A log detection method, characterized in that: include: Acquire multiple current log data, and obtain multiple current encoding information groups based on the multiple current log data; The multiple current coding information groups include: multiple log information in the multiple current log data; the log information represents the event information in the corresponding log data; forming a first column of a first matrix based on categories of a plurality of log information in the plurality of current encoded information groups; Based on each log information in the n-1th current coded information group and the preset label value of the corresponding log information category, forming the nth column of the first matrix; the number of columns of the first matrix is ​​n; n is a positive integer greater than 1; According to the number of times the multiple log information appear in the first matrix, a plurality of current log information groups corresponding to the multiple current log data are obtained; the multiple current log information groups are overlapping log information groups in a frequent pattern tree; the frequent pattern tree is constructed according to the number of times the multiple log information appear; The multiple current log information groups are matched with multiple pre-stored log information groups in a pattern library to obtain matching results of the multiple current log information groups, and detection results for the multiple current log data are determined based on the matching results.

2. The log detection method according to claim 1, characterized in that: After matching the multiple current log information groups with multiple pre-stored log information groups in the pattern library to obtain matching results for the multiple current log information groups, and determining detection results for the multiple current log data based on the matching results, the method further includes: If the matching result indicates that at least one current log information group matches at least one pre-stored log information group, the current log data corresponding to the at least one current log information group is added to the multiple sample log data extracted when determining the pattern library to obtain the updated multiple pre-stored log information groups, and the pattern library updated based on the updated multiple pre-stored log information groups.

3. The log detection method according to claim 1, characterized in that: The obtaining, according to the number of times the multiple log information appear in the first matrix, multiple current log information groups corresponding to the multiple current log data includes: Calculate the support of each log information category based on the number of occurrences of each log information in the multiple current coded information groups in the first matrix; Determine the target log information category corresponding to the target support greater than the support threshold; In a second matrix constructed based on the target log information category, determining the first log information group corresponding to each of the multiple current coded information groups; Constructing a frequent pattern tree based on attribute information corresponding to each of the plurality of first-place log information groups; the attribute information includes: category name information, parent node information, total support, parent path information, log set information, and log count information corresponding to each of the plurality of first-place log information groups; the total support represents the sum of the support of the log information category corresponding to the first-place log information group; In the frequent pattern tree, a first frequent pattern of a left node in the frequent pattern tree and a second frequent pattern of a right node in the frequent pattern tree are determined, and overlapping log information groups of the first frequent pattern and the second frequent pattern are determined as the multiple current log information groups.

4. The log detection method according to claim 3, characterized in that: The method of constructing a frequent pattern tree based on attribute information corresponding to each of the plurality of first log information groups includes: In the second matrix, the first log information group with the minimum number of rows is determined as the root node; The frequent pattern tree is constructed based on the known attribute information corresponding to the root node and the plurality of first log information groups.

5. The log detection method according to claim 3, characterized in that: The method further comprises: determining, in the frequent pattern tree, a first frequent pattern of a left node in the frequent pattern tree and a second frequent pattern of a right node in the frequent pattern tree, and determining overlapping log information groups of the first frequent pattern and the second frequent pattern as the multiple current log information groups, including: Performing a post-order traversal in the frequent pattern tree, and determining the first frequent pattern based on the total support of the father node and the left node in the frequent pattern tree; the first frequent pattern includes the first log information group of the corresponding father node; Performing a pre-order traversal in the frequent pattern tree, determining a second frequent pattern based on the log count information of the father node and the total support of the right node in the frequent pattern tree; the second frequent pattern includes the first log information group of the corresponding father node and the right node; The overlapping log information groups of the first frequent pattern and the second frequent pattern are determined as the multiple current log information groups.

6. The log detection method according to claim 1, characterized in that: Before obtaining a plurality of current log data and obtaining a plurality of current coding information groups based on the plurality of current log data, the method further includes: Acquire a plurality of sample log data before a current moment, and obtain a plurality of sample coding information groups based on the plurality of sample log data; constructing a sample matrix based on the sample log information and the sample log information categories in the plurality of sample coding information groups; The plurality of pre-stored log information groups are obtained according to the number of times the sample log information appears in the sample matrix to form the pattern library.

7. The log detection method according to claim 1, characterized in that: The acquiring of a plurality of current log data, and obtaining a plurality of current coding information groups based on the plurality of current log data, includes: mining the plurality of current log data using a log acquisition tool; Deduplication processing is performed on the multiple current log data to obtain multiple first-screened log data; Deleting missing log data with missing log information from the plurality of first-screened log data to obtain a plurality of second-screened log data; A plurality of preset log information respectively corresponding to the plurality of secondary screening log data are extracted from the plurality of secondary screening log data, and the plurality of preset log information are encoded to obtain the plurality of current encoding information groups respectively corresponding to the plurality of secondary screening log data.

8. The log detection method according to claim 1, characterized in that: The matching of the multiple current log information groups with multiple pre-stored log information groups in the pattern library to obtain matching results of the multiple current log information groups, and determining detection results for the multiple current log data based on the matching results, includes: Matching each current log information group with the plurality of pre-stored log information groups in the pattern library one by one to obtain a matching result for each current log information group; If the matching result of at least one current log information group indicates that the at least one current log information group matches at least one pre-stored log information group, then the detection result of the at least one current log information group is normal; Alternatively, if the matching result of at least one current log information group indicates that the at least one current log information group does not match any pre-stored log information group, then the detection result of the at least one current log information group is abnormal.

9. A log detection device, characterized in that: include: a data acquisition unit, configured to acquire a plurality of current log data, and obtain a plurality of current coding information groups based on the plurality of current log data; The multiple current coding information groups include: multiple log information in the multiple current log data; the log information represents the event information in the corresponding log data; a matrix construction unit, configured to form a first column of a first matrix based on the categories of the plurality of log information in the plurality of current coded information groups, and to form an nth column of the first matrix based on the preset label value of each log information in the n-1th current coded information group and the corresponding log information category; the number of columns of the first matrix is ​​n; n is a positive integer greater than 1; an association extraction unit, configured to obtain, based on the number of occurrences of the plurality of log information in the first matrix, a plurality of current log information groups corresponding to the plurality of current log data; the plurality of current log information groups being overlapping log information groups in a frequent pattern tree; the frequent pattern tree being constructed based on the number of occurrences of the plurality of log information; The matching unit is used to match the multiple current log information groups with multiple pre-stored log information groups in the pattern library to obtain matching results of the multiple current log information groups, and determine detection results for the multiple current log data based on the matching results.

10. A log detection device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the steps in the method according to any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images