Method and device for determining abnormal mode of log piece, equipment and storage medium

Abstract

The embodiment of the invention discloses a method and device for determining an abnormal mode of a log piece, equipment and a storage medium. The method comprises the following steps: determining the type of a log based on a character string of the log contained in a log set; determining an abnormal log piece from a plurality of log pieces, wherein the plurality of log pieces are obtained by dividing the log set based on a sliding time window; determining an abnormal mode of the log piece based on the type of the log contained in the abnormal log piece; comparing the type of the log contained in the log piece to be tested with the log piece abnormal mode; and determining an abnormal mode of the to-be-tested log piece based on a comparison result. The abnormal mode is automatically determined based on the log character string, and the defects of low accuracy and large workload in manual log analysis are overcome. Clustering is carried out based on the similarity between the log character strings, clustering can be completed without manual annotation, and convenience is improved. And log piece division based on the sliding window is wider in application range.

Description

Method, device, apparatus and storage medium for determining abnormal patterns of log files Technical Field

[0001] The present invention relates to the field of log management technology, and in particular to a method, device, equipment and storage medium for determining abnormal patterns of log pieces. Background Art

[0002] Network devices, systems, and service programs typically generate event records called logs during operation. Logs can record the date, time, user, and action, along with other relevant information. System developers and maintenance personnel can use logs to detect abnormal system behavior and errors.

[0003] As computing becomes increasingly complex and applications become more diverse, the types and volume of logs are increasing. Traditional manual log analysis methods are unable to meet the requirements of daily analysis.

[0004] Summary of the Invention

[0005] The embodiments of the present invention provide a method, apparatus, device, and storage medium for determining abnormal patterns of log slices.

[0006] A method for determining an abnormal pattern in a log sheet, comprising:

[0007] Determine the type of the log based on a character string of the log contained in the log set;

[0008] determining an abnormal log slice from a plurality of log slices, wherein the plurality of log slices are obtained by partitioning the log set based on a sliding time window;

[0009] Determining a log slice abnormality pattern based on the type of logs included in the abnormal log slice;

[0010] comparing the type of logs contained in the log sheet to be tested with the log sheet abnormality pattern;

[0011] The abnormality pattern of the log sheet to be tested is determined based on the comparison result.

[0012] Therefore, the present invention automatically identifies abnormal patterns based on log strings in a log collection, overcoming the low accuracy and high workload of manual log analysis. Furthermore, compared to session-based segmentation, the sliding window-based segmentation of logs has a wider range of applications.

[0013] In one embodiment, determining the type of the log based on a character string of the log included in the log set includes:

[0014] extracting log content characterized as unstructured information from the log;

[0015] Divide the log content into word sequences using spaces as delimiters;

[0016] Grouping the logs based on the length of the word sequence;

[0017] For each group:

[0018] Determine based on the characters at the predetermined position of each word in each word sequence contained in the group

[0019] Log string for each word sequence;

[0020] Clustering is performed based on the similarity between any two log strings to obtain the

[0021] The type of logs included;

[0022] The types of all grouped logs are combined into a log type set.

[0023] It can be seen that the embodiment of the present invention also performs clustering based on the similarity between log character strings, and can complete clustering without manual labeling, thereby improving convenience.

[0024] In one embodiment, the predetermined position is the first character or the last character of the word.

[0025] Therefore, the clustering difficulty is reduced by extracting characters at predetermined positions.

[0026] In one embodiment, the any two log strings include a first log string and a second log string: the method further includes: determining the similarity between characters in the same character position between the first log string and the second log string; and determining the similarity between the first log string and the second log string based on the similarity between the characters in the same character position.

[0027] It can be seen that by comparing the log strings one by one, the classification accuracy is guaranteed and the clustering difficulty is reduced.

[0028] In one embodiment, determining an abnormal log slice from a plurality of log slices includes:

[0029] determining, based on principal component analysis, a log slice having an unbalanced log type ratio from the plurality of log slices;

[0030] The log slice with an unbalanced log type ratio is determined as the abnormal log slice.

[0031] It can be seen that abnormal log slices can be easily determined through principal component analysis.

[0032] In one embodiment, determining the log slice abnormality pattern based on the type of logs included in the abnormal log slice includes:

[0033] Determine the number N of abnormal patterns in the log slice;

[0034] When the number m of abnormal log slices is less than or equal to N, determining a corresponding log slice abnormal pattern based on the type sequence of logs contained in each abnormal log slice to obtain m log slice abnormal patterns;

[0035] When the number m of the abnormal log pieces is greater than N, clustering is performed based on the similarity between the type sequences of the logs contained in the abnormal log pieces to obtain N log piece abnormal patterns.

[0036] Therefore, the preset number is taken into consideration to distinctively determine the log sheet abnormality pattern, which improves the implementation convenience.

[0037] In one embodiment, clustering based on the similarity between the type sequences of logs contained in the abnormal log slices to obtain N log slice abnormal patterns includes:

[0038] The type sequence of the logs contained in the abnormal log slice is used as a set element to form a log slice abnormal pattern set;

[0039] In the log slice anomaly pattern set, any two set elements with the greatest similarity are merged to update the log slice anomaly pattern set until the number of set elements in the log slice anomaly pattern set is equal to N.

[0040] It can be seen that by merging the sets to update the log slice exception pattern set, the type sequence of each exception log slice is preserved and the number of exception patterns is reduced.

[0041] In one embodiment, it further includes:

[0042] Determine the longest common subsequence length of a first set of elements of a first abnormal log piece and a second set of elements of a second abnormal log piece, wherein the first abnormal log piece and the second abnormal log piece are any two different abnormal log pieces;

[0043] Based on the number of logs in the first abnormal log piece, the number of logs in the second abnormal log piece, and the longest common subsequence length, the similarity between the first set element of the first abnormal log piece and the second set element of the second abnormal log piece is determined.

[0044] It can be seen that based on the longest common subsequence and hierarchical clustering, the abnormal patterns of log slices can be accurately mined.

[0045] In one embodiment, comparing the type of logs contained in the log piece to be tested with the log piece anomaly pattern includes: determining a similarity between a log type sequence in the log piece to be tested and each log piece anomaly pattern in the log piece anomaly pattern set;

[0046] Determining the abnormal pattern of the log piece to be tested based on the comparison result includes: determining the log piece abnormal pattern in the log piece abnormal pattern set that has the highest similarity with the log type sequence in the log piece to be tested and is greater than a predetermined threshold value as the abnormal pattern of the log piece to be tested.

[0047] Therefore, based on the similarity comparison, the abnormal pattern analysis of the log files to be tested is achieved.

[0048] In one embodiment, when the log slice anomaly pattern in the log slice anomaly pattern set is a set element that has been merged, the method further includes:

[0049] Determining a sub-similarity between a log type sequence included in the log piece to be tested and each log type sequence participating in the set merging in the log piece anomaly pattern;

[0050] The weighted sum of the sub-similarity values ​​is determined as the similarity between the log type sequence in the log piece to be tested and the abnormal pattern of the log piece.

[0051] It can be seen that by weighted summing the similarities of each log type sequence participating in the set merging, each log type sequence participating in the set merging is comprehensively considered, thereby improving the accuracy of the similarity calculation.

[0052] An apparatus for determining an abnormal pattern in a log sheet, comprising:

[0053] A first determination module is configured to determine a type of a log based on a character string of a log included in a log set;

[0054] A second determining module is configured to determine an abnormal log slice from a plurality of log slices, wherein the plurality of log slices are obtained by partitioning the log set based on a sliding time window;

[0055] a third determining module configured to determine a log slice abnormality pattern based on the type of logs included in the abnormal log slice;

[0056] a comparison module configured to compare the type of log contained in the log sheet to be tested with the log sheet abnormality pattern;

[0057] The fourth determining module is configured to determine an abnormal pattern of the log sheet to be tested based on the comparison result.

[0058] Therefore, the embodiments of the present invention automatically identify abnormal patterns based on log strings, overcoming the low accuracy and high workload of manual log analysis. Furthermore, compared to session-based segmentation, the sliding window-based log segmentation has a wider range of applications.

[0059] In one embodiment, the first determining module is configured to:

[0060] extracting log content characterized as unstructured information from the log;

[0061] Divide the log content into word sequences using spaces as delimiters;

[0062] Grouping the logs based on the length of the word sequence;

[0063] For each group:

[0064] Determine based on the characters at the predetermined position of each word in each word sequence contained in the group

[0065] Log string for each word sequence;

[0066] Clustering is performed based on the similarity between any two log strings to obtain the

[0067] The type of logs included;

[0068] The types of all grouped logs are combined into a log type set.

[0069] It can be seen that the embodiment of the present invention also performs clustering based on the similarity between log character strings, and can complete clustering without manual labeling, thereby improving convenience.

[0070] In one embodiment, the arbitrary two log strings include a first log string and a second log string:

[0071] The first determination module is configured to: determine the similarity between the first log string and the second log string, between characters at the same character position; and determine the similarity between the first log string and the second log string based on the similarity between the characters at the same character position.

[0072] It can be seen that by comparing the log strings one by one, the classification accuracy is guaranteed and the clustering difficulty is reduced.

[0073] In one embodiment, the third determination module is configured to: determine the number N of log slice abnormal patterns; when the number m of the abnormal log slices is less than or equal to N, determine the corresponding log slice abnormal pattern based on the type sequence of the logs contained in each abnormal log slice to obtain m log slice abnormal patterns; when the number m of the abnormal log slices is greater than N, perform clustering based on the similarity between the type sequences of the logs contained in the abnormal log slices to obtain N log slice abnormal patterns.

[0074] Therefore, the preset number is taken into consideration to distinctively determine the log sheet abnormality pattern, which improves the implementation convenience.

[0075] In one embodiment, the third determination module is configured to: use the type sequence of logs contained in the abnormal log piece as set elements to combine into a log piece abnormal pattern set; in the log piece abnormal pattern set, any two set elements with the greatest similarity are merged to update the log piece abnormal pattern set until the number of set elements in the log piece abnormal pattern set is equal to N.

[0076] It can be seen that by merging the sets to update the log slice exception pattern set, the type sequence of each exception log slice is preserved and the number of exception patterns is reduced.

[0077] In one embodiment, the third determination module is configured to: determine the longest common subsequence length of a first set of elements in a first abnormal log piece and a second set of elements in a second abnormal log piece, wherein the first abnormal log piece and the second abnormal log piece are any two different abnormal log pieces; and determine the similarity between the first set of elements in the first abnormal log piece and the second set of elements in the second abnormal log piece based on the number of logs in the first abnormal log piece, the number of logs in the second abnormal log piece, and the longest common subsequence length.

[0078] It can be seen that based on the longest common subsequence and hierarchical clustering, the abnormal patterns of log slices can be accurately mined.

[0079] In one embodiment, the comparison module is configured to: determine the similarity between the log type sequence in the log slice to be tested and each log slice anomaly pattern in the log slice anomaly pattern set;

[0080] The fourth determining module is configured to determine, in the log slice anomaly pattern set, a log slice anomaly pattern having the highest similarity with the log type sequence in the log slice to be tested and greater than a predetermined threshold value as the anomaly pattern of the log slice to be tested.

[0081] Therefore, based on the similarity comparison, the abnormal pattern analysis of the log files to be tested is achieved.

[0082] In one embodiment, the comparison module is configured to: when a log slice anomaly pattern in the log slice anomaly pattern set is a set element that has undergone set merging, determine the sub-similarity between the log type sequence contained in the log slice to be tested and each log type sequence participating in the set merging in the log slice anomaly pattern; and determine the weighted sum of the sub-similarity values ​​as the similarity between the log type sequence in the log slice to be tested and the log slice anomaly pattern.

[0083] It can be seen that by weighted summing the similarities of each log type sequence participating in the set merging, each log type sequence participating in the set merging is comprehensively considered, thereby improving the accuracy of the similarity calculation.

[0084] An electronic device, comprising:

[0085] processor;

[0086] a memory for storing executable instructions of the processor;

[0087] The processor is configured to read the executable instructions from the memory and execute the executable instructions to implement any one of the above methods for determining abnormal patterns of log slices.

[0088] A computer-readable storage medium stores computer instructions thereon, wherein when the computer instructions are executed by a processor, the method for determining an abnormal pattern of a log sheet as described in any one of the above items is implemented.

[0089] A computer program product comprises a computer program, wherein when the computer program is executed by a processor, the computer program implements the method for determining abnormal patterns of log slices as described in any one of the above. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, so that those skilled in the art will understand the above and other features and advantages of the present invention more clearly. In the accompanying drawings:

[0091] FIG1 is a flowchart of a method for determining an abnormal pattern of a log sheet according to an embodiment of the present invention.

[0092] FIG2 is an exemplary flow chart of a log grouping process according to an embodiment of the present invention.

[0093] FIG3 is an exemplary flowchart of a log slice preprocessing process according to an embodiment of the present invention.

[0094] FIG. 4 is an exemplary flow chart of a log slice analysis process according to an embodiment of the present invention.

[0095] FIG5 is an exemplary flow chart of determining an abnormal pattern of a log slice according to an embodiment of the present invention.

[0096] FIG6 is a structural diagram of an apparatus for determining an abnormal pattern of a log sheet according to an embodiment of the present invention.

[0097] FIG. 7 is a structural diagram of an electronic device according to an embodiment of the present invention.

[0098] The accompanying drawings are numerals as follows: DETAILED DESCRIPTION

[0099] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail with reference to the following examples.

[0100] For the sake of brevity and intuitiveness in description, the solution of the present invention is explained below by describing several representative implementations. A large number of details in the implementations are only used to help understand the solution of the present invention. However, it is obvious that the technical solution of the present invention may not be limited to these details when implemented. In order to avoid unnecessarily obscuring the solution of the present invention, some implementations are not described in detail, but only a framework is given. Hereinafter, "including" means "including but not limited to", and "according to..." means "at least according to..., but not limited to only according to...". Due to the language habits of Chinese, when the number of a component is not specifically specified below, it means that the component can be one or more, or can be understood as at least one.

[0101] The present invention proposes a method for determining abnormal patterns in log files based on clustering and principal component analysis, overcoming the shortcomings of low accuracy and high workload in manual log analysis. Furthermore, the present invention eliminates the need for labeling log data, enabling unsupervised log analysis and reducing the labeling workload. Furthermore, the sliding window-based log file partitioning employed by the present invention has a wider range of applicability compared to session-based log file partitioning. Furthermore, the present invention utilizes the longest common subsequence and hierarchical clustering to uncover deep-level abnormal patterns in log files, improving the efficiency of abnormal pattern recognition.

[0102] FIG1 is a flow chart of a method for determining an abnormal pattern of a log sheet according to an embodiment of the present invention. As shown in FIG1 , the method includes:

[0103] Step 101: Determine the type of the log based on the character string of the log contained in the log set.

[0104] A log set can contain a large number of logs. For example, a log set can be implemented as a log set obtained based on historical log data. For example, all logs within a predetermined historical time period (such as the past week, month, or year, etc.) can be combined into a log set. i It can usually be represented as m i ={m ih , m ic}, where m ih It is the log header, which generally contains structured information such as IP address and time. ic The specific content of the log is generally unstructured information, usually output by a statement similar to printf("user%s connected",user), such as "user bob connected" or "user bob disconnected", etc.

[0105] In one embodiment, step 101 specifically includes: extracting log content characterized as unstructured information from the log; dividing the log content into word sequences using spaces as delimiters; grouping the logs based on the length of the word sequences; for each group: determining the log string for each word sequence based on the characters at predetermined positions of each word in each word sequence contained in the group; clustering based on the similarity between any two log strings to obtain the type of logs contained in the group; and combining the types of the logs in all groups into a log type set. Preferably, the predetermined position is the first or last character of a word, or a character at any specified position.

[0106] For example: Use space as separator to separate m ic Divide into word sequences T i ,in Among them L i Indicates the length of the sequence, that is, the number of words, t ij Indicates the jth word in the i-th log. Example: When m ic When it is "user bob connected", T i =[user, bob, connected], L i Equal to 3 (i.e. there are 3 words). Divide all logs with the same word sequence length (length L) into their respective groups G L In, G L ={T i |L i =L}, thereby roughly classifying all logs based on their length.

[0107] FIG2 is an exemplary flow chart of a log grouping process according to an embodiment of the present invention. As shown in FIG2 , the method includes:

[0108] Step 201: Get a log set M, where M = {m1, m2, ..., m |M|}. Moreover, the i-th log m in M i Characterized by m i ={m ih , m ic}, where m ih It is the log header, which generally contains structured information such as IP address and time. ic The specific content of the log, generally characterized as unstructured information. The value range of i is [1, |M|].

[0109] Step 202: For each log in the log set M, that is, the i-th log m i :Use space as separator to divide the log content into word sequence T i ,in Among them L i Indicates the length of the sequence, that is, the number of words, t ij Represents the jth word in the i-th log.

[0110] Step 203: Divide word sequences with the same word sequence length L into the same group G L In other words, G L ={T i |L i =L}.

[0111] In addition, the concept of log string is introduced in the embodiment of the present invention. The log string is composed of T i The string of characters at any specified position (such as the first character) of each word in is recorded as where t′ ij Indicates t ij For example, when the specified position is the first character, the log string for "user bob connected" is "ubc." For each group, we can cluster the logs based on the similarity between the log strings of each word sequence within that group to determine the type of logs contained in that group. Then, we combine the log types contained in all groups into a log type set.

[0112] For example, assume that a log collection includes log 1, log 2, log 3, log 4, and log 5. The content of log 1 is "user bob connected," the content of log 2 is "user tom connected," the content of log 3 is "user bob disconnected," the content of log 4 is "user tom disconnected," and the content of log 5 is "client disconnected from the license server." The word sequence in log 1 is "user bob connected," and the length of the sequence is 3. The word sequence in log 2 is "user bob connected," and the length of the sequence is 3. The word sequence in log 3 is "user bob disconnected," and the length of the sequence is 3. The word sequence in log 4 is "user tom disconnected," and the length of the sequence is 3. The word sequence in log 5 is "client disconnected from the license server," and the length of the sequence is 5. Logs 1, 2, 3, and 4 have the same sequence length (all 3), so logs 1, 2, 3, and 4 are grouped together with a sequence length of 3. Log 5 is grouped together with a sequence length of 5. Moreover, the log string of log 1 is "ubc"; the log string of log 2 is "utc"; the log string of log 3 is "ubd"; the log string of log 4 is "utd"; and the log string of log 5 is "cdtls". Moreover, for each group, clustering is performed based on the similarity between any two log strings to obtain the types of logs contained in the group. For example, in a group with a sequence length of 3, 2 log types are clustered (referred to as log type a and log type b respectively). In a group with a sequence length of 5, since it only contains the log string of log 5, 1 log type is clustered (referred to as log type c). Then, the log type set determined based on this log set contains log type a, log type b, and log type c.

[0113] The process of clustering based on the similarity between any two log strings in each group involves a process of determining the similarity between any two log strings.

[0114] For example, any two log strings include a first log string and a second log string. The process of determining the similarity between any two log strings includes: determining the similarity between the first log string and the second log string, and between characters at the same character position; and determining the similarity between the first log string and the second log string based on the similarity between the characters at the same character position. The first log string and the second log string both belong to the same group, so the first log string and the second log string have the same number of characters, and the number of characters is usually plural. The number of character pairs formed by two characters at the same character position between the first log string and the second log string is plural, and the similarity between the characters at the same character position is plural. The similarity between the first log string and the second log string can be determined based on the weighted sum of all similarities between the characters at the same character position (the weight of the similarity between each character can be equal or unequal).

[0115] In a specific implementation of determining the similarity between log strings: any two log strings include the first log string T′ i and the second log string T′ j :The process of determining the similarity between any two log strings includes: determining T′ i and T′ j The similarity between sim(T′ i , T′ j );in t′ ik T′ i The kth character in t′ jk T′ j The kth character in ; L is T′ i and T′ i The number of characters; when t′ ik Equal to t′ ik When F(t′ ik , t′ jk ) is 1; when t′ ik Not equal to t′ jk When F(t′ jk , t′ jk ) is 0.

[0116] Specifically, for each group G corresponding to the length of the respective string sequence L , define any two log strings T′ i and T′ j The similarity is in When sim(T′ i , T′ j)≥λ, the two log strings are considered similar, where λ is the set similarity threshold, for example, λ=0.4. L Create a log string set in the group to store all types of logs in the group (each type can be represented by a log string), expressed as S L . For each G L Each log T in i Generate log string T′ i , and the G L The corresponding S L Calculate the similarity of all log strings in . L There exists a log string s and T′ i Similar, then the log T i Add to the log set G of this type s If it does not exist, create a new log collection And T′ i Join S L Then, all the grouped S corresponding to the length of each string sequence are L , combined into a log type set S, denoted as S = {s1, s2, ..., s |S|}. Among them, s1, s2, ..., s |S| They are implemented as log strings to represent the respective log types.

[0117] The above exemplary description is of a typical example of determining the similarity between log character strings. Those skilled in the art will appreciate that this description is merely exemplary and is not intended to limit the scope of protection of the embodiments of the present invention.

[0118] Step 102: determining an abnormal log slice from a plurality of log slices, wherein the plurality of log slices are obtained by partitioning the log set based on a sliding time window.

[0119] Here, the log slice preprocessing process can be performed in advance. In the log slice preprocessing process, the log set is divided based on the sliding time window to obtain multiple log slices. For example, the log set M is divided into m log slices W1, W2, ..., W using a sliding window with a length of 9 hours and a step size of 1 hour. m The logs contained in each log slice can be the same or different.

[0120] FIG3 is an exemplary flow chart of a log slice preprocessing process according to an embodiment of the present invention. As shown in FIG3 , the method includes:

[0121] Step 301: Divide the log set M into m log pieces W1, W2, ..., W using a sliding window with a predetermined length and step size. m .

[0122] Step 302: For each log slice W i , using ξ ij Characterized in this log sheet, the type is s j The number of times a log entry (of each type in S) appears.

[0123] Step 303: Record w i =(ξ i1 ,ξ i2 ,...,ξ i|S| ) is the vector of the log slice. Let n = |S|, then all w i A matrix A with m rows and n columns is formed.

[0124] In one embodiment, step 102 specifically includes: determining a log piece with an unbalanced log type ratio from a plurality of log pieces based on principal component analysis (PCA); and determining the log piece with an unbalanced log type ratio as an abnormal log piece.

[0125] In statistics, principal component analysis (PCA) is a technique for simplifying a dataset. It is a linear transformation. This transformation transforms the data into a new coordinate system such that the largest variance of any data projection is on the first coordinate (called the first principal component), the second largest variance is on the second coordinate (called the second principal component), and so on. PCA is often used to reduce the dimensionality of a dataset while retaining the features that contribute the most variance.

[0126] Specifically, for each log slice W i , use ξ ij Indicates that in this log file, the type is s j The number of times the log (of each type in S) appears. i =(ξ i1 ,ξ i2 ,...,ξ i|S| ) is the vector of the log slice. Let n = |S|, then all w i This constitutes a matrix A with m rows and n columns. Under normal circumstances, the proportions of various log types tend to be stable. If a significant imbalance in the proportions of various log types occurs within certain time periods (or log slices), it is highly likely that anomalies occurred within these time slices, and these log slices require special attention and analysis. Principal component analysis can be used to identify time periods with imbalances. This involves finding a low-dimensional space that minimizes the sum of distances after projecting data from the high-dimensional space onto the low-dimensional space.

[0127] FIG4 is an exemplary flow chart of a log slice analysis process according to an embodiment of the present invention. As shown in FIG4 , the method includes:

[0128] Step 401: Centralize matrix A to obtain matrix B.

[0129] Step 402: Solve for B T The eigenvalues ​​λi (i=1, 2, ..., n) and eigenvectors V1, V2, ..., V n .

[0130] Step 403: Select the variance ratio after dimensionality reduction as 90%, and calculate k so that

[0131] Step 404: The first k eigenvectors V1, V2, ..., V k Composition matrix P = [V1, V2, ..., V k ], calculate the orthogonal projection matrix of this matrix as V P =P(P T P) -1 P T =PP T .

[0132] Step 405: If the original vector is y, the Euclidean distance from the vector to its mapped subspace can be calculated by calculating the squared prediction error SPE = || y a || 2 Get, where y a is the projection of y onto the subspace, and y a =(IV P )y=(I-PP T )y.

[0133] Step 406: Calculate the distance from each point to the subspace according to the above algorithm and compare it with the detection threshold Q α Compare. That is to say, judge SPE=||y a ||>Q α Is it true? If so, go to step 407; otherwise, go to step 408.

[0134] Step 407: Mark point y as abnormal. α It represents the threshold statistic of the SPE residual function at the (1-α) confidence level, which can be expressed as Get, among them λ j Represents the eigenvalue of the jth principal component of the sample data covariance matrix projected on the subspace, C αRepresents the 1-α percentile of the standard normal distribution and exits this process.

[0135] Step 408: Determine that the log slice is normal, and exit this process.

[0136] At this point, we have been able to obtain normal and abnormal rows and their corresponding log slices.

[0137] The above exemplary description is of a typical example of determining an abnormal log piece from a plurality of log pieces. Those skilled in the art will appreciate that this description is merely exemplary and is not intended to limit the scope of protection of the embodiments of the present invention.

[0138] Step 103: Determine the log slice abnormality pattern based on the type of logs included in the abnormal log slice.

[0139] In one embodiment, step 103 specifically includes: determining the number N of log slice abnormal patterns; when the number m of abnormal log slices is less than or equal to N, determining the corresponding log slice abnormal pattern based on the type sequence of the logs contained in each abnormal log slice to obtain m log slice abnormal patterns; when the number m of abnormal log slices is greater than N, clustering based on the similarity between the type sequences of the logs contained in the abnormal log slices to obtain N log slice abnormal patterns.

[0140] Here, the number N of log slice anomaly patterns is predefined. When the number m of anomaly log slices is less than or equal to N, the type sequence of the logs contained in each anomaly log slice can be determined as the corresponding log slice anomaly pattern. A log slice anomaly pattern comprises the type sequence of the logs contained in the anomaly log slice, where each type in the type sequence can be represented by the log string corresponding to that type. When the number m of anomaly log slices is greater than N, clustering is performed based on the similarity between the type sequences of the logs contained in the anomaly log slices to obtain N log slice anomaly patterns.

[0141] In one embodiment, clustering is performed based on the similarity between type sequences of logs included in abnormal log slices to obtain N log slice abnormal patterns, including: taking the type sequences of logs included in the abnormal log slices as set elements to combine into a log slice abnormal pattern set; in the log slice abnormal pattern set, merging any two set elements with the greatest similarity to update the log slice abnormal pattern set, until the number of set elements in the log slice abnormal pattern set is equal to N.

[0142] Before merging any two set elements with the greatest similarity, a process of determining the similarity between any two set elements is involved.

[0143] Assuming that a first abnormal log piece and a second abnormal log piece are any two different abnormal log pieces, a process of determining the similarity between any two set elements includes: determining a longest common subsequence length of a first set element of the first abnormal log piece and a second set element of the second abnormal log piece, wherein the first abnormal log piece and the second abnormal log piece are any two different abnormal log pieces; and determining the similarity between the first set element of the first abnormal log piece and the second set element of the second abnormal log piece based on the number of logs in the first abnormal log piece, the number of logs in the second abnormal log piece, and the longest common subsequence length.

[0144] Assume that any two set elements include the first log string T′ i and the second log string T′ j In a specific implementation of determining the similarity between any two set elements: determining the first set element y of the first abnormal log piece i and the second set element y of the second abnormal log slice j The similarity d(y i ,y j );in t=LCS(y i ,y j );|y i | is the number of logs in the first abnormal log slice; |y j | is the number of logs in the second abnormal log slice; LCS(y i ,y j ) is the first set element y i and the second set element y j The longest common subsequence length; the first abnormal log piece and the second abnormal log piece are any two different abnormal log pieces.

[0145] Specifically, let the set of abnormal log piece type sequences Y = {y1, y2, ...}. Each element can be represented as a log type sequence contained in an abnormal log piece, such as y1 = [s6, s6, s3, s2, s 10 , s6, s3, s2, s3], y2 = [s6, s1, s5, s3, s2, s 13 , s 10 ], respectively, indicating that the logs in the type sequence y1 of the first abnormal log slice are the 6th, 6th, 3rd, 2nd, 10th, 6th, 3rd, 2nd, and 3rd logs, respectively, and the logs in the type sequence y2 of the second abnormal log slice are the 6th, 1st, 5th, 3rd, 2nd, 13th, and 10th logs, respectively. Each log type in the log type sequence can be represented by the log string corresponding to that log type. Define y i and yj The similarity is: the sum of the number of logs it contains |y i |+|y j |with the longest common subsequence length LCS(y) of the contained log-type sequence i ,y j ), that is, Where t = LCS (y i ,y j Here, we use the longest common subsequence (LCS) to measure the similarity between two log slices. This is because an ordered arrangement of log types can determine an anomalous log slice pattern. Hierarchical clustering, based on the LCS distance metric, yields different categories of anomalous log slices, or log slice anomaly patterns.

[0146] FIG5 is an exemplary flow chart of determining an abnormal pattern of a log slice according to an embodiment of the present invention. As shown in FIG5 , the method includes:

[0147] Step 501: Determine an abnormal log piece set Y = {y1, y2, ...}, where y1, y2, ... are respective abnormal log pieces.

[0148] Step 502: P←{{y1},{y2},{y3}...}; where “←” represents an assignment operation. P is a set of log slice exception patterns. {y1},{y2},{y3}..are the type sequences of the exception log slices y1, y2, y3...respectively. For example, y1=[s6,s6,s3,s2,s 10 , s6, s3, s2, s3], it means that the logs in the type sequence y1 are the 6th, 6th, 3rd, 2nd, 10th, 6th, 3rd, 2nd and 3rd logs respectively. Moreover, each type in the type sequence y1 (i.e. s6, s6, s3, s2, s3) 10 , any one of s6, s3, s2, s3) can be represented by a string that conforms to the log type.

[0149] Step 503: Determine whether |P| > N (where N is a preset value, such as 2), and |P| is the number of y. If so (corresponding to the "Y" branch), proceed to step 505; otherwise (corresponding to the "N" branch), proceed to step 504.

[0150] Step 504: Each set element in P is used as a log slice exception pattern, and the process ends.

[0151] Step 505: Select a, b from P so that Minimum, and perform P←P-{P i}-{P j}+{Pi ∪P j}.

[0152] Step 505 is repeated until |P| is less than or equal to N.

[0153] For example, assume that there are 10 abnormal log pieces, namely abnormal log piece 1 to daily log piece 10. The log type sequences of abnormal log piece 1 to daily log piece 10 are y1 to y 10 .

[0154] (1): When N is greater than or equal to 10, y1~y 10 10 log slice anomaly patterns are determined, and each log slice anomaly pattern contains a corresponding log type sequence. Each log type in the log type sequence can be represented by the log string corresponding to the log type. For example, assuming y1[s6, s6, s3, s2, s 10 , s6, s3, s2, s3], then the log slice anomaly pattern corresponding to y1 is [s6, s6, s3, s2, s 10 , s6, s3, s2, s3], where each log type in the log type sequence can be represented as a log string corresponding to the log type.

[0155] (2): When N is less than 10 (for example, N is equal to 8), the type sequences of the logs contained in the 10 abnormal log slices are respectively used as set elements to form a log slice abnormal pattern set. The log slice abnormal pattern set is {{y1}, {y2}, {y3}...{y 10 In the log slice anomaly pattern set, any two set elements with the greatest similarity are merged to update the log slice anomaly pattern set until the number of set elements in the log slice anomaly pattern set is equal to N. For example, when the similarity between y1 and y2 is the greatest, y1 and y2 are merged. Then the anomaly log slice set is changed to: {{{y1}, {y2}}, {y3}...{y 10}}, it can be seen that the number of elements in the set is reduced by one, and {{y1}, {y2}} becomes a set element in the set. Continue to merge any two set elements with the greatest similarity in the changed abnormal log piece set and continue to update the log piece abnormal pattern set until the final set elements are 8. Assume that the final log piece abnormal pattern set is: {{{{y1}, {y2}}, {y3}}}...{y 10}}. Among them, {{{y1}, {y2}}, {y3}}} corresponds to an abnormal pattern set.

[0156] Step 104: Compare the type of the log contained in the log slice to be tested with the log slice abnormality pattern.

[0157] Here, the log piece to be tested is a log piece for which an abnormal pattern needs to be determined, such as a log piece obtained in real time.

[0158] Step 105: Determine the abnormal pattern of the log sheet to be tested based on the comparison result.

[0159] In one embodiment, step 104 specifically includes determining the similarity between the log type sequence in the log segment to be tested and each log segment anomaly pattern in the log segment anomaly pattern set. Step 105 specifically includes determining the log segment anomaly pattern in the log segment anomaly pattern set that has the highest similarity to the log type sequence in the log segment to be tested and that is greater than a predetermined threshold as the anomaly pattern of the log segment to be tested. If the similarity of the log segment anomaly pattern with the highest similarity to the log type sequence in the log segment to be tested is not greater than the predetermined threshold, the log segment to be tested is determined to be normal.

[0160] Here, the log piece to be tested is obtained, and the unstructured information of each log in the log piece to be tested is divided into word sequences using spaces as delimiters. Then, based on each word sequence, the log string corresponding to the word sequence is determined (the character position is equivalent to the character position in step 203), and then each log string is compared with each type in the log type set for similarity (the similarity determination method can refer to the above calculation sim(T′) i , T′ j ) formula) to determine the type of each log, thereby obtaining the log type sequence in the log slice to be tested.

[0161] If the log type set does not contain a log type corresponding to the log piece to be tested, the log piece to be tested can be added to the log set in step 101, and the process shown in Figure 1 is repeated to update the abnormal log piece model set. Then, the updated abnormal log piece model set is applied and steps 104 and 105 are executed.

[0162] In one embodiment, when a log slice anomaly pattern in a log slice anomaly pattern set is a set element that has undergone set merging, the method further includes: determining a sub-similarity between a log type sequence contained in the log slice to be tested and each log type sequence in the log slice anomaly pattern that participates in the set merging; and determining a weighted sum of the sub-similarity values ​​as the similarity between the log type sequence in the log slice to be tested and the log slice anomaly pattern.

[0163] For example, suppose the log slice exception pattern set is: {{{{y1},{y2}},{y3}}}...{y 10}}{{{y1}, {y2}}, {y3}}}. {{{y1}, {y2}}, {y3}}} are the set elements after set merging. When calculating the similarity between the log type sequence in the log slice to be tested and {{{y1}, {y2}}, {y3}}}, the similarities (called sub-similarity) between the log type sequence in the log slice to be tested and {y1}, {y2}, and {y3} can be calculated separately to obtain three sub-similarity values. The weighted sum of these three sub-similarity values ​​(the weights can be set) is then used to determine the similarity between the log type sequence in the log slice to be tested and the log slice anomaly pattern.

[0164] FIG6 is a block diagram of an apparatus for determining an abnormal pattern of a log sheet according to an embodiment of the present invention. As shown in FIG6 , the apparatus 600 for determining an abnormal pattern of a log sheet includes:

[0165] A first determination module 601 is configured to determine the type of a log based on a character string of a log included in a log set;

[0166] A second determining module 602 is configured to determine an abnormal log slice from a plurality of log slices, wherein the plurality of log slices are obtained by partitioning the log set based on a sliding time window;

[0167] The third determining module 603 is configured to determine a log slice abnormality pattern based on the type of logs included in the abnormal log slice;

[0168] The comparison module 604 is configured to compare the type of log contained in the log piece to be tested with the log piece abnormality pattern;

[0169] The fourth determining module 605 is configured to determine an abnormal pattern of the log slice to be tested based on the comparison result.

[0170] In one embodiment, the first determination module 601 is configured to: extract log content characterized as unstructured information from the log; divide the log content into word sequences using spaces as delimiters; group the logs based on the length of the word sequences; for each group: determine the log string of each word sequence based on the characters at a predetermined position of each word in each word sequence contained in the group; cluster based on the similarity between any two log strings to obtain the type of log contained in the group; and combine the types of the logs in all groups into a log type set.

[0171] In one embodiment, any two log strings include a first log string and a second log string: the first determination module 601 is configured to: determine the similarity between the first log string and the second log string, and between characters at the same character position; and determine the similarity between the first log string and the second log string based on the similarity between the characters at the same character position.

[0172] In one embodiment, the third determination module 603 is configured to: determine the number N of log slice abnormal patterns; when the number m of abnormal log slices is less than or equal to N, determine the corresponding log slice abnormal pattern based on the type sequence of the logs contained in each abnormal log slice to obtain m log slice abnormal patterns; when the number m of abnormal log slices is greater than N, perform clustering based on the similarity between the type sequences of the logs contained in the abnormal log slices to obtain N log slice abnormal patterns.

[0173] In one embodiment, the third determination module 603 is configured to: use the type sequence of logs contained in the abnormal log piece as set elements to combine into a log piece abnormal pattern set; in the log piece abnormal pattern set, any two set elements with the greatest similarity are merged to update the log piece abnormal pattern set until the number of set elements in the log piece abnormal pattern set is equal to N.

[0174] In one embodiment, the third determination module 603 is configured to: determine the longest common subsequence length between the first set elements of the first abnormal log piece and the second set elements of the second abnormal log piece, where the first abnormal log piece and the second abnormal log piece are any two different abnormal log pieces; and determine the similarity between the first set elements of the first abnormal log piece and the second set elements of the second abnormal log piece based on the number of logs in the first abnormal log piece, the number of logs in the second abnormal log piece, and the longest common subsequence length.

[0175] In one embodiment, the comparison module 604 is configured to determine the similarity between the log type sequence in the log piece to be tested and each log piece abnormal pattern in the log piece abnormal pattern set; the fourth determination module 605 is configured to determine the log piece abnormal pattern in the log piece abnormal pattern set that has the highest similarity with the log type sequence in the log piece to be tested and is greater than a predetermined threshold value as the abnormal pattern of the log piece to be tested.

[0176] In one embodiment, the comparison module 604 is configured to: when a log slice anomaly pattern in the log slice anomaly pattern set is a set element that has undergone set merging, determine the sub-similarity between the log type sequence contained in the log slice to be tested and each log type sequence participating in the set merging in the log slice anomaly pattern; and determine the weighted sum of the sub-similarity values ​​as the similarity between the log type sequence in the log slice to be tested and the log slice anomaly pattern.

[0177] The embodiment of the present invention also proposes an electronic device with a processor-memory architecture. Figure 7 is a structural diagram of an electronic device according to an embodiment of the present invention. As shown in Figure 7, the electronic device 700 includes a processor 701, a memory 702, and a computer program stored on the memory 702 and executable on the processor 701. When the computer program is executed by the processor 701, it implements any of the above methods for determining abnormal modes of log slices. Among them, the memory 702 can be specifically implemented as a variety of storage media such as an electrically erasable programmable read-only memory (EEPROM), a flash memory (Flash memory), and a programmable read-only memory (PROM). The processor 701 can be implemented to include one or more central processing units or one or more field programmable gate arrays, wherein the field programmable gate array integrates one or more central processing unit cores. Specifically, the central processing unit or the central processing unit core can be implemented as a CPU, an MCU, or a DSP, etc.

[0178] It should be noted that not all steps and modules in the above processes and structure diagrams are required, and certain steps or modules can be omitted based on actual needs. The execution order of the steps is not fixed and can be adjusted as needed. The division of the modules is merely for the convenience of describing the functional division adopted. In actual implementation, a module can be implemented by multiple modules, and the functions of multiple modules can be implemented by the same module. These modules can be located in the same device or in different devices.

[0179] The hardware modules in each embodiment can be implemented mechanically or electronically. For example, a hardware module may include a specially designed permanent circuit or logic device (such as a dedicated processor, such as an FPGA or ASIC) for performing a specific operation. The hardware module may also include a programmable logic device or circuit (such as a general-purpose processor or other programmable processor) temporarily configured by software to perform a specific operation. As for whether to implement the hardware module mechanically, or using a dedicated permanent circuit, or using a temporarily configured circuit (such as configured by software), it can be decided based on cost and time considerations.

[0180] The above are only preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for determining an abnormal pattern of a log sheet, characterized in that, include: Determine (101) the type of the log based on the character string of the log included in the log set; Determining (102) an abnormal log slice from a plurality of log slices, wherein the plurality of log slices are obtained by partitioning the log set based on a sliding time window; Based on the type of logs included in the abnormal log piece, determining (103) a log piece abnormal mode; Comparing the type of log contained in the log sheet to be tested with the log sheet abnormality pattern (104); Based on the comparison result, the abnormal pattern of the log sheet to be tested is determined (105).

2. The method according to claim 1, characterized in that The determining (101) of the type of the log based on the character string of the log contained in the log set includes: extracting log content characterized as unstructured information from the log; Divide the log content into word sequences using spaces as delimiters; Grouping the logs based on the length of the word sequence; For each group: Determine a log string for each word sequence based on the characters at a predetermined position of each word in each word sequence included in the group; Clustering is performed based on the similarity between any two of the log character strings to obtain the type of logs included in the group; The types of all grouped logs are combined into a log type set.

3. The method according to any one of the preceding claims, characterized in that The predetermined position is the first character or the last character of the word.

4. The method according to any one of the preceding claims, characterized in that The arbitrary two log strings include a first log string and a second log string: the method further includes: Determine the similarity between the first log character string and the second log character string, and between characters at the same character position; Based on the similarity between the characters at the same character position, the similarity between the first log character string and the second log character string is determined.

5. The method according to any one of the preceding claims, characterized in that The determining (102) of an abnormal log piece from a plurality of log pieces comprises: determining, based on principal component analysis, log sheets with an unbalanced log type ratio from the plurality of log sheets; The log slice with an unbalanced log type ratio is determined as the abnormal log slice.

6. The method according to any one of the preceding claims, characterized in that The determining (103) of the log piece abnormality mode based on the type of logs included in the abnormal log piece includes: Determine the number N of abnormal patterns in the log slice; When the number m of the abnormal log slices is less than or equal to N, based on the type sequence of logs contained in each abnormal log slice, a corresponding log slice abnormal pattern is determined to obtain m log slice abnormal patterns; When the number m of the abnormal log pieces is greater than N, clustering is performed based on the similarity between the type sequences of the logs contained in the abnormal log pieces to obtain N log piece abnormal patterns.

7. The method according to any one of the preceding claims, characterized in that The clustering is performed based on the similarity between the type sequences of the logs contained in the abnormal log slices to obtain N log slice abnormal patterns, including: The type sequence of the logs included in the abnormal log piece is taken as a set element to form a log piece abnormal pattern set; In the log slice anomaly pattern set, any two set elements with the greatest similarity are set merged to update the log slice anomaly pattern set until the number of set elements of the log slice anomaly pattern set is equal to N.

8. The method according to any one of the preceding claims, characterized in that Also includes: Determine the longest common subsequence length of a first set element of a first abnormal log piece and a second set element of a second abnormal log piece, wherein the first abnormal log piece and the second abnormal log piece are any two different abnormal log pieces; Based on the number of logs in the first abnormal log piece, the number of logs in the second abnormal log piece, and the longest common subsequence length, the similarity between the first set element of the first abnormal log piece and the second set element of the second abnormal log piece is determined.

9. The method according to any one of the preceding claims, characterized in that The comparing the type of logs contained in the log piece to be tested with the log piece abnormal pattern (104) includes: determining the similarity between the log type sequence in the log piece to be tested and each log piece abnormal pattern in the log piece abnormal pattern set; The determining (105) of the abnormal pattern of the log piece to be tested based on the comparison result comprises: determining the log piece abnormal pattern in the log piece abnormal pattern set, which has the highest similarity with the log type sequence in the log piece to be tested and is greater than a predetermined threshold value, as the abnormal pattern of the log piece to be tested.

10. The method according to any one of the preceding claims, characterized in that When the log slice anomaly pattern in the log slice anomaly pattern set is a set element that has been merged, the method further includes: Determine a sub-similarity between a log type sequence included in the log piece to be tested and each log type sequence participating in the set merging in the log piece anomaly pattern; The weighted sum of the sub-similarity values ​​is determined as the similarity between the log type sequence in the log piece to be tested and the abnormal pattern of the log piece.

11. A device for determining abnormal patterns of log sheets, characterized in that include: A first determination module (601) is configured to determine the type of a log based on a character string of a log included in a log set; A second determination module (602) is configured to determine an abnormal log slice from a plurality of log slices, wherein the plurality of log slices are obtained by partitioning the log set based on a sliding time window; A third determination module (603) is configured to determine a log slice abnormality pattern based on the type of logs included in the abnormal log slice; A comparison module (604) is configured to compare the type of log contained in the log slice to be tested with the log slice abnormality pattern; The fourth determination module (605) is configured to determine the abnormal mode of the log sheet to be tested based on the comparison result.

12. The device according to claim 10, characterized in that The first determining module (601) is configured to: extracting log content characterized as unstructured information from the log; Divide the log content into word sequences using spaces as delimiters; Grouping the logs based on the length of the word sequence; For each group: Determine a log string for each word sequence based on the characters at a predetermined position of each word in each word sequence included in the group; Clustering is performed based on the similarity between any two of the log character strings to obtain the type of logs included in the group; The types of all grouped logs are combined into a log type set.

13. The device according to any one of the preceding claims, characterized in that The arbitrary two log character strings include a first log character string and a second log character string; The first determination module (601) is configured to: determine the similarity between the characters at the same character position between the first log string and the second log string; and determine the similarity between the first log string and the second log string based on the similarity between the characters at the same character position.

14. The device according to any one of the preceding claims, characterized in that The third determination module (603) is configured to: determine the number N of log slice abnormal patterns; when the number m of the abnormal log slices is less than or equal to N, determine the corresponding log slice abnormal pattern based on the type sequence of the logs contained in each abnormal log slice to obtain m log slice abnormal patterns; when the number m of the abnormal log slices is greater than N, perform clustering based on the similarity between the type sequences of the logs contained in the abnormal log slices to obtain N log slice abnormal patterns.

15. The device according to any one of the preceding claims, characterized in that The third determination module (603) is configured to: use the type sequence of the logs contained in the abnormal log piece as set elements to combine into a log piece abnormal pattern set; in the log piece abnormal pattern set, any two set elements with the greatest similarity are combined to update the log piece abnormal pattern set until the number of set elements of the log piece abnormal pattern set is equal to N.

16. The device according to any one of the preceding claims, characterized in that The third determination module (603) is configured to: determine the longest common subsequence length of the first set elements of the first abnormal log piece and the second set elements of the second abnormal log piece, wherein the first abnormal log piece and the second abnormal log piece are any two different abnormal log pieces; based on the number of logs in the first abnormal log piece, the number of logs in the second abnormal log piece and the longest common subsequence length, determine the similarity between the first set elements of the first abnormal log piece and the second set elements of the second abnormal log piece.

17. The device according to any one of the preceding claims, characterized in that The comparison module (604) is configured to: determine the similarity between the log type sequence in the log slice to be tested and each log slice abnormal pattern in the log slice abnormal pattern set; The fourth determination module (605) is configured to: determine the log slice abnormality pattern in the log slice abnormality pattern set, which has the highest similarity with the log type sequence in the log slice to be tested and is greater than a predetermined threshold value, as the abnormality pattern of the log slice to be tested.

18. The device according to any one of the preceding claims, characterized in that The comparison module (604) is configured to: when a log slice abnormality pattern in the log slice abnormality pattern set When it is a set element after set merging, determine the sub-similarity between the log type sequence contained in the log piece to be tested and each log type sequence participating in the set merging in the log piece anomaly pattern; and determine the weighted sum value of each sub-similarity as the similarity between the log type sequence in the log piece to be tested and the log piece anomaly pattern.

19. An electronic device, characterized in that: include: Processor (701); A memory (702), configured to store executable instructions of the processor (701); The processor (701) is configured to read the executable instructions from the memory (702), and execute the executable instructions to implement the method for determining an abnormal pattern of a log slice according to any one of claims 1 to 9.

20. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by a processor, the method for determining abnormal patterns of log sheets according to any one of claims 1 to 9 is implemented.

21. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the method for determining abnormal patterns of log sheets according to any one of claims 1 to 9.

Images