An online log parsing method, system, device and medium

Abstract

The present application relates to the technical field of information processing, and provides an online log analysis method, system, device and medium, comprising: obtaining log messages and corresponding log message lengths; grouping the log messages according to the log message lengths, grouping similar log messages into a category in the same group, counting the number of different marks at each position of the non-public sequence of the same category of log messages in the group, and performing intra-category grouping operation on the group if the number is less than a threshold value; after grouping, frequency analysis is used in each group to construct a log message template tree. The trivial dynamic variable marks are identified through regular expressions; similar log event groups are created by calculating log message similarity; frequency analysis is performed on the instances of the same group to distinguish constant and variable marks; the log message template tree is constructed to improve the template extraction efficiency of the log message. The original log data is analyzed, a log analysis method is given, and log templates in log data can be effectively parsed.

Description

Technical Field

[0001] This invention relates to the field of information processing technology, and in particular to an online log parsing method, system, device, and medium. Background Technology

[0002] Files used to record text information about hardware and software issues during real-time system operation are commonly referred to as system logs. These system logs can be used to monitor various events that occur during system operation, such as timestamps, requested IP addresses, or the status of abnormal tasks. System log analysis and mining techniques help system developers and maintainers better understand system behavior, as well as detect and locate system anomalies, detect abnormal transactions, and find security vulnerabilities during production.

[0003] As software systems become increasingly larger and more complex, system logs are also growing rapidly, with large software systems potentially generating up to 10GB of log data per hour. To analyze such large volumes of log data, fully automated log data analysis and mining techniques are now available, such as anomaly detection, behavioral analysis, program verification, and root cause analysis. Furthermore, with advancements in algorithms and computing power, more and more developers are choosing data mining algorithms for log mining.

[0004] However, because developers typically log system data in a simple and convenient way, system logs are usually unstructured text data. When using data mining algorithms, log parsing techniques are needed to process the system logs, structuring the raw log data, and then using the structured log data to train the data mining algorithm.

[0005] Log parsing technology plays a crucial role in log mining. Traditional log parsing algorithms include regular expression matching, source code-based methods, or rule-based methods. However, modern software systems iterate and evolve very rapidly, making it difficult to obtain the source code defining log messages, and requiring specialists to develop logging rules. These rules also need to be adjusted accordingly when the system changes, a time-consuming and labor-intensive task. Therefore, traditional log parsing algorithms are no longer suitable for today's software logging systems. Summary of the Invention

[0006] The purpose of this invention is to provide an online log parsing method, system, device, and medium to solve the problems that traditional technologies often have difficulty obtaining the source code that defines log messages, and require professionals to compile log rules. When the system changes, these rules also need to be adjusted accordingly, which is a time-consuming and labor-intensive task, resulting in the entire parsing process being time-consuming, labor-intensive, inefficient, and inaccurate.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides an online log parsing method, comprising:

[0009] Get each log message and the length of each log message;

[0010] Log messages are grouped into multiple groups based on their length. Similar log messages within the same group are grouped into one category. The non-common sequence positions of log messages of the same category in each group are determined, and the first number of log messages with different non-common sequence positions is counted. When the first number is less than a threshold, the log messages of the corresponding category in the corresponding group are grouped into categories.

[0011] After grouping, frequency analysis is used within each group to construct a log message template tree.

[0012] Optionally, retrieve the log message and its corresponding length:

[0013] Obtain the raw log message to be parsed; preprocess the raw log message using regular expressions to obtain the preprocessed log message and the log message length.

[0014] Optionally, obtain the raw log message to be parsed and preprocess it using regular expressions:

[0015] The log data to be parsed is processed according to the space delimiter to obtain the log message, which includes multiple elements arranged in sequence;

[0016] Regular expressions are used to define the log header information, which includes the log header timestamp, process ID, log level, log function, IP and MAC address, to obtain the preprocessed log message and its corresponding length.

[0017] Optionally, log messages can be grouped according to their length:

[0018] Log messages with the same length are grouped together. The length of a log message is the number of tags in the log message; where the length of a log message is defined as the number of tags in the log message.

[0019] Optionally, similar log messages within the same group can be grouped into one category, including: using an interactive self-attention Siamese network to extract feature values ​​of each log message within the same group;

[0020] The similarity between the feature values ​​of any two log messages is calculated using the Manhattan distance formula. When the similarity between the feature values ​​of two log messages is less than a predefined threshold, the two log messages are classified into the same category.

[0021] Optionally, the interactive self-attention Siamese neural network includes a Siamese neural network layer and an interactive self-attention layer; the interactive self-attention Siamese network is used to extract feature values ​​of each log message within the same group, including:

[0022] The Siamese neural network layer includes bi-RNN as a branch network, each branch network uses LSTM as a unit, captures more complex patterns through additional gating functions, and each network processes a log message;

[0023] Each log message is represented as a tag sequence x1, ..., x2. t The labeled sequence is passed to the branch network of the Siamese neural network layer. The labeled sequence is encoded in its original order to obtain the forward hidden features. The specific calculation process is as follows:

[0024] f t =σ(W f ·[h t-1 x t ]+b f )

[0025] i t =σ(W i ·[h t-1 x t ]+b i )

[0026]

[0027]

[0028] o t =σ(W o ·[h t-1 x t ]+b o )

[0029] h t =o t *tanh(C t )

[0030] Among them, W i W f W o ∈R d×d It is the weight matrix, b i b f b o ∈R d It is the bias that needs to be learned during the training process of the LSTM, x t ∈R d Here, is the input to time series t, d is the feature dimension of each labeled sequence, σ is the sigmoid function, · is the element-wise product, and C is the input to time series t.t It is a storage unit designed to reduce the risk of gradient vanishing. It is a temporary state of time sequence t, and the forget gate f t Used to reset memory cells, i t and o t These represent the input gate and the output gate, respectively, controlling the input and output of the memory cell; tanh is the activation function.

[0031] The hidden state at each labeled sequence index is updated using the above equation; the log message is represented by the final output of the Siamese neural network as follows:

[0032]

[0033]

[0034] in, and These represent the hidden information in the time sequence t of the two log messages, respectively;

[0035] and The message is passed to the interactive self-attention layer to calculate the interactive self-attention weights. The interactive self-attention layer extracts the semantics of the log message through self-attention and enhances the semantic representation through interactive attention.

[0036] Each input in the interactive self-attention network has a function defined as:

[0037]

[0038] V x =XW x

[0039]

[0040] V y =YW y

[0041] Among them W x and W y These represent the weight matrices of the two sequences, respectively. and It represents the average hidden state of all elements in sequences X and Y, and is the join operator;

[0042] Hidden information in the time sequence t of the two log messages and The attention weight vector is described as follows:

[0043] and

[0044] Among them, a i , β i , It is defined by the following formula:

[0045] q i ,k i ,v i =x i W

[0046]

[0047]

[0048]

[0049] The parameters in the formula are defined as follows:

[0050] d represents the dimensions q and k, and the weight matrix W represents the model's parameters, q i ,k i ,v i Let α be the vector representing the i-th label. i,j and These are the values ​​of the i-th and j-th labels obtained using element-wise multiplication and the softmax function. The output of self-attention is C = (c1, c2, ... cj). n ), where c i =α i v i , representing the enhanced semantic vector of each tag in the log message;

[0051] Remove from matrices α and β and Obtain the interactive self-attention weight vector;

[0052] will sequence and Multiplying by their respective attention weight vectors yields the output vector of the interactive self-attention layer, as described below:

[0053]

[0054]

[0055] Where C x and C y These are the feature values ​​of the log messages X and Y output by the interactive self-attention algorithm.

[0056] Optionally, the similarity between the feature values ​​of any two log messages can be calculated using the Manhattan distance formula, including calculating the similarity between the feature values ​​of any two log messages using the following formula.

[0057] Dis(C x C y )=exp(-||C x -C y ||1).

[0058] Optionally, count the number of different tags at each position in the non-common sequence of log messages of the same type in the group. If the number is less than a threshold, perform a class-based grouping operation on the group.

[0059] The type V marked at each non-public marker position in the statistical log message is determined as follows: when V exceeds a threshold, the corresponding non-public marker position is treated as a variable marker and no division operation is performed; when V does not exceed the threshold, the corresponding non-public marker position is treated as a constant marker and grouped.

[0060] Among them, non-public marker positions refer to marker positions where the number of types is not 1 at the same position.

[0061] Optionally, after grouping, frequency analysis can be used within each group to construct a log message template tree:

[0062] Count the number of constant markers and variable markers in each log message belonging to the same group;

[0063] The number of constant markers and the number of variable markers are compared with the log messages in the corresponding group, and the constant markers and variable markers of the corresponding log messages are adjusted according to the comparison results.

[0064] The constant markers in the adjusted log messages remain unchanged, while the variable markers are replaced with wildcards to obtain the log message template.

[0065] Optionally, the method further includes:

[0066] A fixed-depth tree structure model is used to store log message templates. In the tree structure model, the root node is the message length of the log message template, and the child nodes of the message length are the log message templates.

[0067] Retrieve new log messages and determine the target message length for the new log messages.

[0068] Determine whether the log message templates stored in the tree structure model have a log message template with the same message length as the target message;

[0069] If no template is available, a new log template is built using the new log message and stored in the tree structure model.

[0070] Optionally, the number of constant markers and the number of variable markers are compared with the log messages in the corresponding group, and the constant markers and variable markers of the corresponding log messages are adjusted according to the comparison results. This includes treating the constant markers or variable markers in the log messages as constant markers if the number of constant markers or variable markers in the log messages is not less than the total number of log messages in the corresponding group.

[0071] Secondly, the present invention provides an online log parsing system, comprising:

[0072] The data acquisition module is used to obtain log messages and their corresponding log message lengths;

[0073] The data grouping module is used to group log messages according to their length, classify similar log messages within the same group, count the number of different markers at each position of the non-common sequence of log messages of the same class in the group, and perform class grouping operation on the group if the number is less than the threshold.

[0074] The output module is used to perform frequency analysis within each group after grouping, and to build a log message template tree.

[0075] Optionally, retrieve the log message and its corresponding length:

[0076] Obtain the raw log message to be parsed; preprocess the raw log message using regular expressions, as follows:

[0077] Obtain the raw log message to be parsed, and preprocess the raw log message using regular expressions:

[0078] The log data to be parsed is processed according to the space delimiter to obtain the log message, which includes multiple elements arranged in sequence;

[0079] Regular expressions are used to define the log header information, which includes the log header timestamp, process ID, log level, log function, IP and MAC address, to obtain the preprocessed log message and its corresponding length.

[0080] Thirdly, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of an online log parsing method.

[0081] Thirdly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of an online log parsing method.

[0082] Compared with the prior art, the present invention has the following technical effects:

[0083] This invention preprocesses the acquired log messages using regular expressions, initially groups them based on log message length, creates similar log message groups by applying an interactive self-attention Siamese network to calculate similarity, further groups them by comparing the number of non-common tags with a preset threshold, and performs frequency analysis on tags within the same group to distinguish between constant and variable tags. This allows for the high-precision extraction of log templates that can be used to identify and construct log events. The method of this invention can create an effective and efficient general-purpose log parser, eliminating the need to develop specific parsers for various types of log files. Furthermore, the method of this invention is more accurate than traditional log parsers on representative log datasets. Attached Figure Description

[0084] Figure 1 This is a flowchart of the present invention.

[0085] Figure 2 This is a system structure diagram of the present invention.

[0086] Figure 3 This is a flowchart of the log parsing method provided by the present invention.

[0087] Figure 4 This is a schematic diagram of the interactive self-attention twin neural network provided by the present invention.

[0088] Figure 5 This is a structural diagram of the message template tree provided by the present invention.

[0089] Figure 6 This is a schematic diagram of the original log sequence and log template provided by the present invention.

[0090] Figure 7 This is a schematic diagram of the log structure provided by the present invention. Detailed Implementation

[0091] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0092] Some basic concepts of this invention:

[0093] Log events: Define an event log as a text record of events that occur within the system or as a trace of an application.

[0094] Log messages: Log messages are text sequences that record only the specific event information, excluding information such as timestamps, types, and tags. An event typically contains more than one message; it also includes other information such as time, source, and function.

[0095] Tags: Independent words separated by delimiters in the message section are tags. Generally, the delimiter is a space.

[0096] Log message length: The number of markers in the message section.

[0097] Log message template / event type: The message template refers to the format of the message field in the event log generated by the same print statement, representing the event type. It may consist of fixed and variable tags.

[0098] Log parsing: The process of converting unstructured entries in log messages into structured event types.

[0099] This invention provides an online log parsing method that uses an interactive self-attention Siamese network to calculate the relevance of log messages, extracts message templates from the raw log data, and stores the log templates using a fixed-depth tree structure model. This method can efficiently and accurately extract log templates from unstructured logs, enabling rapid parsing of new log messages and providing system administrators with intuitive summaries and standardized structured data for further analysis and processing.

[0100] Example 1:

[0101] like Figure 1 As shown, an online log parsing method includes:

[0102] S1, obtain each log message and the log message length of each log message;

[0103] S2, group each log message according to the log message length to obtain multiple groups, group similar log messages in the same group into one category, determine the non-common sequence position of each log message in the same category in each group, and count the first number of log messages with different non-common sequence positions. When the first number is less than the threshold, perform class grouping operation on the log messages of the corresponding category in the corresponding group.

[0104] S3, after grouping, uses frequency analysis within each group to build a log message template tree.

[0105] Example 2

[0106] More specifically, such as Figure 3 As shown, this invention provides an online log parsing method, comprising the following steps:

[0107] Step 1: Collect raw log messages from the system with parsing and preprocess the log messages using simple regular expressions;

[0108] like Figure 7 As shown, in log parsing, each log entry can be represented as a log header and a log content. The log header includes trivial variable information such as timestamp, location, log type, and security, while log parsing mainly processes the log content.

[0109] When preprocessing logs, the log parser requires the user to provide simple regular expressions based on domain knowledge representing commonly used variables, such as IP addresses and blk IDs;

[0110] Then the tags matched from the original log message by these regular expressions are replaced. For example, the blk ID will be matched by "blk[O-9]+" and replaced with <*> (i.e., variable tags). The regular expressions used are usually very simple and are used to match common variable tags rather than log messages.

[0111] Step 2: Divide the log messages into different log groups based on their length;

[0112] The preprocessed log messages are divided into different log groups based on their length.

[0113] Log message length is defined as the number of markers in a log message;

[0114] This is based on the assumption that log messages with the same log event may have the same log message length.

[0115] like Figure 6 As shown, during the log parsing process, the constant markers for each log entry remain unchanged, while the variable markers are replaced by wildcards <*>.

[0116] First, the relevant terms in the embodiments of the present invention will be explained:

[0117] Log events: Log events refer to text records of events that occur within the system or traces of applications;

[0118] Log messages: Log messages are text sequences that record only the specific event information, excluding information such as timestamps, types, and tags. An event usually contains more than one message, including other information such as date, source, and tags. For message template extraction, only the message portion of the log event is relevant.

[0119] Log message length: The number of markers in the log message section;

[0120] Tags: Individual words separated by delimiters in the message section are tags. Generally, the delimiter is a space;

[0121] Constant markers: Constant markers are markers in the message part of a log event and are represented by static text in the message template. They are used to describe the function of the current code or the reason for triggering the log printing statement.

[0122] Variable tags: Variable tags are tags that appear in the message portion of a log event and are represented by wildcards in the message template. They reflect dynamic data during system runtime, such as IP addresses and parameter information.

[0123] Step 3: After obtaining the log groups after the above steps, use an interactive self-attention Siamese network to classify similar log messages in the same group into one category.

[0124] An interactive self-attention Siamese neural network consists of a Siamese neural network layer and an interactive self-attention layer;

[0125] The Siamese neural network includes a bi-RNN as a branch network, each branch network uses LSTM as a unit, and captures more complex patterns through additional gating functions. Each network processes a given sentence.

[0126] like Figure 4 Each log is represented as a labeled sequence, which is then passed to the Siamese neural network. The labeled sequence is encoded in its original order to obtain the forward hidden features. The specific calculation process is as follows:

[0127] f t =σ(W f ·[h t-1 x t ]+b f )

[0128] i t =σ(W i ·[h t-1 x t ]+b i )

[0129]

[0130]

[0131] o t =σ(w o ·[h t-1 x t ]+b o )

[0132] h t =ot *tanh(C t )

[0133] Among them, W i W f w o ∈R d×d It is the weight matrix, b i b f b o ∈R d It is the bias that needs to be learned during the training process of the LSTM, x t ∈R d , is the input of time series t, d is the feature dimension of each label, σ is the sigmoid function, · is the element-wise product, and Ct is a storage unit designed to reduce the risk of gradient vanishing / exploding. It is a temporary state of time sequence t, and the forget gate f t Used to reset memory cells, i t and o t These represent the input gate and the output gate, respectively, controlling the input and output of the memory cell; tanh is the activation function.

[0134] The hidden state at each sequence index is updated using the above equation; the log message is represented by the final output of the Siamese neural network as follows:

[0135]

[0136]

[0137] in, and These represent the hidden information in the time sequence t of the two log messages, respectively;

[0138] Then, and The weights are passed to the interactive self-attention layer to calculate the interactive self-attention weights. The interactive self-attention layer extracts sentence semantics through self-attention and enhances the semantic representation through interactive attention.

[0139] Since there are two inputs, each input has a function in the interactive self-attention network, defined as:

[0140]

[0141] V x =XW x

[0142]

[0143] V y =YW y

[0144] Among them W x and W y These represent the weight matrices of the two sequences, respectively. Furthermore, and It represents the average hidden state of all elements in sequences X and Y, and is the concatenation operator.

[0145] Two sequences and The attention weight vector is described as follows:

[0146] and

[0147] Among them, a i , β i , It is defined by the following formula:

[0148] q i k i v i =x i W

[0149]

[0150]

[0151]

[0152] The parameters in the formula are defined as follows:

[0153] d represents the dimensions q and k, and the weight matrix W represents the model's parameters, q i k i v i Let α be the vector representing the i-th label. i,j and These are the values ​​of the i-th and j-th labels obtained using element-wise multiplication and the softmax function. The output of self-attention is C = (c1, c2, ... cj). n ), where c i =α i v i , representing the enhanced semantic vector of each tag in the log message;

[0154] Then, remove from matrices α and β and Obtain the interactive self-attention weight vector;

[0155] Finally, the sequence and Multiplying by their respective attention weight vectors yields the output vector of the interactive self-attention layer, as described below:

[0156]

[0157]

[0158] Where C x and C y It is the output of interactive self-attention.

[0159] Substitute the output vector into the Manhattan distance formula to calculate the similarity, as follows:

[0160] Dis(C x C y )=exp(-||C x -C y ||1)

[0161] For two different log messages, their similarity is first calculated. If it is less than a threshold, they are considered to be related.

[0162] Using the similarity calculation method described above, similar log messages in each log group are clustered into one category.

[0163] Step 4: Within the same category, count the number of distinct words at each position in the non-common part of the log messages and compare it with a preset threshold. If the number is greater than the preset threshold, no further segmentation is performed; if the number is less than the preset threshold, a segmentation is performed.

[0164] The main idea of ​​this step is that, for a log group that has been clustered by similarity, each log message is divided into M positions. At each position, we count how many different label sequences there are, denoted by V. Generally speaking, the larger the value of V, the less likely that the position is to be a label constant. By introducing a threshold, when V exceeds the threshold, we consider this position to be a label variable and no longer perform the division operation. When V does not exceed the threshold, the position is a label constant, and we divide this class into V groups.

[0165] For each log class, count the number of distinct markers at the position of each non-common sequence, and compare them with a threshold. If the number of markers does not exceed the threshold, group the logs.

[0166] For example, the following set of log messages after step 3:

[0167] PacketResponder 1for block blk_38865049064139660terminating

[0168] PacketResponder 0for block blk_-6952295868656571terminating

[0169] PacketResponder 0for block blk_82291938049955061terminating

[0170] PacketResponder 0for block blk_-6670958628987959terminating

[0171] PacketResponder 1for block blk_57249283957299681terminating

[0172] PacketResponder 1for block blk_15149281956729935terminating

[0173] The log message length is 6. The logs are divided into positions 1 to 6, with different markers at positions 2 and 5. The count shows 4 different markers at position 2 and 6 different markers at position 5. Assuming the threshold is half the number of log messages, which is 3, since the number of different markers at position 2 is less than the assumed threshold, grouping is performed directly without further comparison. The grouping results are as follows:

[0174] Group 1:

[0175] PacketResponder 1for block blk_38865049064139660terminating

[0176] PacketResponder 1for block blk_57249283957299681terminating

[0177] PacketResponder 1for block blk_15149281956729935terminating

[0178] Group 2:

[0179] PacketResponder 0for block blk_-6952295868656571terminating

[0180] PacketResponder 0for block blk_82291938049955061terminating

[0181] PacketResponder 0for block blk_-6670958628987959terminating

[0182] Step 5: Use frequency analysis within each group to obtain log templates.

[0183] In this step, the occurrence of tags in each group of log events is analyzed by counting the number of times each tag appears in log events belonging to the same group;

[0184] Constant markers should appear in every log event within the same group, while variable markers are expected to appear only in some log events, not all log events.

[0185] Therefore, any event with a frequency below the maximum is considered a variable marker. Specifically, a set of log messages is as follows:

[0186] PacketResponder 1for block blk_38865049064139660terminating

[0187] PacketResponder 0for block blk_-6952295868656571terminating

[0188] PacketResponder 2for block blk_82291938049955061terminating

[0189] PacketResponder 2for block blk_-6670958628987959terminating

[0190] Frequency analysis shows that the labels PacketResponder, for, block, and terminating appeared four times in this group (the highest frequency), while other labels appeared less than four times. Therefore, they are classified as variable labels.

[0191] The log template obtained by applying frequency analysis to this set of log messages is:

[0192] PacketResponder<*>forblock<*>terminating

[0193] Step 6: Construct the log message template tree

[0194] After obtaining all message templates, a tree structure model is constructed to store the message templates / event types extracted from the log dataset. The structure of the message template tree is as follows: Figure 5 As shown, the child node of the root node is the message length, and the child node of the message length, which is also the leaf node of the tree, is the log message template and the log ID.

[0195] This invention utilizes an interactive self-attention Siamese neural network to learn the semantic information of log messages and provide similarity scores. Furthermore, it incorporates frequency analysis techniques, highlighting the robustness of this invention in parsing log messages.

[0196] This invention combines interactive self-attention twin neural networks and frequency analysis techniques to effectively parse large log files. It uses regular expressions to identify trivial dynamic variable markers, creates similar log event groups by calculating log message similarity, performs frequency analysis on instances within the same group to distinguish between constants and variable markers, and improves the efficiency of log message template extraction by constructing a log message template tree. Based on the main characteristics of system log data, combined with the specific needs of the system, and after analyzing the original log data, this invention presents a log parsing method that can effectively extract log templates from log data.

[0197] Example 3

[0198] Please see Figure 2 As shown, the present invention provides an online log parsing system, comprising:

[0199] The data acquisition module is used to obtain log messages and their corresponding log message lengths;

[0200] The data grouping module is used to group log messages according to their length, classify similar log messages within the same group, count the number of different markers at each position of the non-common sequence of log messages of the same class in the group, and perform class grouping operation on the group if the number is less than the threshold.

[0201] The output module is used to perform frequency analysis within each group after grouping, and to build a log message template tree.

[0202] It should be noted that the specific processing procedures for online log parsing by each functional module of an online log parsing system are the same as those in steps S1-S3 of Embodiment 1, and will not be repeated here.

[0203] The module division in this embodiment of the invention is illustrative and represents only one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of the invention can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0204] In another embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions from the computer storage medium to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used for the operation of an online log parsing method.

[0205] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the online log parsing method in the above embodiments.

[0206] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0207] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0208] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0209] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0210] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. An online log parsing method, characterized in that, include: Get each log message and the length of each log message; Log messages are grouped into multiple groups based on their length. Similar log messages within the same group are grouped into one category. The non-common sequence positions of log messages of the same category in each group are determined, and the first number of log messages with different non-common sequence positions is counted. When the first number is less than a threshold, the log messages of the corresponding category in the corresponding group are grouped into categories. After grouping, frequency analysis is used within each group to construct a log message template tree; Retrieve log messages and their corresponding lengths: Obtain the raw log message to be parsed; preprocess the raw log message using regular expressions to obtain the preprocessed log message and the log message length of the log message; Obtain the raw log message to be parsed, and preprocess the raw log message using regular expressions: The log data to be parsed is processed according to the space delimiter to obtain the log message, which includes multiple elements arranged in sequence; Regular expressions are used to define the log header information, which includes the log header timestamp, process ID, log level, log function, IP and MAC address, to obtain the preprocessed log message and the corresponding log message length. Group log messages according to their length: Log messages with the same length are grouped together, and the length of a log message is equal to the number of tags in the log message. The log message length is defined as the number of log messages that are marked. Grouping similar log messages within the same group into one category includes: using an interactive self-attention Siamese network to extract feature values ​​of each log message within the same group; The similarity between the feature values ​​of any two log messages is calculated using the Manhattan distance formula. When the similarity between the feature values ​​of two log messages is less than a predefined threshold, the two log messages are classified into the same category. If the number of distinct markers at each position in the non-common sequence of log messages of the same type in this group is less than a threshold, then the group is regrouped within the same class. The type V marked at each non-public marker position in the statistical log message is determined as follows: when V exceeds a threshold, the corresponding non-public marker position is treated as a variable marker and no division operation is performed; when V does not exceed the threshold, the corresponding non-public marker position is treated as a constant marker and grouped. Among them, non-public marker positions refer to marker positions where the number of types is not 1 at the same position; After grouping, frequency analysis is used within each group to construct a log message template tree: Count the number of constant markers and variable markers in each log message belonging to the same group; The number of constant markers and the number of variable markers are compared with the log messages in the corresponding group, and the constant markers and variable markers of the corresponding log messages are adjusted according to the comparison results. The constant markers in the log message after adjustment remain unchanged, and the variable markers are replaced with wildcards to obtain the log message template; The method further includes: A fixed-depth tree structure model is used to store log message templates. In the tree structure model, the root node is the message length of the log message template, and the child nodes of the message length are the log message templates. Retrieve new log messages and determine the target message length for the new log messages. Determine whether the log message templates stored in the tree structure model have a log message template with the same message length as the target message; If no template is available, a new log template is built using the new log message and stored in the tree structure model.

2. The online log parsing method according to claim 1, characterized in that, The interactive self-attention Siamese neural network consists of a Siamese neural network layer and an interactive self-attention layer; the interactive self-attention Siamese network is used to extract feature values ​​of log messages within the same group, including: The Siamese neural network layer includes bi-RNN as a branch network, each branch network uses LSTM as a unit, captures more complex patterns through additional gating functions, and each network processes a log message; Each log message is represented as a sequence of tags. The labeled sequence is passed to the branch network of the Siamese neural network layer. The labeled sequence is encoded in its original order to obtain the forward hidden features. The specific calculation process is as follows: in, , , It is a weight matrix. , , It is the bias that the LSTM needs to learn during the training process. The input is time series t, and d is the feature dimension of each labeled sequence. It is the sigmoid function, and • represents element-wise multiplication. It is a storage unit designed to reduce the risk of gradient vanishing. It is a temporary state of time sequence t, the forget gate. Used to reset memory cells and These represent the input gate and the output gate, respectively, controlling the input and output of the memory cell; tanh is the activation function. The hidden state at each labeled sequence index is updated using the above equation; the log message is represented by the final output of the Siamese neural network as follows: in, and These represent the hidden information in the time sequence t of the two log messages, respectively; and The message is passed to the interactive self-attention layer to calculate the interactive self-attention weights. The interactive self-attention layer extracts the semantics of the log message through self-attention and enhances the semantic representation through interactive attention. Each input in the interactive self-attention network has a function defined as: in and These represent the weight matrices of the two sequences, respectively. and It represents the average hidden state of all elements in sequences X and Y, and is the concatenation operator; Hidden information in the time sequence t of the two log messages and The attention weight vector is described as follows: and in, It is defined by the following formula: The parameters in the formula are defined as follows: yes and Dimensions, weight matrix These are the parameters of the model. , The vector representing the i-th label. and The values ​​of the i-th and j-th labels are obtained by using element-wise multiplication and the softmax function. The output of self-attention is... ,in , representing the enhanced semantic vector of each tag in the log message; Remove from matrices α and β and Obtain the interactive self-attention weight vector; will sequence and Multiplying by their respective attention weight vectors yields the output vector of the interactive self-attention layer, as described below: in and These are the feature values ​​of the log messages X and Y output by the interactive self-attention algorithm.

3. The online log parsing method according to claim 2, characterized in that, The similarity between the feature values ​​of any two log messages is calculated using the Manhattan distance formula, including the following formula. 。 4. The online log parsing method according to claim 1, characterized in that, The number of constant markers and the number of variable markers are compared with the log messages in the corresponding group. The constant markers and variable markers of the corresponding log messages are adjusted according to the comparison results. For example, if the number of constant markers or variable markers in the log message is not less than the total number of log messages in the corresponding group, then the constant markers or variable markers in the log message are regarded as constant markers.

5. An online log parsing system that implements the online log parsing method as described in claim 1, characterized in that, include: The data acquisition module is used to obtain log messages and their corresponding lengths. The data grouping module is used to group log messages according to their length. Within the same group, similar log messages are grouped into one category. The number of different markers at each position of the non-common sequence of log messages of the same category in the group is counted. If the number is less than a threshold, the group is grouped into categories. The output module is used to perform frequency analysis within each group after grouping, and to build a log message template tree.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the online log parsing method as described in any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the online log parsing method as described in any one of claims 1 to 4.

Images