Log processing method, apparatus, device, and storage medium

Abstract

The application discloses a log processing method, device and equipment and a storage medium. The method comprises the following steps: intercepting log header data corresponding to unstructured log data; generating a structure analysis formula corresponding to the log header data based on a target conversion rule and the log header data; and performing structure conversion on the unstructured log data based on the structure analysis formula to generate structured log data. In this way, the structure conversion of the unstructured log data is realized through the structure analysis formula of the log header data, the structured log data is generated, the method can be applied to logs of various different sources, the influence of strong assumptions on the log input order is avoided, and the log processing efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of data processing, and more particularly to a log processing method, apparatus, device, and storage medium. Background Technology

[0002] In the IT operations and maintenance field, logs are semi-structured text generated by the operating system or other software systems according to a certain template. Related technologies often require structuring various log message headers to facilitate further log analysis.

[0003] Generally, the following two schemes are mainly adopted: Scheme (1) The scheme of manually writing regular expressions to perform structured parsing of logs. The scheme of manually writing regular expressions refers to using the syntax rules of regular expressions to manually write patterns for matching and parsing log message headers. Scheme (2) The scheme of using the Spell algorithm to perform structured parsing of logs. Specifically, it is based on LCS (Longest Common Subsequence) to parse unstructured log messages into structured log types and parameters in an online streaming manner.

[0004] However, scheme (1) requires manual processing of logs from each source, and when a new source of logs appears, it cannot automatically generate a structured parsing expression for the logs from that source. The spell algorithm output of scheme (2) is affected by the order of the input logs. If the input order changes, it cannot obtain a stable result, resulting in low processing efficiency. Summary of the Invention

[0005] In view of this, embodiments of this application provide a log processing method, apparatus, device, and storage medium, aiming to improve log processing efficiency.

[0006] The technical solution of this application embodiment is implemented as follows:

[0007] In a first aspect, embodiments of this application provide a log processing method, the method comprising:

[0008] Extract the log header data corresponding to the unstructured log data;

[0009] Based on the target transformation rules and the log header data, generate the structural parsing expression corresponding to the log header data;

[0010] The unstructured log data is transformed based on the structure parsing expression to generate structured log data.

[0011] In some embodiments, the unstructured log data includes at least one log record, and the step of extracting the log header data corresponding to the unstructured log data includes:

[0012] For each log record in the at least one log record, preprocessing is performed to determine the first set of data blocks corresponding to the log record;

[0013] The first data block set is parsed based on a pattern parsing algorithm to determine the log pattern corresponding to the log record;

[0014] Based on the log pattern corresponding to each log record, determine the log pattern set corresponding to the at least one log record;

[0015] Based on the first data block set corresponding to each log record, a second data block set corresponding to the at least one log record is determined, wherein the second data block set includes at least one first data block set;

[0016] Based on the log pattern set, the second data block set, and the truncation rules, the at least one log record is truncated to generate the log header data corresponding to the at least one log record.

[0017] In some embodiments, the step of truncating at least one log record based on the log pattern set, the second data block set, and the truncation rule, and generating log header data corresponding to the at least one log record, includes:

[0018] Based on the log pattern set and the second data block set, determine the data to be extracted corresponding to the at least one log record;

[0019] Based on the truncation rules, the data to be truncated is truncated to generate log header data corresponding to the at least one log record.

[0020] In some embodiments, determining the data to be extracted corresponding to the at least one log record based on the log pattern set and the second data block set includes:

[0021] For each log pattern in the log pattern set, based on the mapping relationship between the log pattern and at least one log record and the second data block set, a third data block set corresponding to the log pattern is generated, and each third data block in the third data block set is a second data block corresponding to at least one log record of the log pattern.

[0022] Based on the location information set of the third data block set and the second data block set, a first location information set is determined, wherein the first location information is the location information corresponding to each third data block in the third data block set;

[0023] Based on each of the first location information sets, a log mode location information set is generated, wherein each log mode location information in the log mode location information set is the location information of the third data block set corresponding to the log mode;

[0024] Based on the set of log patterns and the mapping relationship between the log patterns and at least one log record, the target log pattern is determined;

[0025] Based on the target log pattern and the log pattern location information set, determine the data to be extracted corresponding to the at least one log record.

[0026] In some embodiments, determining the target log pattern based on the log pattern set and the mapping relationship between the log pattern and at least one log record includes:

[0027] Based on the log pattern set and the mapping relationship between the log pattern and at least one log record, determine the number of log records corresponding to at least one log pattern in the log pattern set;

[0028] Based on the number of log records corresponding to the at least one log pattern and a preset number threshold, the log pattern in the log pattern set that satisfies the preset number threshold is determined as the target log pattern.

[0029] In some embodiments, determining the data to be extracted corresponding to the at least one log record based on the target log pattern and the at least one log pattern location information set includes:

[0030] Based on the target log pattern and the at least one log pattern location information set, determine the target log pattern location information set;

[0031] For each other log pattern location information set in at least one other log pattern location information set and the target log pattern location information set, a fourth data block set is determined, wherein the fourth data block is a third data block in the other log pattern location information set and the target log pattern location information set with the same location information; the other log pattern location information set is any log pattern location information set in the at least one log pattern location set other than the target log pattern set.

[0032] Based on each of the fourth data block sets, initial truncated data is generated;

[0033] The initial data to be extracted is sorted according to the sorting rules to generate the data to be extracted.

[0034] In some embodiments, the truncation rule includes a truncation length and a log header data type. The step of truncating the data to be truncated based on the truncation rule to generate log header data corresponding to at least one log record includes:

[0035] Based on the specified truncation length, the data to be truncated is truncated to generate initial log header data;

[0036] Determine whether the initial log header data contains the data type of the log header; if so, determine that the initial log header data is the log header data.

[0037] If not, adjust the truncation length until the initial log header data contains the data type of the log header, and then generate the log header data.

[0038] In some embodiments, the preprocessing of each log record in the at least one log record to determine the first set of data blocks corresponding to the log record includes:

[0039] For each log record in the at least one log record, the log record is split according to the splitting rules to generate a set of split data blocks;

[0040] Based on the replacement rules, each segmented data block in the segmented data block set is replaced to generate the first data block set corresponding to the log record.

[0041] In some embodiments, the step of performing structural transformation on the unstructured log data based on the structural parsing expression to generate structured log data includes:

[0042] If not all the unstructured log data is converted into structured log data, then the successfully converted structured logs and the corresponding log patterns are deleted. The unstructured log data that failed to be converted is converted again, and the structure parsing is updated, until all the unstructured log data that failed to be converted is converted and the structured data is generated.

[0043] Secondly, embodiments of this application provide a log processing apparatus, the apparatus comprising:

[0044] The extraction module is used to extract the log header data corresponding to unstructured log data;

[0045] The generation module is used to generate the structural parsing expression corresponding to the log header data based on the target transformation rules and the log header data;

[0046] The transformation module is used to perform structural transformation on the unstructured log data based on the structure parsing expression to generate structured log data.

[0047] Thirdly, embodiments of this application provide a log processing device, including: a processor and a memory for storing a computer program capable of running on the processor, wherein...

[0048] The processor is configured to execute the steps of the method described in the first aspect when running a computer program.

[0049] Fourthly, embodiments of this application provide a computer storage medium storing a computer program, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0050] The technical solution provided in this application extracts log header data corresponding to unstructured log data; generates a structural parsing expression corresponding to the log header data based on the target transformation rules and the log header data; and performs structural transformation on the unstructured log data based on the structural parsing expression to generate structured log data. Thus, by using the structural parsing expression of the log header data, structural transformation of unstructured log data is achieved to generate structured log data. This can be applied to logs from various mixed sources, avoiding the influence of strong assumptions about the log input order and improving processing efficiency. Attached Figure Description

[0051] Figure 1 A schematic flowchart illustrating a log processing method provided in an embodiment of this application;

[0052] Figure 2 This is a flowchart illustrating an application example of the log message header structuring scheme in this application.

[0053] Figure 3 This is a schematic diagram illustrating the process of comparing the Kth token in an application example of this application;

[0054] Figure 4 This is a schematic diagram illustrating the process of generating log message headers in an application example of this application;

[0055] Figure 5 This is a schematic diagram of the structure of the log processing device provided in the embodiments of this application;

[0056] Figure 6 This is a schematic diagram of the structure of the log processing device provided in an embodiment of this application. Detailed Implementation

[0057] The present application will now be described in further detail with reference to the accompanying drawings and embodiments.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0059] Before providing a further detailed description of the embodiments of this application, the nouns and terms used in the embodiments of this application are explained, and the nouns and terms used in the embodiments of this application shall be interpreted as follows:

[0060] 1. Structuring: The process of transforming logs from unstructured text into data types such as "attribute name 1: content 1; attribute name 2: content 2; ...".

[0061] 2. Timestamp: A piece of text that identifies the time an event occurred, such as "SatJul2302:16:572005" or "2007-11-09T11:20UTC". It contains several parts of information such as year, month, day, hour, minute, second, and day of the week, connected by symbols such as spaces, "-", " / ", and "\".

[0062] 3. Log Level: The text that identifies the log level, usually such as "info", "warning", "error" etc.

[0063] 4. Log Pattern: A text block consisting of "<*>" and constants from the log, used to represent the log template, where "<*>" represents variables in the log. For example, the pattern for "[client207.235.226.2]" can be "[client<*>]".

[0064] 5. Token: A piece of text with independent semantics in the log. For example, "[client207.235.226.2]" can be broken down into the following 4 tokens: "[", "client", "207.235.226.2", and "]".

[0065] 6. Regular Expressions: A text string consisting of characters and special symbols that describes a pattern and is used to match text that matches that pattern. For example, the regular expression "\d" represents a single digit from 0 to 9, and the regular expression "\d\d:\d\d" can match text in the form of "12:34".

[0066] 7. Regular expression matching: Use regular expressions for matching.

[0067] 8. grok: A log structuring rule based on regular expressions. Given a regular expression, unstructured logs can be transformed into structured logs after being processed by grok.

[0068] This application provides a log processing method that can be applied to log processing devices, such as... Figure 1 As shown, the method includes the following steps:

[0069] Step 110: Extract the log header data corresponding to the unstructured log data.

[0070] Here, log data is output by the program and is used to help understand the various transactions occurring in the program's backend. Its content is typically unstructured log data. A log consists of two parts: a log header and a log body. Log data can be log messages, which include a message header and a message body. The message header, or log header, includes a timestamp and log level. The log processing device extracts the message header corresponding to the log message and separates the message header from the message body.

[0071] Here, users can trigger the command to start log processing through input devices such as mouse, keyboard and touch screen. After receiving the command to start log processing, the log processing device can start to acquire unstructured log data.

[0072] Step 120: Based on the target transformation rules and log header data, generate the structural parsing expression corresponding to the log header data.

[0073] Here, the target transformation rule is a transformation rule in a preset transformation rule library. The log processing device obtains a determination instruction for the target transformation rule, which includes identification information of the target transformation rule. Based on the identification information in the determination instruction and the preset transformation rule library, the target transformation rule in the preset transformation rule library can be determined. In some embodiments, the rule determination instruction may include a user selection instruction for the transformation rule. The user can select the desired transformation rule (i.e., the target transformation rule) from the preset transformation rule library through a visual operation interface.

[0074] Here, the preset transformation rule library includes at least one of the following: regular expression transformation rules, delimiter transformation rules, grok parsing transformation rules, script transformation rules, JSON parsing transformation rules, and key-value pair transformation rules. Based on the target transformation rules in the above transformation rule library and the log header data, a structure parsing expression corresponding to the log header data is generated. For example, taking the grok parsing transformation rule as the target transformation rule, the log header data is processed based on the grok parsing transformation rule to generate the grok parsing expression corresponding to the log header data.

[0075] Step 130: Perform structural transformation on the unstructured log data based on the structure parsing expression to generate structured log data.

[0076] Here, unstructured log data is transformed using structure parsing to obtain log data in structure parsing format, i.e., structured log data. Taking log messages as an example of unstructured log data, after the above structure transformation, log messages can be transformed into structured logs containing timestamp, log level, and log message body fields.

[0077] In this way, by parsing the structure of the log header data, unstructured log data can be transformed into structured log data, which can be applied to logs from various sources. It can also automatically extract the log header data, avoiding the high cost of manual configuration and the strong assumptions about the log input order, thus improving log processing efficiency.

[0078] In some embodiments, unstructured log data includes at least one log record. Extracting the log header data corresponding to the unstructured log data includes:

[0079] Preprocess each log record in at least one log record to determine the first set of data blocks corresponding to the log record;

[0080] The first data block set is parsed based on the pattern parsing algorithm to determine the log pattern corresponding to the log record.

[0081] Based on the log pattern corresponding to each log record, determine the set of log patterns corresponding to at least one log record.

[0082] Based on the first data block set corresponding to each log record, determine at least one second data block set corresponding to a log record, wherein the second data block set includes at least one first data block set;

[0083] Based on the log pattern set, the second data block set, and the truncation rules, at least one log record is truncated, and log header data corresponding to at least one log record is generated.

[0084] Here, for unstructured log data, which includes at least one log record (which can be a log message), preprocessing is performed on each of the at least one log record to determine the first data block set corresponding to that log record. For example, assuming the at least one log record G includes (G1, G2, G3), for each log record in G (G1, G2, G3), taking G1 as an example, G1 is preprocessed to determine the first data block set 1 corresponding to G1. Similarly, G2 is processed to generate the first data block set 2 corresponding to G2, and G3 is processed to generate the first data block set 3 corresponding to G3.

[0085] Here, log pattern refers to the log model after log clustering. Fixed parts of the log remain unchanged and are represented by their original words, while variable parts are represented by <*> (e.g., the original logs and log patterns in the appendix). The drain (An Online Log Parsing Approach with Fixed Depth Tree) algorithm is a classic pattern parsing algorithm. It's a log structuring algorithm that generates a hierarchical clustering tree while preserving the basic characteristics of the logs to accelerate log clustering. Here, the drain pattern parsing algorithm can be used to parse the first set of data blocks to determine the log pattern corresponding to each log record.

[0086] Here, for at least one log record, there is at least one corresponding log pattern. For each log record within the at least one log record, the first data set corresponding to the log record is parsed using the drain pattern parsing algorithm to determine the corresponding log pattern. Here, the log pattern can be represented by Pi. Taking log record G1 as an example, the log pattern is P1. The first data set corresponding to G1 is parsed using the drain pattern parsing algorithm to determine P1 corresponding to G1. For G2 and G3, the drain pattern parsing algorithm is used to generate P2 corresponding to G2 and P3 corresponding to G3, respectively. Different log records may correspond to the same log pattern.

[0087] Here, based on the log pattern corresponding to each log record, a set of log patterns corresponding to at least one log record is determined. For example, for P1 corresponding to G1, P2 corresponding to G2, and P3 corresponding to G3, P1, P2, and P3 can be the same or different. For example, assuming P1 and P2 are equal, then the set of P corresponding to at least one log record (G1, G2, G3) is (P1, P3).

[0088] Here, the log processing device can store the mapping relationship between at least one log record and the log pattern in real time. For example, the mapping relationship can be represented as (G1, P1; G2, P1; G3, P3).

[0089] Here, a second data block set is determined based on the first data block set corresponding to each log record. This second data block set includes at least one first data block set; that is, the second data block set is a set composed of the first data block sets corresponding to at least one log record. The second data block is the first data block corresponding to at least one log record. Here, preprocessing each log record within the at least one log record generates its corresponding first data block set. When there is only one log record, the first data block set and the second data block set are the same. When there are multiple log records, the second data block set is composed of the first data block sets corresponding to each log record.

[0090] For example, the first data block set can be represented by Un. When at least one log record includes three log records (G1, G2, G3), G1 corresponds to U1, G2 corresponds to U2, and G3 corresponds to U3. U1 can be represented as (token11, token12, ..., token1n), U2 can be represented as (token21, token21, ..., token2n), and U3 can be represented as (token31, token31, ..., token3n). Here, token represents the first data block. Accordingly, based on each first data set, a second data block set Q is generated, i.e., Q is (token11, token12, ..., token1n; token21, token21, ..., token2n; token31, token31, ..., token3n).

[0091] Here, the truncation rules can be preset by the user. Based on the log pattern set, the second data block set, and the truncation rules, at least one log record is truncated, and log header data corresponding to at least one log record is generated.

[0092] In some embodiments, at least one log record is truncated based on a log pattern set, a second data block set, and truncation rules to generate log header data corresponding to at least one log record, including:

[0093] Based on the log pattern set and the second data block set, determine the data to be extracted corresponding to at least one log record;

[0094] Based on the truncation rules, the data to be truncated is truncated, and at least one log header data corresponding to a log record is generated.

[0095] Here, for each log pattern in the log pattern set, it corresponds to at least one log record. For each log record within this set, preprocessing is performed to generate a first data block set. Based on each first data block set, a second data block set is generated. Based on the log pattern set and the second data block set, the data to be truncated corresponding to at least one log record can be determined. This truncated data consists of at least one second data block from the second data block set.

[0096] Here, after generating log header data corresponding to at least one log record, the data to be truncated is truncated based on the truncation rules to generate log header data corresponding to at least one log record.

[0097] In some embodiments, based on a log pattern set and a second data block set, determining the data to be extracted corresponding to at least one log record includes:

[0098] For each log pattern in the log pattern set, a third data block set corresponding to the log pattern is determined based on the mapping relationship between the log pattern and at least one log record and the second data block set;

[0099] Based on the third data block set and the first location information set corresponding to the log mode, the log mode location information set is determined, wherein the first location information is the location information corresponding to each second data block in the second data block set, and the log mode location information is the location information corresponding to each third data block in the third data block set;

[0100] Based on each of the log pattern location information sets, at least one log pattern location information set is generated;

[0101] Based on the set of log patterns and the mapping relationship between the log patterns and at least one log record, the target log pattern is determined;

[0102] Based on the target log pattern and the at least one log pattern location information set, determine the data to be extracted corresponding to the at least one log record.

[0103] Here, for each log pattern, there is at least one corresponding log record. The mapping relationship between log patterns and at least one log record represents the mapping relationship between at least one log record and its corresponding log pattern. In this mapping relationship, the correspondence between log patterns and log records is not necessarily one-to-one; the same log pattern may correspond to at least two log records. Each log record has its corresponding first data block set, and at least one log record has its corresponding second data block set, which is composed of each of the first data block sets.

[0104] For example, the mapping relationship can be (G1, P1; G2, P1; G3, P3). The third data block set is denoted by Dn, where n is the number of log patterns. For each log pattern in the log pattern set, taking P1 as an example, based on this mapping relationship, we can determine that at least one log record corresponding to P1 is G1 and G2. Then, based on the second data block set Q (token11, token12, ..., token1n; token21, token21, ..., token2n; token31, token31, ..., token3n) of at least one log record (G1, G2, G3), we can determine the third data block set D1 (token11, token12, ..., token1n; token21, token21, ..., token2n) corresponding to P1. Each third data block in D1 is a second data block (token11, token12, ..., token1n; token21, token21, ..., token2n) in the second data block set. Similarly, for P3, we can also determine the corresponding D3.

[0105] Here, the log pattern location information set can be represented by Wn, where n is the number of log patterns. The second data block set Q includes the data blocks corresponding to log records 1, 2, and 3. By obtaining the location information corresponding to these data blocks, the first location information set can be determined. This first location information set is the location information set of the second data block set, corresponding to the location information of each second data block in the second data block set. Assuming the first location information set is K, correspondingly, when at least one log record includes G1, G2, and G3, the first location information set K can be represented as (K11, ..., K1n; K21, ..., K2n; K31, ..., K3n). Taking D1 as an example, by finding the location information corresponding to each third data block in D1 within the first location information set K, the log pattern location information W1 is (K11, ..., K1n). Among them, the location information corresponding to the data block tokens (K11, ..., K1n) and D1 (token11, token12, ..., token1n; token21, token21, ..., token2n) is the same. Similarly, for the third data block set 3 corresponding to log mode 3, its corresponding W2 can also be generated as (K31, ..., K3n), (K31, ..., K3n) is (token31, token31, ..., token3n).

[0106] Here, the location information can be the starting location information of the data block token. For each log pattern in the log set, based on the above method, a corresponding log pattern location information set can be generated. For example, for P1 and P3 in the log pattern set (P1, P3), there are corresponding log pattern location information sets W1 and W3, respectively. Here, the log pattern location information is the location information of each third data block in the third data block set corresponding to P1 and the location information of each third data block in the third data block set corresponding to P3.

[0107] Here, at least one log pattern location information set is generated based on each log pattern location information set. Here, at least one log pattern location information set is a collection of log pattern location information sets. At least one log pattern location information set can be represented as (W1, ..., Wn). When the log pattern set is (log pattern 1, log pattern 3), at least one log pattern location information set W is (W1, W2), and W1 is (K11, ..., K1n). Wherein (K11, ..., K1n) is the same as the location information corresponding to the data block tokens of D1 (token11, token12, ..., token1n; token21, token21, ..., token2n). Similarly, for the third data block set 3 corresponding to log pattern 3, its corresponding W2 can also be generated as (K31, ..., K3n), where (K31, ..., K3n) is (token31, token31, ..., token3n).

[0108] Here, the mapping relationship between a log pattern and at least one log record can represent the number or type of log records corresponding to the same log pattern. Based on this mapping relationship and the set of log patterns, the log pattern in the set of log patterns that meets the preset conditions can be determined, and this log pattern is the target log pattern.

[0109] Here, after determining the target pattern, based on the target pattern and at least one log pattern location information set, the data to be intercepted corresponding to at least one log record is determined.

[0110] In some embodiments, determining a target log pattern based on a set of log patterns and a mapping between a log pattern and at least one log record includes:

[0111] Based on the log pattern set and the mapping relationship between log patterns and at least one log record, determine the number of log records corresponding to at least one log pattern in the log pattern set;

[0112] Based on the number of log records corresponding to at least one log pattern and a preset number threshold, the log pattern in the log pattern set that meets the preset number threshold is determined as the target log pattern.

[0113] Here, the preset condition for determining the target log pattern can be a preset quantity threshold. Based on the log pattern set and the mapping relationship between log patterns and at least one log record, the number of log records corresponding to at least one log pattern set in the log pattern set can be determined. For each log pattern in the log pattern set, the number of corresponding log records is different.

[0114] Here, the preset quantity threshold can be set to the maximum number of log records corresponding to the log pattern set. In this case, the target log pattern is the log pattern with the maximum number of log records in the log pattern set. This target pattern includes the most log records, maximizing the integrity of the log data.

[0115] For example, assuming the log pattern set is (P1, P3), based on the mapping relationship between P1 and P3 and at least one log record, assuming the number of log records corresponding to P1 is 2 and the number of log records corresponding to P3 is 1, the preset threshold for the number is 2, that is, P1 is the target log pattern Pm at this time.

[0116] In some embodiments, based on the target log pattern and at least one set of log pattern location information, determining the data to be intercepted corresponding to at least one log record includes:

[0117] Determine the target log pattern location information set based on the target log pattern and at least one log pattern location information set.

[0118] For each other log pattern location information set and the target log pattern location information set in at least one other log pattern location information set, a fourth data block set is determined. The fourth data block is a third data block whose location information is the same in the other log pattern location information set and the target log pattern location information set. The other log pattern location information set is any log pattern location information set other than the target log pattern set in at least one log pattern location set.

[0119] Based on each fourth data block set, generate initial truncation data;

[0120] The initial data to be extracted is sorted according to the sorting rules to generate the data to be extracted.

[0121] Here, at least one log pattern location information set includes at least one log pattern location information set. After determining the target pattern, the location information set corresponding to the target pattern can be found in the log pattern location information set. For example, at least one log pattern location information set is (W1, W2). The target log pattern is P1, and the target log pattern location information set is W1. The location information sets of other log patterns besides the target pattern P1 are W2.

[0122] Here, for each other log pattern location information set other than the target log pattern, at least one fourth data block and the number of fourth data blocks in the other log pattern location information set that have the same location information as the target log pattern location information set are determined. For example, for the other pattern location information set W2 (K31, ..., K3n), at least one fourth data block in W2 has the same location information as the target log pattern location information set W1 (K11, ..., K1n). Assuming that only W11 has the same location information as W31, the fourth data block is the third data block token11 corresponding to W11. This operation is performed for each other log pattern location information set. In this way, at least one fourth data block in at least one log pattern location information set that has the same location information as the target pattern location information set, and the number of fourth data blocks in that set, can be selected.

[0123] For example, suppose that at least one log pattern location information set includes two other log pattern location information sets besides the target log pattern location information set, namely, other log pattern location information set W2 and other location pattern information set W3. Suppose that the target log pattern location information set is W1, and the location information in W1 that is the same as W2 is (K11, K12, K13). In this case, the corresponding at least one fourth data block is (token11, token12, token13). The location information that is the same as W3 is (token11, token12, token14), and token11 and token12 are repeated, with a repetition count of 1. Thus, the initial truncated data is (token11, token12, token13, token14).

[0124] Here, the sorting rule can be the number of repetitions of the fourth data block. For example, in the initial data segmentation above, token11 and token12 are repeated 2 times, and token13 and token14 are repeated 0 times. Thus, the sorting rule can be set to sort according to the number of repetitions of the fourth data block in at least one set of fourth data blocks to generate the data to be segmented (token11, token12, token13, token14).

[0125] In some embodiments, the truncation rules include truncation length and the data type of the log header data. Based on the truncation rules, the data to be truncated is truncated to generate log header data corresponding to at least one log record, including:

[0126] Based on the truncation length, the data to be truncated is truncated to generate the initial log header data;

[0127] Determine whether the initial log header data contains the data type of a log header; if so, then determine that the initial log header data is log header data.

[0128] If not, adjust the truncation length until the initial log header data contains the data type of the log header, and then generate the log header data.

[0129] Here, the preset truncation rules include the truncation length and the data type of the log header. Generally, the log header includes timestamps, log levels, and components. Users can determine the data type of the log header in the truncation rules based on the data type of the log header they need to truncate. Users can set the truncation length in the truncation rules based on experience, thereby reducing the number of times unstructured data is truncated and improving the accuracy of truncation.

[0130] Here, the data to be truncated is first truncated based on the truncation length to generate initial log header data; it is then determined whether the initial log header data contains the data type of the log header. If so, the initial log header data is determined to be log header data; otherwise, the truncation length is adjusted, i.e., a new truncation length is determined. The data to be truncated is then truncated based on the new truncation length until the initial log header data contains the data type of the log header, at which point the log header data is generated.

[0131] In some embodiments, preprocessing is performed on each log record in at least one log record to determine a first set of data blocks corresponding to the log record, including:

[0132] For each log record in at least one log record, the log record is split according to the splitting rules to generate a set of split data blocks;

[0133] Based on the replacement rules, each data block in the set of split data blocks is replaced to generate the first set of data blocks corresponding to the log record.

[0134] Here, preprocessing is performed on each log record within at least one log record. Preprocessing may include determining the first set of data blocks corresponding to the log record based on splitting and replacement rules. These splitting and replacement rules can be predefined by the user.

[0135] Here, for each log record in at least one log record, the log record is first cut into data blocks based on the cutting rules to generate a set of cut data blocks; then, the cut data blocks are replaced based on the replacement rules, which can be regular expressions. Each cut data in the set of cut data is replaced based on the regular expressions to generate the first set of data blocks corresponding to the log record.

[0136] For example, taking log message 1 as an example, log message 1 is segmented based on segmentation rules to generate a set of segmented data blocks. Regular expressions are then used to replace the content of each segmented data block, generating a first set of data blocks. Here, the first data block is the replaced segmented data block. In this way, logs can be structured without relying on manually writing regular expressions, allowing the extraction of log messages containing timestamps, log levels, and log message body fields, which is beneficial for efficient further analysis of the log messages.

[0137] In some embodiments, unstructured log data is transformed into structured log data based on structural parsing to generate structured log data, including:

[0138] If not all unstructured log data is converted into structured log data, then the successfully converted structured logs and their corresponding log patterns are deleted. The unstructured log data that failed to convert is converted again, and the structure parsing is updated, until all the unstructured log data that failed to convert is converted and structured data is generated.

[0139] Here, the unstructured log data is transformed based on the structure parsing formula. If it is determined that not all unstructured log data has been transformed into structured log data, the successfully transformed structured logs and the corresponding log patterns are deleted. The unstructured log data generated by the transformation is transformed again. The structure parsing formula is updated based on the unstructured data that failed to be transformed. The unstructured data that failed to be transformed is transformed based on the updated structure parsing formula until all the unstructured data that failed to be transformed is transformed and structured data is generated.

[0140] The present application will be further described in detail below with reference to application examples.

[0141] In the IT operations and maintenance field, logs are semi-structured text generated by the operating system or other software systems according to a certain template. A log entry consists of two parts: a message header and a message content. The format of the message header is determined by the operating system or software system that generated the log, and typically includes information such as a timestamp, log level, and components. Logs generated from the same source have a consistent message header format. The message content is usually a text containing constants and variables, and its expression is flexible. Logs generated from the same source can contain message content in various different formats. Message header structuring refers to identifying the message header portion from the raw log text and extracting information such as the timestamp and log level from the message header for further log analysis.

[0142] Existing solutions for structuring message headers include: manually writing regular expressions, drain pattern parsing algorithms, and longest common subsequence (spell).

[0143] (1) Manually write regular expressions

[0144] Manually writing regular expressions refers to using the syntax rules of regular expressions to manually write patterns for matching and parsing log message headers. This method is typically used for customizing logging systems or meeting the needs of existing logging systems. Specifically, manually writing regular expressions includes the following steps: 1. Determine the fields of the log message header: Based on business requirements and system design, determine which fields need to be included in the log message header, such as timestamp, level, application name, etc. 2. Define the regular expression pattern: Based on the fields of the log message header, use the syntax rules of regular expressions to define a pattern that can match the log message header.

[0145] For example, you can use `\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}\\.\\d{3}Z` to match timestamps in ISO format. 3. Test Regular Expression Patterns: Use testing tools or write test code to test regular expression patterns to ensure they accurately match and parse log message headers. 4. Integrate Regular Expression Patterns: Integrate regular expression patterns into the logging system for matching and parsing log message headers.

[0146] Manually writing regular expressions can be applied to various types of log parsing scenarios, but it has the following drawbacks: it requires a high level of technical expertise, as the syntax of regular expressions is relatively complex and requires a long period of learning and practice to master; at the same time, it is slow in performance, and its performance is poor in some cases of regular expression matching, which can slow down the program's running speed when processing large amounts of data.

[0147] (2) Drain mode parsing algorithm

[0148] This method is based on the following assumptions: original log messages with the same log pattern have the same length; and the first few words of original log messages with the same log pattern are identical. Based on these assumptions, a tree-structured data method is used to quickly group and cluster logs, and templates are extracted from the clustering results.

[0149] The advantages of this algorithm are its speed and efficiency, and its good adaptability to well-segmented original log information. However, it has the following drawbacks: firstly, it makes a strong assumption about the length of the log; secondly, when the words used for log grouping search are parameters, the grouping results are incorrect.

[0150] (3) Longest common subsequence (spell)

[0151] Spell is an algorithm based on LCS (Largest Common Subsequence) to efficiently extract message types from raw system logs. Its observation is that if we consider the input of log printing statements (i.e., log entries) as a sequence, then in most log printing statements, constants representing the message type typically constitute the majority of the sequence, while parameter values ​​constitute only a small portion. If two log entries are generated by the same log printing statement but differ only due to different parameter values, then the LCS of these two sequences is likely a constant in the code.

[0152] However, in practical applications, its efficiency is very low. For each new log entry, it must calculate the LCS with every pattern in the log pattern set, and the time complexity of calculating the LCS is O(N^2).

[0153] In summary, the above-mentioned schemes (1), (2), and (3) also have the following disadvantages:

[0154] (1) Manually writing regular expressions requires manual operation for logs from each source. When a new source of logs appears, it is not possible to automatically generate a structured parsing expression for the message header of the logs from that source.

[0155] (2) Drain makes strong assumptions about log formats: original log messages with the same log pattern have the same length, and the first few words of original log messages with the same log pattern are the same. Logs that do not meet these assumptions are difficult to parse correctly.

[0156] (3) The output of the spell algorithm is affected by the order of the input logs. For the same batch of logs, if parallel computing or group computing is used, causing a change in the input order, a relatively stable result cannot be obtained. In addition, algorithms such as drain and spell can only distinguish between variables and constants, but cannot identify the specific meaning of each variable, and therefore cannot automatically extract information such as timestamps and log levels.

[0157] Based on this, this application example provides a log message header structuring scheme, aiming to solve the problems of existing log platforms' inability to automatically parse log headers when processing unstructured log data, resulting in high learning costs and slow processing efficiency due to the need for manual configuration. Furthermore, it can be applied to logs from various mixed sources and makes no strong assumptions regarding log input order or whether the starting tokens of the logs are the same.

[0158] The following details the implementation process of the log message header structuring scheme provided in this application example. This structuring scheme can be applied to a unified log analysis and processing platform to process and parse unstructured log data.

[0159] In this application example, the unstructured log data includes at least one log message (i.e., the aforementioned at least one log message record). The number of log messages is set to four, meaning the at least one log message G is (G1, G2, G3, G4). The first data block set of G1 is U1, the first data block set of G2 is U2, the first data block set of G3 is U3, and the first data block set of G4 is U4. The pattern parsing algorithm uses the Drain algorithm, and the target transformation rule is the GROOK transformation rule. This application example mainly includes log segmentation, message header truncation, and GROOK parsing generation. Figure 2 As shown, the specific steps are as follows:

[0160] Step 210: Preprocess at least one log message (i.e., log segmentation).

[0161] For each log message in at least one log message, the log message is segmented based on the segmentation rules to generate a set of segmented data blocks; each segmented data block in the set of segmented data blocks is replaced based on the replacement rules to generate a first set of data blocks corresponding to the log message.

[0162] Here, for each log message in at least one log message, each log message is divided into several tokens according to the following rules, and these tokens form the first data block set. The tokens here refer to the first data block mentioned above.

[0163] For each log message, its data type includes: timestamp, log level, Chinese characters, punctuation marks, and other content. The segmentation rules here can be used to segment the log message based on its data type, generating a set of segmented data blocks. Each segment in the set of segmented data blocks has a different data type.

[0164] After the log message is segmented, each segmented data block in the segmented data block set can be replaced based on replacement rules, which include regular expressions. Specifically, taking log message G1 as an example...

[0165] a) Use regular expressions to replace the content of data type "\d\d:\d\d:\d\d" in log message G1 with "% <time>The symbol "\d" is used as a token, where "\d" represents a single digit from 0 to 9.

[0166] b) Using regular expressions, replace content in log message G1 with data types in the form of "(year)(separator)(month)(separator)(day)" with "% <date>The year is recorded as a token. The year can be either a 4-digit number (e.g., 2022) or a 2-digit number (e.g., 99).

[0167] The separator can be one of " / ", "-", "\", ".", or a space, and the two separators must be the same type; "(month)" can be an integer not less than 1 and not greater than 12, or the English abbreviation of the month (such as "Jan" or "Feb"); "(day)" can be an integer not less than 1 and not greater than 31.

[0168] c) Using regular expressions, replace data types in log message G1 that are the first three abbreviations of the weekday (such as "Mon" or "Tue") with "%". <week>And record it as a token.

[0169] d) Using regular expressions, replace the words "info", "error", "warn", "warning", and "notice" in log message G1, where the data type is log level, along with the first letter capitalized and all letters capitalized forms of these five words, with "%". <geveg>And record it as a token.

[0170] Here, the replacement rules can also be set by the user, for example:

[0171] e) When the data type of log message G1 is Chinese characters, each Chinese character is treated as a separate token.

[0172] f) For log message G1, the data type is punctuation marks, and each punctuation mark is a separate token.

[0173] g) For the remainder of log message G1, each space (or the space between the beginning / end of the log) constitutes a token. Each space represents one token.

[0174] Thus, log message G1 is divided into several tokens according to the above operation, and these tokens form the first data block set U1. U1 can be represented as (token11, token21, ..., token1n), where n is the number of tokens. The same operation is performed on log messages G2, G3, and G4 to generate the first data block set U2, U3, and U4, respectively.

[0175] Step 220: Perform pattern parsing on at least one log message.

[0176] The first data block set is parsed using the drain pattern parsing algorithm to determine the log pattern corresponding to the log message.

[0177] The log messages after the above word segmentation and replacement, i.e., the first data set U1, are input into the drain algorithm for pattern parsing, resulting in several log patterns, i.e., log pattern P1. The drain algorithm is a classic pattern parsing algorithm. Although it has two drawbacks mentioned above—strong assumptions about the logs and inability to identify the meaning of each variable—the drain algorithm is only used as one step in the log message structuring process here. That is, only the drain algorithm is used to parse the log messages for pattern parsing, and the parsed log patterns are used for subsequent operations. Subsequent operations do not continue to use the drain algorithm. The aforementioned drawbacks can be overcome by other steps in this application, and the process remains feasible.

[0178] Based on the log pattern corresponding to each log message, determine the set of log patterns corresponding to at least one log message.

[0179] In this application example, the log patterns corresponding to the first data block set U1, first data block set U2, first data block set U3, and first data block set U4 are P1, P2, P3, and P4, respectively. If all log patterns are different, the log pattern set P is (P1, P2, P3, P4), and the mapping relationship between log messages and log patterns is (G1: P1; G2: P2; G3: P3; G4: P4). In this application example, P1 and P2 are the same, so the set P in this application example is (P1, P3, P4), and the mapping relationship is (G1, G2: P1; G3: P3; G4: P4).

[0180] Based on the first data block set corresponding to each log message, determine at least one second data block set corresponding to a log message, wherein the second data block set includes at least one first data block set.

[0181] In this application example, as shown above, U1 can be represented as (token11, token12, ..., token1n), where n is the number of tokens. U2 can be represented as (token21, token21, ..., token2n), U3 can be represented as (token31, token31, ..., token3n), and U4 can be represented as (token41, token41, ..., token4n). The second data block set here includes (token11, token12, ..., token1n; token21, token21, ..., token2n; token31, token31, ..., token3n; token41, token41, ..., token4n). U1, U2, U3, and U4 generate the second data block set Q (token11, token12, ..., token4n) corresponding to the log messages (G1, G2, G3, G4).

[0182] For each log pattern in each log pattern set (P1, P2, P3, P4), a third data block set corresponding to the log pattern is determined based on the mapping relationship between the log pattern and at least one log message and the second data block set.

[0183] Here, taking P1 as an example, based on log mode 1 and the mapping relationship (G1, G2: P1; G3: P3; G4: P4), we can determine that the log messages corresponding to P1 are G1 and G2. Then, based on the second data block set Q (token11, token12, ..., token4n), we can determine that the third data block set D1 corresponding to log mode 1 is (token11, token12, ..., token2n). Similarly, the third data block set D2 corresponding to P3 is (token31, token32, ..., token3n), and the third data block set D3 corresponding to P4 is (token41, token42, ..., token4n).

[0184] Based on the third data block set and the first location information set corresponding to the log mode, the log mode location information set is determined. The first location information is the location information corresponding to each second data block in the second data block set, and the log mode location information is the location information corresponding to each third data block in the third data block set.

[0185] Here, the location information refers to the starting location information of the tokens. The location information set of each token in the second data block set Q (token11, token12, ..., token4n), i.e., the first location information, is represented by K. Taking the third data block set D1 as an example, the log mode location information of log mode P1 is the location information of each third data block in D1. The log mode location information set W1 includes (K11, K12, ..., K2n), which respectively represent the starting location information corresponding to (token11, token12, ..., token2n). The log mode location information set W2 is (K21, K22, ..., K2n), and the log mode location information set W3 is (K31, K32, ..., K3n).

[0186] Based on each first location information set, at least one log pattern location information set is generated.

[0187] Here, based on W1, W2, and W3, at least one log pattern location information set (W1, W2, W3) can be generated. Here, W1 is the location information set corresponding to each token in (token11, token12, ..., token2n), W2 is the location information set of each token in (token31, token32, ..., token3n), and W3 is the location information set of each token in (token41, token42, ..., token4n).

[0188] Step 230: Extract the message header (i.e., the log header data mentioned above) corresponding to at least one log message.

[0189] a) Determine the target log pattern based on the set of log patterns and the mapping relationship between the log pattern and at least one log message.

[0190] Here, assuming N log patterns are obtained after pattern parsing, the number of log messages corresponding to each log pattern is counted based on the above mapping relationship. The log pattern with the most corresponding log messages is denoted as Pm, which is the target log pattern. In this application example, P1 has the most corresponding log messages, which is 2, so the target log pattern Pm is P1.

[0191] Based on the target log pattern P1 and at least one log pattern location information set (W1, W2, W), the location information set of the target log pattern P1 is determined to be W1.

[0192] b) Let Pi be the i-th log pattern in at least one log pattern location information set Wi, where i is an integer from 1 to N. In this application example, i is 3. For each Pi other than P1 (i.e., any log pattern location information set other than the target pattern location information set mentioned above), Pi includes P2 and P3 mentioned above. Compare the tokens (i.e., the fourth data block set) in the starting location information set of Pi (P2, P3) that have the same starting position as Pm, and the number of such tokens, denoted as Li. Taking the process of comparing the K-th token in the location information set of log pattern Pi with the K-th token in the location information set of P1 to determine the fourth data block set as an example, ... Figure 3 As shown:

[0193] Step 310: Start (K=0).

[0194] Step 320: Retrieve the Kth token of log pattern Pi and the Kth token of log pattern Pm(P1).

[0195] Extract the revelation position information of the Kth token of log pattern Pi and the starting position information of the Kth token of P1.

[0196] Step 330: Are the two tokens the same?

[0197] Determine if the starting position information of the Kth token in log pattern Pi is the same as the starting position information of the Kth token in P1. If yes, proceed to step 340; otherwise, proceed to step 350.

[0198] Step 340: Increment the K value by 1.

[0199] If it is determined that the starting position information of the Kth token in log pattern Pi is the same as the starting position information of the Kth token in P1, then the value of K is incremented by 1, K+1 is updated based on K=K+1, and the starting position information of the (K+1)th token in log pattern Pi is checked to see if it is the same as the starting position information of the (K+1)th token in P1.

[0200] Step 350: End (Li = K).

[0201] When it is determined that the starting position information of the Kth token in log pattern Pi is different from the starting position information of the Kth token in P1, the comparison of the token position information between log pattern Pi and P1 ends. At this time, the value of the same number of times Li is K.

[0202] c) Generate initial truncation data based on each fourth data block set;

[0203] The initial data to be extracted is sorted according to the sorting rules to generate the data to be extracted.

[0204] For each fourth data block set, initial truncation data is generated and the occurrence frequency of all values ​​of Li is counted. The initial truncation data generated for each possible value, i.e., each fourth data block set, is sorted in descending order of occurrence frequency to generate truncation data T(T(1), T(2), ..., T(n)), where T(n) is the fourth data block.

[0205] d) Figure 4 A flowchart illustrating the process of generating log message headers, as follows: Figure 4 As shown, based on the truncation rules, the data T(T(1), T(2), ..., T(n)) to be truncated is truncated to generate message header data corresponding to at least one log message. The specific steps are as follows:

[0206] Step 410: Start capturing.

[0207] Here, the truncation length of the truncation rule is set to K, where K is the Kth fourth data block of the data to be truncated. K is set to 1, and the number of truncation loops is M.

[0208] Step 420: Generate initial message header data H(K).

[0209] Here, the tokens of the location information set of the target log pattern Pm are extracted, that is, the data to be extracted T(T(1), T(2), ..., T(n)) is extracted based on the extraction length K. The first T(K) tokens are used to generate the initial message header data, denoted as H(K).

[0210] Step 430: Does H(K) contain both a timestamp and a log level?

[0211] Determine whether the initial message header data H(K) contains the data type of the message header, namely timestamp and log level. If so, determine that the initial message header data is message header data and proceed to step 440.

[0212] If not, proceed to step 450.

[0213] For example, take the first T(1) tokens of Pm and denote them as H(1). If H(1) contains both timestamp and log level, then H(1) is the token corresponding to the message header; otherwise, take the first T(2) tokens of Pm and denote them as H(2), and check whether H(2) contains both timestamp and log level.

[0214] Step 440: Output H(K).

[0215] At this point, H(K) is the log message header data. Output H(K) to generate the subsequent structure parsing expression.

[0216] Step 450: Update the K value.

[0217] Adjust the truncation length, i.e., update the value of K based on K = K + 1.

[0218] Step 460: Does the value of K ≤ M and does T(K) exist?

[0219] Then adjust the truncation length until the initial message header data contains the data type of the message header, and generate the message header data.

[0220] Determine if the updated K value is ≤ M and if T(K) exists. If yes, re-truncate based on the updated K value and execute step 420 until the initial message header data contains the data type of the message header. Repeat this process until a T(k) (k < the set threshold) is found such that H(k) contains both the timestamp and the log level. If no such T(k) is found, execute step 470 to generate the final log message header data H(1). If no matching result is found after the number of repetitions reaches the threshold, take H(1) as the final result of this step.

[0221] Step 470: Output H(1).

[0222] Thus, this application example can structure logs without relying on manually written regular expressions, extracting timestamp, log level, and log message body fields, which is beneficial for efficient further analysis of the logs.

[0223] Step 240: Generate the grok parser.

[0224] Based on the target transformation rules and log header data, a structural parsing expression corresponding to the log header data is generated. In the application example, the target transformation rule is the grok parsing transformation rule. Taking the message header as H(1) as an example, H(1) mainly includes the timestamp and log level.

[0225] Initially, the grok parsing is an empty string. For each token in H(1), the corresponding grok parsing is generated according to the following rules (grok parsing conversion rules).

[0226] If the current token content is "% <level>If the expression is incomplete, then add "%{LOGLEVEL:Level}" to the end of the grok parsing expression; here "% <level>"Generated by applying the above replacement rules to the log level in the log message."

[0227] The specified timestamp token includes: "% <date> ”、"％ <time>”、"％ <week>The specified separator tokens include: " / ", "-", "\", ".", and space.

[0228] If the current token is any of the timestamp tokens, concatenate this token with the subsequent consecutive timestamp tokens or separator tokens into a single variable named `timestamp`. Replace the token in `timestamp` with the corresponding regular expression according to Table 1 below, and then add "(?" to the end of the grok parsing expression. <timestamp>The first "Timestamp" is a constant text representing the field name; the second "timestamp" is a variable, and this position needs to be replaced with the content of that variable.

[0229] Table 1

[0230]

[0231] For the remaining tokens, escape them and append them directly to the end of the grok parser.

[0232] After iterating through all tokens in H(1), add "%{GREEDYDATA:message}" to the end of the grok parsing to obtain the grok parsing result. Thus, this application example generates a grok parsing based on the truncated message header, which solves the problem that spell and drain algorithms, when performing structured log parsing, can only identify variables and constants in the log, but cannot generate grok parsing, and cannot specifically identify information such as timestamp and level.

[0233] Step 250: Match the input logs with the generated grok parsing.

[0234] Structured log data is generated by performing structural parsing on unstructured log data.

[0235] If all input logs can be matched by the grok parsing expression, the algorithm process ends; otherwise, the matched logs and their corresponding patterns are removed, and the process returns to step 230 to re-extract the message header and generate a new grok parsing expression, until all logs can be matched. In other words, if not all unstructured log data is converted into structured log data, the successfully converted structured logs and their corresponding log patterns are deleted. The unstructured log data that failed to convert is converted again, and the structure parsing expression is updated, until all the unstructured log data that failed to convert is converted and structured data is generated.

[0236] This application example utilizes the drain algorithm and grok parsing to identify and structure log message headers. It can structure logs without relying on manually written regular expressions, transforming unstructured logs into structured logs containing timestamps, log levels, and log message body fields. This facilitates efficient further analysis of the logs and can be applied to logs from various sources without making strong assumptions about the order of log input or whether the starting tokens of the logs are the same.

[0237] To implement the method of this application embodiment, this application embodiment also provides a log processing apparatus, which corresponds to the above-described log processing method. The steps in the above-described log processing method embodiment are also fully applicable to this log processing apparatus embodiment. For example... Figure 5 As shown, the log processing device 500 includes: an extraction module 510, a generation module 520, and a conversion module 530. The extraction module 510 is used to extract log header data corresponding to unstructured log data; the generation module 520 is used to generate a structural parsing expression corresponding to the log header data based on the target conversion rules and the log header data; and the conversion module 530 is used to perform structural conversion on the unstructured log data based on the structural parsing expression to generate structured log data.

[0238] In some embodiments, the log processing apparatus further includes a determining module 540, configured to preprocess each log record in at least one log record to determine a first data block set corresponding to the log record; perform pattern parsing on the first data block set based on a pattern parsing algorithm to determine a log pattern corresponding to the log record; determine a log pattern set corresponding to at least one log record based on the log pattern corresponding to each log record; and determine a second data block set corresponding to at least one log record based on the first data block set corresponding to each log record, wherein the second data block set includes at least one first data block set; the truncation module 510 is further configured to truncate at least one log record based on the log pattern set, the second data block set, and truncation rules to generate log header data corresponding to at least one log record.

[0239] In some embodiments, the determining module 540 is further configured to determine the data to be truncated corresponding to at least one log record based on the log pattern set and the second data block set; the truncating module 510 is further configured to truncate the data to be truncated based on the truncating rules and generate log header data corresponding to at least one log record.

[0240] In some embodiments, the determining module 540 is further configured to, for each log pattern in the log pattern set, determine a third data block set corresponding to the log pattern based on the mapping relationship between the log pattern and at least one log record and the second data block set; determine a log pattern location information set based on the third data block set corresponding to the log pattern and a first location information set, wherein the first location information is the location information corresponding to each second data block in the second data block set, and the log pattern location information is the location information corresponding to each third data block in the third data block set; the generating module 520 is further configured to generate at least one log pattern location information set based on each log pattern location information set; the determining module 540 is further configured to, based on the log pattern set and the mapping relationship between the log pattern and at least one log record, determine a target log pattern; and determine the data to be intercepted corresponding to at least one log record based on the target log pattern and at least one log pattern location information set.

[0241] In some embodiments, the determining module 540 is further configured to determine the number of log records corresponding to at least one log pattern in the log pattern set based on the log pattern set and the mapping relationship between the log pattern and at least one log record; and to determine the log pattern in the log pattern set that satisfies the preset number threshold as the target log pattern based on the number of log records corresponding to at least one log pattern and a preset number threshold.

[0242] In some embodiments, the determining module 540 is further configured to determine a target log pattern location information set based on the target log pattern and at least one log pattern location information set; for each of the other log pattern location information sets and the target log pattern location information set in at least one other log pattern location information set, determine a fourth data block set, wherein the fourth data block is a third data block in the other log pattern location information set that has the same location information as the target log pattern location information set; the other log pattern location information set is any log pattern location information set other than the target log pattern set in at least one log pattern location set; the generating module 520 is further configured to generate initial truncated data based on each fourth data block set; and sort the initial truncated data according to the sorting rules to generate truncated data.

[0243] In some embodiments, the truncation rules include the truncation length and the data type of the log header data. The truncation module 510 is further configured to truncate the data to be truncated based on the truncation length to generate initial log header data. The determination module 540 is further configured to determine whether the initial log header data contains the data type of the log header. If yes, the initial log header data is determined to be log header data. If no, the truncation length is adjusted until the initial log header data contains the data type of the log header, and then the log header data is generated.

[0244] In some embodiments, the generation module 520 is further configured to, for each log record in at least one log record, divide the log record according to the dividing rule to generate a set of divided data blocks; and replace each divided data block in the set of divided data blocks according to the replacement rule to generate a first set of data blocks corresponding to the log record.

[0245] In some embodiments, the log processing module further includes an update module 550, which is used to delete the successfully converted structured logs and the log patterns corresponding to the successfully converted structured logs if not all unstructured log data has been converted into structured log data, and to re-convert the unstructured log data that failed to be converted, update the structure parsing expression, until all the unstructured log data that failed to be converted is converted and structured data is generated.

[0246] In practical applications, the interception module 510, generation module 520, conversion module 530, determination module 540, and update module 550 can be implemented by the processor in the log processing device. Of course, the processor needs to run the computer program in the memory to implement its functions.

[0247] It should be noted that the log processing device provided in the above embodiments is only illustrated by the division of the above program modules. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. In addition, the log processing device and the log processing method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0248] Based on the hardware implementation of the above program modules, and in order to implement the method of the embodiments of this application, the embodiments of this application also provide a log processing device. Figure 6 This is only an exemplary structure of the log processing device, not the entire structure; implementation is possible as needed. Figure 6 The structure shown may be part or all of the structure.

[0249] like Figure 6 As shown, the log processing device 600 provided in this embodiment includes at least one processor 601, a memory 602, a user interface 603, and at least one network interface 604. The various components in the log processing device 600 are coupled together via a bus system 605. It can be understood that the bus system 605 is used to implement communication between these components. In addition to a data bus, the bus system 605 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 6 The general designated all buses as Bus System 605.

[0250] The user interface 603 may include a monitor, keyboard, mouse, trackball, click wheel, buttons, touchpad, or touch screen.

[0251] The memory 602 in this embodiment is used to store various types of data to support the operation of the log processing device. Examples of such data include any computer program used to operate on the log processing device.

[0252] The log processing method disclosed in this application can be applied to or implemented by the processor 601. The processor 601 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the log processing method can be completed by the integrated logic circuitry in the hardware of the processor 601 or by instructions in software form. The processor 601 can be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 601 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software modules can be located in a storage medium, specifically in memory 602. The processor 601 reads information from memory 602 and, in conjunction with its hardware, completes the steps of the log processing method provided in the embodiments of this application.

[0253] In an exemplary embodiment, the log processing device may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned methods.

[0254] It is understood that memory 602 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memories described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable types of memories.

[0255] In an exemplary embodiment, this application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a memory 602 that stores a computer program. This computer program can be executed by the processor 601 of the log processing device to complete the steps of the method described in this application. The computer-readable storage medium can be a ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM, etc.

[0256] It should be noted that "first," "second," etc., are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0257] Furthermore, the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0258] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.< / timestamp> < / week> < / time> < / date> < / level> < / level> < / geveg> < / week> < / date> < / time>

Claims

1. A log processing method characterized by, The method includes: Extract the log header data corresponding to the unstructured log data; Based on the target transformation rules and the log header data, generate the structural parsing expression corresponding to the log header data; Based on the aforementioned structural parsing expression, the unstructured log data is transformed to generate structured log data; The unstructured log data includes at least one log record, and the extraction of the log header data corresponding to the unstructured log data includes: For each log record in the at least one log record, preprocessing is performed to determine the first set of data blocks corresponding to the log record; The first data block set is parsed based on a pattern parsing algorithm to determine the log pattern corresponding to the log record; Based on the log pattern corresponding to each log record, determine the log pattern set corresponding to the at least one log record; Based on the first data block set corresponding to each log record, a second data block set corresponding to the at least one log record is determined, wherein the second data block set includes at least one first data block set; Based on the log pattern set, the second data block set, and the truncation rules, the at least one log record is truncated to generate the log header data corresponding to the at least one log record.

2. The method of claim 1, wherein, The step of extracting at least one log record based on the log pattern set, the second data block set, and the truncation rule, and generating log header data corresponding to the at least one log record, includes: Based on the log pattern set and the second data block set, determine the data to be extracted corresponding to the at least one log record; Based on the truncation rules, the data to be truncated is truncated to generate log header data corresponding to the at least one log record.

3. The method of claim 2, wherein, The step of determining the data to be extracted corresponding to the at least one log record based on the log pattern set and the second data block set includes: For each log pattern in the log pattern set, a third data block set corresponding to the log pattern is determined based on the mapping relationship between the log pattern and at least one log record and the second data block set; Based on the third data block set and the first location information set corresponding to the log mode, the log mode location information set is determined, wherein the first location information is the location information corresponding to each second data block in the second data block set, and the log mode location information is the location information corresponding to each third data block in the third data block set; Based on each of the log pattern location information sets, at least one log pattern location information set is generated; Based on the set of log patterns and the mapping relationship between the log patterns and at least one log record, the target log pattern is determined; Based on the target log pattern and the at least one log pattern location information set, determine the data to be extracted corresponding to the at least one log record.

4. The method of claim 3, wherein, The step of determining the target log pattern based on the log pattern set and the mapping relationship between the log pattern and at least one log record includes: Based on the log pattern set and the mapping relationship between the log pattern and at least one log record, determine the number of log records corresponding to at least one log pattern in the log pattern set; Based on the number of log records corresponding to the at least one log pattern and a preset number threshold, the log pattern in the log pattern set that satisfies the preset number threshold is determined as the target log pattern.

5. The method of claim 3, wherein, The step of determining the data to be extracted corresponding to the at least one log record based on the target log pattern and the at least one log pattern location information set includes: Based on the target log pattern and the at least one log pattern location information set, determine the target log pattern location information set; For each other log pattern location information set in at least one other log pattern location information set and the target log pattern location information set, a fourth data block set is determined, wherein the fourth data block is a third data block in the other log pattern location information set and the target log pattern location information set with the same location information; the other log pattern location information set is any log pattern location information set in the at least one log pattern location set other than the target log pattern set. Based on each of the fourth data block sets, initial truncated data is generated; The initial truncated data is sorted according to the sorting rules to generate the data to be truncated.

6. The method of claim 2, wherein, The truncation rules include the truncation length and the data type of the log header data. The step of truncating the data to be truncated based on the truncation rules to generate log header data corresponding to at least one log record includes: Based on the specified truncation length, the data to be truncated is truncated to generate initial log header data; Determine whether the initial log header data contains the data type of the log header; if so, determine that the initial log header data is the log header data. If not, adjust the truncation length until the initial log header data contains the data type of the log header, and then generate the log header data.

7. The method of claim 1, wherein, The step of preprocessing each log record in the at least one log record to determine the first set of data blocks corresponding to the log record includes: For each log record in the at least one log record, the log record is split according to the splitting rules to generate a set of split data blocks; Based on the replacement rules, each segmented data block in the segmented data block set is replaced to generate the first data block set corresponding to the log record.

8. The method of claim 1, wherein, The process of transforming the unstructured log data based on the structure parsing expression to generate structured log data includes: If not all the unstructured log data is converted into structured log data, then the successfully converted structured logs and the corresponding log patterns are deleted. The unstructured log data that failed to convert is converted again, and the structure parsing is updated, until all the unstructured log data that failed to convert is converted and the structured log data is generated.

9. A log processing apparatus characterized by comprising: The device includes: The interception module is used to intercept the log header data corresponding to unstructured log data; the unstructured log data includes at least one log record; The generation module is used to generate the structural parsing expression corresponding to the log header data based on the target transformation rules and the log header data; The conversion module is used to perform structural conversion on the unstructured log data based on the structural parsing expression to generate structured log data; The determination module is configured to: preprocess each log record in the at least one log record to determine a first data block set corresponding to the log record; perform pattern parsing on the first data block set based on a pattern parsing algorithm to determine the log pattern corresponding to the log record; determine a log pattern set corresponding to the at least one log record based on the log pattern corresponding to each log record; determine a second data block set corresponding to the at least one log record based on the first data block set corresponding to each log record, wherein the second data block set includes at least one first data block set; and truncate the at least one log record based on the log pattern set, the second data block set, and truncation rules to generate log header data corresponding to the at least one log record.

10. A log processing device characterized by comprising: include: A processor and memory for storing computer programs that can run on the processor, wherein, The processor, when running a computer program, performs the steps of the method according to any one of claims 1 to 8.

11. A computer storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images