Online parsing method, system, electronic device and storage medium for multi-source logs

Abstract

The present invention discloses an online parsing method, system, electronic device, and storage medium for multi-source logs. The method comprises the following steps: collecting multi-source logs; classifying the collected logs and extracting key field information from the logs using a log tree; preprocessing the key field information, namely, the message field; grouping the logs using the first word in the preprocessed log message field; extracting verb part-of-speech features from the grouped log message field content; and extracting log event templates and update log templates from logs with similar verb part-of-speech features using a longest common subsequence (LCS) algorithm. The method distinguishes execution-type logs from status-type logs using the verb part-of-speech features, and, combined with the multi-source information of the logs, can provide an important data foundation for constructing workflow diagrams and root cause analysis in subsequent log analysis, effectively resolving the overfitting problem in which logs with similar character structures but different semantics are mistakenly identified as identical log events.

Description

Technical Field

[0001] The present invention relates to the technical field of log analysis, and in particular to an online parsing method, system, electronic device and storage medium for multi-source logs. Background Art

[0002] As software scale and complexity increase, the volume of logs generated by software also becomes vastly larger. According to statistics, large cloud applications generate approximately 10GB of logs per hour. Traditional manual analysis of this log information is no longer sufficient, leading to the emergence of automated log analysis technology.

[0003] In daily operations and maintenance, using unstructured logs for anomaly detection and root cause analysis is a challenging analytical task, and performing real-time and effective anomaly detection from massive amounts of log data is even more challenging. The content in a log can generally be divided into a constant portion and a variable portion. The constant portion is fixed text content that represents a log event template, while the variable portion reflects information during system runtime, such as status values ​​and parameters (IP address, duration, file path, etc.). The purpose of log parsing is to separate log event information and parameter information, converting the original unstructured log into a structured log that can be easily recognized and processed by computers, thereby resolving the problem of unstructured log data being difficult to analyze. As the primary step in log analysis, it plays a critical role in the effectiveness of subsequent log analysis.

[0004] Currently, in the field of log parsing research, there are similarity-based clustering methods, frequent item mining methods, and heuristic methods. Existing methods mainly perform well for logs in specific fields. Furthermore, most current log parsing methods are only suitable for offline log parsing and cannot meet the computational requirements of online log parsing. Typical methods that support online log parsing include Drain and Spell. However, experimental results show that when encountering log event templates with similar structures, they are prone to log parsing overfitting. Such log event templates with similar structures are often common in large software systems. For example, the following are two logs from OpenStack:

[0005] Instance spawned successfully.

[0006] Instance destroyed successfully.

[0007] When extracting event templates and finding common subsequences, the two logs mentioned above are often mistakenly treated as the same event template (Instance*successfully.), but in fact they are two types of log events with completely opposite meanings.

[0008] Another example is the event log with similar results in the nova component of OpenStack.

[0009] 2022-04-21 18:14:29.104 19134 INFO nova.compute.manager[req-67a9597b-4d98-486d-a8dc-b2b3718357d5-----][instance:ac8ca295-9d99-4b5f-9a73-694c87af0f3c]VM Started(Lifecycle Event)

[0010] 2022-04-21 18:14:29.172 19134 INFO nova.compute.manager[req-67a9597b-4d98-486d-a8dc-b2b3718357d5-----][instance:ac8ca295-9d99-4b5f-9a73-694c87af0f3c]VM Paused(Lifecycle Event)

[0011] 2022-04-21 18:14:31.339 19134 INFO nova.compute.manager[req-67a9597b-4d98-486d-a8dc-b2b3718357d5-----][instance:ac8ca295-9d99-4b5f-9a73-694c87af0f3c]VM Resumed(Lifecycle Event)

[0012] During log parsing, the key words "Started," "Paused," and "Resumed" in log events can easily be marked as variables * when searching for common subsequences. This can cause several log events with different semantic features to be mistakenly parsed as a single event VM<*> (Lifecycle Event). Summary of the Invention

[0013] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the existing technology and provide an online parsing method for multi-source logs. The method can quickly classify, query and retrieve logs through a log tree, and classify verb part-of-speech features, effectively solving the problem that log events with different semantic features are mistakenly classified as one type of event.

[0014] In order to achieve the above object, the present invention adopts the following technical solutions:

[0015] In one aspect, the present invention provides an online parsing method for multi-source logs, comprising the following steps:

[0016] Collect logs from multiple different application components in a distributed system in the form of log streams;

[0017] Use the log tree to classify and retrieve the collected logs and extract the key field information of the logs, which includes: log identification ID, host, path, timestamp, level and log content message;

[0018] Preprocess the message field in the key field information of the log. Use regular expressions to replace the common variable markers IP addresses, numeric variables, and URL address characters in the message field with the wildcard character *.

[0019] For the pre-processed log message field, group the logs using the first word;

[0020] Verb part-of-speech features are extracted from the message field content of the grouped logs. Logs are classified based on the verb part-of-speech features. Logs containing verb part-of-speech features are classified as execution logs, while logs without verb part-of-speech features are classified as status logs.

[0021] Logs with similar verb part-of-speech features are subjected to the longest common subsequence (LCS) algorithm to extract log event templates and update log templates.

[0022] Preferably, the collection is performed in the form of log streams, specifically:

[0023] Adopt the open source log management solution of ElasticStack, collect server logs through the lightweight component of filebeat, and configure the corresponding directory files for collection.

[0024] Preferably, the log tree is used to classify and retrieve the collected logs and extract key field information of the logs, specifically:

[0025] Classification and retrieval are performed through a log tree. The first layer of the log tree is the host node, the second layer is the program module node, and the third layer is the log key field information. When a new log is input, the host node is matched first, and then the program module node is matched. If the match is not successful, a branch node is added, and then the log key field is extracted. After the above steps, log key field information in a unified format is extracted from logs from different application components. The log key field information includes a log identifier, log source information, and log field information. The log identifier is the unique ID number of the log, and the log source information includes: the host name host where the log is located and the path path of the log; the log field information includes: the time series feature timestamp of the log, the specific log content message of the log, and the importance feature level of the log. For logs that do not contain the level field, the default value is set.

[0026] Preferably, the logs are grouped using the first word, specifically:

[0027] When selecting the first word, the corpus method is used for judgment. When the first word selected is a word outside the corpus range, the second word is selected as the first word identifier, and so on. When the first word is a word in the corpus, it is directly used as the first word identifier of the log; after determining the first word, if there is already a first word identifier group for the log, the log ID is directly added to the group. If there is no first word identifier group for the log, a first word group is automatically created and the log ID is added to the group.

[0028] Preferably, the verb part-of-speech features are extracted from the message field content of the grouped logs, and classified according to the verb part-of-speech features. Logs containing verb part-of-speech features are classified as execution logs and grouped according to specific verb features; logs without verb part-of-speech features are classified as status logs, specifically:

[0029] Use the NLTK part-of-speech tagging toolkit to tag the logs with parts of speech, extract the verb part-of-speech features in the logs, and classify the logs with verb part-of-speech features as execution logs, and those without verb part-of-speech features as status logs; use the verb part-of-speech features of the logs to classify the logs. If there is already a verb part-of-speech feature classification corresponding to the log, the ID of the log is added; if there is no verb part-of-speech feature classification for the log, a verb part-of-speech feature classification is automatically created, and the log ID is added to the classification; if the input log does not contain verb part-of-speech features, the input log is classified as a status log.

[0030] Preferably, the logs with similar verb part-of-speech features are subjected to the longest common subsequence (LCS) algorithm to extract the log event template and update the log template, specifically:

[0031] The existing log event templates in the same verb part-of-speech feature group are arranged in descending order from long to short. The newly added logs are calculated in sequence with the log event templates in the group for the longest common subsequence. If the length of the longest common subsequence is more than half of the length of the existing log template, the match is successful and the log ID is directly added to the existing log template group. When the length of the original log template is greater than the length of the longest common subsequence, the log template needs to be updated with the existing longest common subsequence; if the length of the longest common subsequence is less than half of the length of the existing template, the match fails and a new independent log template needs to be created and the ID added to the template.

[0032] Another aspect of the present invention provides an online parsing system for multi-source logs, which is applied to the online parsing method for multi-source logs, including: a log collection module, a key field extraction module, a log preprocessing module, a first word grouping module, a verb part-of-speech feature classification module, and a log event template extraction module;

[0033] The log collection module is used to collect logs from multiple different application components in a distributed system in the form of log streams;

[0034] The key field extraction module is used to classify and retrieve the collected logs using the log tree and extract the log key field information, which includes: log identification ID, host, path, timestamp, level and log content message;

[0035] The log preprocessing module is used to preprocess the message field in the log key field information and replace the special marks in the message field with wildcards through regular expressions;

[0036] The first word grouping module is used to group the logs using the first word in the pre-processed log message field;

[0037] The verb part-of-speech feature classification module is used to extract verb part-of-speech features from the message field content of the log, classify the logs according to the verb part-of-speech features, and classify the logs containing verb part-of-speech features as execution logs, and the logs without verb part-of-speech features as status logs;

[0038] The log event template extraction module is used to extract log event templates and update log templates from logs with similar verb part-of-speech features through the longest common subsequence (LCS) algorithm.

[0039] Another aspect of the present invention provides an electronic device, characterized in that the electronic device includes:

[0040] at least one processor; and,

[0041] a memory communicatively coupled to the at least one processor, wherein:

[0042] The memory stores computer program instructions that can be executed by the at least one processor. The computer program instructions are executed by the at least one processor to enable the at least one processor to perform the online parsing method for multi-source logs.

[0043] In another aspect, the present invention provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the online parsing method for multi-source logs.

[0044] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0045] The present invention combines the characteristics of distributed software system logs to design a data structure and general method that supports online parsing of multi-source logs. By distinguishing execution logs from status logs based on the verb part-of-speech features of the logs, and combining the multi-source information of the logs, it can provide an important data foundation for constructing workflow diagrams and root cause analysis in subsequent log analysis, effectively solving the overfitting problem in which logs with the same structure but different semantics are mistakenly treated as the same log event during the template extraction process. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0047] Figure 1 This is a flow chart of an online parsing method for multi-source logs according to an embodiment of the present invention.

[0048] Figure 2 A schematic diagram of a search tree for an online parsing method for multi-source logs according to an embodiment of the present invention.

[0049] Figure 3 The figure is a schematic diagram of the structure of an online parsing system for multi-source logs according to an embodiment of the present invention.

[0050] Figure 4 2 is a structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0052] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0053] like Figure 1 As shown, Figure 2 As shown, this embodiment provides an online parsing method for multi-source logs, including the following steps:

[0054] S1. Collect logs from multiple different application components in a distributed system in the form of log streams.

[0055] Furthermore, the collection is performed in the form of log streams, specifically:

[0056] Adopt the open source log management solution of ElasticStack, collect server logs through the lightweight component of filebeat, and configure the corresponding directory files for collection.

[0057] Furthermore, the input log stream comes from the control service log / var / log / nova / nova-api.log of the control01 control host and the computing service log / var / log / nova / nova-compute.log of the compute node host.

[0058] [1]control01 / var / log / nova / nova-api.log 2022-04-27 18:33:47.956 1343INFO nova.api.openstack.compute.server_external_events

[0059] [req-288b9b4f-cda5-4e29-a3a1-ede2f4073674 ba82731d70654e39bc5e832454a6fd06 d4d0d97635944209a9093fcd443531d7-default default]Creating eventnetwork-changed:830ec68b-e073-4932-bcec-94c12afe1377 for instance b9000564-fe1a-409b-b8cc-1e88b294cd1d on compute1.

[0060] [2]control01 / var / log / nova / nova-api.log 2022-04-27 18:34:44.264 1317INFOnova.api.openstack.compute.server_external_events

[0061] [req-ae103bb1-d263-4fd5-b425-ce546d38f655 ba82731d70654e39bc5e832454a6fd06 d4d0d97635944209a9093fcd443531d7-default default]Creating eventnetwork-vif-plugged:830ec68b-e073-4932-bcec-94c12afe1377 for instance9069c8d7-cfd7-4440-88b4-0e1577cddd76 on compute3.

[0062] [3]compute3 / var / log / nova / nova-compute.log 2022-04-27 18:34:44.31519134 INFO nova.virt.libvirt.driver[-][instance:9069c8d7-cfd7-4440-88b4-0e1577cddd76]Instance spawned successfully.

[0063] [4]compute3 / var / log / nova / nova-compute.log 2022-04-27 19:02:40.63219134 INFO nova.virt.libvirt.driver[-][instance:9069c8d7-cfd7-4440-88b4-0e1577cddd76]Instance destroyed successfully.

[0064] S2. Classify and retrieve the collected logs using the log tree and extract key field information of the logs, including log identification ID, host, path, timestamp, level and log content message;

[0065] Furthermore, the first layer of the log tree is the host node, the second layer is the program module node, and the third layer is the log key field information. When a new log is input, the host node is matched first, and then the program module node is matched. If the match is not successful, a branch node is added, and then the log key field is extracted; after the above steps, log key field information in a unified format is extracted from logs from different application components, and the log key field information includes a log identifier, log source information, and log field information, wherein the log identifier is a unique ID number of the log, and the log source information includes: the host name host where the log is located and the path path of the log; the log field information includes: the time series feature timestamp of the log, the specific log content message of the log, and the importance feature level of the log. For logs that do not contain the level field, it is set to the default value default.

[0066] Furthermore, we extract field information in a unified format from logs from different components, as shown in the following table.

[0067]

[0068]

[0069] S3. Preprocess the message field in the key field information of the log, and replace the common variable markers IP address, numeric variable, and URL address characters in the message field with the wildcard * through regular expressions.

[0070] [1]Creating event network-changed:*for instance*on compute1

[0071] [2]Creating event network-vif-plugged:*for instance*on compute3

[0072] [3]Instance spawned successfully.

[0073] [4]Instance destroyed successfully.

[0074] S4. Group the logs using the first word in the message field of the preprocessed logs.

[0075] Furthermore, the corpus method is used to make a judgment when selecting the first word. When the first word selected is not in the corpus, the second word is selected as the first word identifier, and so on. When the first word is a word in the corpus, it is directly used as the first word identifier of the log; after determining the first word, if there is already a first word identifier group for the log, the ID of the log is directly added to the group. If there is no first word identifier group for the log, a first word group is automatically created and the ID of the log is added to the group.

[0076] Creating: {1,2}

[0077] Instance:{3,4}

[0078] S5. Extract verb part-of-speech features from the message field content of the grouped logs, classify them according to the verb part-of-speech features, and classify logs containing verb part-of-speech features as execution logs. Group them according to the specific verb features, and logs without verb part-of-speech features are classified as status logs.

[0079] Furthermore, the NLTK part-of-speech tagging toolkit is used to perform part-of-speech tagging on the logs, and the verb part-of-speech features in the logs are extracted. The logs containing verb part-of-speech features are classified as execution logs, and the logs without verb part-of-speech features are classified as status logs; the logs are classified using the verb part-of-speech features of the logs. If there is already a verb part-of-speech feature classification corresponding to the log, the ID of the log is added; if there is no verb part-of-speech feature classification for the log, a verb part-of-speech feature classification is automatically created, and the log ID is added to the classification; if the input log does not contain verb part-of-speech features, the input log is classified as a status log classification.

[0080] ('Creating','VBG'): {1,2}

[0081] ('spawned','VBD'): {3}

[0082] ('destroyed','VBD'):{4}

[0083] S6. Extract log event templates and update log templates from logs with the same verb part-of-speech features using the longest common subsequence (LCS) algorithm.

[0084] Furthermore, the existing log event templates within the same verb part-of-speech feature group are arranged in descending order from long to short, and the newly added logs are calculated in order with the log event templates in the group for the longest common subsequence. If the length of the longest common subsequence is more than half of the length of the existing log template, the match is successful, and the log ID is directly added to the existing log template group. When the length of the original log template is greater than the length of the longest common subsequence, the log template needs to be updated with the existing longest common subsequence; if the length of the longest common subsequence is less than half of the length of the existing template, the match fails, and a new independent log template needs to be created, and the ID is added to the template.

[0085] Creating event*for instance*on*{1,2}

[0086] Instance spawned successfully.{3}

[0087] Instance destroyed successfully.{4}

[0088] Based on the same concept as the online parsing method for multi-source logs in the above embodiment, the present invention also provides an online parsing system for multi-source logs, which can be used to execute the above-mentioned online parsing method for multi-source logs. For ease of explanation, the structural diagram of an embodiment of an online parsing system for multi-source logs only shows the parts related to the embodiment of the present invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation of the device, and it may include more or fewer components than shown in the diagram, or combine certain components, or arrange the components differently.

[0089] like Figure 3 As shown, in another embodiment of the present invention, an online parsing system 100 for multi-source logs is provided, which includes a log collection module 101, a key field extraction module 102, a log preprocessing module 103, a first word grouping module 104, a verb part-of-speech feature classification module 105, and a log event template extraction module 106;

[0090] The log collection module 101 is used to collect logs from multiple different application components in a distributed system in the form of log streams;

[0091] The key field extraction module 102 is used to classify and retrieve the collected logs using the log tree and extract the key field information of the logs, wherein the key field information of the logs includes: log identification ID, host, path, timestamp, level and log content message;

[0092] The log preprocessing module 103 is used to preprocess the message field in the log key field information and replace the special marks in the message field with wildcards through regular expressions;

[0093] The first word grouping module 104 groups the logs using the first word in the pre-processed message field of the logs;

[0094] The verb part-of-speech feature classification module 105 is used to extract verb part-of-speech features from the message field content of the log, classify the logs according to the verb part-of-speech features, and classify the logs containing verb part-of-speech features as execution logs, and classify the logs without verb part-of-speech features as status logs;

[0095] The log event template extraction module 106 is used to extract log event templates and update log templates from logs with similar verb part-of-speech features through the longest common subsequence (LCS) algorithm.

[0096] It should be noted that the online parsing system for multi-source logs of the present invention corresponds one-to-one to the online parsing method for multi-source logs of the present invention. The technical features and beneficial effects described in the embodiment of the online parsing method for multi-source logs are applicable to the embodiment of the online parsing system for multi-source logs. For specific details, please refer to the description in the embodiment of the method of the present invention. No further details will be given here. This is hereby declared.

[0097] In addition, in the implementation of the online parsing system for multi-source logs in the above embodiment, the logical division of each program module is only an example. In actual application, the above functions can be assigned to different program modules as needed, for example, for the convenience of corresponding hardware configuration requirements or software implementation. That is, the internal structure of the online parsing system for multi-source logs is divided into different program modules to complete all or part of the functions described above.

[0098] like Figure 4 As shown, in another embodiment of the present invention, an electronic device for online parsing of multi-source logs is provided. The electronic device 200 may include a first processor 201, a first memory 202 and a bus, and may also include a computer program stored in the first memory 202 and executable on the first processor 201, such as an online parsing program 203 for multi-source logs.

[0099] Among them, the first memory 202 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example, SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the first memory 202 can be an internal storage unit of the electronic device 200, such as a mobile hard disk of the electronic device 200. In other embodiments, the first memory 202 can also be an external storage device of the electronic device 200, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card, etc. equipped on the electronic device 200. Furthermore, the first memory 202 can also include both an internal storage unit of the electronic device 200 and an external storage device. The first memory 202 can not only be used to store application software and various types of data installed on the electronic device 200, such as the code of the online parsing program 203 of the multi-source log, but can also be used to temporarily store data that has been output or is to be output.

[0100] In some embodiments, the first processor 201 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including a combination of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The first processor 201 is the control core (Control Unit) of the electronic device, connecting the various components of the entire electronic device using various interfaces and lines, and executing or executing programs or modules stored in the first memory 202, as well as calling data stored in the first memory 202, to perform various functions of the electronic device 200 and process data.

[0101] Figure 4 Only the electronic device with components is shown, and it can be understood by those skilled in the art that Figure 4 The structure shown does not constitute a limitation on the electronic device 200 , and the electronic device 200 may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0102] The online parsing program 203 of the multi-source log stored in the first memory 202 of the electronic device 200 is a combination of multiple instructions. When executed in the first processor 201, it can achieve the following:

[0103] Collect logs from multiple different application components in a distributed system in the form of log streams;

[0104] Use the log tree to classify and retrieve the collected logs and extract the key field information of the logs, which includes: log identification ID, host, path, timestamp, level and log content message;

[0105] Preprocess the message field in the key field information of the log, and replace the special tokens in the message field with wildcards through regular expressions;

[0106] For the pre-processed log message field, group the logs using the first word;

[0107] Verb part-of-speech features are extracted from the message field content of the grouped logs. Logs are classified based on the verb part-of-speech features. Logs containing verb part-of-speech features are classified as execution logs, while logs without verb part-of-speech features are classified as status logs.

[0108] Logs with similar verb part-of-speech features are subjected to the longest common subsequence (LCS) algorithm to extract log event templates and update log templates.

[0109] Furthermore, if the modules / units integrated in the electronic device 200 are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0110] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0111] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0112] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A multi-source log online parsing method, characterized in that: The steps include: Collect logs from multiple different application components in a distributed system in the form of log streams; Use the log tree to classify and retrieve the collected logs and extract the key field information of the logs, which includes: log identification ID, host name host, log path path, log time series feature timestamp, log importance feature level and log content message; Preprocess the message field in the key field information of the log. Use regular expressions to replace the common variable markers IP addresses, numeric variables, and URL address characters in the message field with the wildcard character *. For the pre-processed log message field, group the logs using the first word; Verb part-of-speech features are extracted from the message field content of the grouped logs. Classification is performed based on the verb part-of-speech features. Logs containing verb part-of-speech features are classified as execution logs, while logs without verb part-of-speech features are classified as status logs. Specifically: Use the NLTK part-of-speech tagging toolkit to perform part-of-speech tagging on the logs, extract the verb part-of-speech features in the logs, and classify the logs containing verb part-of-speech features as execution logs, and those without verb part-of-speech features as status logs. Use the verb part-of-speech features of the logs to classify the logs. If a verb part-of-speech feature classification already exists for the log, the log ID is added. If a verb part-of-speech feature classification does not already exist for the log, a verb part-of-speech feature classification is automatically created and the log ID is added to the classification. If the input log does not contain verb part-of-speech features, the input log is classified as a status log. The logs with similar verb part-of-speech features are extracted using the longest common subsequence (LCS) algorithm to extract log event templates and update log templates. Specifically: The existing log event templates in the same verb part-of-speech feature group are arranged in descending order from long to short. The newly added logs are calculated in sequence with the log event templates in the group for the longest common subsequence. If the length of the longest common subsequence is more than half of the length of the existing log template, the match is successful and the log ID is directly added to the existing log template group. When the length of the original log template is greater than the length of the longest common subsequence, the log template needs to be updated with the existing longest common subsequence; if the length of the longest common subsequence is less than half of the length of the existing template, the match fails and a new independent log template needs to be created and the ID added to the template.

2. The online parsing method for multi-source logs according to claim 1, characterized in that: The collection is performed in the form of log streams, specifically: Adopt the open source log management solution of ElasticStack, collect server logs through the lightweight component of filebeat, and configure the corresponding directory files for collection.

3. The online parsing method for multi-source logs according to claim 1, characterized in that: The log tree is used to classify and retrieve the collected logs and extract key field information of the logs, specifically: Classification and retrieval are performed through the log tree. The first layer of the log tree is the host node, the second layer is the program module node, and the third layer is the log key field information. When a new log is input, the host node is matched first, and then the program module node is matched. If no match is successful, a branch node is added, and then the log key field is extracted; After the above steps, log key field information in a unified format is extracted from logs from different application components. The log key field information includes log identifier, log source information, and log field information. The log identifier is the unique ID number of the log, and the log source information includes: the host name host where the log is located and the log path path; the log field information includes: the log's time series feature timestamp, the log's specific log content message, and the log's importance feature level. For logs that do not contain the level field, the default value is set.

4. The online parsing method for multi-source logs according to claim 1, characterized in that: The logs are grouped using the first word, specifically: When selecting the first word, the corpus method is used for judgment. When the first word selected is a word outside the corpus range, the second word is selected as the first word identifier, and so on. When the first word is a word in the corpus, it is directly used as the first word identifier of the log; after determining the first word, if there is already a first word identifier group for the log, the log ID is directly added to the group. If there is no first word identifier group for the log, a first word group is automatically created and the log ID is added to the group.

5. An online parsing system for multi-source logs, used to implement the online parsing method for multi-source logs according to any one of claims 1 to 4, characterized in that: Includes: log collection module, key field extraction module, log preprocessing module, first word grouping module, verb part-of-speech feature classification module and log event template extraction module; The log collection module is used to collect logs from multiple different application components in a distributed system in the form of log streams; The key field extraction module is used to classify and retrieve the collected logs using the log tree and extract the log key field information, which includes: log identification ID, host, path, timestamp, log importance feature level and log content message; The log preprocessing module is used to preprocess the message field in the log key field information and replace the special marks in the message field with wildcards through regular expressions; The first word grouping module is used to group the logs using the first word in the pre-processed log message field; The verb part-of-speech feature classification module is used to extract verb part-of-speech features from the message field content of the log, classify the logs according to the verb part-of-speech features, and classify the logs containing verb part-of-speech features as execution logs, and the logs without verb part-of-speech features as status logs; The log event template extraction module is used to extract log event templates and update log templates from logs with similar verb part-of-speech features through the longest common subsequence (LCS) algorithm.

6. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively coupled to the at least one processor, wherein: The memory stores computer program instructions executable by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can perform the online parsing method for multi-source logs according to any one of claims 1 to 4.

7. A computer-readable storage medium storing a program, characterized in that: When the program is executed by a processor, the online parsing method for multi-source logs according to any one of claims 1 to 4 is implemented.

Images