Method for generating log parsing template, log parsing method, device and equipment

Abstract

The present disclosure relates to a method for generating a log parsing template, a log parsing method, an apparatus and a device. The method includes: obtaining a target log to be parsed; determining the format of the target log and one or more feature fields of the target log; determining an initial parsing template based on the one or more feature fields; and correcting the initial parsing template based on the format to obtain a log parsing template. By determining the format of the target log and one or more feature fields of the target log, and determining an initial parsing template based on the one or more feature fields, the present disclosure is used to complete the parsing of logs in different formats, and corrects the initial parsing template based on the format of the target log, making the initial parsing template more perfect, enabling the logs in the same format to be parsed more quickly, reducing the writing cost of log parsing personnel, improving the writing efficiency, and reducing the later operation and maintenance cost.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technologies, and in particular, to a method for generating a log parsing template, a log parsing method, an apparatus, and a device. Background Art

[0002] With the continuous maturity of big data technologies, big data log audit systems have become increasingly popular. A big data log audit system can collect in real time logs generated by various network devices, security devices, hosts, operating systems, and various application systems of different manufacturers in a user network. However, such information needs to be subjected to certain escape encoding to form a specific data format so that users can perform operations such as querying, statistics, and correlation analysis on the collected parsed logs on an analysis platform. Since the current systems are relatively complex and huge in volume, and the types of devices included in a system are numerous, the data formats of log source information are also diverse. The traditional approach is to write a corresponding set of parsing codes for each type of device to convert the log source information into a data format that can be used in the presentation layer.

[0003] Generally, log parsing personnel often spend a great deal of effort writing the parsing of such log source information and debugging it so as to connect the logs to the platform for further use. However, the written parsing rules are not centrally managed, and for the same type of data, the rules need to be rewritten to complete the parsing, and cannot be reused. Summary of the Invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a method for generating a log parsing template, a log parsing method, an apparatus, and a device, so as to reduce the writing cost of log parsing personnel and improve the writing efficiency.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for generating a log parsing template, which is characterized by including:

[0006] Obtain a target log to be parsed;

[0007] Determine the format of the target log and one or more characteristic fields of the target log;

[0008] Based on the one or more characteristic fields, determine an initial parsing template;

[0009] Based on the format, correct the initial parsing template to obtain a log parsing template.

[0010] In a second aspect, an embodiment of the present disclosure provides a log parsing method, which is characterized by including:

[0011] Obtain a target log to be parsed;

[0012] Determine the format of the target log and one or more characteristic values of the target log;

[0013] Based on the format, the one or more characteristic values, and a pre-established association relationship, select a log parsing template; wherein, the association relationship is an association relationship among a log parsing template, a log format, and at least one characteristic value;

[0014] Parse the target log based on the log parsing template.

[0015] In a third aspect, an embodiment of the present disclosure provides a log parsing template generation device, which is characterized by including:

[0016] An acquisition unit, configured to acquire a target log to be parsed;

[0017] A first determination unit, configured to determine the format of the target log and one or more characteristic fields of the target log;

[0018] A second determination unit, configured to determine an initial parsing template based on the one or more characteristic fields;

[0019] A correction unit, configured to correct the initial parsing template based on the format to obtain a log parsing template.

[0020] In a fourth aspect, an embodiment of the present disclosure provides a log parsing device, which is characterized by including:

[0021] An acquisition unit, configured to acquire a target log to be parsed;

[0022] A first determination unit, configured to determine the format of the target log and one or more characteristic values of the target log;

[0023] A selection unit, configured to select a log parsing template based on the format, the one or more characteristic values, and a pre-established association relationship; wherein, the association relationship is an association relationship among a log parsing template, a log format, and at least one characteristic value;

[0024] A parsing unit, configured to parse the target log based on the log parsing template.

[0025] In a fifth aspect, an embodiment of the present disclosure provides an electronic device, including:

[0026] A memory;

[0027] A processor; and

[0028] A computer program;

[0029] Among them, the computer program is stored in the memory and is configured to be executed by the processor to implement the methods described in the first aspect and the second aspect.

[0030] A method for generating a log parsing template, a log parsing method, a device, and a device provided by an embodiment of the present disclosure determine the format of a target log and one or more feature fields of the target log, and determine an initial parsing template based on the one or more feature fields for completing the parsing work of logs in different formats, and correct the initial parsing template based on the format of the target log, so that the initial parsing template is more perfect, logs in the same format can be parsed more quickly, reducing the writing cost of log parsing personnel, improving the writing efficiency, and reducing the later operation and maintenance cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure and used together with the specification to explain the principles of the present disclosure.

[0032] To more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the following will briefly introduce the accompanying drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0033] Figure 1 A schematic flowchart of a method for generating a log parsing template provided by an embodiment of the present disclosure;

[0034] Figure 2 A schematic flowchart of a method for determining the format of a target log provided by an embodiment of the present disclosure;

[0035] Figure 3 A schematic flowchart of a method for determining an initial parsing template provided by an embodiment of the present disclosure;

[0036] Figure 4 A schematic flowchart of a method for correcting an initial parsing template provided by an embodiment of the present disclosure;

[0037] Figure 5 A schematic flowchart of a method for parsing a log provided by an embodiment of the present disclosure;

[0038] Figure 6 A schematic diagram of a device for generating a log parsing template provided by an embodiment of the present disclosure;

[0039] Figure 7 A schematic diagram of a device for parsing a log provided by an embodiment of the present disclosure;

[0040] Figure 8Structural schematic diagram of the electronic device provided by the embodiments of the present disclosure. Detailed implementation manners

[0041] In order to more clearly understand the above objects, features and advantages of the present disclosure, the solutions of the present disclosure will be further described below. It should be noted that, without conflict, the embodiments of the present disclosure and the features in the embodiments may be combined with each other.

[0042] In the following description, many specific details are set forth in order to fully understand the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only a part of the embodiments of the present disclosure, rather than all the embodiments.

[0043] Logstash is an open-source server-side data processing pipeline that can simultaneously obtain data from multiple data sources, transform it, and then send it to a preset storage location, such as Es (Elasticsearch, a distributed full-text search engine) or different types of databases.

[0044] During the process of data transmission from the source end to the storage location, the filters in Logstash can parse each event, and the events come from the logs and alerts of security devices, network devices, hosts, operating systems, database systems, user business systems of different manufacturers, etc. After the filters in Logstash parse the events, they can identify the named fields to construct a structure and convert them into a common format to improve the efficiency of data analysis.

[0045] Logstash can dynamically transform and parse data, regardless of format or complexity. The specific steps are as follows 1) to 4):

[0046] 1) Manually judge the data format, consult the relevant filter syntax, and derive the structure of data of types such as JSON (JavaScript Object Notation) and xml (Extensible Markup Language) through configuration.

[0047] 2) Manually find the data of the IP type field according to the data with the extracted structure (such as "172.16.213.132"), and obtain other geographical location information through configuration again.

[0048] 3) The time type field needs to be converted to the standard format.

[0049] 4) Complete the parsing of this type of data.

[0050] However, the filter using Logstash has the following disadvantages:

[0051] 1) When writing data parsing using the filter of Logstash, relevant documents need to be queried on the Internet. When extracting json-type data, parameters such as setting the character set encoding of the data and the field names to be read need to be configured according to the method provided by Logstash;

[0052] 2) The written filter rules are not centrally managed. For the same type of data, the rules need to be rewritten to complete the parsing and cannot be reused;

[0053] 3) The labor cost is high, and continuous debugging is required to obtain the desired log results.

[0054] Therefore, at least one embodiment of the present disclosure provides a method for generating a log parsing template, a log parsing method, a device, and a device. By determining the format of the target log and one or more characteristic fields of the target log, an initial parsing template is determined based on the one or more characteristic fields to complete the parsing work of logs in different formats, and the initial parsing template is corrected based on the format of the target log, so that the initial parsing template is more perfect, the logs in the same format can be parsed more quickly, the writing cost of log parsing personnel is reduced, the writing efficiency is improved, and the later operation and maintenance cost is reduced.

[0055] Figure 1 It is a schematic flowchart of a method for generating a log parsing template provided by an embodiment of the present disclosure. As Figure 1 shown, the method for generating the log parsing template includes but is not limited to the following steps S101 to S104:

[0056] S101. Obtain the target log to be parsed.

[0057] Network devices, systems, and service programs, etc., will generate logs when operating. Logs are used to record events; each line of log records descriptions of related operations such as date, time, user, and action.

[0058] Network operating systems are designed with various log files, such as application logs, security logs, system logs, Scheduler service logs, FTP (File Transfer Protocol) logs, WWW (World Wide Web) logs, DNS (Domain Name System) server logs, etc. These vary according to the services enabled on the system. When operating on the system, the log files usually record some relevant content of the operation.

[0059] Optionally, the log can be structured data or unstructured data.

[0060] S102. Determine the format of the target log and one or more characteristic fields of the target log.

[0061] There are various formats of logs. For example, they include the key-value (kv) format, the json format, the syslog format, and the long text format. When the target log to be parsed is obtained, the format of the target log and one or more characteristic fields it contains can be determined.

[0062] For example, if the log is: dev_type = "waf" time = "2021 / 08 / 01 33:12:11" ip = "194.2.3.2", it can be determined that the format of the log is the kv format, and its characteristic fields include the device type (device type or dev_type), time, and address (Internet Protocol or ip).

[0063] S103. Determine an initial parsing template based on one or more characteristic fields.

[0064] In the prior art, in order to facilitate the parsing of the characteristic values of different characteristic fields, a corresponding field parsing template is set for each characteristic field. The field parsing template is used to parse the characteristic value of the corresponding characteristic field. Therefore, after determining one or more characteristic fields of the target log, an initial parsing template for parsing the target log can be obtained based on the field parsing templates corresponding to these characteristic fields respectively. In some embodiments, the initial parsing template is a template obtained by merging the field parsing templates corresponding to one or more characteristic fields respectively.

[0065] For example, the log is: dev_type = "waf" time = "2021 / 08 / 01 33:12:11" ip = "194.2.3.2". The characteristic fields of this log are dev_type, time, and ip. The field parsing templates corresponding to dev_type, time, and ip can be merged into an initial parsing template. When the log received again includes the characteristic fields: dev_type, time, and ip, this initial parsing template can be used for parsing.

[0066] S104. Modify the initial parsing template based on the format to obtain a log parsing template.

[0067] Considering that other information related to logs (such as geographical location information, asset information, etc.) can be provided to the operation and maintenance personnel together with the log parsing results, and different log formats may have different ways of obtaining other information related to logs. Therefore, it is necessary to correct the initial parsing template based on the log format to obtain a log parsing template. The log parsing template can not only parse the information contained in the log itself, but also parse the information related to the log.

[0068] It can be seen that the embodiments of the present disclosure complete the parsing work of logs in different formats by obtaining the target log to be parsed, determining the format of the target log and one or more characteristic fields of the target log, and determining the initial parsing template based on the one or more characteristic fields. And correct the initial parsing template based on this format, making the initial parsing template more perfect. Subsequently, logs in the same format can directly use the log parsing template for parsing, thereby reducing the code workload of log parsing personnel, improving the writing efficiency, and reducing the subsequent operation and maintenance costs.

[0069] Figure 2 FIG. is a flowchart for determining the format of a target log provided by an embodiment of the present disclosure. As Figure 2 shown, determining the format of the target log includes the following steps S201 to S203:

[0070] S201. Based on a third-party Java (computer programming language) class library, determine whether the format of the target log is a key-value (kv) format or a JavaScript Object Notation (json) format; if not, execute step S202;

[0071] S202. Based on the RFC3164 data standard, determine whether the format of the target log is a syslog format; if not, execute step S203;

[0072] S203. Determine that the format of the target log is a long text format.

[0073] The conventional format types of target logs include: kv format, json format, syslog format, and long text format. To determine which format the target log belongs to, first use an existing third-party Java class library to determine whether it is a kv format or a json format. If so, mark it as the kv format or the json format and end the step; if not, use the RFC3164 data standard to determine whether it is a syslog format. If so, mark it as the syslog format and end the step. If not, mark it as the long text format. Determining the format type of the target log can quickly find the corresponding template and complete the parsing based on its format and the characteristics it contains, improving the parsing efficiency.

[0074] Figure 3 A schematic flowchart of a process for determining an initial parsing template provided by an embodiment of the present disclosure. This process can be applied to Figure 1 step 103 in Figure 3 As shown, based on one or more feature fields, determining the initial parsing template includes the following steps S301 to S304:

[0075] S301. Determine the field parsing templates preset for one or more feature fields.

[0076] In the prior art, in order to facilitate the parsing of the feature values of different feature fields, a corresponding field parsing template is set for each feature field, and the field parsing template is used to parse the feature value of the corresponding feature field.

[0077] For example: The log is: dev_type = "waf" time = "2021 / 08 / 01 33:12:11" ip = "194.2.3.2". The feature fields of this log are dev_type (device type), time (time), and ip (address). Among them, the field parsing template corresponding to the feature field dev_type can parse this feature field, and the obtained feature value is waf; the field parsing template corresponding to the feature field time can parse this feature field, and the obtained feature value is: 2021 / 08 / 01 33:12:11; the field parsing template corresponding to the feature field ip can parse this feature field, and the obtained feature value is: 194.2.3.2.

[0078] S302. Select at least one field parsing template to parse the corresponding feature field to obtain at least one feature value.

[0079] For example: The log is: dev_type = "waf" time = "2021 / 08 / 01 33:12:11" ip = "194.2.3.2". The feature fields of this log are dev_type (device type), time (time), and ip (address). Select the field parsing template corresponding to the feature field dev_type to parse this feature field, and the obtained feature value is waf.

[0080] S303. Merge the unselected field parsing templates into an initial parsing template.

[0081] For example, the log is: dev_type = "waf" time = "2021 / 08 / 01 33:12:11" ip = "194.2.3.2". The characteristic fields of this log are dev_type (device type), time (time), and ip (address). If the characteristic field dev_type is selected, the unselected characteristic fields are time and ip. Therefore, the field parsing templates corresponding to the characteristic field time and the field parsing templates corresponding to the characteristic field ip are merged into the initial parsing template. Similarly, if the characteristic field dev_type and the characteristic field ip are selected, the unselected characteristic field is time. Therefore, the field parsing template corresponding to the characteristic field time is used as the initial parsing template.

[0082] S304. Establish an association relationship between the initial parsing template and at least one characteristic value.

[0083] For example, the log is: dev_type = "waf" time = "2021 / 08 / 01 33:12:11" ip = "194.2.3.2". The characteristic fields of this log are dev_type (device type), time (time), and ip (address).

[0084] In some embodiments, if the field parsing template corresponding to the characteristic field dev_type is used to parse this characteristic field and the obtained characteristic value is waf, the unselected characteristic fields are time and ip. Therefore, the field parsing templates corresponding to the characteristic field time and the field parsing templates corresponding to the characteristic field ip are merged into the initial parsing template. Furthermore, an association relationship is established between this initial parsing template and the characteristic value waf of the characteristic field dev_type.

[0085] Correspondingly, when a new log is obtained subsequently, if this log includes the characteristic field dev_type and the corresponding characteristic value is waf, the initial parsing template associated with waf can be directly selected based on the established association relationship between waf and the initial parsing template, and this log can be quickly parsed without determining the initial parsing template of this log again, improving the parsing efficiency.

[0086] In other embodiments, if the field parsing template corresponding to the characteristic field dev_type is used to parse this characteristic field and the obtained characteristic value is waf, and the field parsing template corresponding to the characteristic field ip is used to parse this characteristic field and the obtained characteristic value is 194.2.3.2, the unselected characteristic field is time. Therefore, the field parsing template corresponding to the characteristic field time is used as the initial parsing template. Furthermore, an association relationship is established between this initial parsing template and the characteristic value waf and the characteristic value 194.2.3.2.

[0087] Accordingly, when obtaining new logs subsequently, if a log includes the feature fields dev_type and ip, and the corresponding feature values are waf and 194.2.3.2 respectively, then based on the established association relationship between waf, 194.2.3.2 and the initial parsing template, the initial parsing template associated with waf and 194.2.3.2 can be directly selected to quickly parse the log, without having to determine the initial parsing template of the log again, thus improving the parsing efficiency.

[0088] It can be seen that in this embodiment, by establishing an association relationship between one or more feature values and the initial parsing template, so that after obtaining new logs subsequently, the associated initial parsing template can be directly selected based on the one or more feature values to quickly parse the log, without having to determine the initial parsing template of the log again, thus improving the parsing efficiency.

[0089] Figure 4 FIG. is a schematic flowchart of a process for modifying an initial parsing template provided by an embodiment of the present disclosure. As Figure 4 shown, modifying the initial parsing template based on the format to obtain a log parsing template may include, but is not limited to, the following steps S401 and S402:

[0090] S401. Based on the format being a long text format, use one or more pre-set regular expressions to perform regular matching with the target log to obtain one or more target regular expressions that match successfully.

[0091] Among them, a regular expression (Regular Expression, a pattern used to describe a set of string features, used to match specific strings. A tool for pattern description through special characters + ordinary characters to achieve the purpose of text matching).

[0092] Different regular expressions are used to match different objects, and the objects may include, but are not limited to: time, host, ip, MAC (Media Access Control) address, port, etc. By using one or more regular expressions to perform regular matching with the target log, one or more target regular expressions that match successfully can be obtained.

[0093] S402. Add a regular extraction command to the initial parsing template to obtain a log parsing template.

[0094] Among them, the regular extraction command is used to: based on one or more target regular expressions, perform regular matching with the target log, and mark the field name of the log data that matches successfully as the feature field name identified by the corresponding target regular expression.

[0095] For example, when the target log is in the long text format, first use one or more pre-set regular expressions to perform regular matching with the target log (such as regular matching for time, host, IP, MAC, port, etc.) to obtain one or more target regular expressions that match successfully; add the commands for these regular extractions to the initial parsing template, so that the long text format log performs regular matching with the above regular expressions, and mark the field names of the log data that match successfully as the characteristic field names identified by the corresponding target regular expressions, thereby obtaining the log parsing template.

[0096] It can be seen that in this embodiment, for logs in the long text format, the log parsing template is obtained by adding regular extraction commands to the initial parsing template. In this way, after obtaining a new long text format log subsequently, when using the log parsing template to parse the log, the regular extraction commands can be executed to achieve regular matching with the log, obtain more characteristic fields of the log, and improve the parsing effect.

[0097] In some embodiments, Figure 4 the process of modifying the initial parsing template shown may further include Figure 4 step S403 not shown in

[0098] S403. Based on the format being the kv format or the json format, add commands for obtaining geographical location information and asset information to the log parsing template; wherein, the commands for obtaining geographical location information and asset information are configured to be executed after the regular extraction commands; and the commands for obtaining geographical location information and asset information are configured to: match the names of one or more characteristic fields with the string 'ip', perform ip format regular verification on the characteristic values of the matching characteristic fields, and if the verification passes, obtain the corresponding geographical location information and asset information based on the characteristic values.

[0099] In this embodiment, after performing regular verification on the characteristic field ip in the kv format or the json format, geographical location information and asset information are supplemented, which facilitates subsequent analysis and use of the log and improves the log analysis efficiency.

[0100] In some embodiments, Figure 4 the process of modifying the initial parsing template shown may further include Figure 4 step S404 not shown in

[0101] S404. Based on the format being the syslog format or the long text format, add commands for obtaining geographical location information and asset information to the log parsing template; wherein, the commands for obtaining geographical location information and asset information are configured to be executed after the regular extraction commands; and the commands for obtaining geographical location information and asset information are configured to: obtain the corresponding geographical location information and asset information based on the characteristic value corresponding to the characteristic field being the ip field.

[0102] In this embodiment, when the feature field is an IP in the syslog format or the long text format, geographical location information and asset information are supplemented to facilitate subsequent analysis and use of the log, improving the log analysis efficiency.

[0103] In some embodiments, the method for generating a log parsing template may further include: establishing an association relationship among the log parsing template, the log format, and at least one feature value based on the association relationship between the initial parsing template and at least one feature value.

[0104] For example, when the log is: dev_type = "waf" time = "2021 / 08 / 01 33:12:11" ip = "194.2.3.2", an association relationship among the log parsing template, the kv format, and the feature value waf can be established. Since this log has 3 feature values, the feature value can be any one of the three values, or a combination of two of them, or all three values.

[0105] It can be seen that in this embodiment, by establishing an association relationship among the log parsing template, the log format, and at least one feature value, for logs with the same log format and including the same feature values subsequently, the log parsing template can be directly selected according to this association relationship to quickly parse the log, without having to determine the log parsing template of the log again, improving the parsing efficiency.

[0106] Figure 5 It is a schematic flowchart of a log parsing method provided by an embodiment of the present disclosure. The method may include but is not limited to the following steps S501 to S504:

[0107] S501. Obtain the target log to be parsed.

[0108] The implementation principles and specific methods of S501 and the above-mentioned 5101 are the same, and will not be elaborated here.

[0109] S502. Determine the format of the target log and one or more feature values of the target log;

[0110] When the log to be parsed is obtained, determine the format of the log and one or more features it contains. For example: when the log is: dev_type = "waf" time = "2021 / 08 / 01 33:12:11" ip = "194.2.3.2", the format of the log is the kv format, and the feature values it contains are waf, 2021 / 08 / 01 33:12:11, and 194.2.3.2.

[0111] S503. Select a log parsing template based on the format, one or more feature values, and a pre-established association relationship, where the association relationship is the association relationship between the log parsing template, the log format, and at least one feature value.

[0112] For example, the log format, one or more feature values, and the pre-established association relationship are: the association relationship between kv, waf, and the log parsing template. Then, if the new log obtained is: dev_type = "waf" time = "2021 / 08 / 01 33:12:11" ip = "194.2.3.2", it can be determined that the log format is kv, and the feature value of dev_type is waf. Then, based on the foregoing association relationship, the log parsing template used for this new log can be directly determined.

[0113] S504. Parse the target log based on the log parsing template.

[0114] In the embodiments of the present disclosure, by obtaining the target log to be parsed; determining the format of the target log and one or more feature values of the target log; selecting a log parsing template based on the format, one or more feature values, and a pre-established association relationship, where the association relationship is the association relationship between the log parsing template, the log format, and at least one feature value; and parsing the target log based on the log parsing template, the parsing work of the target log is completed, and there is no need to determine the log parsing template of this log again, improving the parsing efficiency.

[0115] Figure 6 FIG. is a schematic diagram of a device for generating a log parsing template provided by an embodiment of the present disclosure. The device for generating the log parsing template can execute the processing flow provided by the method embodiment for generating the log parsing template. As Figure 6 shown, the device 60 for generating the log parsing template includes: an obtaining unit 61, a first determining unit 62, a second determining unit 63, and a correcting unit 64. The obtaining unit 61 is used to obtain the target log to be parsed. The first determining unit 62 is used to determine the format of the target log and one or more feature fields of the target log. The second determining unit 63 is used to determine an initial parsing template based on one or more feature fields. The correcting unit 64 is used to correct the initial parsing template based on the format to obtain the log parsing template.

[0116] Optionally, the first determining unit 62 determines the format of the target log as follows: based on a Java third-party class library, determine whether the format of the target log is the kv format or the json format; if not, based on the RFC3164 data standard, determine whether the format of the target log is the syslog format; if not, determine that the format of the target log is the long text format.

[0117] Optionally, the second determination unit 63 is configured to: determine one or more field parsing templates preset for the feature fields; select at least one field parsing template to parse the corresponding feature fields to obtain at least one feature value; merge the unselected field parsing templates into an initial parsing template; and establish an association relationship between the initial parsing template and the at least one feature value.

[0118] Optionally, the correction unit 64 is configured to: based on the format being a long text format, use one or more preset regular expressions to perform regular matching with the target log to obtain one or more target regular expressions that match successfully; add a regular extraction command to the initial parsing template to obtain a log parsing template; wherein the regular extraction command is configured to: based on the one or more target regular expressions, perform regular matching with the target log, and mark the field name of the log data that matches successfully as the feature field name identified by the corresponding target regular expression.

[0119] Optionally, the correction unit 64 is further configured to: based on the format being a kv format or a json format, add a geographical location information and asset information acquisition command to the log parsing template; wherein the geographical location information and asset information acquisition command is configured to be executed after the regular extraction command; and the geographical location information and asset information acquisition command is configured to: match the name of one or more feature fields with the string "ip", perform ip format regular verification on the feature value of the matching feature field, and if the verification passes, obtain the corresponding geographical location information and asset information based on the feature value.

[0120] Optionally, the correction unit 64 is further configured to: based on the format being a syslog format or a long text format, add a geographical location information and asset information acquisition command to the log parsing template; wherein the geographical location information and asset information acquisition command is configured to be executed after the regular extraction command; and the geographical location information and asset information acquisition command is configured to: obtain the corresponding geographical location information and asset information based on the feature value corresponding to the feature field being an ip field.

[0121] Optionally, the log parsing template generation device further includes Figure 6 a building unit (not shown in the figure) configured to establish an association relationship between the log parsing template, the log format, and the at least one feature value based on the association relationship between the initial parsing template and the at least one feature value.

[0122] Figure 6 The log parsing template generation device of the illustrated embodiment can be used to execute the technical solution of the above-mentioned log parsing template generation method embodiment, and its implementation principle and technical effect are similar, which will not be elaborated here.

[0123] Figure 7Schematic diagram of the log parsing device provided by an embodiment of the present disclosure. The log parsing device can execute the processing flow provided by the embodiment of the log parsing method, such as Figure 7 As shown, the log parsing device 70 includes: an acquisition unit 71, a first determination unit 72, a selection unit 73, and a parsing unit 74; wherein, the acquisition unit 71 is configured to acquire a target log to be parsed; the first determination unit 72 is configured to determine the format of the target log and one or more characteristic values of the target log; the selection unit 73 is configured to select a log parsing template based on the format, one or more characteristic values, and a pre-established association relationship; wherein, the association relationship is an association relationship among the log parsing template, the log format, and at least one characteristic value; the parsing unit 74 is configured to parse the target log based on the log parsing template.

[0124] Figure 7 The log parsing device in the illustrated embodiment can be used to execute the technical solution of the above-mentioned log parsing method embodiment, and its implementation principle and technical effect are similar, which will not be elaborated here.

[0125] Figure 8 Schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. The embodiment of the present disclosure provides a method for generating a log parsing template and a processing flow provided by the method embodiment of log parsing that can be executed. As Figure 8 shown, the electronic device 80 includes: a memory 81, a processor 82, and a communication interface 83; wherein, the computer program is stored in the memory 81 and is configured to be executed by the processor 82 to perform the method for generating the log parsing template and the log parsing method as described above.

[0126] In addition, an embodiment of the present disclosure further provides a computer program product, which includes a computer program or instruction, and when the computer program or instruction is executed by a processor, it implements the method for generating the log parsing template and the log parsing method as described above.

[0127] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including the element.

[0128] The above are only specific embodiments of the present disclosure, enabling those skilled in the art to understand or implement the present disclosure. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but rather will be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating a log parsing template, characterized in that, Including: Obtain a target log to be parsed; Determine the format of the target log and one or more characteristic fields of the target log, wherein the format of the target log includes at least one of the kv format, json format, syslog format, and long text format; Determine an initial parsing template based on the one or more characteristic fields; Modify the initial parsing template based on the format to obtain a log parsing template; The determining the initial parsing template based on the one or more characteristic fields includes: Determine the field parsing templates preset for the one or more characteristic fields; Select at least one field parsing template to parse the corresponding characteristic field to obtain at least one characteristic value; Merge the unselected field parsing templates into an initial parsing template; Establish an association relationship between the initial parsing template and the at least one characteristic value; The modifying the initial parsing template based on the format to obtain a log parsing template includes: When the format is the long text format, obtain the log parsing template by adding a regular extraction command to the initial parsing template; When the format is the kv format or json format, add geographical location information and asset information to the log parsing template; When the format is the syslog format or the characteristic field in the long text format is ip, add geographical location information and asset information to the log parsing template.

2. The method according to claim 1, characterized in that The determining the format of the target log includes: Judge whether the format of the target log is the kv format or json format based on a Java third-party class library; If not, judge whether the format of the target log is the syslog format based on the RFC3164 data standard; If not, determine that the format of the target log is the long text format.

3. The method according to claim 2, wherein The modifying the initial parsing template based on the format to obtain a log parsing template includes: Based on the format being the long text format, use one or more preset regular expressions to perform regular matching with the target log to obtain one or more target regular expressions that match successfully; Add a regular extraction command to the initial parsing template to obtain a log parsing template; Wherein, the regular extraction command is used for: performing regular matching with the target log based on the one or more target regular expressions, and marking the field names of the log data that match successfully as the characteristic field names identified by the corresponding target regular expressions.

4. The method according to claim 3, wherein The modifying the initial parsing template based on the format to obtain a log parsing template further includes: Based on the format being the kv format or json format, add a geographical location information and asset information acquisition command to the log parsing template; Wherein, the geographical location information and asset information acquisition command is configured to be executed after the regular extraction command; and the geographical location information and asset information acquisition command is configured to: match the names of the one or more characteristic fields with the string 'ip', perform ip format regular verification on the characteristic values of the matched characteristic fields, and if the verification passes, obtain the corresponding geographical location information and asset information based on the characteristic values.

5. The method according to claim 3, characterized in that, Correcting the initial parsing template based on the format to obtain the log parsing template further includes: Based on the format being the syslog format or the long text format, adding a geographical location information and asset information acquisition command to the log parsing template; Wherein, the geographical location information and asset information acquisition command is configured to be executed after the regular extraction command; and the geographical location information and asset information acquisition command is configured to: obtain corresponding geographical location information and asset information based on the feature value corresponding to the feature field being the ip field.

6. The method according to claim 1, wherein The method further includes: Based on the association relationship between the initial parsing template and the at least one feature value, establish an association relationship between the log parsing template, the log format, and the at least one feature value.

7. A log parsing method, characterized in that, Including: Obtain the target log to be parsed; Determine the format of the target log and one or more feature values of the target log, wherein the format of the target log includes at least one of the kv format, the json format, the syslog format, and the long text format; Based on the format, the one or more feature values, and the pre-established association relationship, select a log parsing template; wherein the association relationship is an association relationship between the log parsing template, the log format, and at least one feature value; Parse the target log based on the log parsing template; The selecting a log parsing template based on the format, the one or more feature values, and the pre-established association relationship includes: Determine the field parsing templates preset for the one or more feature fields; Select at least one field parsing template to parse the corresponding feature field to obtain at least one feature value; Merge the unselected field parsing templates into an initial parsing template; Establish an association relationship between the initial parsing template and the at least one feature value; When the format is the long text format, obtain the log parsing template by adding a regular extraction command to the initial parsing template; When the format is the kv format or the json format, add geographical location information and asset information to the log parsing template; When the format is the syslog format or the long text format and the feature field is ip, add geographical location information and asset information to the log parsing template.

8. A device for generating a log parsing template, characterized in that, Including: An acquisition unit for acquiring the target log to be parsed; A first determination unit for determining the format of the target log and one or more feature fields of the target log, wherein the format of the target log includes at least one of the kv format, the json format, the syslog format, and the long text format; A second determination unit for determining an initial parsing template based on the one or more feature fields; A correction unit for correcting the initial parsing template based on the format to obtain the log parsing template; The determining an initial parsing template based on the one or more feature fields includes: Determine the field parsing templates preset for the one or more feature fields; Select at least one field parsing template to parse the corresponding feature field to obtain at least one feature value; Merge the unselected field parsing templates into an initial parsing template; Establish an association relationship between the initial parsing template and the at least one eigenvalue; The correcting the initial parsing template based on the format to obtain a log parsing template includes: When the format is a long text format, obtaining the log parsing template by adding a regular extraction command to the initial parsing template; When the format is a kv format or a json format, adding geographical location information and asset information to the log parsing template; When the format is a syslog format or the characteristic field in the long text format is an ip, adding geographical location information and asset information to the log parsing template.

9. A log parsing device, characterized in that, Including: An obtaining unit, configured to obtain a target log to be parsed; A first determining unit, configured to determine the format of the target log and one or more eigenvalues of the target log, wherein the format of the target log includes at least one of a kv format, a json format, a syslog format, and a long text format; A selecting unit, configured to select a log parsing template based on the format, the one or more eigenvalues, and a pre-established association relationship; wherein the association relationship is an association relationship among a log parsing template, a log format, and at least one eigenvalue; A parsing unit, configured to parse the target log based on the log parsing template; The selecting a log parsing template based on the format, the one or more eigenvalues, and a pre-established association relationship includes: Determining a field parsing template preset for the one or more characteristic fields; Selecting at least one field parsing template to parse the corresponding characteristic field to obtain at least one eigenvalue; Merging the unselected field parsing templates into an initial parsing template; Establishing an association relationship between the initial parsing template and the at least one eigenvalue; When the format is a long text format, obtaining the log parsing template by adding a regular extraction command to the initial parsing template; When the format is a kv format or a json format, adding geographical location information and asset information to the log parsing template; When the format is a syslog format or the characteristic field in the long text format is an ip, adding geographical location information and asset information to the log parsing template.

10. An electronic device, characterized in that, Including: A memory; A processor; And A computer program; Wherein, the computer program is stored in the memory and is configured to be executed by the processor to implement the log parsing template generation method as described in any one of claims 1 to 6 or the log parsing method as described in claim 7.

Images