Rapid log analysis method and device based on field pre-matching

Abstract

The invention provides a quick log analysis method based on field pre-matching. The quick log analysis method comprises the following steps: acquiring a to-be-analyzed original log and an asset identifier of the to-be-analyzed original log; a rule tree is loaded, multiple layers of predefined rule nodes are arranged in the rule tree, and the rule tree comprises a generalization rule layer, a pre-matching field layer, a content matching layer, a regular layer and an event layer; inputting the to-be-analyzed original log carrying the asset identifier into a rule tree of a preset rule node, performing log analysis based on a generalization rule, and converting the to-be-analyzed original log into a standardized format; performing field pre-matching based on the log generalization result to obtain a field matching type; based on the field matching type, a next input layer is selected, and the input layer is an event layer or a regular layer; acquiring corresponding event information according to the field mapping or regular matching result; and outputting the original log of the to-be-analyzed log and an analysis result thereof, wherein the analysis result comprises the asset identifier, the to-be-analyzed log, the standardized log and the event.

Description

Technical Field

[0001] The present invention relates to the technical field of log parsing, and in particular to a method and device for rapid log parsing based on field pre-matching. Background Art

[0002] Logs are text records generated by various infrastructure components such as operating systems, security devices, and applications. They contain information such as system operations, user activities, and error reports. In daily R&D or operation and maintenance, logs play an extremely important role. When a system or program fails, the time and cause of the problem can be located by observing the logs, and the problem can be investigated and solved based on the cause. In business systems, a large number of user logs can be obtained, and key information can be extracted from the log content to calculate the number of active users of the business system, page views, and other data. Further analysis of these data will help business growth and sustainable development.

[0003] Log parsing refers to the process of extracting and analyzing data from log content. The main goal of log parsing is to extract useful information from raw log data and convert it into a structured format for further data analysis, monitoring, or reporting. After parsing the raw logs into structured data, users can more conveniently perform log query, analysis, and visualization operations without having a professional technical background, which improves the efficiency of troubleshooting and troubleshooting. By generating time through log parsing, users do not need to search and query from massive logs. They can directly locate problems through generated events, discover potential security threats, and help operation and maintenance personnel respond to problems quickly.

[0004] Generally, logs are unstructured and contain a large amount of free text and variable-length fields, so they cannot be parsed through a unified framework. Currently, the commonly used log parsing methods include regular expression parsing, pattern parsing, and JSON parsing. Regular expression parsing has high accuracy, and regular expressions need to be configured for each type of log, which requires a lot of manpower and time; pattern parsing has high efficiency, but due to the large number of variables in the log and the uncertain length of the log, the standardized data parsed by this method has low accuracy and poor usability; JSON parsing is targeted, but this method is only accurate when parsing logs in JSON format, and has no effect in parsing unstructured logs. Summary of the invention

[0005] In view of the problems of low efficiency and poor accuracy of parsing methods such as regular parsing, the embodiment of the present application aims to provide a fast and accurate log parsing method and device, which can complete log parsing and generate events more quickly while ensuring the accuracy of the parsing.

[0006] The present invention provides a log fast parsing method based on field pre-matching, comprising:

[0007] S1. Obtain the log to be parsed and its asset identifier;

[0008] S2. Load a rule tree, wherein the rule tree has multiple layers of rules built in, including a generalization rule layer, a pre-matching field layer, a content matching layer, a regularization layer, and an event layer;

[0009] S3: Input the log to be parsed that carries the asset identifier into the rule tree, perform log parsing based on the generalized rules, and convert the log to be parsed into a standardized format;

[0010] S4. Based on the log generalization result, perform field pre-matching to obtain the field matching type;

[0011] S5. Based on the field matching type, select the next input layer, which is the event layer or the regularization layer;

[0012] S6. Obtain corresponding event information according to the field mapping or regular matching result;

[0013] S7. Output the logs to be parsed and the parsing results thereof, including asset identification, logs to be parsed, standardized logs, and events.

[0014] Furthermore, the log to be parsed is the original log

[0015] Further, in the S1, before obtaining the log to be parsed, the system built-in asset information table is loaded, and the content of the asset information table includes asset identification, asset type, manufacturer, system version, and subdivision version; the log to be parsed is obtained, and the asset information of the log to be parsed is obtained, and the asset information includes asset type, manufacturer, system version, subdivision version, etc.; the asset information of the log to be parsed is matched with the system built-in asset information table to obtain the asset identification of the log to be parsed; if the asset information matches successfully, the original log to be parsed is bound to the obtained asset identification; if the asset information matches unsuccessfully, the asset information table is updated, and in updating the asset information table, it includes: incrementally generating the asset identification; adding a new row of records in the asset information table to store the asset type, manufacturer, system version, and subdivision version of the log to be parsed;

[0016] Further, in said S2:

[0017] In the generalization layer, given the log generalization rule, a rule tree corresponds to an integrated generalization rule, and the integrated generalization rule contains multiple sub-rules for parsing different types of logs;

[0018] The pre-matching field layer includes the pre-matching field names under the corresponding asset identification, and the pre-matching field layer only includes a set of field names, that is, the pre-matching field names are unique;

[0019] The content matching layer is given a field mapping for pre-matching, including a field name and a field value, the content matching layer includes multiple groups of field mappings, and the field name in each group of field mappings corresponds to the field name of the pre-matching field layer;

[0020] The regular layer is given a regular expression corresponding to a generated event, and multiple regular record tables are built in the regular layer, each of which uniquely corresponds to a set of field mappings in the content matching layer; the event layer is used to give the basic information of the event, including event ID, event classification, event name, event description, impact, and disposal suggestions.

[0021] Furthermore, in S3:

[0022] The log parsing based on generalized rules refers to parsing the original log into a standardized or structured log, inputting the log to be parsed carrying the asset identifier into the rule tree, parsing the log to be parsed according to the generalized rules, and generating a standardized log through the generalized rule layer.

[0023] Furthermore, the log to be parsed enters the generalization layer, firstly matches the integrated generalization rule, matches the generalization sub-rule according to the type or format of the log to be parsed, and parses the log to be parsed into a standardized log based on the matched generalization sub-rule.

[0024] Furthermore, in S4: after the log to be parsed is output from the generalization layer, it enters the pre-matching field layer. In the rule tree, the pre-matching field exists uniquely. According to the pre-matching field name, the field value corresponding to the pre-matching field is filtered out from the standardized log, the pre-matching field and its field value are bound to generate a field group to be matched in JSON format.

[0025] Furthermore, the log to be parsed is output from the pre-matching field layer and enters the content matching layer, and the field group to be matched matches the field mapping preset in the content matching layer in sequence; if the match is successful, the matching type corresponding to the successfully matched field mapping is obtained, and then the log to be parsed, the asset identifier, the standardized log, the field mapping, and the field matching type are bound; if the match fails, it is determined that the parsing fails, and the log to be parsed, the asset identifier, the standardized log, and the field group to be matched are output.

[0026] Further, in said S5:

[0027] Select the input layer based on the asset identifier and field matching type;

[0028] If the field match type output from the content matching layer is an exact match, it directly enters the event layer;

[0029] If the field matching type is fuzzy matching, then input the regular layer and obtain the corresponding regular record table according to the field mapping;

[0030] Match the logs to be parsed one by one with the regular expressions in the regular record table;

[0031] After matching each item one by one, determine whether the log to be parsed can successfully match the regular expression;

[0032] If the original log still fails to successfully match the regular expression in the regular record table after full matching, it is determined that the parsing fails, and the original log, asset identifier, standardized log, field mapping, and field matching type are output;

[0033] If after matching one by one, there is a regular expression that can match the log to be parsed, the parsing is determined to be successful, and the log to be parsed, asset identifier, standardized log, field mapping, field matching type, regular expression are bound, and then entered into the event layer.

[0034] Further, in said S6:

[0035] According to the field mapping or regular matching results, that is, based on the field mapping or regular matching that succeeds in the original log, the corresponding event information is obtained, including event ID, event classification, event name, event description, impact, and disposal suggestion information.

[0036] On the other hand, the present invention provides a parsing device using a log fast parsing method based on field pre-matching, the parsing device comprising an acquisition module, a loading module, a parsing module, and an output module, wherein:

[0037] The acquisition module includes a first acquisition module and a second acquisition module; the first acquisition module is used to obtain the original log content to be parsed, and obtain the asset information corresponding to the log, including asset IP, asset type, manufacturer, system version, subdivision version, etc.; the second acquisition module is used to match the asset information of the log to be parsed with the asset information built into the system, and obtain the asset identifier corresponding to the log to be parsed;

[0038] A loading module is used to load the parsing rule tree under the asset identifier corresponding to the log to be parsed, that is, to load the generalization rules, pre-matching fields, matching content, regular record table, and event information under the corresponding asset identifier for use in the parsing module;

[0039] The parsing module is used to parse the original logs. Based on the asset identifier and the built-in information of the rule tree, the original logs are parsed into standardized logs and events are generated;

[0040] The output module includes a first output module, a second output module, a third output module, and a fourth output module;

[0041] Furthermore, the first output module is used to output information on rule tree loading failure. When loading the rule tree according to the asset identifier of the log to be parsed, if the loading fails, that is, there is no parsing rule tree corresponding to the asset identifier, then the result is output, including the original log and the asset identifier;

[0042] The second output module is used to output the information of the field matching failure in the parsing module. If the field group to be matched fails to match the built-in field mapping in the content matching layer, the original log, asset identification, standardized log, and field group to be matched are output;

[0043] The third output module is used to output the information of regular expression matching failure. If the corresponding regular expression is not matched after the full matching of the log to be parsed is completed during the matching at the regular expression layer, the original log, asset identification, standardized log, field mapping, and field matching type are output;

[0044] The fourth output module is used to output completely parsed log information, including asset identification, original log, standardized log, and events.

[0045] On the other hand, the present invention further provides a computer-readable storage medium storing computer program instructions. When the computer program instructions are executed by a computer, the computer executes the steps of the aforementioned method for rapid log parsing based on field pre-matching.

[0046] The beneficial effects of the embodiments of the present invention are as follows:

[0047] 1. Establish the correspondence between asset identifiers and log generalization rules, and directly load the corresponding rule tree according to the asset identifier of the log to be parsed, which improves the speed of the first stage of log parsing.

[0048] 2. By using the field pre-matching method, after the log generalization is completed, there is no need to directly match the regular expression. Instead, field matching is performed based on the results obtained from the first stage of generalization. This narrows the scope of regular expression matching to a certain extent and improves the speed of log parsing.

[0049] 3. The relationship between asset identification, field mapping, field matching type and regular table is established. After the log completes the field pre-matching, there is no need to search the entire volume, and the regular record table of the corresponding asset identification is directly entered for matching, which improves the speed of regular matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A flowchart of a log fast parsing method provided by the present invention is provided in an embodiment of the present invention;

[0051] Figure 2 A schematic diagram of a rule tree provided by the present invention;

[0052] Figure 3 A flow chart of log generalization provided by the present invention;

[0053] Figure 4 A flow chart of the field pre-matching method provided by the present invention;

[0054] Figure 5 A flow chart of the regular matching method provided by the present invention;

[0055] Figure 6 A schematic diagram of a log parsing device provided by the present invention. DETAILED DESCRIPTION

[0056] Certain embodiments of the present invention will be described more fully hereinafter with reference to the accompanying drawings, in which some, but not all, embodiments are shown. Indeed, various embodiments of the present invention may be implemented in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure may satisfy applicable legal requirements.

[0057] In order to make the objectives, features and advantages of the embodiments of the present application more obvious and understandable, the specific implementation methods of the present invention are described in detail below with reference to the accompanying drawings.

[0058] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from the description, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0059] The present application provides a method for rapid log parsing based on field pre-matching:

[0060] like Figure 1 As shown, the specific steps include:

[0061] Step S1, obtaining the log to be parsed and its asset identification.

[0062] Obtain the log to be parsed, that is, obtain the log to be parsed by Syslog forwarding or client reporting, and obtain the asset information that generates the log, including asset IP, asset type, manufacturer, system version, subdivision version, etc.

[0063] Based on the asset information table built into the log system, the corresponding asset identification is obtained according to the asset information of the log to be parsed. Specifically, the asset information table built into the system includes asset identification, asset type, manufacturer, system version, subdivision version, adaptation status and other information.

[0064] If the asset information of the log to be parsed is not matched in the system's built-in asset information table, it is determined that the parsing has failed, and the asset information table is updated at the same time.

[0065] Furthermore, the asset information table is updated, specifically including adding a new row of data in the asset information table, incrementally generating an asset identifier, and inserting the asset information corresponding to the log to be parsed, including the asset type, manufacturer, system version, subdivision version, and adaptation status, where the adaptation status is not adapted.

[0066] Furthermore, the log to be parsed is bound to the asset identifier and the next step is entered.

[0067] Step S2, loading a rule tree, that is, matching the rule tree according to the asset identifier of the log to be parsed in step 1.

[0068] If the corresponding rule tree can be loaded according to the asset identifier, proceed to the next step; if the loading fails, the original log and asset identifier are output.

[0069] The rule tree and the asset identifier are in a one-to-many relationship, that is, the rule tree matched by the asset identifier is unique, but one rule tree corresponds to one or more asset identifiers. The following table gives an example of the corresponding relationship between the asset identifier and the rule tree:

[0070] Table 1

[0071] Asset Identification Rule Tree id1 T1 id2 T1 id3 T2 id4 T2 id5 T2 id6 T3 id7 T4 ... ... idn Tn-i

[0072] The rule tree has multiple layers of built-in rules, including generalization layer, pre-matching field layer, content matching layer, regularization layer, and event layer. Figure 2 .

[0073] S201 is a generalization layer. Given a log generalization rule, a rule tree corresponds to an integrated generalization rule. The integrated generalization rule includes multiple sub-rules and is used to parse different types of logs.

[0074] S202 is a pre-matching field layer, which includes pre-matching field names under the corresponding asset identification. The pre-matching field layer only includes a set of field names, that is, the pre-matching field names are unique.

[0075] S203 is the content matching layer, which provides a field mapping for pre-matching, including field names and field values. The content matching layer includes multiple sets of field mappings, and the field names in each set of field mappings correspond to the field names of the pre-matching field layer. An example of a single set of field mappings is as follows:

[0076] {"moduleName":"SHELL","logAbstract":"CMD_RECORD"}.

[0077] S204 is the regularization layer. Given a regular expression corresponding to a generated event, multiple regularization record tables are built into the regularization layer. Each regularization record table uniquely corresponds to a set of field mappings in the content matching layer. The specific corresponding relationship is as follows:

[0078] Table 2

[0079] Field Mapping Field Match Type Regular record table KV1 1 null KV2 0 RRT1 KV3 1 null KV4 1 null KV5 0 RRT2 ... ... ... VxD 0 RRT

[0080] KVi represents the i-th group of field mappings, and the content example is:

[0081] {"moduleName":"SHELL","logAbstract":"CMD_RECORD"};

[0082] RRTj represents the jth regular record table, which contains multiple regular expressions;

[0083] The field matching type is used to determine the event acquisition method corresponding to the field mapping, and the value is 0 or 1;

[0084] If the field matching type is 1, it means an exact match. After the log to be parsed completes content matching, it does not need to go through regular matching and directly enters the event layer to obtain the event information under the field mapping.

[0085] If the field matching type is 0, it indicates fuzzy matching. After the log to be parsed is output from the content matching layer, it enters the regular layer to obtain the regular record table under the field mapping. Based on the regular record table, the log to be parsed sequentially matches the regular expression in the regular record table. If the match is successful, it enters the event layer to obtain the event information corresponding to the regular expression that successfully matched; if the match fails, it binds the log to be parsed, field mapping, and field matching type, and outputs the result.

[0086] S205 is the event layer, which is used to provide basic information of a given event, including event ID, event classification, event name, event description, impact, and disposal suggestions.

[0087] Figure 2 In , the field mapping layer is the content matching layer.

[0088] Step S3, log generalization, refers to parsing the original log into a standardized or structured log, inputting the log to be parsed with the asset identifier into the rule tree, and parsing the log to be parsed according to the generalized rule. The log to be parsed passes through the generalized rule layer to generate a standardized log.

[0089] For the specific generalization process, see Figure 3, the log to be parsed enters the generalization layer, first matches the integrated generalization rule, matches the generalization sub-rule according to the type or format of the log to be parsed, and parses the log to be parsed into a standardized log based on the matched generalization sub-rule.

[0090] The following table shows examples of original logs and generalized standardized logs:

[0091] Table 3

[0092]

[0093] Step S4: Based on the log generalization result, perform field pre-matching to obtain pre-matching fields, successfully matched field mappings, and types.

[0094] Specifically, see Figure 4 , after the log to be parsed is output from the generalization layer, it enters the pre-matching field layer. In the rule tree, the pre-matching field exists uniquely. According to the pre-matching field name, the field value corresponding to the pre-matching field is filtered out from the standardized log, the pre-matching field and its field value are bound, and a group of fields to be matched in JSON format is generated. For example, the built-in fields of the pre-matching field layer are moduleName and logAbstract. Based on the original log in Table 3, the generated group of fields to be matched is as follows: {"moduleName":"HTTPD","logAbstract":"PASS"}.

[0095] Furthermore, the log to be parsed is output from the pre-matching field layer and enters the content matching layer, and the field group to be matched sequentially matches the field mapping preset in the content matching layer. If the match is successful, the matching type corresponding to the field mapping of the successful match is obtained, and then the log to be parsed, asset identification, standardized log, field mapping, and field matching type are bound. If the match fails, it is determined that the parsing failed, and the log to be parsed, asset identification, standardized log, and field group to be matched are output. For example, the content matching layer has multiple groups of field mappings, one of which is

[0096] {"moduleName":"HTTPD","logAbstract":"PASS"}, based on the original log in Table 3, the match is successful and the next layer is entered.

[0097] Step S5: Select the next input layer based on the field matching type. Figure 5 .

[0098] Step S501, selecting an input layer according to the asset identifier and the field matching type.

[0099] Step S502: If the field matching type output from the content matching layer is an exact match, directly enter the event layer.

[0100] Step S503: If the field matching type is fuzzy matching, the regularization layer is input, and the corresponding regularization record table is obtained according to the field mapping.

[0101] Step S504: Match the logs to be parsed one by one with the regular expressions in the regular record table.

[0102] Step S505: After matching one by one, determine whether the log to be parsed can successfully match the regular expression.

[0103] Step S506: if the original log still fails to successfully match the regular expression in the regular record table after full matching, it is determined that the parsing fails, and the original log, asset identification, standardized log, field mapping, and field matching type are output.

[0104] Step S507: if after matching one by one, there is a regular expression that can match the log to be parsed, it is determined that the parsing is successful, and the log to be parsed, asset identification, standardized log, field mapping, field matching type, and regular expression are bound, and then input into the event layer.

[0105] Step S6, according to the field mapping or regular matching result, that is, based on the field mapping or regular matching successfully matched in the original log, obtain the corresponding event information, including event ID, event classification, event name, event description, impact and disposal suggestions and other information.

[0106] Step S7, outputting the original log and its parsing results, including asset identification, original log, standardized log and event information.

[0107] An embodiment of the present application provides a log fast parsing device based on field pre-matching, and the device is used to execute the log fast parsing method provided by the embodiment of the present application.

[0108] Figure 6 The log fast parsing device 3 provided in the embodiment of the present application mainly includes a first acquisition module 311, a second acquisition module 312, a loading module 32, a parsing module 33, a first output module 341, a second output module 342, a third output module 343, and a fourth output module 344.

[0109] The first acquisition module 311 is used to acquire the original log content to be parsed, and acquire the asset information corresponding to the log, including asset IP, asset type, manufacturer, system version, subdivision version, etc.;

[0110] The second acquisition module 312 is used to match the asset information of the log to be parsed with the asset information built into the system to obtain the asset identifier corresponding to the log to be parsed;

[0111] The loading module 32 is used to load the parsing rule tree under the asset identification corresponding to the log to be parsed, that is, to load the generalization rule, pre-matching field, matching content, regular record table, and event information under the corresponding asset identification for use in the parsing module;

[0112] The parsing module 33 is used to parse the original log, parse the original log into a standardized log and generate an event based on the asset identifier and the built-in information of the rule tree;

[0113] The first output module 341 is used to output information about rule tree loading failure. When loading a rule tree according to the asset identifier of the log to be parsed, if the loading fails, that is, there is no parsing rule tree corresponding to the asset identifier, the result is output, including the original log and the asset identifier.

[0114] The second output module 342 is used to output the information of the field matching failure in the parsing module. If the to-be-matched field group fails to match the built-in field mapping in the content matching layer, the original log, asset identification, standardized log, and to-be-matched field group are output.

[0115] The third output module 343 is used to output information about regular expression matching failure. If the regular expression is not matched after the parsed log is fully matched during matching at the regular expression layer, the original log, asset identifier, standardized log, field mapping, and field matching type are output.

[0116] The fourth output module 344 is used to output the completely parsed log information, including asset identification, original log, standardized log, and events.

[0117] The log to be parsed is the original log. The event layer stores events.

[0118] In addition, the above-mentioned figures are only schematic illustrations of the processing included in the method according to the embodiment of the present invention, and are not intended to limit the purpose. The order in the figures does not limit the time sequence of these operations, and these operations can be performed simultaneously or independently in multiple modules. Moreover, multiple steps can be combined into one step for execution, or one step can be disassembled into multiple steps for execution.

[0119] The above description is only a specific implementation mode of the present invention and does not limit the protection scope of the present invention. For those skilled in the art, within the scope of the present invention, the above embodiments can be changed or modified, and these changes and modifications also fall within the protection scope of the claims of the present invention.

Claims

1. A log fast parsing method based on field pre-matching, characterized in that: include: S1. Obtain the log to be parsed and its asset identifier; S2. Load a rule tree, wherein the rule tree has multiple layers of rules built in, including a generalization rule layer, a pre-matching field layer, a content matching layer, a regularization layer, and an event layer; S3: Input the log to be parsed that carries the asset identifier into the rule tree, perform log parsing based on the generalized rules, and convert the log to be parsed into a standardized format; S4. Based on the log generalization result, perform field pre-matching to obtain the field matching type; S5. Based on the field matching type, select the next input layer, which is the event layer or the regularization layer; S6. Obtain corresponding event information according to the field mapping or regular matching result; S7. Output the logs to be parsed and the parsing results thereof, including asset identification, logs to be parsed, standardized logs, and events.

2. The method for rapid log parsing based on field pre-matching according to claim 1, characterized in that: The log to be parsed is the original log.

3. The method for rapid log parsing based on field pre-matching according to claim 2 is characterized in that: In S1, before obtaining the log to be parsed, load the system's built-in asset information table, the contents of which include asset identification, asset type, manufacturer, system version, and subdivision version; obtain the log to be parsed, and obtain the asset information of the log to be parsed, the asset information includes asset type, manufacturer, system version, subdivision version, etc.; match the asset information of the log to be parsed with the system's built-in asset information table, and obtain the asset identification of the log to be parsed; If the asset information matches successfully, the original log to be parsed is bound to the obtained asset identifier; If the asset information fails to match, the asset information table is updated, and the updating of the asset information table includes: incrementally generating an asset identifier; Add a new row of records in the asset information table to store the asset type, manufacturer, system version, and subdivision version of the log to be parsed.

4. The method for rapid log parsing based on field pre-matching according to claim 1, characterized in that: In S2: In the generalization layer, given the log generalization rule, a rule tree corresponds to an integrated generalization rule, and the integrated generalization rule contains multiple sub-rules for parsing different types of logs; The pre-matching field layer includes the pre-matching field names under the corresponding asset identification, and the pre-matching field layer only includes a set of field names, that is, the pre-matching field names are unique; The content matching layer is given a field mapping for pre-matching, including a field name and a field value, the content matching layer includes multiple groups of field mappings, and the field name in each group of field mappings corresponds to the field name of the pre-matching field layer; The regular layer is given a regular expression corresponding to a generated event, and multiple regular record tables are built in the regular layer, each of which uniquely corresponds to a set of field mappings in the content matching layer; the event layer is used to give the basic information of the event, including event ID, event classification, event name, event description, impact, and disposal suggestions.

5. The method for rapid log parsing based on field pre-matching according to claim 4 is characterized in that: In the S3: The log parsing based on generalized rules refers to parsing the original log into a standardized or structured log, inputting the log to be parsed carrying the asset identifier into the rule tree, parsing the log to be parsed according to the generalized rules, and generating a standardized log through the generalized rule layer.

6. The method for rapid log parsing based on field pre-matching according to claim 5, characterized in that: The log to be parsed enters the generalization layer, first matches the integrated generalization rule, matches the generalization sub-rule according to the type or format of the log to be parsed, and parses the log to be parsed into a standardized log based on the matched generalization sub-rule.

7. The method for rapid log parsing based on field pre-matching according to claim 6, characterized in that: In S4: after the log to be parsed is output from the generalization layer, it enters the pre-matching field layer. In the rule tree, the pre-matching field is unique. According to the pre-matching field name, the field value corresponding to the pre-matching field is screened out from the standardized log, the pre-matching field and its field value are bound, and a field group to be matched in JSON format is generated.

8. The method for rapid log parsing based on field pre-matching according to claim 7, characterized in that: After the log to be parsed is output from the pre-matching field layer, it enters the content matching layer. The field group to be matched sequentially matches the field mapping preset in the content matching layer. If the match is successful, the matching type corresponding to the successfully matched field mapping is obtained, and then the log to be parsed, asset identifier, standardized log, field mapping, and field matching type are bound. If the match fails, it is considered as parsing failure, and the log to be parsed, asset ID, standardized log, and field group to be matched are output.

9. The method for rapid log parsing based on field pre-matching according to claim 8, characterized in that: In S5: Select the input layer based on the asset identifier and field matching type; If the field match type output from the content matching layer is an exact match, it directly enters the event layer; If the field matching type is fuzzy matching, then input the regular layer and obtain the corresponding regular record table according to the field mapping; Match the logs to be parsed one by one with the regular expressions in the regular record table; After matching each item one by one, determine whether the log to be parsed can successfully match the regular expression; If the original log still fails to successfully match the regular expression in the regular record table after full matching, it is determined that the parsing fails, and the original log, asset identifier, standardized log, field mapping, and field matching type are output; If after matching one by one, there is a regular expression that can match the log to be parsed, the parsing is determined to be successful, and the log to be parsed, asset identifier, standardized log, field mapping, field matching type, regular expression are bound, and then entered into the event layer.

10. The method for rapid log parsing based on field pre-matching according to claim 9, characterized in that: In S6: According to the field mapping or regular matching results, that is, based on the field mapping or regular matching that succeeds in the original log, the corresponding event information is obtained, including event ID, event classification, event name, event description, impact, and disposal suggestion information.

11. A parsing device using the log fast parsing method based on field pre-matching according to any one of claims 1 to 10, characterized in that: The analysis device includes an acquisition module, a loading module, a parsing module, and an output module, wherein: The acquisition module includes a first acquisition module and a second acquisition module; the first acquisition module is used to obtain the original log content to be parsed, and obtain the asset information corresponding to the log, including asset IP, asset type, manufacturer, system version, subdivision version, etc.; the second acquisition module is used to match the asset information of the log to be parsed with the asset information built into the system, and obtain the asset identifier corresponding to the log to be parsed; A loading module is used to load the parsing rule tree under the asset identifier corresponding to the log to be parsed, that is, to load the generalization rules, pre-matching fields, matching content, regular record table, and event information under the corresponding asset identifier for use in the parsing module; The parsing module is used to parse the original logs. Based on the asset identifier and the built-in information of the rule tree, the original logs are parsed into standardized logs and events are generated; The output module includes a first output module, a second output module, a third output module, and a fourth output module.

12. The log fast parsing device based on field pre-matching according to claim 10, characterized in that: The first output module is used to output information about rule tree loading failure. When loading the rule tree according to the asset identifier of the log to be parsed, if the loading fails, that is, there is no parsing rule tree corresponding to the asset identifier, then the result is output, including the original log and the asset identifier; The second output module is used to output the information of the field matching failure in the parsing module. If the field group to be matched fails to match the built-in field mapping in the content matching layer, the original log, asset identification, standardized log, and field group to be matched are output; The third output module is used to output the information of regular expression matching failure. If the corresponding regular expression is not matched after the full matching of the log to be parsed is completed during the matching at the regular expression layer, the original log, asset identification, standardized log, field mapping, and field matching type are output; The fourth output module is used to output completely parsed log information, including asset identification, original log, standardized log, and events.

13. A computer-readable storage medium storing computer program instructions, wherein when the computer program instructions are executed by a computer, the computer executes the steps of the log fast parsing method based on field pre-matching as described in any one of claims 1 to 10.

Images