Log analysis method and system
Through analysis rules filtering and segmented parsing based on log features, combined with parsing trees and prefilters, the performance waste and error parsing rules of complex log text are solved, and efficient and flexible log parsing is achieved.
Patent Information
- Application Number
- CN202510474705.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-16
- Publication Date
- 2025-08-01
AI Technical Summary
The prior art has problems of performance waste and parsing rules errors when dealing with complex nested log texts, and lacks flexibility and grouping of parsing rules for different requirements.
By filtering parsing rules based on log features, using parsing trees and prefilters for segmented parsing, combining parsing rule cache pools and parsing engine layers, flexible configuration and performance optimization of parsing rules are achieved.
Improves the flexibility and performance of log parsing, reduces the frequency of using error resolution rules, and simplifies the difficulty of configuring analysis rules.
Smart Images
Figure CN120409459A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of data analysis, and in particular, relates to a log parsing method and system. Background Art
[0002] Log parsing, which can also be text parsing, is a technology for processing the content of a string after receiving the data string through a receiver. The technology closest to the present invention currently is multi-layer parsing. Text formats include fixed rules or fixed rule parsing methods such as json, csv, key-value, xml, etc. For irregular formats, only regular expressions can be used for parsing. The technical principle of multi-layer parsing is to first perform overall parsing according to the log format. If the analysis requirements for the log are met, the parsing process is completed. If not, then according to the need, corresponding rule parsing is performed on a part of the parsed content. If a log contains multiple different format contents, multiple multi-layer parses are required to ensure correct extraction of the required content. For example, if the overall content of a log is in json format, but some of its fields are in csv format or key-value format and regular expressions, then when extracting the internal fields, a first-layer json parse and a second-layer csv parse are required to obtain the content in the csv. If all the contents in the csv, key-value, and regular expressions also need to be extracted at this time, multiple multi-layer parses are required, that is, multiple json parses, and then the results of each json parse are further parsed accordingly to obtain the specified log field content.
[0003] The inventors found that the existing technical solutions mainly have the following disadvantages during the implementation of this embodiment:
[0004] (1) Inflexible use. When performing complex nested parsing on a log text, multiple parses may occur in the first layer, resulting in performance waste.
[0005] (2) There is no handling for incorrect parsing rule files.
[0006] (3) There is no grouping for parsing rules with different requirements. Summary of the Invention
[0007] In view of the problems existing in the prior art, the present invention provides a log parsing method and system.
[0008] In a first aspect, an embodiment of the present disclosure provides a log parsing method, including:
[0009] Obtaining log features based on the acquired log file;
[0010] Filtering out parsing rules that meet the log features according to the log features;
[0011] Segment the log through the parsing process defined by the parsing rules to obtain the parsed content;
[0012] Encapsulate the parsed content.
[0013] Optionally, the parsing rules include a parsing tree, which uses the data structure characteristics of the tree to represent the hierarchical association relationship of parsing. The parsing tree enables the child node to parse the source data obtained from the parsing result of the parent node.
[0014] Optionally, the parsing tree includes rule description information, which is used to explain the data source and parsing method of this node.
[0015] In a second aspect, an embodiment of the present disclosure further provides a log parsing system for any of the log parsing methods in the first aspect, including: an interface layer, a rule layer, an engine layer, and an assignment layer;
[0016] The interface layer is used to provide data interfaces for the rule layer and the engine layer;
[0017] The rule layer is used to define the parsing method of the log, describe the parsing unit corresponding to the log parsing, describe the cooperation method of different parsing units in the log parsing process, and bind the rules to the corresponding attributes for positioning and searching;
[0018] The engine layer is used to build an execution engine, call execution rules, and encapsulate parsing objects;
[0019] The assignment layer is used to assign values to and encapsulate the parsing objects.
[0020] Optionally, the interface layer is used for loading parsing rules, parsing log texts, testing and parsing log text samples, and parsing individual parsing units.
[0021] Optionally, the rule layer includes a rule description object, a rule parser, a rule cache entity object, a rule cache pool, and a rule sequence optimizer;
[0022] The rule description object includes a parsing rule set, a bound device IP rule mapping set, and a bound topic rule mapping set;
[0023] The parsing rule set includes a pre-filter object set, a parsing unit, and a field assignment description object;
[0024] The pre-filter object set is used to distinguish the logs corresponding to the parsing rules. When all the logs meet the filtering conditions of the pre-filter object set, the corresponding parsing rules are used to parse all the logs that meet the pre-filter object set;
[0025] The parsing unit includes a parsing unit ID, a parent node parsing unit ID, a parent node parsing unit position encoding, a parent node parsing unit position name, a parsing unit parser type, a parsing rule parameter object, and a set of child parsing unit IDs;
[0026] The field assignment description object is used to describe the fields to which the data parsed by each parsing unit is assigned, and is used to explain the parsing result object;
[0027] The field assignment description object includes a field name, a parsing unit information, a format conversion relationship information object, a value mapping relationship information object, and a value calculation assignment information object.
[0028] Optionally, the rule parser is used to receive a rule object, then parse the rule object to generate a corresponding rule cache object, and add the rule cache object to the cache pool;
[0029] The parsing rule cache entity object is an object obtained by the parser to directly use the parsing rule at the engine layer;
[0030] The rule cache pool is used to save the parsing rule cache objects, save and manage the cache objects, and the engine layer obtains the rule cache objects through the cache pool interface.
[0031] Optionally, the rule cache pool includes a general rule cache pool, a device IP cache pool, and a topic cache pool;
[0032] The device IP cache pool and the topic cache pool are both set with corresponding timing tasks. The timing tasks sort the data in the corresponding cache pools by hit rate, and place the rule caches with high hit rates in front of the cache pool for preferential matching.
[0033] Optionally, the engine layer includes a construction part and an execution part. The construction part includes a rule object parser, a parsing engine builder, and a parsing resource sharing cache pool; the execution part includes a unit parser engine, a pre-filtering parser, and a field parser engine;
[0034] After receiving the parsing unit data of the parsing rule, the construction part generates a corresponding parsing engine according to the description information of the parsing unit, assigns it to the parsing engine field of the rule cache object for the executor to call, and at the same time assigns the assignment engine to the assignment information field of the rule cache according to the field information of the parsing rule for the data parsed by the parsing executor;
[0035] After the engine layer receives the log text, it determines whether the log text includes attached information. If the log text includes attached information, it obtains the corresponding data information from the cache pool corresponding to the attached information. If the corresponding data information cannot be obtained from the cache pool corresponding to the attached information, it obtains the parsing rule cache information set from the rule cache pool, and then traverses the cache information set. When traversing the cache information set, it first performs pre-filtering parsing. If the pre-filtering is hit during the pre-filtering parsing, it calls the parsing unit tree of the parsing rule set and executes the parsing logic according to the order of the constructed parsing unit tree. If all the parsing is successful, the traversal stops; after the parsing is successful, it calls the rule hit count interface of the rule cache pool and sorts the corresponding rule cache pool, and performs parsing again according to the received log text and the sorting result of the rule cache pool.
[0036] Optionally, the data object parsed and executed by the engine layer is a parsing set. The data object parsed and executed has a corresponding parsing unit id, the time required for parsing, and the parsed data. The parsing result includes positioning information, and the positioning information is used to provide positioning for obtaining data by the next parsing unit.
[0037] The log parsing method and system provided by the present invention. Among them, the log parsing method can split the originally very complex log text object into multiple segments through the parsing process defined by the parsing rules, which can reduce the configuration difficulty of the parsing rules, thereby achieving the purpose of improving the parsing flexibility. By screening out the parsing rules that meet the log characteristics according to the log characteristics, while improving the parsing performance, the situation of using the wrong parsing rule file is reduced. Brief Description of the Drawings
[0038] By describing the exemplary embodiments of the present disclosure in more detail in conjunction with the accompanying drawings, the above and other objects, features, and advantages of the present disclosure will become more obvious. Among them, in the exemplary embodiments of the present disclosure, the same reference numerals generally represent the same components.
[0039] Figure 1 It is a flowchart of the log parsing method provided by the embodiment of the present disclosure;
[0040] Figure 2 It is a schematic block diagram of the principle of the log parsing system provided by the embodiment of the present disclosure;
[0041] Figure 3 It is a schematic structural diagram of the parsing rule object provided by the embodiment of the present disclosure;
[0042] Figure 4 It is a business flowchart of the rule layer provided by the embodiment of the present disclosure;
[0043] Figure 5 It is a schematic structural diagram of the cache object provided by the embodiment of the present disclosure;
[0044] Figure 6 This is the business process diagram of the rule parsing engine provided by the embodiments of the present disclosure;
[0045] Figure 7 This is the principle block diagram of the parsing rule cache pool provided by the embodiments of the present disclosure;
[0046] Figure 8 This is the logic design diagram of the execution layer provided by the embodiments of the present disclosure;
[0047] Figure 9 This is the schematic diagram of the parsing object structure provided by the embodiments of the present disclosure. Detailed implementation manners
[0048] The following describes the embodiments of the present disclosure in detail with reference to the accompanying drawings.
[0049] It should be clear that the following illustrates the implementation manners of the present disclosure through specific specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments. The present disclosure can also be implemented or applied through other different specific implementation manners, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that, without conflict, the following embodiments and the features in the embodiments can be combined with each other. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present disclosure without creative efforts belong to the scope of protection of the present disclosure.
[0050] It should be noted that the following describes various aspects of the embodiments within the scope of the appended claims. It should be obvious that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is illustrative only. Based on the present disclosure, those skilled in the art should understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement the device and / or practice the method. In addition, this device can be implemented and this method can be practiced using other structures and / or functions in addition to one or more of the aspects described herein.
[0051] It should also be noted that the diagrams provided in the following embodiments only illustrate the basic concept of the present disclosure schematically, and only show the components related to the present disclosure in the diagrams, rather than being drawn according to the number, shape and size of the components in actual implementation. The types, quantities and proportions of the actual components in implementation may be arbitrarily changed, and the component layout type may also be more complex.
[0052] In addition, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.
[0053] For ease of understanding, as Figure 1 shown, this embodiment discloses a log parsing method, including:
[0054] Obtaining log features based on the acquired log file;
[0055] Filtering out parsing rules that meet the log features according to the log features;
[0056] Segmenting and parsing the log through the parsing process defined by the parsing rules to obtain the parsed content;
[0057] Encapsulating the parsed content.
[0058] Optionally, the parsing rules include a parsing tree, and the parsing tree uses the data structure characteristics of the tree to represent the hierarchical association relationship of the parsing. The parsing tree enables the child node to parse the source data obtained from the parsing result of the parent node.
[0059] Optionally, the parsing tree includes rule description information, which is used to explain the data source and parsing method of this node.
[0060] Define a parsing rule object, which must include a parsing tree (a data structure of a multi-way tree) and a rule mapping relationship object (which can be a map structure or an object containing a key field for rule binding information and a rule set automatically). Each parsing rule has a parsing tree, which uses the data structure characteristics of the tree to represent the hierarchical association relationship of the parsing, so that the child node parses the source data obtained from the parsing result of the parent node to solve the problem of multiple parses.
[0061] Each parsing rule has a pre-filter, which is used to quickly determine whether this rule is the best parsing rule for this text. The pre-filtering rules perform a relational AND operation on multiple rules. The rules include string containment, string non-containment, regular match satisfaction, regular match non-satisfaction, etc.
[0062] Each parsing tree has rule description information, which is used to explain the data source and parsing method of this node. Then, a corresponding parsing engine is constructed according to the description information. The parsing engine is uniformly abstracted as an interface, and the underlying implementations include regular parsing, JSON parsing, CSV parsing, and key-value parsing. Each parser uniformly returns the same parsing result object.
[0063] After receiving the log, according to the log characteristics, the parsing rules that meet the characteristics are filtered out by the pre-filter, and then the log is segmented and parsed through the parsing process defined by the parsing rule object. Finally, the parsed content is encapsulated to complete the entire parsing process.
[0064] As Figure 2 shown, the embodiment of the present disclosure also provides a log parsing system for the log parsing method disclosed in this embodiment, including: an interface layer, a rule layer, an engine layer, and an assignment layer;
[0065] The interface layer is used to provide data interfaces for the rule layer and the engine layer;
[0066] The rule layer is used to define the log parsing method, describe the parsing units corresponding to the log parsing, describe the cooperation method of different parsing units during the log parsing process, and bind the rules to the corresponding attributes for positioning and searching;
[0067] The engine layer is used to build an execution engine, call execution rules, and encapsulate parsing objects;
[0068] The assignment layer is used to assign values and encapsulate the parsing objects.
[0069] The overall system architecture is divided into an interface layer that provides business function services externally; a rule layer that defines, loads, parses, and adds parsing rules to the rule cache pool. An engine layer that completes the parsing and assignment of log texts; an assignment layer that completes the assignment and encapsulation of log fields.
[0070] Optionally, the interface layer is used for loading parsing rules, parsing log texts, testing and parsing log text samples, and parsing individual parsing units.
[0071] The interface layer mainly exposes some business interfaces of the rule layer and some business interfaces of the engine layer externally. It completes the business support of the system and ensures that the external can use the system functions quickly and simply. The interface functions mainly include interfaces such as loading parsing rules, overall parsing of log texts, testing and parsing of log text samples, and parsing of individual parsing units.
[0072] Optionally, the rule layer includes a rule description object, a rule parser, a rule cache entity object, a rule cache pool, and a rule sequence optimizer;
[0073] The rule description object includes a parsing rule set, a bound device IP rule mapping set, and a bound topic rule mapping set;
[0074] The parsing rule set includes a pre-filter object set, parsing units, and a field assignment description object;
[0075] The pre-filter object set is used to distinguish the logs corresponding to the parsing rules. When all the logs meet the filtering conditions of the pre-filter object set, the corresponding parsing rules are used to parse the logs that all meet the pre-filter object set.
[0076] The parsing unit includes a parsing unit id, a parent parsing unit id, a parent parsing unit position code, a parent parsing unit position name, a parsing unit parser type, a parsing rule parameter object, and a set of child parsing unit ids.
[0077] The field assignment description object is used to describe the fields to which the data parsed by each parsing unit is assigned, and is used to explain the parsing result object.
[0078] The field assignment description object includes a field name, a parsing unit information, a format conversion relationship information object, a value mapping relationship information object, and a value calculation and assignment information object.
[0079] Optionally, the rule parser is used to receive a rule object, then parse the rule object to generate a corresponding rule cache object, and add the rule cache object to the cache pool.
[0080] The parsing rule cache entity object is an object obtained by the parser to transform the parsing rule for direct use in the engine layer.
[0081] The rule cache pool is used to save the parsing rule cache objects, save and manage the cache objects, and the engine layer obtains the rule cache objects through the cache pool interface.
[0082] Optionally, the rule cache pool includes a general rule cache pool, a device IP cache pool, and a topic cache pool.
[0083] The device IP cache pool and the topic cache pool are both set with corresponding timing tasks. The timing tasks sort the data in the corresponding cache pools by hit rate, and place the rule caches with high hit rates in front of the cache pool for priority matching.
[0084] Parsing rule description object: The parsing rule defines the parsing method for log samples, and is used to describe which parsing units a log text goes through for parsing, how each parsing unit should cooperate with each other to split and parse the log, and on which device IPs or topics the rule is bound for quick positioning and searching.
[0085] The description result of the parsing rule object is as Figure 3 shown:
[0086] The main part of the parsing rule object is the combination of the pre-filter object and the set of parsing units. The pre-filter set is used to distinguish which type of log the parsing rule belongs to. Multiple pre-filters are in an "and" relationship. Only when all the filtering conditions are met can it be shown that this log is to be parsed by this rule, preventing too many parsing rules in the parsing rule pool and the situation of using the wrong parsing rule file.
[0087] The parsing rule set is used to describe the parsing process. Multiple parsing units form a parsing tree (the parsing tree structure is a multi-way tree). A parsing unit can have one or more sub-parsing units. Each sub-parsing unit locates the content of the parsing result of the parent parsing unit through the parentUnitId and parentIndexNum or parentIndexName for the next parsing operation. Each parsing unit has parsing types of regular expression, JSON, CSV, and KV (KEY-VALUE). In this way, the complex log can be parsed segmentally through the combination of multiple parsing units. The parsing rule parameter object is used for supplementary use in different types of parsing. For the regular expression parsing type, rule parameter 1 stores the definition string of the regular expression. For the JSON type, no parsing rule parameters are required. For the CSV parsing type, rule parameter 1 stores the delimiter information, and parsing parameter 2 stores the information of the symbols to be ignored, which is used for parsing the CSV type. For the KV parsing type, rule parameter 1 stores the delimiter information, and parsing parameter 2 stores the delimiter information between the key and the value, completing the description of the KV data parsing logic.
[0088] The bound device IP rule mapping set and the bound topic rule mapping set are both saved using a data structure similar to a map. The key is the device IP or the topic string, and the value is the parsing rule set object, which stores the set of rule data objects bound to this key.
[0089] The field assignment description object is used to describe which field the data parsed by each parsing unit is assigned to, which is used to explain the final parsing result object. The parsing engine will perform format conversion, value mapping processing, and value operation operations on the parsed fields according to this description information, and finally complete the assignment operation.
[0090] As Figure 4 shown, for rule parsing and caching, the rule parser is used to receive the rule object, then parse the rule object, generate the corresponding rule cache object, and add it to the cache pool. This process is the overall business process of the rule layer.
[0091] The parsing rule cache object, the parsing rule cache entity object is an object that the parser turns the parsing rule into directly available in the engine layer. The main field design of the cache object is as Figure 5 shown.
[0092] The rule parser transforms the parsed rule object into a rule object that can be directly used by the engine layer. The business flow chart of the rule parsing engine is as Figure 6 shown below.
[0093] The parsed rule cache pool, as Figure 7 shown below, is used to save the parsed rule cache objects, save and manage the cache objects. The engine layer obtains the rule cache objects through the cache pool interface. There are three map structures in the parsing cache pool: the overall rule cache pool map, where the key is the rule ID and the value is the cache object; the device IP cache pool, which is used to save the set of cache objects for specific device IPs; and the topic cache pool, which is used to save the set of cache objects for specific topics.
[0094] Both the device IP cache pool and the topic cache pool have a scheduled task to sort the data in the cache pool by hit rate, placing the rule caches with high hit rates in the front for preferential matching, thus improving the parsing performance.
[0095] Optionally, the engine layer includes a construction part and an execution part. The construction part includes a rule object parser, a parsing engine builder, and a parsing resource sharing cache pool; the execution part includes a unit parsing engine, a pre-filtering parser, and a field parsing engine.
[0096] After receiving the parsed unit data of the parsing rule, the construction part generates a corresponding parsing engine according to the description information of the parsed unit, assigns it to the parsing engine field of the parsed unit of the rule cache object for the executor to call, and at the same time assigns the assignment engine to the assignment information field of the rule cache according to the field information of the parsing rule for the data parsed by the parsing executor.
[0097] After receiving the log text, the engine layer determines whether the log text includes additional information. If the log text includes additional information, it obtains the corresponding data information from the cache pool corresponding to the additional information. If the corresponding data information cannot be obtained from the cache pool corresponding to the additional information, it obtains the set of parsed rule cache information from the rule cache pool, and then traverses the cache information set. When traversing the cache information set, pre-filtering parsing is first performed. If the pre-filtering is hit during the pre-filtering parsing, the parsing unit tree of the parsed rule set is called, and the parsing logic is executed according to the order of the constructed parsing unit tree. If all the parsing is successful, the traversal stops; after the parsing is successful, the rule hit count interface of the rule cache pool is called, and the corresponding rule cache pool is sorted. When parsing again, the parsing is performed according to the received log text and the sorting result of the rule cache pool.
[0098] Optionally, the data object parsed and executed by the engine layer is a parsed set. The data object parsed and executed has a corresponding parsing unit ID, parsing time required, and parsed data. The parsing result includes positioning information, which is used to provide positioning for obtaining data for the next parsing unit.
[0099] The engine layer mainly completes the construction of the execution engine, the invocation and execution of rules, and the operation of encapsulating parsing objects. The parsing layer then encapsulates the intermediate objects parsed by the engine layer into parsing objects, assigns values to the content of the objects according to fields, encapsulates the data, and completes the data transfer of the parser to the outside.
[0100] The engine layer is divided into a construction part and an execution part. The rule object parser, parsing engine constructor, etc. are the construction part, providing interfaces for the rule parsers in the rule layer to use. The execution part is to obtain the data in the cache pool and call the constructed parsing engine for execution.
[0101] After receiving the parsing unit data of the parsing rule, the construction part will produce a corresponding parsing engine according to the description information of the parsing unit, assign it to the parsing engine field of the parsing unit of the rule cache object, and be used by the executor in the execution layer for invocation. At the same time, according to the field information of the parsing rule, the assignment engine is assigned to the assignment information field of the rule cache, used to parse and process the data after being parsed by the executor, and then passed to the assignment layer for assignment processing.
[0102] The logical design diagram of the execution layer is as Figure 8 shown. Parsing and execution receive the log text. If the text is attached with device IP or topic information, the corresponding data information is obtained from the cache pool according to the attached information. If it cannot be obtained, the parsing rule cache information set is obtained from the rule cache pool including all cache information. Then, after traversing the obtained rule cache information set, the pre-filter is executed first. If the pre-filter is hit, the parsing unit tree in the parsing cache is called, and the parsing logic is executed according to the order of the parsing unit tree constructed by the parsing layer. If all parsing is successful, the traversal stops. If the parsing is not successful, the next rule is continued to be traversed. After hitting, the rule hit counting interface of the rule pool is called. For the rule that is hit through which IP or topic feature information pool, the rule is incremented by 1. If the rule is obtained through the full-scale pool, it will be added to the corresponding feature information pool and then incremented by 1 to ensure that the rules in the rule pool are sorted by the hit rate. The next time it is obtained, it is the data sorted by the hit rate, which improves the execution efficiency. This design is a solution obtained through practice because the number of device IPs or topics generally received by a collector device is not too large.
[0103] The data object for parsing execution is a parsing set. Each parsing object has a parsing unit id, the time required for parsing (in microseconds), and the combined data to be parsed. Since the number of data contents parsed by each parsing unit cannot be determined, a set is needed to store the parsing results. Because there are multiple data, each parsing result needs to locate which position of the data it is, providing positioning information for the next parsing unit to obtain data. The position information is the parsing group subscript and the parsing group name. If it is regular parsing, CSV, or KV, the parsing group subscript can locate the content. If it is JSON parsing, the parsing group name is required. The parsing object structure is as follows Figure 9 as shown.
[0104] The assignment layer only needs to perform assignment encapsulation on the parsing object to obtain the final parsing result. The final parsing result
[0105] mainly includes the field name and the field content. Among them, the metadata field parser describes the field information corresponding to the field name. The description content includes the field name, the field default value, the field data type. After the assignment engine calculates the value, it also needs to convert the data into the specified type of data according to the field data type.
[0106] This embodiment provides a pre-filtering function and a specific information rule binding function to avoid the problem of processing with an incorrect parsing rule file as much as possible.
[0107] This embodiment groups the rule sets by characteristics such as device ip or topic as the source or differentiation conditions of the characteristics, so as to group the parsing rules with different requirements.
[0108] This embodiment splits the parsing process of the originally complex log text object into multiple simple parsing units through the description information of the parsing rule object, and finally completes the overall parsing logic through the parsing tree relationship between each parsing unit, which can effectively reduce the configuration difficulty of the parsing rules and improve the flexibility of parsing. At the same time, by adding the design idea of pre-filtering and the innovative idea of binding the feature content rule pool, the parsing performance is improved while the situation of using an incorrect parsing rule file is reduced.
[0109] The basic principles of the present disclosure have been described above in combination with specific embodiments. However, it should be noted that the advantages, advantages, effects, etc. mentioned in the present disclosure are only examples and not limitations. It cannot be considered that these advantages, advantages, effects, etc. are essential for each embodiment of the present disclosure. In addition, the above-mentioned specific details are only for the purpose of illustration and easy understanding, rather than limitations. The above details do not limit the present disclosure to necessarily adopt the above specific details to implement.
[0110] In this disclosure, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. The block diagrams of devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and do not intend to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any way. Words such as "including", "comprising", "having", etc. are open-ended words, meaning "including but not limited to", and can be used interchangeably with each other. The words "or" and "and" used herein refer to the word "and / or" and can be used interchangeably with it, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to" and can be used interchangeably with it.
[0111] In addition, as used herein, "or" in the listing of items starting with "at least one" indicates a disjunctive listing, so that for example, the listing of "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the term "exemplary" does not mean that the examples described are preferred or better than other examples.
[0112] It should also be noted that in the systems and methods of this disclosure, each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent solutions of this disclosure.
[0113] Various changes, substitutions, and alterations to the technologies described herein can be made without departing from the teachings defined by the appended claims. In addition, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, compositions of events, means, methods, and acts described above. Current or later-developed processes, machines, manufactures, compositions of events, means, methods, or acts that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein can be utilized. Thus, the appended claims include such processes, machines, manufactures, compositions of events, means, methods, or acts within their scope.
[0114] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to the broadest scope consistent with the principles and novel features disclosed herein.
[0115] The foregoing description has been presented for purposes of illustration and description. Furthermore, this description is not intended to limit embodiments of the present disclosure to the form disclosed herein. Although several example aspects and embodiments have been discussed above, those skilled in the art will recognize some variations, modifications, alterations, additions, and subcombinations thereof.
Claims
1. A log parsing method, characterized in that, It includes: Obtain log features based on the acquired log files; Filter out parsing rules that meet the log features according to the log features; Perform segmented parsing on the log through the parsing process defined by the parsing rules to obtain the parsed content; Encapsulate the parsed content.
2. The log parsing method according to claim 1, characterized in that The parsing rules include a parsing tree, and the parsing tree uses the characteristics of the tree data structure to represent the hierarchical association relationship of parsing. The parsing tree enables the child node to parse the source data obtained from the parsing result of the parent node.
3. The log parsing method according to claim 2, wherein The parsing tree includes rule description information, which is used to explain the data source and parsing method of this node.
4. A log parsing system for the log parsing method according to any one of claims 1 to 3, characterized in that, It includes: An interface layer, a rule layer, an engine layer, and an assignment layer; The interface layer is used to provide data interfaces for the rule layer and the engine layer; The rule layer is used to define the parsing method of the log, describe the parsing units corresponding to the log parsing, describe the cooperation method of different parsing units during the log parsing process, and bind the rules to the corresponding attributes for positioning and searching; The engine layer is used to build an execution engine, call execution rules, and encapsulate parsing objects; The assignment layer is used to assign values to and encapsulate the parsing objects.
5. The log parsing system according to claim 4, wherein The interface layer is used for loading parsing rules, parsing log texts, testing and parsing log text samples, and parsing individual parsing units.
6. The log parsing system according to claim 4, wherein The rule layer includes a rule description object, a rule parser, a rule cache entity object, a rule cache pool, and a rule sequence optimizer; The rule description object includes a parsing rule set, a bound device IP rule mapping set, and a bound topic rule mapping set; The parsing rule set includes a pre-filter object set, a parsing unit, and a field assignment description object; The pre-filter object set is used to distinguish the logs corresponding to the parsing rules. When all the logs meet the filtering conditions of the pre-filter object set, the corresponding parsing rules are used to parse all the logs that meet the pre-filter object set; The parsing unit includes a parsing unit ID, a parent parsing unit ID, a parent parsing unit position code, a parent parsing unit position name, a parsing unit parser type, a parsing rule parameter object, and a set of child parsing unit IDs; The field assignment description object is used to describe the fields to which the data parsed by each parsing unit is assigned, and is used to explain the parsing result object; The field assignment description object includes a field name, parsing unit information, a format conversion relationship information object, a value mapping relationship information object, and a value calculation assignment information object.
7. The log parsing system according to claim 6, wherein The rule parser is used to receive a rule object, then parse the rule object, generate a corresponding rule cache object, and add the rule cache object to the cache pool; The parsing rule cache entity object is an object obtained by the parser turning the parsing rules into an object directly used by the engine layer; The rule cache pool is used to save the rule cache objects, save and manage the cache objects, and the engine layer obtains the rule cache objects through the cache pool interface.
8. The log parsing system according to claim 7, wherein The rule cache pool includes a general rule cache pool, a device IP cache pool, and a topic cache pool; Timing tasks are set for both the device IP cache pool and the topic cache pool. The timing tasks sort the data in the corresponding cache pools by hit rate, place the rule caches with high hit rates in front of the cache pools for preferential matching.
9. The log parsing system according to claim 8, wherein The engine layer includes a construction part and an execution part. The construction part includes a rule object parser, a parsing engine builder, and a parsing resource sharing cache pool; the execution part includes a unit parsing engine, a pre-filtering parser, and a field parsing engine. After the construction part receives the parsing unit data of the parsing rule, it generates a corresponding parsing engine according to the description information of the parsing unit, assigns it to the parsing engine field of the parsing unit of the rule cache object for the executor to call, and at the same time assigns the assigned engine to the assignment information field of the rule cache according to the field information of the parsing rule for parsing the data after the parsing executor parses. After the engine layer receives the log text, it determines whether the log text includes additional information. If the log text includes additional information, it obtains the corresponding data information from the cache pool corresponding to the additional information. If the corresponding data information cannot be obtained from the cache pool corresponding to the additional information, it obtains the parsing rule cache information set from the rule cache pool, and then traverses the cache information set. When traversing the cache information set, pre-filtering parsing is performed first. If the pre-filtering is hit during the pre-filtering parsing, the parsing unit tree of the parsing rule set is called, and the parsing logic is executed according to the order of the constructed parsing unit tree. If all parses are successful, the traversal stops; after successful parsing, the rule hit count interface of the rule cache pool is called, and the corresponding rule cache pool is sorted, and the parsing is performed again according to the received log text and the sorting result of the rule cache pool.
10. The log parsing system according to claim 9, wherein The data object parsed and executed by the engine layer is a parsing set. The data object parsed and executed has a corresponding parsing unit id, parsing required time, and parsed data. The parsing result includes positioning information, and the positioning information is used to provide positioning for obtaining data for the next parsing unit.