Log analysis method and device, equipment, storage medium

Abstract

The application discloses a log analysis method and device, equipment and a storage medium. The method comprises the following steps: replacing variables in a first log to be analyzed with identifiers to obtain a second log; wherein the variables comprise semantic variables and / or non-semantic variables; different semantic variables correspond to different first identifiers, and different non-semantic variables correspond to the same second identifier; searching a target log template matched with the second log from a first log template library constructed in advance; wherein the first log template library is obtained by performing a deduplication process on a second log template library; the second log template library comprises at least one second log template; the second log template is obtained by performing the identifier replacement on variables in a third log; and determining an analysis result of the first log according to the target log template. In this way, the number of log templates that need to be matched by the second log is reduced, and the log analysis speed is improved.

Description

Technical Field

[0001] This application relates to language processing technology, including but not limited to log parsing methods, apparatus, devices, and storage media. Background Technology

[0002] System logs are text output by programs written by developers, recording information about each task during the current device operation, such as the event type of user requests, user identity (ID), and event level. Mining system logs can be used for anomaly detection, attack tracing, fault diagnosis, and user profiling. However, raw log data is usually unstructured or semi-structured, so appropriate methods need to be designed to parse logs into structured data. Therefore, log parsing is a key step in achieving automated and effective log analysis. As software systems are continuously updated, the size and complexity of logs gradually increase; however, some log parsing methods have low parsing speeds. Summary of the Invention

[0003] In view of this, the log parsing method, apparatus, device, and storage medium provided in this application, after replacing the identifier of the first log to be parsed to obtain the second log, match it with the deduplicated first log template library; thus, the number of log templates that need to be matched for the second log is reduced, and the log parsing speed is improved.

[0004] According to one aspect of the embodiments of this application, a log parsing method is provided, comprising: replacing the identifiers of variables in a first log to be parsed to obtain a second log; wherein the variables include semantic variables and / or non-semantic variables; different semantic variables correspond to different first identifiers, and different non-semantic variables correspond to the same second identifier; searching for a target log template that matches the second log from a pre-built first log template library; wherein the first log template library is obtained by deduplicating a second log template library, the second log template library includes at least one second log template, and the second log template is obtained by replacing the identifiers of variables in a third log; and determining the parsing result of the first log based on the target log template.

[0005] The log parsing apparatus provided in this application includes: a replacement module, used to replace the identifiers of variables in a first log to be parsed to obtain a second log; wherein the variables include semantic variables and / or non-semantic variables; different semantic variables correspond to different first identifiers, and different non-semantic variables correspond to the same second identifier; a search module, used to search for a target log template that matches the second log from a pre-built first log template library; wherein the first log template library is obtained by deduplicating a second log template library, the second log template library includes at least one second log template, and the second log template is obtained by replacing the identifiers of the variables in a third log; and a first determination module, used to determine the parsing result of the first log based on the target log template.

[0006] The electronic device provided in this application includes a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements the method described in this application.

[0007] The computer-readable storage medium provided in this application embodiment stores a computer program thereon, which, when executed by a processor, implements the method provided in this application embodiment.

[0008] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0009] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application. Obviously, the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0010] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0011] Figure 1 A schematic diagram illustrating the implementation flow of the log parsing method provided in this application embodiment;

[0012] Figure 2 A schematic diagram illustrating the construction process of the template mapping relationship between the first log template library and the third log template library provided in this application embodiment;

[0013] Figure 3a A schematic diagram of the similar log template and the merged third log template provided in the embodiments of this application;

[0014] Figure 3b A schematic diagram of the similar log template and the merged fourth log template provided in the embodiments of this application;

[0015] Figure 4 This is a schematic diagram of the log parsing process provided in an embodiment of this application;

[0016] Figure 5 This is a schematic diagram of multi-source logs provided in an embodiment of this application;

[0017] Figure 6 This is a schematic diagram illustrating the parsing time corresponding to different log parsing methods provided in the embodiments of this application;

[0018] Figure 7 This is a schematic diagram of the log parsing device provided in an embodiment of this application;

[0019] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the specific technical solutions of this application will be further described in detail below with reference to the accompanying drawings of the embodiments of this application. The following embodiments are used to illustrate this application, but are not intended to limit the scope of this application.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0022] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0023] This application provides a log parsing method applied to an electronic device. This electronic device can be of various types with information processing capabilities, such as a mobile phone, tablet computer, or desktop computer. The functionality achieved by this method can be implemented by a processor in the electronic device calling program code. The program code can be stored in a computer storage medium. Therefore, the electronic device includes at least a processor and a storage medium.

[0024] Figure 1 This is a schematic diagram illustrating the implementation process of the log parsing method provided in the embodiments of this application, as shown below. Figure 1 As shown, the method may include the following steps 101 to 103:

[0025] Step 101: Replace the identifiers of the variables in the first log to be parsed to obtain the second log; wherein, the variables include semantic variables and / or non-semantic variables; different semantic variables correspond to different first identifiers, and different non-semantic variables correspond to the same second identifier.

[0026] Network devices, electronic devices, systems, modules, and service programs all generate logs during runtime, which at least record events. These logs are mostly unstructured or semi-structured text data. Before analysis, structured fields need to be extracted from the logs, i.e., log templates need to be extracted; this process is called log parsing. The extracted log templates provide a much better overview of the logs than the original logs, enabling subsequent operations such as anomaly detection and user behavior analysis.

[0027] In some embodiments, logs may contain at least constants and variables. Constants may be text written by developers to describe the behavior or function of the current program, and may appear in the form of words. Variables reflect dynamic information during program execution, such as IP address, website domain name, file path, Uniform Resource Locator (URL), software name, and user ID.

[0028] In some embodiments, the variables may include semantic variables and non-semantic variables. Semantic variables may be variables that can characterize the semantic information of the first log, such as IP address, website domain name, URL, file path, and software name. Non-semantic variables may be other variables that are different from the semantic variables and have no special meaning or specific rules, such as userid and garbled characters.

[0029] In some embodiments, corresponding regular expressions can be designed for different variables so that the regular expressions can be used to identify the variables in the first log and / or the third log.

[0030] In some embodiments, the first log can be separated by spaces, that is, the first log is regarded as a sentence composed of multiple words. The semantic variables in the first log are identified by a pre-designed regular expression, and different semantic variables are replaced with the corresponding first identifier. For example, Table 1 shows the regular expressions and first identifiers corresponding to different semantic variables provided in the embodiments of this application.

[0031] Table 1. Regular expressions and first identifiers corresponding to different semantic variables.

[0032]

[0033] In some embodiments, each non-semantic variable in the first log can be identified by a pre-designed regular expression, and different non-semantic variables can be replaced with the same second identifier; for example, the second identifier can be a uniform identifier <*>.

[0034] Understandably, in this embodiment of the application, different semantic variables in the log sample library are replaced with different first identifiers, and non-semantic variables are replaced with a unified identifier. Compared with the traditional log parsing method that replaces all variables with a unified identifier, this achieves fine-grained identification of variables and preserves the original log semantic information to a greater extent.

[0035] Step 102: Search for a target log template that matches the second log from a pre-built first log template library; wherein the first log template library is obtained by deduplicating the second log template library, the second log template library includes at least one second log template, and the second log template is obtained by replacing the identifier in the variables of the third log.

[0036] In some embodiments, the second log template library can be deduplicated by discarding log templates in the second log template library that are identical to any of the second log templates, thus obtaining the first log template library.

[0037] In some embodiments, pre-designed regular expressions can be used to identify semantic and non-semantic variables in the third log, and different semantic variables can be replaced with corresponding first identifiers, while different non-semantic variables can be replaced with second identifiers to obtain a second log template.

[0038] In some embodiments, the third log may be a log from a large-scale log library, which includes some or all logs collected from various systems, software, and modules. Accordingly, the variables in each third log entry in the large-scale log library are replaced with identifiers to obtain a second log template corresponding to each third log entry, thereby obtaining a second log template library. The second log template is obtained by performing the identifier replacement on the variables in the third log; that is, the second log template library is obtained by performing the identifier replacement on the variables of each third log entry in the large-scale log library.

[0039] In some embodiments, the third log may also be a log from a log library corresponding to the system that generated the first log, wherein the log library includes some or all logs collected on the system that generated the first log. Accordingly, the variables in each third log entry in the log library corresponding to the system that generated the first log are replaced with identifiers to obtain a second log template corresponding to each third log entry, thereby obtaining a second log template library. The second log template is obtained by performing the identifier replacement on the variables in the third log; that is, the second log template library is obtained by performing the identifier replacement on the variables in each third log entry in the log library corresponding to the system that generated the first log.

[0040] Understandably, in this embodiment, a pre-designed regular expression is used to identify different variables for each third log in the log database, generating a second log template for matching with the second log after deduplication. That is, only regular expressions need to be designed for the variables. Compared to setting a regular expression for each log entry and matching the log to be parsed based on the regular expression, this avoids using a large number of regular expressions when there are many log samples to be matched, thereby reducing computational complexity.

[0041] Furthermore, in this embodiment, after the identifier replacement is performed on the third log in the log library, the generated second log template consists of an identifier and a constant. Thus, some third logs with the same structure but different log content may generate the same second log template after the identifier replacement. That is, the identifier replacement can generate a second log template library containing a large number of duplicate templates. Therefore, the second log template library can be deduplicated to obtain a first log template library, and then the first log template library can be used to match the second log, which can reduce the number of log templates that the second log needs to match, thereby improving the log parsing speed.

[0042] Step 103: Determine the parsing result of the first log based on the target log template.

[0043] In some embodiments, a third log template that maps to the target log template can be found in the third log template library based on the template mapping relationship between the first log template library and the pre-built third log template library, and the mapped third log template can be used as the parsing result; wherein, the third log template library is obtained by merging log templates in the first log template library.

[0044] In some embodiments, the target log template can be used as the parsing result of the first log.

[0045] In some embodiments, the template mapping relationship can be constructed through steps 201 to 203 of the following embodiments, which will not be described in detail here.

[0046] In some embodiments, the electronic device that constructs the template mapping relationship may be an electronic device that performs log parsing on the first log, or it may be another electronic device that is different from the electronic device that performs log parsing on the first log.

[0047] In some embodiments, if the mapped third log template cannot be found in the third log template library according to the template mapping relationship, the first log is added to the log library where the third log is located, the log library where the third log is located is updated, and the template mapping relationship is updated based on the updated log library corresponding to the third log, so that the log parsing is performed according to the updated template mapping relationship.

[0048] In some embodiments, such as Figure 2 As shown in the diagram, the process of constructing the template mapping relationship between the first log template library and the third log template library includes the following steps 201 to 203:

[0049] Step 201: Determine the similarity between any two log templates in the first log template library.

[0050] In some embodiments, a pre-trained text similarity model can be used to determine the similarity between pairs of log templates in the first log template library; wherein the text similarity model is trained based on the first log template library.

[0051] In some embodiments, the training process of the text similarity model includes the following steps 2011 to 2012 (not shown in the figure):

[0052] Step 2011: Determine the term frequency-inverse document frequency (TF-IDF) of each word in one of the log templates in the first log template library based on the following formula (1), so as to determine the log template vector of one of the log templates according to the TF-IDF of each word.

[0053]

[0054] Wherein, #word represents the number of times the first word appears in one of the log templates; #total represents the number of times the first word appears in the first log template library; #L represents the number of log templates in the first log template library; and #Lword represents the number of log templates in the first log template library that include the first word.

[0055] Step 2012: Determine the log template vector of each log template in the first log template library, calculate the similarity between any two log template vectors in the first log template library, and determine the text similarity between any two log templates in the first log template library based on the similarity between the two log template vectors.

[0056] Understandably, in the embodiments of this application, the text similarity model is applied to the log parsing method. The similarity between pairs of templates in the first log template library can be determined based on the text similarity model, so as to identify structurally similar templates. These structurally similar templates are then merged to obtain the third log template library.

[0057] Step 202: Merge the log templates in the first log template library.

[0058] In some embodiments, two log templates in the first log template library with a similarity greater than a first threshold can be merged to obtain the third log template library;

[0059] In some other embodiments, two log templates in the first log template library with a similarity greater than a second threshold can be merged to obtain a fourth log template library; and the similarity between any two log templates in the fourth log template library can be determined, and two log templates in the fourth log template library with a similarity greater than a second threshold can be merged until the similarity of each log template in the obtained log template library is less than or equal to the second threshold, to obtain the third log template library;

[0060] In some embodiments, the first threshold and the second threshold can be preset according to actual needs; the first threshold and the second threshold can be the same or different.

[0061] In some embodiments, two log templates in the first log template library whose similarity is greater than a first threshold or a second threshold can be understood as similar log templates, or two log templates in the first log template library whose similarity is greater than a first threshold or a second threshold can be referred to as a similar template pair.

[0062] Understandably, in the embodiments of this application, on the one hand, two log templates in the first log template library with a similarity greater than the first threshold can be merged once to obtain a third log template library; on the other hand, two log templates in the first log template library with a similarity greater than the second threshold can be merged once to obtain a fourth log template library, and then the fourth log template library can be merged a second time, or the templates after the second merge can be merged again, until the similarity of each of the log templates in the obtained log template library is less than or equal to the second threshold, thereby obtaining a third log template library; in this way, after multiple merges, while ensuring that log templates with similar structures in the first log template library can be merged, the number of third log template libraries is reduced, thereby reducing the number of third log templates that the second log needs to match, and improving the parsing speed.

[0063] In some embodiments, two log templates with a similarity greater than a first threshold are merged to obtain the third log template library, including one of the following methods (1) to (2):

[0064] Method (1) is as follows: if N constants are inconsistent in two log templates with a similarity greater than the first threshold, the N constants in one of the log templates are replaced with the second identifier and used as the third log template; wherein N is less than or equal to the third threshold.

[0065] In some embodiments, the third threshold can be preset according to actual log parsing needs.

[0066] For example, Figure 3a This is a schematic diagram of similar log templates and merged third log templates provided in the embodiments of this application. The first type of example is an example of merging two log templates with a similarity greater than a first threshold to obtain a third log template library by means of method (1).

[0067] exist Figure 3a In the first type of example, assuming the first threshold is 1 and the second identifier is <*>, the two log templates with a similarity greater than the first threshold are template X1 and template Y1; where X1 is: Received block blk_src: <path>, Y1 is: Receiving block blk_src: <path>If there is a discrepancy between constants in X1 and Y1 (i.e., the constant Received in X1 is different from the constant Receiving in Y1), then replace Received in X1 or Receiving in Y1 with the second identifier <*>. The resulting third log template Z1 is: <*>block blk_src: <path>.

[0068] Method (2) is as follows: when there is at least one overlapping sequence between two log templates with similarity greater than the first threshold, the overlapping sequence whose length meets the condition among the at least one overlapping sequence is used as the third log template.

[0069] In some embodiments, the first content in two log templates with a similarity greater than a first threshold can be identified first, the content including constants and / or variables. Then, the second content that conforms to the original arrangement order of one of the two log templates with a similarity greater than the first threshold can be identified in the first content. Then, the first i third contents of different lengths can be identified based on the second content, and each third content is taken as an overlapping sequence of different lengths. Wherein, 1≤i≤the sum of the number of constants and variables in the second content.

[0070] For example, consider two log templates with a similarity greater than the first threshold, A1 and A2. Template A1 consists of the sequence a1 b1 c1 d1 e1, and template A2 consists of the sequence a1 f1 d1 e1 c1. Here, a1, b1, c1, d1, e1, and f1 are constants or variables, separated by spaces. The common first content in A1 and A2 is a1, d1, e1, and c1. The order of a1, d1, e1, and c1 in A1 is a1c1 d1 e1, and the order in A2 is a1 d1 e1 c1. Assuming the content conforming to the original order of A1 is taken as the second content, then the second content is a1 d1 c1. The sum of the number of constants and variables in the second content is 3. Therefore, the third content, i.e., the overlapping sequence of the two log templates with a similarity greater than the first threshold, is a1, a1 d1, and a1 d1c1.

[0071] In some embodiments, the longest overlapping sequence can be used as the third log template; or, the overlapping sequence with a length equal to a first preset value among the at least one overlapping sequence can be used as the third log template; wherein, the first preset value can be preset according to actual log parsing requirements;

[0072] Figure 3a The second and third examples are examples of merging two log templates with a similarity greater than the first threshold to obtain a third log template library by means of method (2); wherein, the second example is an example of using the longest overlapping sequence as the third log template, and the third example is an example of using the overlapping sequence with a length equal to the first preset value in the at least one overlapping sequence as the third log template.

[0073] exist Figure 3a In the second type of example, assume that the two log templates with a similarity greater than the first threshold are template X2 and template Y2; where X2 is: BLOCK*ask <ip>to repli cate blk_<*>to datanode(s) <ip>Y2 is BLOCK*ask <ip>to repli cate blk_<*>to datanode(s) <ip> <ip>X2 and Y2 have at least one overlapping sequence, for example, the overlapping sequence BLOCK*ask. <ip>to repli, overlapping sequence BLOCK*ask, BLOCK*ask <ip>torepli cate blk_<*>to datanode(s) <ip>and overlapping sequences to datanode(s) <ip>etc.; the longest overlapping sequence can be BLOCK*ask <ip>to repli cate blk_<*>to datanode(s) <ip>Z2 serves as the third log template.

[0074] exist Figure 3a In the third type of example, assume that the two log templates with a similarity greater than the first threshold are X3 and Y3, respectively; where X3 is: PacketResponder blk_<*><*>Exception java.io.IOException:Brokenpipe, and Y3 is writeBlock blk_<*>received Exception java.io.IOException:Brokenpipe. X3 and Y3 have at least one overlapping sequence, for example, overlapping sequence bk_<*>Exceptionjava.io.IOException:Broken pipe, overlapping sequence IOException:Broken pipe, etc.; if the first preset value is 6, then the third log template Z3 is the first 6 constants or variables in the third content determined according to X3 and Y3: that is, Z3 is: blk_<*>Exception java.io.IOException:Broken pipe, where the 6 constants or variables in Z3 are: blk, <*>, Exception, java.io.IOException:, Broken, and pipe.

[0075] In some embodiments, the method of merging two log templates with a similarity greater than a second threshold to obtain a fourth log template is the same as the method of merging two log templates with a similarity greater than a first threshold to obtain a third log template. Merging two log templates with a similarity greater than a second threshold in the first log template library to obtain a fourth log template library includes one of the following methods (3) to (4):

[0076] Method (3) is as follows: if there are M constants that are inconsistent between two log templates with a similarity greater than the second threshold, the M constants in one of the log templates are replaced with the second identifier and then used as the third log template; wherein M is less than or equal to the fourth threshold.

[0077] In some embodiments, the fourth threshold can be preset according to actual log parsing requirements.

[0078] For example, Figure 3b This is a schematic diagram of similar log templates and the merged fourth log template provided in the embodiments of this application. The first type of example is an example of merging two log templates with a similarity greater than the second threshold by means of method (3) to obtain the fourth log template library.

[0079] exist Figure 3b In the first type of example, assuming the first threshold is 1 and the second identifier is <*>, the two log templates with a similarity greater than the first threshold are template X1 and template Y1; where X1 is: Received block blk_src: <path>, Y1 is: Receiving block blk_src: <path>If there is a discrepancy between constants in X1 and Y1 (i.e., the constant Received in X1 is different from the constant Receiving in Y1), then replace Received in X1 or Receiving in Y1 with the second identifier <*>. The resulting third log template Z1 is: <*>block blk_src: <path>.

[0080] Method (4) is as follows: when there is at least one overlapping sequence between two log templates with similarity greater than the second threshold, the overlapping sequence whose length meets the condition among the at least one overlapping sequence is used as the fourth log template.

[0081] In some embodiments, the same fourth content in two log templates with a similarity greater than a second threshold can be identified first. The content includes constants and / or variables. Then, the fifth content that conforms to the original arrangement order of one of the two log templates with a similarity greater than a first threshold can be identified in the fourth content. Then, the first i sixth contents of different lengths can be identified based on the fifth content, and each sixth content is regarded as an overlapping sequence of different lengths. Wherein, 1≤i≤the sum of the number of constants and variables in the fifth content.

[0082] For example, consider two log templates with a similarity greater than the second threshold, A1 and A2. Template A1 consists of the sequence a1 b1 c1 d1 e1, and template A2 consists of the sequence a1 f1 d1 e1 c1. Here, a1, b1, c1, d1, e1, and f1 are constants or variables, separated by spaces. The fourth content shared by A1 and A2 is a1, d1, e1, and c1. In A1, the order of a1, d1, e1, and c1 is a1c1 d1 e1, and in A2, the order is a1 d1 e1 c1. Assuming the content conforming to the original order of A1 is taken as the fifth content, then the fifth content is a1 d1 c1. The sum of the number of constants and variables in the fifth content is 3. Therefore, the sixth content, i.e., the overlapping sequence of the two log templates with a similarity greater than the second threshold, is a1, a1 d1, and a1 d1c1.

[0083] In some embodiments, the longest overlapping sequence can be used as the fourth log template; or, the overlapping sequence with a length equal to a second preset value among the at least one overlapping sequences can be used as the fourth log template; wherein, the second preset value can be preset according to actual log parsing requirements;

[0084] Figure 3a The second and third examples are examples of merging two log templates with a similarity greater than the second threshold to obtain a third log template library by means of method (2); wherein, the second example is an example of using the longest overlapping sequence as the fourth log template, and the third example is an example of using the overlapping sequence with a length equal to the second preset value in the at least one overlapping sequence as the fourth log template.

[0085] exist Figure 3b In the second type of example, assume that the two log templates with a similarity greater than the second threshold are template X2 and template Y2; where X2 is: BLOCK*ask <ip>to repli cate blk_<*>to datanode(s) <ip>Y2 is BLOCK*ask <ip>to repli cate blk_<*>to datanode(s) <ip> <ip>X2 and Y2 have at least one overlapping sequence, for example, the overlapping sequence BLOCK*ask. <ip>to repli, overlapping sequence BLOCK*ask, BLOCK*ask <ip>torepli cate blk_<*>to datanode(s) <ip>and overlapping sequences to datanode(s) <ip>Wait, the longest overlapping sequence can be BLOCK*ask <ip>to repli cate blk_<*>to datanode(s) <ip>Z2 is the fourth log template.

[0086] exist Figure 3b In the third type of example, assume that the two log templates with a similarity greater than the second threshold are X3 and Y3, respectively; where X3 is: PacketResponder blk_<*><*>Exception java.io.IOException:Brokenpipe, and Y3 is writeBlock blk_<*>received Exception java.io.IOException:Brokenpipe. X3 and Y3 have at least one overlapping sequence, for example, overlapping sequence bk_<*>Exceptionjava.io.IOException:Broken pipe, overlapping sequence IOException:Broken pipe, etc. If the second preset value is 6, then the fourth log template Z3 is the first 6 constants or variables in the sixth content determined according to X3 and Y3: that is, Z3 is: blk_<*>Exception java.io.IOException:Broken pipe, where the 6 constants or variables in Z3 are: blk, <*>, Exception, java.io.IOException:, Broken, and pipe.

[0087] Step 203: Establish a template mapping relationship between the merged log template and the original log template.

[0088] In some embodiments, if a third log template is obtained after merging a pair of similar log templates once, a template mapping relationship is established between each log template in the pair of log templates before merging and the merged third log template. If a third log template is obtained after merging N pairs of similar templates multiple times, a template mapping relationship is established between each log template in the N pairs of similar templates and the merged third log template.

[0089] For example, if template A and template B are merged to obtain template C, then a template mapping relationship is established between A and C, and between B and C; if A and B are merged to obtain C, template D and template E are merged to obtain template F, and C and F are merged to obtain template G, then a template mapping relationship is established between A and G, between B and G, between D and G, and between E and G.

[0090] In some embodiments, establishing a template mapping relationship between the merged log template and the log template before merging can be understood as the log template before merging and the log template after merging satisfying the template mapping relationship; or, it can be understood as the log template in the third log template library that matches or corresponds to the log template before merging is the merged log template.

[0091] It should be noted that each pair of similar log templates in the first log template library may contain different log templates; for example, the log templates A, B, C and D contained in the first similar template pair A and B are different from those contained in the second similar template pair C and D; or, the first log template library contains at least two similar log template pairs with the same log template; for example, the first log template library contains a third similar template pair and a fourth similar template pair, wherein the third similar template pair A and B contains the same log template A as the fourth similar template pair A and C.

[0092] In some embodiments, the accuracy of the template mapping relationship obtained after the merging process is determined; and the first threshold or the second threshold is adjusted according to the accuracy.

[0093] In some embodiments, the accuracy of the template mapping relationship between the merged log template and the log template before merging can be determined based on the pre-annotated mapping relationship between the first log template library and the third log template library.

[0094] In some embodiments, inappropriate template mapping relationships in the first log template library and the third log template library can be marked, and the accuracy of the template mapping relationship can be determined based on the ratio of the number of inappropriate template mapping relationships to the total number of mapping relationships in the first log template library and the third log template library, so as to evaluate the effect of log parsing based on the accuracy.

[0095] Understandably, when evaluating the effectiveness of log parsing, it is sufficient to mark the inappropriate first mapping relationship between the first log template library and the third log template library, rather than marking whether a large number of third logs in the log library match the third log templates after identifier replacement and merging; thus, the evaluation scheme for log parsing effectiveness is simplified.

[0096] In some embodiments, the first threshold or the second threshold may be adjusted based on the accuracy rate.

[0097] Understandably, if the accuracy of the template mapping relationship is low, it indicates that the classification of the third log in the log library is inappropriate or that the classification of the log library containing the third log cannot be converged. Therefore, new similar template pairs can be determined by adjusting the first threshold and / or the second threshold, thereby obtaining a more suitable merged third log template.

[0098] Based on this, the exemplary application of the embodiments of this application in a practical application scenario will be described below.

[0099] System logs are text output by programs written by developers (such as printf, logging.log, and logger.info programs), recording information about each task during the current device operation, such as the event type, user ID, and event level of user requests. Mining system logs can be used for anomaly detection, attack tracing, fault diagnosis, and user profiling. However, raw log data is usually unstructured or semi-structured, requiring appropriate methods to parse it into structured data. Log parsing is a key step in achieving automated and effective log analysis. In some embodiments, log parsing methods are shown in Table 2 below.

[0100] Table 2 Log parsing methods

[0101]

[0102] Mode is a crucial characteristic of log parsing methods. Depending on the scenario in which the parser is used, it can be categorized into two modes: offline and online. Offline log parsing methods require pre-parsing all log data in batches. To cope with frequent system updates, developers need to periodically rerun the offline parser to obtain the latest log templates. In contrast, online parsers parse log data in a streaming manner, allowing for better integration with subsequent log mining tasks.

[0103] Preprocessing refers to the steps of processing all variables in logs based on certain domain knowledge. Developers need to design appropriate regular expressions to process each log entry individually. For example, IP addresses (e.g., 10.86.169.121:62260) are common variables in various logs. Existing parsing schemes use the uniform identifier <*> to replace all variables in the logs. While this improves the efficiency of extracting log templates, the obtained log templates lack some semantic information.

[0104] The key factor determining the effectiveness of log parsing methods is the technique for extracting log templates, which directly affects the efficiency and accuracy of template extraction. The Simple Log Clustering Tool (SLCT) allows for variable-length parameters between two log entries using a clustering algorithm. In some embodiments, log parsing methods include: Log Key Extraction (LKE), which uses finite state automata to extract system task flows from log sequences and applies a hierarchical clustering algorithm based on a custom weighted edit distance to identify anomalous log sequences. The Sigmoid function (LogSig) improves upon LKE by applying heuristic rules to optimize the clustering algorithm, representing system logs in the form of multiple word pairs, and extracting log templates based on identical word pairs. The AECID-PG and Drain algorithms are tree-based log parsing algorithms. Drain uses log length to generate partitions, using a log word as a tree node to generate a tree of log templates corresponding to the original log content. The Spell algorithm parses logs in an online streaming mode, applying the longest common subsequence algorithm to match log templates. The IPLoM algorithm groups log data in a hierarchical manner.

[0105] In related technologies, log parsing relies heavily on regular expressions to extract specific log templates. As scale and complexity increase, software systems often generate a large number of logs with different templates. Therefore, manually creating and maintaining regular expressions requires significant manpower. Furthermore, with software updates, many new log templates are generated, necessitating frequent updates to the log parsing method. This has three main drawbacks, A) through C):

[0106] A) A large number of regular expressions need to be designed to match the corresponding log templates, and coarse-grained identification of variable parameters in logs will lead to a large loss of semantics;

[0107] B) Generating a template library for logs with tens of millions of records takes a lot of time, and some methods are difficult to handle massive log data.

[0108] C) When evaluating the effectiveness of log parsing, it is necessary to manually annotate the original logs, which is a time-consuming and repetitive task.

[0109] This application provides a log parsing method based on a text similarity model. It applies regular expressions to identify common variable types in third-party logs and the first log to be parsed, and performs fine-grained segmentation, such as accurately identifying IP addresses, URLs, and file paths in the third-party logs and the first log to be parsed. Duplicate log templates in the second log template library are removed. Since the log library containing the third logs consists of a large number of third logs with the same or similar structures, the number of log templates in the second log template library can be reduced from tens of millions to about one hundred. This significantly improves the parsing speed; the log parsing and evaluation work can be completed based on the first log template after deduplication.

[0110] This application's embodiments utilize text similarity recognition to identify structured logs, representing a data-driven log parsing method capable of parsing multi-source heterogeneous logs. Figure 4 A flowchart of log parsing provided for embodiments of this application:

[0111] refer to Figure 4 The log parsing implementation scheme includes the following steps 401 to 406:

[0112] Step 401: Fine-grained matching of variables in the third log is performed using regular expressions. The third log is separated by spaces and treated as a sentence composed of multiple words. Regular expressions are designed to match common variables (i.e., semantic variables) in the third log, such as IP addresses, URLs, and file paths, as shown in Table 1. Finally, other variables without special meaning (i.e., non-semantic variables) are replaced with uniform identifiers (<*>) to obtain the second log template. Unlike traditional log parsing methods that replace all variables with uniform identifiers, this scheme achieves fine-grained variable identification, preserving the original log semantic information to the greatest extent.

[0113] Step 402: Remove duplicate logs from the second log template library (i.e., preprocess the first log template) to obtain the first log template library. Taking the third log collected on the Hadoop Distributed File System (HDFS) as an example, the original 11 million logs can be compressed to 54.

[0114] Step 403: Using the first log template library as the corpus, train a TF-IDF-based text similarity model to filter structurally similar log templates. The calculation formula is shown in Equation 1. The entire training process can be completed within 5 seconds. When system updates cause the log library corresponding to the third log to need to be updated, the text similarity model can be updated online.

[0115] Step 404: Apply the trained text similarity model to calculate the similarity between any two log templates in the first log template library, and determine whether template merging is possible by setting a first threshold. Log templates with similarity below the first threshold do not need to be merged and are directly used as the third log template.

[0116] Step 405: Divide the templates to be merged into three categories, such as... Figure 3a or Figure 3b As shown. The first category involves a single inconsistent word (an example of a constant), which is replaced with a second identifier. The second category involves two log templates having a subsequence relationship, in which case the subsequence is used as the merged third log template. The third category uses the result of the longest common subsequence algorithm as the merged third log template. A hash table (an example of a template mapping relationship) is constructed corresponding to the first and third log template libraries, using log templates from the first library as keys and templates from the third library as values.

[0117] Step 406: When the first log to be parsed is parsed as a stream, the second log is obtained after the identifier replacement. A target log template matching the second log is searched from the first log template library, and then a third log template corresponding to the target log template is found based on the hash table. This determines the parsing result of the first log. If a system update prevents the parsing result of the first log from being determined based on the hash table, steps 403 to 405 are repeated to update the hash table.

[0118] In some embodiments, when evaluating the log parsing performance, inappropriate template mapping relationships in the hash table are marked, and the number of inappropriate template mapping relationships and all template mapping relationships is counted to complete the calculation of indicators such as accuracy.

[0119] Third-party logs generated by different systems and software vary significantly in structure. Figure 5 The multi-source logs provided in the embodiments of this application, such as Figure 5 As shown. The bolded part represents the main content of the third log, which contains variables of different types, and even variables of the same type, such as time, vary significantly in form. Therefore, for multi-source logs, it is necessary to establish a corresponding log template library and hash table for each source's third log, and the first threshold needs to be flexibly adjusted based on the evaluation of the parsing effect.

[0120] The embodiments of this application can identify variables in logs with fine granularity to preserve semantics;

[0121] This application embodiment uses a text similarity model to find first log templates with similar structures in the first log template library, which is something that has not been considered or involved in the relevant scenarios.

[0122] This application embodiment constructs hash tables corresponding to the first log template library and the third log template library.

[0123] In evaluating the log parsing performance, this application embodiment requires marking inappropriate template mapping relationships in the hash table and then counting the number of inappropriate template mapping relationships and all template mapping relationships to complete the calculation of indicators such as accuracy.

[0124] The embodiments of this application are applicable to parsing large-scale logs, and the efficiency is improved by more than four times when parsing more than ten million logs. Figure 6 The parsing time corresponds to the different log parsing methods provided in the embodiments of this application.

[0125] Logs are frequently updated data. In some embodiments, it is necessary to repeatedly annotate the third log template corresponding to the third log after each update. The embodiments of this application reduce the workload of manually annotating logs during the evaluation stage, thereby reducing manpower consumption.

[0126] The embodiments of this application can flexibly adjust the first threshold according to the template mapping relationship to adapt to different types of logs, which facilitates the optimization of the entire solution.

[0127] The embodiments of this application are applicable to online multi-source log parsing, and the semantic information retained in the first log template is rich, which is beneficial to the subsequent log mining work.

[0128] It should be noted that although the steps of the method in this application are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps; or steps from different embodiments may be combined into a new technical solution.

[0129] Based on the foregoing embodiments, this application provides a log parsing device, which includes various modules and units included in each module, and can be implemented by a processor; of course, it can also be implemented by specific logic circuits; in the implementation process, the processor can be a central processing unit (CPU), microprocessor (MPU), digital signal processor (DSP) or field programmable gate array (FPGA), etc.

[0130] Figure 7 This is a schematic diagram of the log parsing device provided in the embodiments of this application, as shown below. Figure 7 As shown, the log parsing device 70 includes a replacement module 71, a search module 72, and a first determination module 73, wherein:

[0131] Replacement module 71 is used to replace the identifiers of variables in the first log to be parsed to obtain the second log; wherein, the variables include semantic variables and / or non-semantic variables; different semantic variables correspond to different first identifiers, and different non-semantic variables correspond to the same second identifier;

[0132] The lookup module 72 is used to search for a target log template that matches the second log from a pre-built first log template library; wherein the first log template library is obtained by deduplicating the second log template library, the second log template library includes at least one second log template, and the second log template is obtained by replacing the identifier of the variable in the third log.

[0133] The first determining module 73 determines the parsing result of the first log based on the target log template.

[0134] In some embodiments, the log parsing device 70 further includes a deduplication module, which is used to discard log templates in the second log template library that are identical to any of the second log templates, in order to obtain the first log template library.

[0135] In some embodiments, the log parsing device 70 further includes a second determining module, which is configured to find a third log template that maps to the target log template from the third log template library according to the template mapping relationship between the first log template library and the pre-built third log template library; wherein the third log template library is obtained by merging log templates in the first log template library; and the mapped third log template is used as the parsing result.

[0136] In some embodiments, the log parsing device 70 further includes a third determining module, a merging module, and an establishing module. The third determining module is used to determine the similarity between any two log templates in the first log template library. The merging module is used to merge two log templates in the first log template library whose similarity is greater than a first threshold to obtain the third log template library; or, to merge two log templates in the first log template library whose similarity is greater than a second threshold to obtain a fourth log template library; and to determine the similarity between any two log templates in the fourth log template library, merge two log templates in the fourth log template library whose similarity is greater than the second threshold, until each of the similarities in the obtained log template library is less than or equal to the second threshold, to obtain the third log template library. The establishing module is used to establish a template mapping relationship between the merged log templates and the log templates before merging.

[0137] In some embodiments, the merging module is configured to, when two log templates with a similarity greater than a first threshold have N constants that are inconsistent, replace the N constants in one of the log templates with the second identifier and use it as the third log template; wherein N is less than or equal to the third threshold; and when two log templates with a similarity greater than the first threshold have at least one overlapping sequence, use the overlapping sequence whose length meets the condition among the at least one overlapping sequence as the third log template.

[0138] In some embodiments, the third determining module is used to determine the similarity between pairs of log templates in the first log template library using a pre-trained text similarity model; wherein the text similarity model is trained based on the first log template library.

[0139] In some embodiments, the log parsing device 70 further includes a fourth determining module, which is used to determine the accuracy of the template mapping relationship obtained after the merging process; the log parsing device 70 further includes an adjusting module, which is used to adjust the first threshold or the second threshold according to the accuracy.

[0140] The descriptions of the above device embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0141] It should be noted that, in the embodiments of this application... Figure 7 The module division shown in the decoding device is illustrative and represents only one logical functional division; in actual implementation, other division methods may be used. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, exist as separate physical units, or be integrated into one unit with two or more units. The integrated units can be implemented in hardware, as software functional units, or a combination of both.

[0142] It should be noted that, in the embodiments of this application, if the above-described log parsing method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0143] This application provides an electronic device. Figure 8 This is a schematic diagram of the hardware entity of the electronic device according to an embodiment of this application, such as... Figure 8 As shown, the electronic device 80 includes a memory 81 and a processor 82. The memory 81 stores a computer program that can run on the processor 82. When the processor 82 executes the program, it implements the steps in the method provided in the above embodiments.

[0144] It should be noted that the memory 81 is configured to store instructions and applications executable by the processor 82, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data and video communication data) in the processor 82 and various modules in the electronic device 80, which can be implemented by flash memory or random access memory (RAM).

[0145] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps in the log parsing method provided in the above embodiments.

[0146] This application provides a computer program product containing instructions that, when run on a computer, cause the computer to execute the steps in the log parsing method provided in the above-described method embodiments.

[0147] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium, storage medium, and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0148] It should be understood that the phrases "one embodiment," "an embodiment," or "some embodiments" throughout the specification mean that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment," "in one embodiment," or "in some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments. The descriptions of the various embodiments above tend to emphasize the differences between the various embodiments; their similarities or commonalities can be referred to mutually, and for the sake of brevity, they will not be repeated here.

[0149] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three kinds of relationships. For example, object A and / or object B can represent three situations: object A exists alone, object A and object B exist simultaneously, and object B exists alone.

[0150] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one…" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0151] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple modules or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or modules can be electrical, mechanical, or other forms.

[0152] The modules described above as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules. They may be located in one place or distributed across multiple network units. Some or all of the modules may be selected to achieve the purpose of this embodiment according to actual needs.

[0153] In addition, each functional module in the various embodiments of this application can be integrated into one processing unit, or each module can be a separate unit, or two or more modules can be integrated into one unit; the integrated modules can be implemented in hardware or in the form of hardware plus software functional units.

[0154] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0155] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0156] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0157] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0158] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0159] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.< / ip> < / ip> < / ip> < / ip> < / ip> < / ip> < / ip> < / ip> < / ip> < / ip> < / ip> < / path> < / path> < / path> < / ip> < / ip> < / ip> < / ip> < / ip> < / ip> < / ip> < / ip> < / ip> < / ip> < / ip> < / path> < / path> < / path>

Claims

1. A log parsing method, characterized in that, The method includes: The variables in the first log to be parsed are replaced with identifiers to obtain the second log; wherein, the variables include semantic variables and / or non-semantic variables; different semantic variables correspond to different first identifiers, and different non-semantic variables correspond to the same second identifier; The first log template library is used to find the target log template that matches the second log from a pre-built first log template library. The first log template library is obtained by deduplicating the second log template library. The second log template library includes at least one second log template. The second log template is obtained by replacing the identifier of the variable in the third log. Based on the target log template, determine the parsing result of the first log.

2. The method according to claim 1, characterized in that, The deduplication process for the second log template library includes: Discard any log template in the second log template library that is identical to any of the second log templates to obtain the first log template library.

3. The method according to claim 1, characterized in that, Determining the parsing result of the first log based on the target log template includes: Based on the template mapping relationship between the first log template library and the pre-built third log template library, a third log template that maps to the target log template is found from the third log template library; wherein, the third log template library is obtained by merging log templates in the first log template library; The third log template mapped to the phase is used as the parsing result.

4. The method according to claim 3, characterized in that, The merging process for log templates in the first log template library includes: Determine the similarity between any two log templates in the first log template library; and merge two log templates in the first log template library whose similarity is greater than a first threshold to obtain the third log template library; or... The first log template library is merged with two log templates whose similarity is greater than the second threshold to obtain a fourth log template library; and the similarity between any two log templates in the fourth log template library is determined, and two log templates whose similarity is greater than the second threshold in the fourth log template library are merged until the similarity of each log template in the obtained log template library is less than or equal to the second threshold to obtain the third log template library; in, The method further includes: establishing a template mapping relationship between the merged log template and the log template before merging.

5. The method according to claim 4, characterized in that, The step of merging two log templates in the first log template library whose similarity is greater than a first threshold to obtain the third log template library includes one of the following: If two log templates with a similarity greater than the first threshold have N constants that are inconsistent, the N constants in one of the log templates are replaced with the second identifier and used as the third log template; wherein N is less than or equal to the third threshold. If two log templates with a similarity greater than the first threshold have at least one overlapping sequence, the overlapping sequence whose length meets the condition among the at least one overlapping sequence shall be used as the third log template.

6. The method according to claim 4, characterized in that, Determining the similarity between any two log templates in the first log template library includes: The similarity between any two log templates in the first log template library is determined using a pre-trained text similarity model; wherein the text similarity model is trained based on the first log template library.

7. The method according to claim 4, characterized in that, The method further includes: Determine the accuracy of the template mapping relationship obtained after the merging process; The first threshold or the second threshold is adjusted based on the accuracy rate.

8. A log parsing device, characterized in that, include: The replacement module is used to replace the identifiers of variables in the first log to be parsed to obtain the second log; wherein, the variables include semantic variables and / or non-semantic variables; different semantic variables correspond to different first identifiers, and different non-semantic variables correspond to the same second identifier; The search module is used to search for a target log template that matches the second log from a pre-built first log template library; wherein, the first log template library is obtained by deduplicating the second log template library, the second log template library includes at least one second log template, and the second log template is obtained by replacing the identifier of the variable in the third log; The first determining module is used to determine the parsing result of the first log based on the target log template.

9. An electronic device comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the program, it implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images