Device log template extraction method, apparatus, device, and medium

Abstract

The application provides a device log template extraction method, device, equipment and medium, and relates to the field of artificial intelligence. The device log template extraction method comprises the following steps: obtaining original log information of a device and at least one first mask which is a model of log variables irrelevant to a device log template; shielding log variables irrelevant to the device log template in the original log information according to the at least one first mask to obtain first log information; and extracting a log template corresponding to the first log information. In the embodiment of the application, the model of log variables irrelevant to the device log template is set, and the log variables irrelevant to the device log template in the original log information are shielded, so that the rationality of log template extraction can be improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and in particular to a method, apparatus, device, and medium for extracting device log templates. Background Technology

[0002] Equipment, especially large-scale equipment deployed in network systems (e.g., cloud-based network equipment), generates massive amounts of logs during operation, such as 50Gb (approximately 120-200 million lines) of device logs per hour. These logs record the operational status of each device. By collecting and analyzing these logs, useful information can be extracted from this massive amount of data, such as identifying existing or potential faults. However, due to the lack of standardized log specifications, log data is unstructured and voluminous, making manual analysis inefficient. To achieve fast, automated, efficient, and accurate log analysis, it is essential to introduce AI algorithms for automatic log parsing, thereby reducing operational costs and significantly improving the business experience.

[0003] Because unstructured log data is very difficult to process directly, it is often necessary to parse the logs to obtain a template, and then perform log analysis based on the template. For example, log anomaly detection and root cause analysis can be performed based on the log template. Therefore, the accuracy of log analysis is closely related to the accuracy and reasonableness of the extracted log template.

[0004] Existing log templates directly extract and process raw log information, which can lead to problems such as a large number of log messages with similar business functions corresponding to different templates, or low differentiation between different templates. In other words, existing log templates have low aggregation of similar templates or low differentiation between dissimilar templates, resulting in low extraction accuracy and poor rationality. Summary of the Invention

[0005] This invention provides a method, apparatus, device, and medium for extracting device log templates to solve the problem of low accuracy in existing log template extraction methods.

[0006] To solve the above-mentioned technical problems, the present invention is implemented as follows:

[0007] In a first aspect, embodiments of the present invention provide a method for extracting device log templates, the method comprising:

[0008] Obtain the device's raw log information and at least one pre-set first mask, wherein the first mask is a model of log variables that is independent of the device log template;

[0009] Based on the at least one first mask, log variables in the original log information that are unrelated to the device log template are masked to obtain the first log information;

[0010] Extract the log template corresponding to the first log information.

[0011] Optionally, the first mask includes at least one of the following:

[0012] Models for general log variables and network log variables.

[0013] Optionally, the first mask includes at least one of the following:

[0014] First submask and second submask;

[0015] The first sub-mask is a model of general log variables that are independent of the device log template, and the second sub-mask is a model of network log variables that are independent of the device log template.

[0016] Optionally, the general log variables include at least one of the following: date, HTTP address, image name, IP address, and file name;

[0017] The network log variables include at least one of the following: instance identifier, Python execution statement, and request identifier.

[0018] Optionally, when the first mask includes at least two, the step of masking irrelevant variables in the original log information according to the at least one first mask to obtain the first log information includes:

[0019] The masking order of the at least two first masks is determined based on the length and / or complexity of the at least two first masks;

[0020] Irrelevant variables in the original log information are masked according to the masking order to obtain the first log information.

[0021] Optionally, determining the masking order of the at least two first masks based on their length and / or complexity includes the following:

[0022] Arrange the at least two first masks in descending order of mask length to determine the masking order of the at least two first masks; arrange the at least two first masks in descending order of mask complexity to determine the masking order of the at least two first masks.

[0023] The masking order of the at least two first masks is determined by arranging them in order of the weighted sum of mask length and complexity.

[0024] Optionally, before obtaining the first log information by masking irrelevant variables in the original log information according to the at least one first mask, the method further includes:

[0025] Similar statements in the original log information are processed for consistency to obtain second log information;

[0026] The step of masking irrelevant variables in the original log information according to the at least one first mask to obtain the first log information includes:

[0027] Based on the at least one first mask, irrelevant variables in the second log information are masked to obtain the first log information.

[0028] Optionally, the process of performing consistency processing on similar statements in the original log information to obtain second log information includes:

[0029] If semantic analysis is performed on the original log information and a first statement and a second statement with the same semantics are obtained, the first statement will be replaced with the second statement.

[0030] Use the replaced original log information as the second log information.

[0031] Optionally, the method further includes:

[0032] Using the log template and the log variables, a structured log is obtained;

[0033] Feature extraction is performed on the structured logs to obtain log feature vectors;

[0034] Based on the log feature vector, perform anomaly detection and root cause localization of the original log information.

[0035] Secondly, embodiments of the present invention provide a device for extracting device log templates, comprising:

[0036] The acquisition module is used to acquire the device's raw log information and at least one pre-set first mask, wherein the first mask is a model of log variables that is independent of the device log template;

[0037] A masking module is used to mask log variables in the original log information that are unrelated to the device log template according to the at least one first mask, so as to obtain the first log information;

[0038] The extraction module is used to extract the log template corresponding to the first log information.

[0039] Optionally, the first mask includes at least one of the following:

[0040] Models for general log variables and network log variables.

[0041] Optionally, the first mask includes at least one of the following:

[0042] First submask and second submask;

[0043] The first sub-mask is a model of general log variables that are independent of the device log template, and the second sub-mask is a model of network log variables that are independent of the device log template.

[0044] Optionally, the general log variables include at least one of the following: date, HTTP address, image name, IP address, and file name;

[0045] The network log variables include at least one of the following: instance identifier, Python execution statement, and request identifier.

[0046] Optionally, when the first mask comprises at least two, the shielding module comprises:

[0047] The sequence determination module is used to determine the masking order of the at least two first masks based on the length and / or complexity of the at least two first masks;

[0048] The first masking submodule is used to mask irrelevant variables in the original log information according to the masking order to obtain the first log information.

[0049] Optionally, the sequence determination module includes one of the following:

[0050] The first sequence determination submodule is used to arrange the at least two first masks in descending order of mask length to determine the shielding order of the at least two first masks;

[0051] The second sequence determination submodule is used to arrange the at least two first masks in descending order of mask complexity and determine the masking order of the at least two first masks.

[0052] The third sequence determination submodule is used to arrange the at least two first masks in order of the weighted sum of mask length and complexity, and to determine the masking order of the at least two first masks.

[0053] Optionally, the device further includes:

[0054] The processing module is used to perform consistency processing on similar statements in the original log information to obtain second log information;

[0055] The shielding module includes:

[0056] The second masking submodule masks irrelevant variables in the second log information according to the at least one first mask, thereby obtaining the first log information.

[0057] Optionally, the processing module includes:

[0058] The replacement module is used to replace the first statement with the second statement when semantic analysis of the original log information yields a first statement and a second statement with the same semantics.

[0059] Use the replaced original log information as the second log information.

[0060] Optionally, the device further includes:

[0061] The detection module is used to obtain structured logs using the log template and the log variables;

[0062] Feature extraction is performed on the structured logs to obtain log feature vectors;

[0063] Based on the log feature vector, perform anomaly detection and root cause localization of the original log information.

[0064] Thirdly, embodiments of the present invention provide an electronic device, including a processor, the processor being configured to:

[0065] Obtain the device's raw log information and at least one pre-set first mask, wherein the first mask is a model of log variables that is independent of the device log template;

[0066] Based on the at least one first mask, log variables in the original log information that are unrelated to the device log template are masked to obtain the first log information;

[0067] Extract the log template corresponding to the first log information.

[0068] Optionally, the first mask includes at least one of the following:

[0069] Models for general log variables and network log variables.

[0070] Optionally, the first mask includes at least one of the following:

[0071] First submask and second submask;

[0072] The first sub-mask is a model of general log variables that are independent of the device log template, and the second sub-mask is a model of network log variables that are independent of the device log template.

[0073] Optionally, the general log variables include at least one of the following: date, HTTP address, image name, IP address, and file name;

[0074] The network log variables include at least one of the following: instance identifier, Python execution statement, and request identifier.

[0075] Optionally, when the first mask includes at least two, irrelevant variables in the original log information are masked according to the at least one first mask to obtain the first log information, including:

[0076] The masking order of the at least two first masks is determined based on the length and / or complexity of the at least two first masks;

[0077] Irrelevant variables in the original log information are masked according to the masking order to obtain the first log information.

[0078] Optionally, determining the masking order of the at least two first masks based on their length and / or complexity includes the following:

[0079] Arrange the at least two first masks in descending order of mask length to determine the masking order of the at least two first masks; arrange the at least two first masks in descending order of mask complexity to determine the masking order of the at least two first masks.

[0080] The masking order of the at least two first masks is determined by arranging them in order of the weighted sum of mask length and complexity.

[0081] Optionally, before obtaining the first log information by masking irrelevant variables in the original log information according to the at least one first mask, the processor is further configured to:

[0082] Similar statements in the original log information are processed for consistency to obtain second log information;

[0083] The step of masking irrelevant variables in the original log information according to the at least one first mask to obtain the first log information includes:

[0084] Based on the at least one first mask, irrelevant variables in the second log information are masked to obtain the first log information.

[0085] Optionally, the process of performing consistency processing on similar statements in the original log information to obtain second log information includes:

[0086] If semantic analysis is performed on the original log information and a first statement and a second statement with the same semantics are obtained, the first statement will be replaced with the second statement.

[0087] Use the replaced original log information as the second log information.

[0088] Optionally, the processor is further configured to:

[0089] Using the log template and the log variables, a structured log is obtained;

[0090] Feature extraction is performed on the structured logs to obtain log feature vectors;

[0091] Based on the log feature vector, perform anomaly detection and root cause localization of the original log information.

[0092] Fourthly, embodiments of the present invention provide an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, it implements the steps of the device log template extraction method as described in the first aspect above.

[0093] Fifthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the device log template extraction method described in the first aspect above.

[0094] The device log template extraction method in this embodiment obtains the original log information of the device and at least one pre-set first mask, where the first mask is a model of log variables unrelated to the device log template; based on the at least one first mask, log variables unrelated to the device log template in the original log information are masked to obtain first log information; and the log template corresponding to the first log information is extracted. In this embodiment, by setting a model of log variables unrelated to the device log template, log variables unrelated to the device log template in the original log information are masked. That is, before extracting the template, the mask is used to efficiently identify and mask the "template-irrelevant variables" in the original log information, thereby improving the rationality of log template extraction. Attached Figure Description

[0095] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0096] Figure 1 This is a schematic diagram of an anomaly detection based on a log template provided in an embodiment of the present invention;

[0097] Figure 2 This is a flowchart of a device log template extraction method provided in an embodiment of the present invention;

[0098] Figure 3This is a flowchart of another device log template extraction method provided in an embodiment of the present invention;

[0099] Figure 4 This is a schematic diagram of a device log template extraction architecture provided in an embodiment of the present invention;

[0100] Figure 5 This is a schematic diagram of an operation and maintenance system based on log analysis provided in an embodiment of the present invention;

[0101] Figure 6 This is a schematic diagram of a device log template extraction device provided in an embodiment of the present invention;

[0102] Figure 7 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0103] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0104] For ease of understanding, the following describes some aspects of the embodiments of the present invention:

[0105] Logs, as a type of time-series text data, consist of timestamps and text messages, recording the real-time operational status of a system. Specifically, logs record the working status of devices (e.g., network systems), the information flow of handshakes, and the workflow between ports. Logs from systems experiencing anomalies often differ significantly from normal logs in the order of events (i.e., workflow or sequence) and the frequency of events. Therefore, the key to anomaly detection based on logs is to extract the time-series information (i.e., the order in which events occur) and frequency information contained within the logs. This requires first converting the log text information into event templates and then further into numerical information before anomaly detection. Direct processing of unstructured log data is very difficult; often, log templates are obtained through log parsing, and then anomaly detection is performed based on these templates.

[0106] See also Figure 1The exemplary description includes logs such as "2021.05.21 00:05:20.001 port Asend 256 bytes request to B", "2021.05.21 00:05:20.002 port B received 256 bytes request from A", "2021.05.21 00:05:20.003 establish connection between A and B", and "2021.05.21 00:05:20.004 port A received 2.5M bytes from B", which include timestamps and text messages, recording information such as the working status of network devices, the information flow of handshakes, and the workflow of port status. The logs mentioned above correspond to templates E1-E4. When the templates appear in the order of E1, E2, E3, E4 or E1, E1, E2, E3, E4, it can be considered a normal communication state. When the templates appear in the order of E1, E1, E1, E1, E1 (the frequency of events differs significantly from normal logs) or E1, E2, E4 (the order of events differs significantly from normal logs), it can be considered an abnormal communication state. The accuracy of log analysis (e.g., anomaly detection) is closely related to the reasonableness of the extracted log templates.

[0107] A template is essentially a "summary" of a log entry. Log parsing is template extraction, which is the extraction of this log "summary." A log entry can be viewed as a template with parameters. For example, if the template is "send *bytes to *" and the parameters are "size = 120, block = blk_4612", the combination of the template and parameters will produce the log entry "send 120 bytes to blk_4612". Log parsing is essentially the reverse process of log printing; it involves processing the log entry in reverse to obtain the template.

[0108] Among existing automatic log parsing methods, the Drain algorithm is quite common. This method assumes that logs with the same length, or in other words, log templates with the same number of tokens, have similar business meanings. Therefore, length is an important criterion for template extraction. Furthermore, specific keywords also represent specific business meanings. Variables are generally considered to be pure numbers or device codes. Conventional template extraction methods, such as... Figure 2As shown, the variable `token` (purely numeric or device code) is first converted to `*`, then categorized by length, and finally categorized by keyword, resulting in a template. For example, `Receive from node blk_3587` is processed to obtain the template `Receive from node*`. After extracting the template content, a unique ID is assigned. Figure 2 For example, the input layer collects raw log messages; the number matching layer identifies and masks numeric variables before extracting templates; the length matching layer performs length matching on log messages; the keyword matching layer matches keywords (receive, send) in log messages; and the output layer assigns and outputs template IDs.

[0109] Existing automatic log parsing solutions directly extract templates from raw logs, which leads to a problem where many log messages with similar business logic correspond to different templates. At the same time, the differentiation between different templates is low, meaning that the aggregation of similar templates and the differentiation between dissimilar templates are both very low. Anomaly detection based on such unreasonable templates will greatly reduce the accuracy of log analysis.

[0110] In this embodiment of the invention, a method for extracting device log templates is proposed to solve the problems of low accuracy and poor rationality in existing log template extraction.

[0111] See Figure 3 , Figure 3 This is a flowchart of a device log template extraction method provided in an embodiment of the present invention, such as... Figure 3 As shown, the method includes the following steps:

[0112] Step 301: Obtain the device's raw log information and at least one pre-set first mask, wherein the first mask is a model of log variables that is unrelated to the device log template;

[0113] In this embodiment of the invention, the original log information of the aforementioned device can be understood as log information with an extraction template. The first mask can be constructed based on the pattern or format of the log variables. For example, a date variable can be in the pattern "xxxx year / xx month / xx day" or "xxxx-xx-xx", and a corresponding first mask can be formed for that date variable.

[0114] Different raw log information can correspond to different first masks, and the type of the first mask can be determined based on the type of the raw log information. For example, for network device logs, at least some of the first masks can correspond to specific network log variables.

[0115] Optionally, the first mask includes at least one of the following:

[0116] Models for general log variables and network log variables.

[0117] Alternatively, the first mask may be understood to include at least one of the following:

[0118] First submask and second submask;

[0119] The first sub-mask is a model of general log variables that are independent of the device log template, and the second sub-mask is a model of network log variables that are independent of the device log template.

[0120] The construction of the first mask mentioned above can refer to common log variables, such as date, address, and filename. Furthermore, considering the application scenario of automatic parsing of network cloud device logs, and combining the characteristics of logs generated by network cloud devices in actual production, a first mask can be set for network log-specific variables, such as instance identifiers, Python execution statements, and request identifiers. This means specifically considering the characteristic patterns of network logs and identifying statements closely related to the business scenario in the logs generated by network cloud devices. By combining the business semantic features in intelligent network operation and maintenance scenarios with the characteristics of logs generated by network cloud devices in actual production, the effectiveness of network device log template extraction can be improved, thereby improving the accuracy of network log analysis, such as improving the accuracy of fault analysis and root cause localization.

[0121] Optionally, the general log variables include at least one of the following: date, HTTP address, image name, IP address, and file name;

[0122] The network log variables include at least one of the following: instance identifier, Python execution statement, and request identifier.

[0123] The general log variables include at least one of the following: date, HTTP address, image name, IP address, and file name. This can be understood as the first sub-mask including at least one of the following: date mask, HTTP address mask, image name mask, IP address mask, and file name mask.

[0124] The network log variables include at least one of the following: instance identifier, Python execution statement, and request identifier. It can be understood that the second sub-mask includes at least one of the following: instance identifier mask, Python execution statement mask, and ReqID.

[0125] Python execution statements typically end with ".py", and a Python execution statement mask can also be called a ".py mask".

[0126] Step 302: Shield log variables in the original log information that are unrelated to the device log template according to the at least one first mask, to obtain first log information.

[0127] In the embodiment of the present invention, when log variable information corresponding to the first mask and unrelated to the device log template is identified in the original log information, the information may be shielded, so that the log variables unrelated to the device log template are not taken as objects for template extraction in the template process. For example, "xxxx year xx month xx day" or "xxxx-xx-xx" in the original log information is uniformly replaced with "<*>", to obtain new log information (the first log information), or when the original log information is "Date: Fri, June 18, 2021 08:23:10 GMT Connection:keep-alive" and the first mask is a date mask in the format of "week date month year time", that is, the target information corresponding to the first mask — the date information " Fri, June 18, 2021 08:23:10 " is shielded and uniformly replaced with "<*>", to obtain the first log information "Date:<*>GMT Connection:keep-alive", and subsequent log template extraction is based on the first log information. The foregoing shielding of the target information may also be understood as no longer reading the target information in the log template extraction process; the foregoing replacement with "<*>" is only an optional implementation, and the embodiment of the present invention is not limited to the foregoing shielding manner, as long as the shielded target information is not read in the log template extraction process.

[0128] In the embodiment of the present invention, template extraction is performed on the first log information after mask shielding, which can reduce the interference of log variables on template extraction. Compared with the simple shielding mode for numeric variables in the prior art, in the embodiment of the present invention, the shielding operation is specifically performed with a variable mask in a specific format, which can improve the effectiveness and rationality of shielding.

[0129] In a case where there are a plurality of first masks, shielding may be performed in a preset order.

[0130] Optionally, in a case where the at least two first masks are included, the shielding of irrelevant variables in the original log information according to the at least one first mask to obtain the first log information includes:

[0131] determining a shielding order of the at least two first masks according to the length and / or complexity of the at least two first masks;

[0132] shielding the irrelevant variables in the original log information according to the shielding order to obtain the first log information.

[0133] Optionally, the different first masks are not simply randomly arranged during implementation. Instead, they must be executed sequentially based on the statement length and the decreasing richness of the alphanumeric combinations. This can be understood as stacking different mask functional modules. This is because matching shorter statements before longer ones will prevent the program from recognizing the longer statements. For example, a date statement in a log might be "Date:Fri,18Jun 2021 08:23:10". The Date Mask functional module can recognize it as "Date:<*>". However, if numbers are matched first, the statement becomes "Date:Fri,<*>Jun<*><*>:<*>:<*>", in which case the Date Mask functional module will be unable to recognize the date. Therefore, the mask execution order needs to be arranged appropriately.

[0134] In this embodiment of the invention, the sorting priority can be determined based on the length and / or complexity of at least two first masks. For example, the longer the mask length, the higher the sorting priority and the earlier the masking order; the higher the mask complexity, the higher the sorting priority and the earlier the masking order. Alternatively, a comprehensive sorting can be performed using the length and complexity of the first masks, for example, by weighted summing of the mask length and complexity, and determining the order based on the sum. Alternatively, sorting can be performed separately by length and complexity, and then a comprehensive order adjustment can be made based on the two sorting results.

[0135] Optionally, determining the masking order of the at least two first masks based on their length and / or complexity includes the following:

[0136] Arrange the at least two first masks in descending order of mask length to determine the masking order of the at least two first masks; arrange the at least two first masks in descending order of mask complexity to determine the masking order of the at least two first masks.

[0137] The masking order of the at least two first masks is determined by arranging them in order of the weighted sum of mask length and complexity.

[0138] Due to the lack of standardized logging specifications, log data often appears unstructured. For example, statements with the same meaning are expressed in different ways. For instance, "send B to device A" is also expressed as "send B to device A" or "send B to device A", which leads to synonymous log information being divided into different templates.

[0139] In an embodiment of the present invention, optionally, before obtaining the first log information by masking irrelevant variables in the original log information according to the at least one first mask, the method further includes:

[0140] Similar statements in the original log information are processed for consistency to obtain second log information;

[0141] The step of masking irrelevant variables in the original log information according to the at least one first mask to obtain the first log information includes:

[0142] Based on the at least one first mask, irrelevant variables in the second log information are masked to obtain the first log information.

[0143] This involves unifying different expressions of synonymous statements. For example, "send B to device A," "send B to device A," and "send B to device A" are unified into "send B to device A." This simultaneously resolves the inconsistency in templates for statements with similar business meanings in the logs, improves the aggregation of similar templates, and results in more reasonable templates, achieving a "similarity-compatibility" effect.

[0144] Optionally, the process of performing consistency processing on similar statements in the original log information to obtain second log information includes:

[0145] If semantic analysis is performed on the original log information and a first statement and a second statement with the same semantics are obtained, the first statement will be replaced with the second statement.

[0146] Use the replaced original log information as the second log information.

[0147] The first statement and the second statement can be understood as different statements with the same semantics.

[0148] It is understandable that the above consistency processing can be obtained through semantic analysis, unifying the expression of semantically identical statements before applying masking. This consistency processing can also be understood as a form of consistency masking, such as... Figure 4 The Consistency Mask shown.

[0149] Step 302 above can be understood as, for example Figure 4 As shown, this embodiment of the invention innovatively sets up a "MASK layer" compared to existing template extraction methods, which improves existing automatic log parsing methods and enables the identification and masking of "template-irrelevant variables" (i.e., frequently occurring numbers and / or letter combinations in the log that need to be masked before template extraction) in the form of a mask (model) before template extraction.

[0150] Specifically, it includes:

[0151] 1) Before extracting the template, the general masking of "template-irrelevant variables" is performed, specifically including: Date Mask, Http Mask, Image Mask, IP Mask, File Mask. The date, HTTP address, image name, IP address, and file name are automatically identified and replaced with <*>. The above Date Mask, Http Mask, Image Mask, IP Mask, and FileMask can be understood as the first sub-mask.

[0152] The aforementioned general masking differs from existing numeric or device ID masking methods. Taking date masking as an example, when the original log information is "August 15, 2022", the existing variable masking method will result in "<*> year <*> month <*> day", which will interfere with the template length and content. However, based on the embodiments of the present invention, when the first mask including the date mask is used for masking, the date information will be replaced with <*> as a whole.

[0153] 2) Targeted masking of "template-irrelevant variables" during template extraction, specifically for network cloud device log scenarios. This includes: Instance Mask, .py Mask, and Req Mask. The instance ID, .py statement, and req ID are automatically identified and replaced with <*>. The Instance Mask, .py Mask, and Req Mask can be understood as a second sub-mask. This allows for the setting of masks based on the business semantic features of intelligent network operations and maintenance scenarios and the characteristics of logs generated by network cloud devices in actual production, thereby improving the effectiveness of network device log template extraction.

[0154] Understandably, a first mask can also be set specifically for other log extraction scenarios. Optionally, at least part of the type of the first mask is related to the type of the original log information.

[0155] 3) By adding Consistency Mask, the problem of inconsistent templates for similar business statements in the logs of network cloud devices is solved. The aforementioned Consistency Mask is used to implement the consistency processing of the aforementioned similar statements.

[0156] The masking of the template-independent variables mentioned above can be achieved by designing regular expressions for keywords of each category (different first masks) or by training a deep learning model to match and extract parameters.

[0157] By masking irrelevant variables in the original log information, the aggregation of similar templates and the distinguishability of dissimilar templates can be enhanced, achieving the effect of "similarity is compatible, dissimilarity is exclusive". For example, the specific architecture, implementation process, and effects of this invention can be found in [reference needed]. Figure 4 As shown.

[0158] The embodiments of the present invention add a MASK layer compared to the prior art, specifically including the following modules:

[0159] 1. Consistency Mask: Enables consistent processing of statements with similar business meanings;

[0160] 2. Date Mask: Enables automatic identification and replacement of dates appearing in log messages with constants;

[0161] 3. Instance Mask: Enables automatic identification and replacement of instance IDs (instance identifiers) created by virtual machines with constants;

[0162] 4..py Mask: Enables automatic identification and replacement of Python execution statements ending in .py with constants;

[0163] 5. HTTP Mask: Enables the automatic identification and replacement of HTTP addresses appearing in log messages with constants;

[0164] 6. Image Mask: Enables automatic identification and replacement of image IDs (image names) with constants;

[0165] 7. Request Mask: Enables automatic identification and replacement of Request IDs with constants;

[0166] 8. IP Mask: Enables automatic identification and replacement of IP addresses appearing in log messages with constants;

[0167] 9. File Mask: Enables automatic identification and replacement of filenames appearing in log messages with constants.

[0168] This invention, by adding a mask layer, achieves the identification and masking of "template-irrelevant variables" before template extraction, solving the problems of excessive differences in log message statements containing similar business logic and insufficient differences in log message statements from different business logics. For example, existing template extraction methods do not mask HTTP addresses; the same template may be identified as different templates due to different HTTP addresses, leading to misclassification. In another case, different templates may be classified as the same template because they share the same HTTP address, also leading to misclassification. This application, through an HTTP mask, automatically identifies and replaces HTTP addresses appearing in log messages with constants, solving the problems of excessive differences in log message statements containing similar business logic and insufficient differences in log message statements from different business logics.

[0169] Step 303: Extract the log template of the first log information.

[0170] In this embodiment of the invention, before extracting the log template, the original log is first masked to obtain the processed first log information. The template is then extracted using the processed first log information. Compared with the prior art, which directly extracts templates from the original logs, this method can extract templates after reasonably masking irrelevant variables, thereby improving the accuracy of template extraction.

[0171] The specific methods for extracting log templates can refer to existing extraction methods, such as further employing steps like number matching, length matching, and keyword matching.

[0172] The aforementioned MASK layer is embedded in the underlying log parsing algorithm and can be implemented using programming languages ​​such as Python, MATLAB, and Java. This allows it to automatically complete all the mask function modules, and the results will be automatically input into the number matching layer.

[0173] The extracted log templates can be further used for log analysis. The specific flowchart for template extraction and application is shown in Figure 5. Steps 301-303 of this embodiment mainly address problems in the log parsing process. The log templates obtained through log parsing can be further used for log mining through feature extraction.

[0174] Specifically, in practical applications of network intelligent operation and maintenance scenarios based on log analysis, it typically includes, for example: Figure 5 The three main modules shown are: log parsing, feature extraction, and log mining. The specific functions of each module are as follows:

[0175] 1. Log parsing module: This module automatically extracts event templates and parses raw, unstructured log messages into a structured form with event templates (constants) and specific parameters (variables).

[0176] 2. Feature Extraction Module: This module further encodes structured log data into numerical feature vectors, enabling the application of AI models.

[0177] 3. Log Mining Module: This module trains an AI model using feature sequences to enable log anomaly detection and root cause localization.

[0178] Optionally, the method further includes:

[0179] Using the log template and the log variables, a structured log is obtained;

[0180] Feature extraction is performed on the structured logs to obtain log feature vectors;

[0181] Based on the log feature vector, perform anomaly detection and root cause localization of the original log information.

[0182] In network intelligent operation and maintenance scenarios based on log analysis, the functions of anomaly detection and root cause localization are mainly implemented by the last module: the log mining module. Its effectiveness depends significantly on the effectiveness of log parsing. The embodiments of the present invention are as an automatic log parsing optimization method, which improves the existing automatic log parsing technology. Based on this solution, a more reasonable template can be obtained, thereby improving the accuracy of subsequent anomaly detection and root cause localization.

[0183] Identifying and masking "template-irrelevant variables" before template extraction can improve the rationality of log template extraction, thereby improving the accuracy of subsequent anomaly detection and root cause localization.

[0184] Experimentally, raw log text generated by a distributed system during its runtime over the past 6 days (nearly 480,000 log messages) was collected from the massive logs generated by the network cloudification devices in the live network system. The existing solution (before optimization) and the solution proposed in this embodiment of the invention (after optimization) were tested. The results showed that the number of templates obtained after log parsing using this solution was reduced by 10%. It can be seen that this solution increases the aggregation of similar templates, while the distinction between different templates is greater. As shown in Table 1, the left side of the table shows the extraction results of the existing template extraction algorithm, and the right side of the table shows the extraction results of the template extraction algorithm of this embodiment of the invention.

[0185] Table 1

[0186]

[0187] The device log template extraction method in this embodiment obtains the original log information of the device and at least one pre-set first mask, where the first mask is a model of log variables unrelated to the device log template; based on the at least one first mask, log variables unrelated to the device log template in the original log information are masked to obtain first log information; and the log template of the first log information is extracted. In this embodiment, by setting a model of log variables unrelated to the device log template, log variables unrelated to the device log template in the original log information are masked, i.e., before template extraction, the "template-irrelevant variables" of the original log information are efficiently identified and masked using a mask, thereby improving the rationality of log template extraction.

[0188] See Figure 6 , Figure 6 This is a schematic diagram of the structure of a device log template extraction device provided in an embodiment of the present invention, as shown below. Figure 6 As shown, the device log template extraction device 600 includes:

[0189] The acquisition module 601 is used to acquire the device's original log information and at least one pre-set first mask, wherein the first mask is a model of log variables that is independent of the device log template;

[0190] The masking module 602 is used to mask log variables in the original log information that are unrelated to the device log template according to the at least one first mask, so as to obtain the first log information;

[0191] Extraction module 603 is used to extract the log template corresponding to the first log information.

[0192] It should be noted that the source device of the original log information and the execution device in the embodiments of the present invention can be different devices or the same device.

[0193] Optionally, the first mask includes at least one of the following:

[0194] Models for general log variables and network log variables.

[0195] Optionally, the first mask includes at least one of the following:

[0196] First submask and second submask;

[0197] The first sub-mask is a model of general log variables that are independent of the device log template, and the second sub-mask is a model of network log variables that are independent of the device log template.

[0198] Optionally, the general log variables include at least one of the following: date, HTTP address, image name, IP address, and file name;

[0199] The network log variables include at least one of the following: instance identifier, Python execution statement, and request identifier.

[0200] Optionally, when the first mask comprises at least two, the shielding module 602 includes:

[0201] The sequence determination module is used to determine the masking order of the at least two first masks based on the length and / or complexity of the at least two first masks;

[0202] The first masking submodule is used to mask irrelevant variables in the original log information according to the masking order to obtain the first log information.

[0203] Optionally, the sequence determination module includes one of the following:

[0204] The first sequence determination submodule is used to arrange the at least two first masks in descending order of mask length to determine the shielding order of the at least two first masks;

[0205] The second sequence determination submodule is used to arrange the at least two first masks in descending order of mask complexity and determine the masking order of the at least two first masks.

[0206] The third sequence determination submodule is used to arrange the at least two first masks in order of the weighted sum of mask length and complexity, and to determine the masking order of the at least two first masks.

[0207] Optionally, the device further includes:

[0208] The processing module is used to perform consistency processing on similar statements in the original log information to obtain second log information;

[0209] The shielding module 602 includes:

[0210] The second masking submodule masks irrelevant variables in the second log information according to the at least one first mask, thereby obtaining the first log information.

[0211] Optionally, the processing module includes:

[0212] The replacement module is used to replace the first statement with the second statement when semantic analysis of the original log information yields a first statement and a second statement with the same semantics.

[0213] Use the replaced original log information as the second log information.

[0214] Optionally, the device further includes:

[0215] The detection module is used to obtain structured logs using the log template and the log variables;

[0216] Feature extraction is performed on the structured logs to obtain log feature vectors;

[0217] Based on the log feature vector, perform anomaly detection and root cause localization of the original log information.

[0218] It should be noted that the apparatus provided in the embodiments of the present invention is an apparatus capable of executing the above-described device log template extraction method. Therefore, all implementation methods in the embodiments of the above-described device log template extraction method are applicable to the electronic device and can achieve the same or similar beneficial effects.

[0219] For details, see Figure 7 As shown, this embodiment of the invention also provides an electronic device, including a bus 701, a transceiver 702, an antenna 703, a bus interface 704, a processor 705, and a memory 706.

[0220] The processor 705 is used for:

[0221] Obtain the device's raw log information and at least one pre-set first mask, wherein the first mask is a model of log variables that is independent of the device log template;

[0222] Based on the at least one first mask, log variables in the original log information that are unrelated to the device log template are masked to obtain the first log information;

[0223] Extract the log template corresponding to the first log information.

[0224] Optionally, the first mask includes at least one of the following:

[0225] Models for general log variables and network log variables.

[0226] Optionally, the first mask includes at least one of the following:

[0227] First submask and second submask;

[0228] The first sub-mask is a model of general log variables that are independent of the device log template, and the second sub-mask is a model of network log variables that are independent of the device log template.

[0229] Optionally, the general log variables include at least one of the following: date, HTTP address, image name, IP address, and file name;

[0230] The network log variables include at least one of the following: instance identifier, Python execution statement, and request identifier.

[0231] Optionally, when the first mask includes at least two, irrelevant variables in the original log information are masked according to the at least one first mask to obtain the first log information, including:

[0232] The masking order of the at least two first masks is determined based on the length and / or complexity of the at least two first masks;

[0233] Irrelevant variables in the original log information are masked according to the masking order to obtain the first log information.

[0234] Optionally, determining the masking order of the at least two first masks based on their length and / or complexity includes the following:

[0235] Arrange the at least two first masks in descending order of mask length to determine the masking order of the at least two first masks; arrange the at least two first masks in descending order of mask complexity to determine the masking order of the at least two first masks.

[0236] The masking order of the at least two first masks is determined by arranging them in order of the weighted sum of mask length and complexity.

[0237] Optionally, before obtaining the first log information by masking irrelevant variables in the original log information according to the at least one first mask, the processor 705 is further configured to:

[0238] Similar statements in the original log information are processed for consistency to obtain second log information;

[0239] The step of masking irrelevant variables in the original log information according to the at least one first mask to obtain the first log information includes:

[0240] Based on the at least one first mask, irrelevant variables in the second log information are masked to obtain the first log information.

[0241] Optionally, the process of performing consistency processing on similar statements in the original log information to obtain second log information includes:

[0242] If semantic analysis is performed on the original log information and a first statement and a second statement with the same semantics are obtained, the first statement will be replaced with the second statement.

[0243] Use the replaced original log information as the second log information.

[0244] Optionally, the processor 705 is further configured to:

[0245] Using the log template and the log variables, a structured log is obtained;

[0246] Feature extraction is performed on the structured logs to obtain log feature vectors;

[0247] Based on the log feature vector, perform anomaly detection and root cause localization of the original log information.

[0248] exist Figure 7 In this document, a bus architecture (represented by bus 701) is used. Bus 701 can include any number of interconnected buses and bridges, linking various circuits including one or more processors represented by processor 705 and memory represented by memory 706. Bus 701 can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. Bus interface 704 provides an interface between bus 701 and transceiver 702. Transceiver 702 can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by processor 705 is transmitted over a wireless medium via antenna 703, which further receives data and transmits data to processor 705.

[0249] Processor 705 manages bus 701 and general processing, and also provides various functions, including timing, peripheral interface, voltage regulation, power management, and other control functions. Memory 706 can be used to store data used by processor 705 during operation.

[0250] Optionally, the processor 705 can be a CPU, ASIC, FPGA, or CPLD.

[0251] It should be noted that the electronic device provided in the embodiments of the present invention is a device capable of executing the above-described device log template extraction method. Therefore, all implementation methods in the embodiments of the above-described device log template extraction method are applicable to the electronic device and can achieve the same or similar beneficial effects.

[0252] This invention also provides an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor. When the program is executed by the processor, it implements the various processes of the above-described device log template extraction method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0253] This invention also provides a computer-readable storage medium storing a computer program. When executed by a processor, this computer program implements the various processes of the above-described device log template extraction method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0254] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0255] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or device, etc.) to execute the methods described in the various embodiments of the present invention.

[0256] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.

Claims

1. A method for extracting device log templates, characterized in that, The method includes: Obtain the device's raw log information and at least one pre-set first mask, wherein the first mask is a model of log variables that is independent of the device log template; Based on the at least one first mask, log variables in the original log information that are unrelated to the device log template are masked to obtain the first log information; Extract the log template corresponding to the first log information; When the first mask includes at least two, the step of masking irrelevant variables in the original log information according to the at least one first mask to obtain the first log information includes: The masking order of the at least two first masks is determined based on the length and / or complexity of the at least two first masks; Irrelevant variables in the original log information are masked according to the masking order to obtain the first log information.

2. The method according to claim 1, characterized in that, The first mask includes at least one of the following: Models for general log variables and network log variables.

3. The method according to claim 1, characterized in that, Determining the masking order of the at least two first masks based on their length and / or complexity includes the following: Arrange the at least two first masks in descending order of mask length to determine the masking order of the at least two first masks; arrange the at least two first masks in descending order of mask complexity to determine the masking order of the at least two first masks. The masking order of the at least two first masks is determined by arranging them in order of the weighted sum of mask length and complexity.

4. The method according to claim 1, characterized in that, Before obtaining the first log information by masking irrelevant variables in the original log information according to the at least one first mask, the method further includes: Similar statements in the original log information are processed for consistency to obtain second log information; The step of masking irrelevant variables in the original log information according to the at least one first mask to obtain the first log information includes: Based on the at least one first mask, irrelevant variables in the second log information are masked to obtain the first log information.

5. The method according to claim 4, characterized in that, The process of performing consistency processing on similar statements in the original log information to obtain second log information includes: If semantic analysis is performed on the original log information and a first statement and a second statement with the same semantics are obtained, the first statement will be replaced with the second statement. Use the replaced original log information as the second log information.

6. The method according to claim 1, characterized in that, The method further includes: Using the log template and the log variables, a structured log is obtained; Feature extraction is performed on the structured logs to obtain log feature vectors; Based on the log feature vector, perform anomaly detection and root cause localization of the original log information.

7. A device for extracting equipment log templates, characterized in that, include: The acquisition module is used to acquire the device's raw log information and at least one pre-set first mask, wherein the first mask is a model of log variables that is independent of the device log template; A masking module is used to mask log variables in the original log information that are unrelated to the device log template according to the at least one first mask, so as to obtain the first log information; The extraction module is used to extract the log template corresponding to the first log information; When the first mask comprises at least two, the shielding module includes: The sequence determination module is used to determine the masking order of the at least two first masks based on the length and / or complexity of the at least two first masks; The first masking submodule is used to mask irrelevant variables in the original log information according to the masking order to obtain the first log information.

8. An electronic device, characterized in that, Includes a processor, said processor being used for: Obtain the device's raw log information and at least one pre-set first mask, wherein the first mask is a model of log variables that is independent of the device log template; Based on the at least one first mask, log variables in the original log information that are unrelated to the device log template are masked to obtain the first log information; Extract the log template corresponding to the first log information; When the first mask includes at least two, the step of masking irrelevant variables in the original log information according to the at least one first mask to obtain the first log information includes: The masking order of the at least two first masks is determined based on the length and / or complexity of the at least two first masks; Irrelevant variables in the original log information are masked according to the masking order to obtain the first log information.

9. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the device log template extraction method as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the device log template extraction method as described in any one of claims 1 to 6.

Images