Log processing method, device, equipment and storage medium in data center network

Abstract

The disclosed embodiments disclose a log processing method, apparatus, device, and storage medium within a data center network. The method comprises: obtaining multiple original logs; preprocessing the original logs to obtain valid content of the original logs, wherein the valid content includes relevant content used to determine log analysis results; filtering out domain keywords related to the domain of the original log from the valid content to obtain a backup log; and extracting log templates from the backup logs to obtain log templates corresponding to the backup logs. This technical solution can efficiently and comprehensively extract high-quality log templates.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a log processing method, apparatus, device, and storage medium in a data center network. Background Art

[0002] System Event Log (SEL) is a type of data that is widely present in various server systems. It contains various event logs generated by the server system baseboard manager, heat exchanger, etc. Warnings or potential major problems can be discovered by analyzing the SEL information. However, the scale of logs generated by the server system is huge, and the original logs are often unstructured data. Whether it is manual analysis or analysis through analytical models, structured data is required. Therefore, how to structure the original logs, that is, to extract log templates through log parsing technology, and to extract log templates that are conducive to analysis is particularly important. At present, the ability to effectively template the server system event log mainly depends on the experience of experts. Due to limited manpower, the expert mining and analysis of log templates is not only inefficient but also has problems such as incomplete template library and insufficient template expressiveness. As a result, the manually mined log templates cannot accurately judge the status of the server or server components. Summary of the Invention

[0003] In order to solve the problems in the related art, the embodiments of the present disclosure provide a log processing method, apparatus, device and storage medium in a data center network.

[0004] In a first aspect, an embodiment of the present disclosure provides a log processing method within a data center network.

[0005] Specifically, the log processing method includes:

[0006] Obtain multiple original logs within the data center network;

[0007] Preprocessing the original log to obtain valid content of the original log, wherein the valid content includes relevant content for determining a log analysis result;

[0008] Filtering out domain keywords of the domain in which the original log is located in the valid content to obtain a backup log;

[0009] A log template is extracted from the backup log to obtain a log template corresponding to the backup log.

[0010] In a second aspect, an embodiment of the present disclosure provides a log processing method.

[0011] Specifically, the log processing method includes:

[0012] Get multiple original logs;

[0013] Preprocessing the original log to obtain valid content of the original log, wherein the valid content includes relevant content for determining a log analysis result;

[0014] Filtering out domain keywords of the domain in which the original log is located in the valid content to obtain a backup log;

[0015] A log template is extracted from the backup log to obtain a log template corresponding to the backup log.

[0016] In a possible implementation, obtaining multiple original logs includes:

[0017] Get the original log whose length does not exceed the preset length.

[0018] In a possible implementation, preprocessing the original log to obtain valid content of the original log includes:

[0019] Invalid content in the original log is filtered out by using a preset regular expression library to obtain valid content of the original log, where the invalid content includes content irrelevant to the log analysis result.

[0020] In one possible implementation, the method further includes:

[0021] The format of the valid content is converted according to the preset conversion rules to obtain valid content that meets the format requirements.

[0022] In a possible implementation, converting the format of the valid content according to a preset conversion rule to obtain valid content that meets the format requirements includes:

[0023] Converting the writing format of letters in the valid content into a preset format;

[0024] and / or,

[0025] According to a preset word correspondence, the domain words in the valid content are replaced with preset words corresponding to the domain words. The word correspondence stores the domain words of the domain where the original log is located and the corresponding preset words.

[0026] In a possible implementation, extracting a log template from the backup log to obtain a log template corresponding to the backup log includes:

[0027] Matching the backup log with the extracted log template;

[0028] In response to the backup log failing to match the extracted log template, performing log template extraction on the backup log to generate a log template corresponding to the backup log;

[0029] In response to the backup log matching the extracted log template, the log template of the backup log is determined to be the extracted log template matching the backup log.

[0030] In one possible implementation, the method further includes:

[0031] Obtaining a device status tag corresponding to the original log;

[0032] Based on the device status label, the log template and the number of backup logs corresponding to the log template, an importance evaluation is performed on the log template to obtain the importance of the log template in determining the device status;

[0033] The log template whose importance exceeds the first preset threshold is recorded as an important log template.

[0034] In one possible implementation, the method further includes:

[0035] Obtaining a device status tag corresponding to the original log;

[0036] Based on the device status tag and the backup log corresponding to the log template, a log template feature is obtained, where the log template feature includes at least one of the following features: a time series statistical feature, an association rule of the log template, and a frequent sequence pattern of the log template;

[0037] The acquiring of the log template feature based on the device status tag and the backup log corresponding to the log template includes:

[0038] Counting multiple time series statistical features of the domain keywords of the log template, where the domain keywords of the log template include domain keywords corresponding to the backup log matching the log template, and the time series statistical features include statistical features of time series data of the domain keywords of the log template for the same device within a preset time period;

[0039] Using an association rule mining algorithm, based on the device status tag and the backup log corresponding to the log template, performing association rule mining on the log template to obtain multiple log template association rules;

[0040] A frequent sequence pattern mining algorithm is used to mine the log template based on the device status label and the backup log corresponding to the log template, so as to obtain a plurality of frequent sequence patterns of the log template.

[0041] In one possible implementation, the method further includes:

[0042] Based on the device status label, importance evaluation is performed on multiple time series statistical features, multiple log template association rules, and multiple log template frequent sequence patterns of the log template respectively, to obtain the importance of the multiple time series statistical features of the log template, the importance of the multiple log template association rules, or the importance of the multiple log template frequent sequence patterns in determining the device status;

[0043] Determining, among the multiple time series statistical features of the log template, a time series statistical feature whose importance exceeds a second preset threshold as an important time series feature of the log template;

[0044] Determining a log template association rule whose importance exceeds a third preset threshold among the multiple log template association rules as an important log template association rule;

[0045] The log template frequent sequence pattern whose importance exceeds a fourth preset threshold among the multiple log template frequent sequence patterns is determined as an important log template frequent sequence pattern.

[0046] In a third aspect, an embodiment of the present disclosure provides a log processing device.

[0047] Specifically, the log processing device includes:

[0048] A first acquisition module is configured to acquire multiple original logs;

[0049] a preprocessing module configured to preprocess the original log to obtain valid content of the original log, wherein the valid content includes relevant content for determining a log analysis result;

[0050] A filtering module configured to filter out domain keywords of the domain where the original log is located in the valid content to obtain a backup log;

[0051] The extraction module is configured to extract the log template from the backup log to obtain the log template corresponding to the backup log.

[0052] In a possible implementation, the first acquisition module is configured to:

[0053] Get the original log whose length does not exceed the preset length.

[0054] In a possible implementation, preprocessing the original log to obtain valid content of the original log includes:

[0055] Invalid content in the original log is filtered out by using a preset regular expression library to obtain valid content of the original log, where the invalid content includes content irrelevant to the log analysis result.

[0056] In a possible implementation, the device further includes:

[0057] The conversion module is configured to perform format conversion on the valid content according to a preset conversion rule to obtain valid content that meets the format requirements.

[0058] In a possible implementation, the conversion module is configured to:

[0059] Converting the writing format of letters in the valid content into a preset format;

[0060] and / or,

[0061] According to a preset word correspondence, the domain words in the valid content are replaced with preset words corresponding to the domain words. The word correspondence stores the domain words of the domain where the original log is located and the corresponding preset words.

[0062] In a possible implementation, the extraction module is configured to:

[0063] Matching the backup log with the extracted log template;

[0064] In response to the backup log failing to match the extracted log template, performing log template extraction on the backup log to generate a log template corresponding to the backup log;

[0065] In response to the backup log matching the extracted log template, the log template of the backup log is determined to be the extracted log template matching the backup log.

[0066] In a possible implementation, the device further includes:

[0067] A second acquisition module is configured to obtain a device status tag corresponding to the original log;

[0068] a first evaluation module configured to perform an importance evaluation on the log template based on the device status tag, the log template and the number of backup logs corresponding thereto, and obtain the importance of the log template in determining the device status;

[0069] The recording module is configured to record the log template whose importance exceeds a first preset threshold as an important log template.

[0070] In a possible implementation, the device further includes:

[0071] a third acquisition module, configured to acquire a device status tag corresponding to the original log;

[0072] a fourth acquisition module configured to acquire a log template feature based on the device status tag and the backup log corresponding to the log template, the log template feature comprising at least one of the following features: a time series statistical feature, an association rule of the log template, and a frequent sequence pattern of the log template;

[0073] The acquiring of the log template feature based on the device status tag and the backup log corresponding to the log template includes:

[0074] Counting multiple time series statistical features of the domain keywords of the log template, where the domain keywords of the log template include domain keywords corresponding to the backup log matching the log template, and the time series statistical features include statistical features of time series data of the domain keywords of the log template for the same device within a preset time period;

[0075] Using an association rule mining algorithm, based on the device status tag and the backup log corresponding to the log template, performing association rule mining on the log template to obtain multiple log template association rules;

[0076] A frequent sequence pattern mining algorithm is used to mine the log template based on the device status label and the backup log corresponding to the log template, so as to obtain a plurality of frequent sequence patterns of the log template.

[0077] In a possible implementation, the device further includes:

[0078] a second evaluation module configured to respectively evaluate the importance of the multiple time series statistical features, the multiple log template association rules, and the multiple log template frequent sequence patterns of the log template based on the device status label, and obtain the importance of the multiple time series statistical features, the multiple log template association rules, or the multiple log template frequent sequence patterns in determining the device status;

[0079] The determination module is configured to determine the time series statistical features of the log template whose importance exceeds the second preset threshold among the multiple time series statistical features of the log template as the important time series features of the log template; determine the log template association rules of the multiple log template association rules whose importance exceeds the third preset threshold as the important log template association rules; and determine the log template frequent sequence patterns of the multiple log template frequent sequence patterns whose importance exceeds the fourth preset threshold as the important log template frequent sequence patterns.

[0080] In a fourth aspect, an embodiment of the present disclosure provides an electronic device comprising a memory and a processor, wherein the memory is used to store one or more computer instructions, and wherein the one or more computer instructions are executed by the processor to implement a method as described in any one of the first aspects.

[0081] In a fifth aspect, an embodiment of the present disclosure provides a computer-readable storage medium on which computer instructions are stored. When the computer instructions are executed by a processor, the method as described in any one of the first aspects is implemented.

[0082] In a sixth aspect, an embodiment of the present disclosure provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the method steps described in any one of the first aspects.

[0083] According to the technical solution provided by the embodiment of the present disclosure, unimportant content in the original log can be filtered out to obtain the effective content of the original log; domain keywords of the domain where the original log is located in the effective content can be filtered out to obtain a backup log; log template extraction is performed on the backup log to obtain a log template corresponding to the backup log. In this way, log templates are automatically extracted from the effective content of the original log, and high-quality log templates can be extracted efficiently and comprehensively. In addition, by filtering out domain keywords and then performing log template extraction, the number of templates can be greatly reduced.

[0084] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Other features, objectives and advantages of the present disclosure will become more apparent through the following detailed description of non-limiting embodiments in conjunction with the accompanying drawings. In the accompanying drawings:

[0086] Figure 1 A flowchart of a log processing method according to an embodiment of the present disclosure is shown.

[0087] Figure 2 A schematic diagram illustrating the flow structure of a log processing process according to an embodiment of the present disclosure is shown.

[0088] Figure 3 A structural block diagram of a log processing device according to an embodiment of the present disclosure is shown.

[0089] Figure 4 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0090] Figure 5 A schematic diagram showing the structure of a computer system suitable for implementing the method according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0091] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement them. In addition, for the sake of clarity, parts not related to the description of the exemplary embodiments are omitted in the accompanying drawings.

[0092] In the present disclosure, it should be understood that terms such as "include" or "have" are intended to indicate the presence of features, numbers, steps, actions, components, parts, or combinations thereof disclosed in the present specification, and are not intended to exclude the possibility that one or more other features, numbers, steps, actions, components, parts, or combinations thereof exist or are added.

[0093] It should also be noted that, in the absence of conflict, the embodiments and features of the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0094] In this disclosure, the acquisition of user information or user data is an operation authorized and confirmed by the user, or actively selected by the user.

[0095] As mentioned above, the System Event Log (SEL) is a type of data that is widely present in various server systems. It includes various event logs generated by the server system baseboard manager, heat exchanger, etc. Warnings or potential major problems can be discovered by analyzing the SEL information. However, the scale of logs generated by the server system is huge, and the original logs are often unstructured data. Whether it is manual analysis or analysis through analytical models, structured data is required. Therefore, how to structure the original logs, that is, to extract log templates through log parsing technology, and to extract log templates that are conducive to analysis is particularly important. At present, the ability to effectively template the server system event log mainly depends on the experience of experts. Due to limited manpower, the expert mining and analysis of log templates is not only inefficient but also has problems such as incomplete template libraries and insufficient template expressiveness. As a result, the manually mined log templates cannot accurately judge the status of the server or server components.

[0096] The present disclosure provides a log processing method, which can filter out unimportant content in the original log to obtain the effective content of the original log; filter out domain keywords of the domain where the original log is located in the effective content to obtain a backup log; extract log templates from the backup log to obtain log templates corresponding to the backup log. In this way, log templates are automatically extracted from the effective content of the original log, and high-quality log templates can be extracted efficiently and comprehensively. In addition, by filtering out domain keywords and then extracting log templates, the number of templates can be greatly reduced.

[0097] Figure 1 1 shows a flow chart of a log processing method according to an embodiment of the present disclosure. Figure 1 As shown, the log processing method includes the following steps S101-S104:

[0098] In step S101, multiple original logs are obtained;

[0099] In step S102, the original log is pre-processed to obtain the valid content of the original log;

[0100] In step S103, the domain keywords of the domain of the original log in the valid content are filtered out to obtain a backup log;

[0101] In step S104, a log template is extracted from the backup log to obtain a log template corresponding to the backup log.

[0102] In one possible implementation, the log processing method may be applicable to computers, computing devices, electronic devices, servers, service clusters, and the like that can perform log processing.

[0103] In one possible implementation, the original log refers to the original information of various events generated by the device during operation. For example, the original log may be a server system event log, which is used to record various events generated by the server system's baseboard management controller (BMC), heat exchanger controller (HSC), and BOIS (Basic Input Output System). The system event log is recorded in a buffer on the server. The system event log is used to analyze the cause of server anomalies or failures. Moreover, the buffer is independent of the OS (Operating System). Even if the OS is unavailable, the SEL log can still be read through an out-of-band system.

[0104] In one possible implementation, the log content recorded in the original log is often complex and lengthy, and usually includes unimportant content such as the time when the event occurred, IP address, long characters, garbled characters, etc. Therefore, in order to extract a more effective log template, the original log can be preprocessed to filter out these unimportant contents and retain the valid content in the original log. The valid content includes relevant content used to determine the log analysis results. These valid contents will have a certain impact on the log analysis results, mainly including the events that occurred and information related to the events. For example, if a fault event occurs, the valid content can be the fault type, fault location, fault level, functional information affected by the fault, etc.

[0105] In one possible implementation, original logs of different fields record the domain keywords of the fields. In the same field and the same type of original logs, except for the domain keywords, the rest of the data structure is the same. For example, for server system event logs, the domain keywords include asserted and deasserted. Due to the special structure of system event logs, the difference between original logs of the same type only exists in asserted and deasserted. Removing these two words can significantly reduce the number of log templates.

[0106] For example, the original log 1 in the server system event log is: 1e5|08 / 22 / 2021|03:14:51|Drive Slot #0x15|Drive Fault|Asserted;

[0107] Original log 2 is: 1e6|08 / 22 / 2021|03:14:53|Drive Slot#0x15|Drive Fault|Deasserted;

[0108] Among them, the unimportant content in the original log is irrelevant information in the time field, such as the unimportant content in the original log 1 is "1e5|08 / 22 / 2021|03:14:51|", and the unimportant content in the original log 2 is "1e6|08 / 22 / 2021|03:14:53|"; preprocessing the original log 1 and the original log 2, the valid content of the original log 1 is obtained as "Drive Slot #0x15|Drive Fault |Asserted", and the valid content of the original log 2 is "Drive Slot #0x15|Drive Fault|Deasserted". It can be seen that the two original logs have the same data except for the keywords "Asserted" and "Deasserted" in the field. After filtering out "Asserted" in the valid content of the original log 1 and "Deasserted" in the valid content of the original log 2, the backup log 1 corresponding to the original log 1 is "Drive Slot #0x15|Drive Fault", and the backup log 2 corresponding to the original log 2 is "Drive Slot #0x15|Drive Fault", a log template "Drive Slot*|Drive Fault|*" can be extracted from backup log 1 and backup log 2. If the domain keywords "Asserted" and "Deasserted" are not filtered out, a log template "Drive Slot*|Drive Fault|*Asserted|*" can be extracted from backup log 1, and another log template "Drive Slot*|Drive Fault|*Deasserted|*" can be extracted from backup log 2. This shows that if domain keywords are not filtered out, the number of extracted log templates will increase exponentially with the increase in original log types.

[0109] In this embodiment, in scenarios where the original log needs to be analyzed, such as analyzing the original log to determine the device status or fault location, the log template can be used as a template feature, and the corresponding domain keyword can be extracted as another feature. The two features are combined to determine the device status or fault location.

[0110] This implementation can filter out unimportant content in the original log to obtain the valid content of the original log; filter out domain keywords of the domain where the original log is located in the valid content to obtain a backup log; perform log template extraction on the backup log to obtain a log template corresponding to the backup log. In this way, log templates are automatically extracted from the valid content of the original log, and high-quality log templates can be extracted efficiently and comprehensively. In addition, filtering out domain keywords before performing log template extraction can greatly reduce the number of templates.

[0111] In a possible implementation, step S101 in the above log processing method may further include the following steps:

[0112] Get the original log whose length does not exceed the preset length.

[0113] In this embodiment, the original logs with a longer length exceeding the preset length can be filtered out, and the original logs with a length not exceeding the preset length can be obtained. The original logs with a longer length are filtered out, which can improve the efficiency of generating log templates.

[0114] In a possible implementation, step S102 in the log processing method, i.e., preprocessing the original log to obtain valid content of the original log, includes:

[0115] Invalid content in the original log is filtered out using a preset regular expression library to obtain valid content of the original log.

[0116] In this embodiment, the invalid content includes content that is irrelevant to the log analysis result and has no effect on the log analysis result, such as IP, time, long numbers, garbled characters, etc.

[0117] In this implementation, a regular expression (RE), also known as a regular expression, is a logical formula that operates on character strings (including common characters (e.g., letters from a to z) and special characters (called "metacharacters"). It uses predefined specific characters and combinations of these specific characters to form a "regular string" that is used to express a filtering logic for character strings. A regular expression is a text pattern that describes one or more character strings to be matched when searching text, and can be used to filter out the matching one or more character strings.

[0118] In this embodiment, the preset regular expression library stores at least one regular expression, each of which is used to filter one or more invalid contents in the original log, such as IP address, time, long numbers, garbled characters, etc. These regular expressions can be used to retrieve and filter out the invalid contents in the original log to obtain the valid contents of the original log.

[0119] In a possible implementation, the log processing method may further include the following steps:

[0120] According to preset rules, the format of the valid content is converted to obtain valid content that meets the format requirements.

[0121] In this embodiment, in order to extract the log template more accurately, the valid content of each original log can be converted into valid content in a unified format according to a preset conversion rule.

[0122] In one possible implementation, the log processing method converts the format of the valid content according to a preset conversion rule to obtain valid content that meets the format requirements, including:

[0123] Converting the writing format of letters in the valid content into a preset format;

[0124] and / or,

[0125] According to a preset word correspondence, the relevant words in the valid content are replaced with preset words corresponding to the relevant words, and the word correspondence stores the domain words of the domain where the original log is located and the corresponding preset words.

[0126] In this embodiment, the preset conversion rule may be a conversion rule for the writing format of letters, for example, all letters in the valid content may be converted into uppercase letters or lowercase letters, or the first letter may be uppercase and the rest of the letters may be lowercase, etc., various preset formats.

[0127] In this embodiment, the preset conversion rule can be a word replacement rule, and domain words with the same semantics can be set to correspond to unified preset words based on abbreviations, synonyms, antonyms, possible similar phrases, etc., so as to establish a word correspondence relationship. In this way, domain words with the same semantics can be replaced with unified preset words according to the preset word correspondence relationship.

[0128] In this embodiment, the storage form of the preset word correspondence includes but is not limited to a dictionary form, a database, a tree structure, and the like.

[0129] In a possible implementation, the log processing method described above extracting a log template from the backup log to obtain a log template corresponding to the backup log includes:

[0130] Matching the backup log with the extracted log template;

[0131] In response to the backup log failing to match the extracted log template, performing log template extraction on the backup log to generate a log template corresponding to the backup log;

[0132] In response to the backup log matching the extracted log template, the log template of the backup log is determined to be the extracted log template matching the backup log.

[0133] In this implementation, for the first backup log, since there is no extracted log template at this time, log template extraction can be performed directly on the first backup log to generate log template 1 corresponding to the first backup log. For the second backup log, the second backup log can be matched with the extracted log template 1. If there is no match, log template extraction is performed on the second backup log to generate log template 2 corresponding to the second backup log. If there is a match, the log template corresponding to the second backup log is log template 1. The corresponding log templates can be generated for each subsequent backup log in the same manner. In this way, multiple log templates can be generated based on the backup logs corresponding to these original logs.

[0134] For example, using original log 1 and original log 2 as examples, the backup log 1 corresponding to original log 1 is "Drive Slot#0x15|Drive Fault", and the extracted log template 1 is "Drive Slot*|Drive Fault|*". The backup log 2 corresponding to original log 2 is "Drive Slot#0x15|Drive Fault", which matches the extracted log template 1. Therefore, the log template of original log 2 is log template 1. Log template 1 is extracted based on original logs 1 and 2. Assuming that the original log 3 is "33fd|08 / 22 / 2021|15:07:13|System ACPI Power StateACPI_PWR_Status|S0 / G0:working|Asserted", by filtering out the unimportant content "33fd|08 / 22 / 2021|15:07:13|" and the field keyword "Asserted", we can obtain the backup log 3 "System ACPI Power State ACPI_PWR_Status|S0 / G0:working", which does not match the extracted log template 1. Log template extraction is performed on the backup log 3 to generate the log template "System ACPI Power State ACPI_PWR_Status" corresponding to the backup log 3.

[0135] It's important to note that in the example raw logs 1, 2, and 3 above, in addition to the extracted log template and filtered domain keywords, the valid content of these raw logs also contains "#0x15" in raw logs 1 and 2, and "S0 / G0:working" in raw log 3. These variables vary within raw logs of the same type and can be extracted and recorded separately. These state variables can be used in subsequent training of the analysis model, combined with the extracted log template and domain keywords.

[0136] In a possible implementation, in the above log processing method, the method further includes:

[0137] Obtaining a device status tag corresponding to the original log;

[0138] Based on the device status tag and the log template, performing an importance evaluation on the log template to obtain the importance of the log template in determining the device status;

[0139] Log templates whose importance exceeds a first preset threshold are recorded in an important log template library.

[0140] In this embodiment, the device status tag of the original log refers to the device status when the original log is generated. The device status can be a downtime state, a fault state, a normal state, or the like.

[0141] In this embodiment, a first importance evaluation algorithm can be used to calculate the importance of the log template when determining the device status. The first importance evaluation algorithm can be an importance evaluation algorithm based on statistical features, a supervised importance evaluation algorithm based on a model, an importance evaluation algorithm based on significance analysis, an important template extraction algorithm based on a word graph model, or an unsupervised feature selection algorithm.

[0142] In this embodiment, the important template extraction algorithm based on statistical features refers to determining the importance of the log template to the device status based on the statistical features of the log template. It can be based on the log template and its corresponding number of spare logs, and statistically analyze various statistical features such as part of speech, term frequency (TF), inverse document frequency (IDF), TF-IDF value, word position information, mutual information, etc. in the log template when the device status label is normal state and fault state. The importance of the log template to the device status is determined based on these statistical features. For example, if the term frequency of log template A with the device status label as normal state is not much different from the term frequency of log template A with the device status label as fault state, it means that the log template A has little influence on determining the device status and has low importance. If the term frequency of log template A with the device status label as normal state is significantly different from the term frequency of log template A with the device status label as fault state, it means that the log template A has a greater influence on determining the device status and has high importance. In this way, the importance of each log template in determining the device status can be determined.

[0143] In this embodiment, the model-based supervised importance assessment algorithm refers to the use of model training to obtain the importance of each log template. The input of the model is the log template and its corresponding number of backup logs, the device status label of the backup log corresponding to each original log, and the output is the importance of each log template. For example, the model can be a tree model, a linear model, or a neural network model, and there is no limitation here.

[0144] In this embodiment, the importance evaluation algorithm based on significance analysis refers to analyzing the significance of each log template, where the significance is the importance of the log template. The importance evaluation algorithm based on significance analysis may include but is not limited to the KS-test (Kolmogorov Smirnov test, KS test) algorithm.

[0145] In this embodiment, the importance evaluation algorithm based on the word graph model refers to constructing a graph structure of the log template based on the device status label, the log template and its corresponding number of backup logs, and then performing model analysis based on the graph structure to obtain the importance of each log template.

[0146] In this embodiment, the unsupervised feature selection algorithm refers to selecting log templates important for determining the device status from various log templates without the need for device status labels, and may include three feature selection algorithms: filtering, embedded, and wrapping.

[0147] In this embodiment, after calculating the importance of each log template in determining the device status, the log template whose importance exceeds the first preset threshold can be selected as the important log template and recorded in the important log template library. In this way, when performing subsequent log analysis, the important log template can be used to extract the features of the original log for analysis, reducing the number of log templates and extracting the required features more efficiently.

[0148] It should be noted here that when recording an important log template, the importance of the important log template may also be recorded.

[0149] In a possible implementation, in the above log processing method, the method further includes:

[0150] Obtaining a device status tag corresponding to the original log;

[0151] Based on the device status tag and the backup log corresponding to the log template, a log template feature is obtained, where the log template feature includes at least one of the following features: a time series statistical feature, an association rule of the log template, and a frequent sequence pattern of the log template;

[0152] The acquiring of the log template feature based on the device status tag and the backup log corresponding to the log template includes:

[0153] Counting multiple time series statistical features of the domain keywords of the log template, where the domain keywords of the log template include domain keywords corresponding to the backup log matching the log template, and the time series statistical features include statistical features of time series data of the domain keywords of the log template for the same device within a preset time period;

[0154] Using an association rule mining algorithm, based on the device status tag and the backup log corresponding to the log template, performing association rule mining on the log template to obtain multiple log template association rules;

[0155] A frequent sequence pattern mining algorithm is used to mine the log template based on the device status label and the backup log corresponding to the log template, so as to obtain a plurality of frequent sequence patterns of the log template.

[0156] In this embodiment, the time series statistical features of the domain keywords are for the same log template, and it is necessary to count the time series statistical features of the domain keywords of log template A, the time series statistical features of the domain keywords of log template B, and so on.

[0157] In this embodiment, the backup logs corresponding to the original logs generated by the same device within a sliding window (5 minutes) that match the log template can be obtained, and the number of domain keywords corresponding to the matched backup logs is counted as the time series data of the domain keywords of the log template. The time series statistical features can be obtained by performing statistical feature calculations on the time series data within a preset time period. The statistical feature calculations include the calculation of statistical features such as sum, difference, entropy, kurtosis, skew, median, and standard derivation.

[0158] In this embodiment, the association rule mining algorithm is used to mine various concurrent association relationships between log templates, and may include but is not limited to Apriori (association rule algorithm), FP-Growth (Frequent Pattern Growth, frequent pattern mining), Eclat (depth-first algorithm), etc. These association rule mining algorithms are all existing data mining algorithms and are not described in detail here. The log template association rules can include, for example, when log template A and log template B are generated simultaneously, the device status label is faulty, or when a composite feature such as the Cartesian product or multiplication of two log templates is generated, the device status label is faulty, and so on.

[0159] In this embodiment, the frequent sequence pattern mining algorithm is used to mine the sequential relationship between each log template, which may include but is not limited to AprioriAll (sequential pattern mining), GSP (Generalized Sequential Pattern mining algorithm), FreeSpan (frequent pattern projection sequence pattern mining algorithm), PrefixSpan (Prefix-Projected Pattern Growth, prefix projection pattern mining) and other sequence pattern mining algorithms. These frequent sequence pattern mining algorithms are all existing data mining algorithms and are not described in detail here. The frequent sequence pattern of the log template can be such as when log template A, log template B and log template C are generated in sequence, the device status label is fault, or when log template A and log template D are generated in sequence, the device status label is fault, etc.

[0160] It should be noted here that the log template described in this embodiment may also be an important log template whose importance exceeds a first preset threshold.

[0161] In a possible implementation, in the above log processing method, the method further includes:

[0162] Based on the device status label, importance evaluation is performed on multiple time series statistical features, multiple log template association rules, and multiple log template frequent sequence patterns of the log template respectively, to obtain the importance of the multiple time series statistical features of the log template, the importance of the multiple log template association rules, or the importance of the multiple log template frequent sequence patterns in determining the device status;

[0163] Determining, among the multiple time series statistical features of the log template, a time series statistical feature whose importance exceeds a second preset threshold as an important time series feature of the log template;

[0164] Determining a log template association rule whose importance exceeds a third preset threshold among the multiple log template association rules as an important log template association rule;

[0165] The log template frequent sequence pattern whose importance exceeds a fourth preset threshold among the multiple log template frequent sequence patterns is determined as an important log template frequent sequence pattern.

[0166] In this embodiment, the second importance evaluation algorithm can be an importance evaluation algorithm based on statistical features, a supervised importance evaluation algorithm based on a model, an importance evaluation algorithm based on significance analysis, an important template extraction algorithm based on a word graph model, or an unsupervised feature selection algorithm.

[0167] In this embodiment, the second importance evaluation algorithm can be used. For each log template, the importance of multiple time series statistical features of the log template can be evaluated, and the importance of multiple time series statistical features of the log template in determining the device status can be calculated. The importance of each time series statistical feature of different log templates is different. For example, the importance of the peak value of log template A can be calculated to be 0.8, the importance of the mean value is 0.7, and the importance of entropy is 0.2. The importance of the peak value of log template B can be calculated to be 0.2, the importance of the mean value is 0.9, the importance of entropy is 0.6, and so on. Assuming that the second preset is 0.5, the important time series features of log template A are the peak value and the mean value, and the important time series features of log template B are the mean value and the entropy.

[0168] In this embodiment, the second importance evaluation algorithm can be used to evaluate the importance of the multiple log template association rules, calculate the importance of the multiple log template association rules, and select the log template association rules whose importance exceeds the third preset threshold from the multiple log template association rules as important log template association rules.

[0169] In this embodiment, the second importance evaluation algorithm can be used to evaluate the importance of the multiple log template frequent sequence patterns, calculate the importance of the multiple log template frequent sequence patterns, and select the log template frequent sequence pattern whose importance exceeds the fourth preset threshold from the multiple log template frequent sequence patterns as the log template frequent sequence pattern.

[0170] It should be noted here that the first preset threshold, the second preset threshold, the third preset threshold and the fourth preset threshold may be the same or different and may be set according to actual needs.

[0171] For example, Figure 2 A flow chart showing the log processing process according to an embodiment of the present disclosure is shown as follows: Figure 2As shown, an original log whose log length does not exceed a preset length can be obtained from the original log library 201, and then the original log is input into the processing module 202 for a series of processing, such as filtering out invalid content in the original log through a preset regular expression library to obtain valid content of the original log, performing format conversion on the valid content according to preset conversion rules to obtain valid content that meets the format requirements, filtering out the domain keywords of the domain where the original log is located in the valid content to obtain a backup log, and inputting the backup log into the log template extraction module 203 to extract the log template corresponding to the backup log. In combination with the device status tag in the device status tag library 204 and the log template, the first importance evaluation algorithm in the importance evaluation module 205 is used to evaluate the importance of the log template, and an important log template whose importance exceeds a first preset threshold is obtained, and recorded in the important log template library 206; the feature mining module 207 can mine the time series statistical features, log template association rules and log template frequent sequence patterns of the important log template, and use the second importance evaluation algorithm in the importance evaluation module 205 to respectively evaluate the importance of the time series statistical features, log template association rules and log template frequent sequence patterns of the important log template, and obtain important features that meet the importance requirements, namely the important time series features, important log template association rules and important log template frequent sequence patterns of the important log template, and record them in the important feature library 208. The important log template, important time series features, important log template association rules and important log template frequent sequence patterns obtained by the above log processing process are used to form the template features of the system event log, and combined with the server status tag, model training is performed. The trained model can improve the accuracy of server status diagnosis from the original 60% to 90%.

[0172] The present disclosure also provides a log processing method in a data center network, the method comprising the following steps:

[0173] Obtain multiple original logs within the data center network;

[0174] Preprocessing the original log to obtain valid content of the original log, wherein the valid content includes relevant content for determining a log analysis result;

[0175] Filtering out domain keywords of the domain in which the original log is located in the valid content to obtain a backup log;

[0176] A log template is extracted from the backup log to obtain a log template corresponding to the backup log.

[0177] The log processing method provided by the present disclosure can be applied to a data center network to process the original logs of the servers in the data center network. Detailed descriptions refer to the descriptions in the above embodiments and will not be repeated here.

[0178] The following are embodiments of the apparatus disclosed herein, which can be used to execute embodiments of the method disclosed herein.

[0179] Figure 3 The block diagram of the log processing device according to the embodiment of the present disclosure is shown. The device can be implemented as part or all of the electronic device through software, hardware or a combination of both. Figure 3 As shown, the log processing device includes:

[0180] A first acquisition module 301 is configured to acquire multiple original logs;

[0181] A preprocessing module 302 is configured to preprocess the original log to obtain valid content of the original log, wherein the valid content includes relevant content for determining a log analysis result;

[0182] A filtering module 303 is configured to filter out domain keywords of the domain in which the original log is located in the valid content to obtain a backup log;

[0183] The extraction module 304 is configured to extract a log template from the backup log to obtain a log template corresponding to the backup log.

[0184] In a possible implementation, the first acquisition module 301 is configured to:

[0185] Get the original log whose length does not exceed the preset length.

[0186] In a possible implementation, the pre-processing module 302 is configured to:

[0187] Invalid content in the original log is filtered out by using a preset regular expression library to obtain valid content of the original log, where the invalid content includes content irrelevant to the log analysis result.

[0188] In a possible implementation, the device further includes:

[0189] The conversion module is configured to perform format conversion on the valid content according to a preset conversion rule to obtain valid content that meets the format requirements.

[0190] In a possible implementation, the conversion module is configured to:

[0191] Converting the writing format of letters in the valid content into a preset format;

[0192] and / or,

[0193] According to a preset word correspondence, the domain words in the valid content are replaced with preset words corresponding to the domain words. The word correspondence stores the domain words of the domain where the original log is located and the corresponding preset words.

[0194] In a possible implementation, the extraction module 304 is configured to:

[0195] Matching the backup log with the extracted log template;

[0196] In response to the backup log failing to match the extracted log template, performing log template extraction on the backup log to generate a log template corresponding to the backup log;

[0197] In response to the backup log matching the extracted log template, the log template of the backup log is determined to be the extracted log template matching the backup log.

[0198] In a possible implementation, the device further includes:

[0199] A second acquisition module is configured to obtain a device status tag corresponding to the original log;

[0200] a first evaluation module configured to perform an importance evaluation on the log template based on the device status tag, the log template and the number of backup logs corresponding thereto, and obtain the importance of the log template in determining the device status;

[0201] The recording module is configured to record the log template whose importance exceeds a first preset threshold as an important log template.

[0202] In a possible implementation, the device further includes:

[0203] a third acquisition module, configured to acquire a device status tag corresponding to the original log;

[0204] a fourth acquisition module configured to acquire a log template feature based on the device status tag and the backup log corresponding to the log template, the log template feature comprising at least one of the following features: a time series statistical feature, an association rule of the log template, and a frequent sequence pattern of the log template;

[0205] The acquiring of the log template feature based on the device status tag and the backup log corresponding to the log template includes:

[0206] Counting multiple time series statistical features of the domain keywords of the log template, where the domain keywords of the log template include domain keywords corresponding to the backup log matching the log template, and the time series statistical features include statistical features of time series data of the domain keywords of the log template for the same device within a preset time period;

[0207] Using an association rule mining algorithm, based on the device status tag and the backup log corresponding to the log template, performing association rule mining on the log template to obtain multiple log template association rules;

[0208] A frequent sequence pattern mining algorithm is used to mine the log template based on the device status label and the backup log corresponding to the log template, so as to obtain a plurality of frequent sequence patterns of the log template.

[0209] In a possible implementation, the device further includes:

[0210] a second evaluation module configured to respectively evaluate the importance of the multiple time series statistical features, the multiple log template association rules, and the multiple log template frequent sequence patterns of the log template based on the device status label, and obtain the importance of the multiple time series statistical features, the multiple log template association rules, or the multiple log template frequent sequence patterns in determining the device status;

[0211] The determination module is configured to determine the time series statistical features of the log template whose importance exceeds the second preset threshold among the multiple time series statistical features of the log template as the important time series features of the log template; determine the log template association rules of the multiple log template association rules whose importance exceeds the third preset threshold as the important log template association rules; and determine the log template frequent sequence patterns of the multiple log template frequent sequence patterns whose importance exceeds the fourth preset threshold as the important log template frequent sequence patterns.

[0212] Figure 3 The technical terms and technical features involved in the embodiments shown and related Figure 1-Figure 2 The technical terms and technical features mentioned in the embodiments shown and related are the same or similar. Figure 3 The explanation and description of the technical terms and technical features involved in the embodiments shown and related can refer to the above Figure 1-Figure 2 The explanations of the illustrated and related embodiments will not be repeated here.

[0213] The present disclosure also discloses an electronic device, Figure 4 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0214] like Figure 4As shown, the electronic device 400 includes a memory 401 and a processor 402, wherein the memory 401 is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor 402 to implement the method according to the embodiment of the present disclosure.

[0215] Figure 5 A schematic diagram showing the structure of a computer system suitable for implementing the method according to an embodiment of the present disclosure is shown.

[0216] like Figure 5 As shown, the computer system 500 includes a processing unit 501, which can execute various processes in the above-mentioned embodiments according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage unit 508 into a random access memory (RAM) 503. Various programs and data required for the operation of the computer system 500 are also stored in the RAM 503. The processing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0217] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed so that a computer program read therefrom can be installed into the storage section 508 as needed. Among them, the processing unit 501 can be implemented as a processing unit such as a CPU, a GPU, a TPU, an FPGA, an NPU, etc.

[0218] In particular, according to embodiments of the present disclosure, the methods described above can be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising computer instructions that, when executed by a processor, implement the method steps described above. In such embodiments, the computer program product can be downloaded and installed from a network via the communication portion 509 and / or installed from removable media 511.

[0219] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of code, and the module, program segment or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or can be implemented using a combination of dedicated hardware and computer instructions.

[0220] The units or modules described in the embodiments of the present disclosure may be implemented in software or programmable hardware. The units or modules described may also be provided in a processor, and the names of these units or modules do not, in certain circumstances, limit the units or modules themselves.

[0221] As another aspect, the present disclosure further provides a computer-readable storage medium. This computer-readable storage medium may be included in the electronic device or computer system described in the above embodiments, or may be a standalone computer-readable storage medium not incorporated into the device. The computer-readable storage medium stores one or more programs, which are used by one or more processors to execute the methods described in the present disclosure.

[0222] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the invention herein is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also encompasses other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the inventive concept. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

Claims

1. A log processing method within a data center network, comprising: Obtain multiple original logs within the data center network; Preprocessing the original log to obtain valid content of the original log, wherein the valid content includes relevant content for determining a log analysis result; Filtering out domain keywords of the domain in which the original log is located in the valid content to obtain a backup log; Extracting a log template from the backup log to obtain a log template corresponding to the backup log; The method further comprises: Obtaining a device status tag corresponding to the original log; Based on the device status tag and the backup log corresponding to the log template, a log template feature is obtained, where the log template feature includes at least one of the following features: a time series statistical feature, an association rule of the log template, and a frequent sequence pattern of the log template; Based on the device status label, importance evaluation is performed on multiple time series statistical features, multiple log template association rules, and multiple log template frequent sequence patterns of the log template respectively, to obtain the importance of the multiple time series statistical features of the log template, the importance of the multiple log template association rules, or the importance of the multiple log template frequent sequence patterns in determining the device status; Determining, among the multiple time series statistical features of the log template, a time series statistical feature whose importance exceeds a second preset threshold as an important time series feature of the log template; Determining a log template association rule whose importance exceeds a third preset threshold among the multiple log template association rules as an important log template association rule; Determining a log template frequent sequence pattern whose importance exceeds a fourth preset threshold among the multiple log template frequent sequence patterns as an important log template frequent sequence pattern; The important time series features, the important log template association rules, and the important log template frequent sequence patterns are used for server status diagnosis.

2. A log processing method, comprising: Get multiple original logs; Preprocessing the original log to obtain valid content of the original log, wherein the valid content includes relevant content for determining a log analysis result; Filtering out domain keywords of the domain in which the original log is located in the valid content to obtain a backup log; Extracting a log template from the backup log to obtain a log template corresponding to the backup log; The method further comprises: Obtaining a device status tag corresponding to the original log; Based on the device status tag and the backup log corresponding to the log template, a log template feature is obtained, where the log template feature includes at least one of the following features: a time series statistical feature, an association rule of the log template, and a frequent sequence pattern of the log template; Based on the device status label, importance evaluation is performed on multiple time series statistical features, multiple log template association rules, and multiple log template frequent sequence patterns of the log template respectively, to obtain the importance of the multiple time series statistical features of the log template, the importance of the multiple log template association rules, or the importance of the multiple log template frequent sequence patterns in determining the device status; Determining, among the multiple time series statistical features of the log template, a time series statistical feature whose importance exceeds a second preset threshold as an important time series feature of the log template; Determining a log template association rule whose importance exceeds a third preset threshold among the multiple log template association rules as an important log template association rule; Determining a log template frequent sequence pattern whose importance exceeds a fourth preset threshold among the multiple log template frequent sequence patterns as an important log template frequent sequence pattern; The important time series features, the important log template association rules, and the important log template frequent sequence patterns are used for server status diagnosis.

3. The method according to claim 2, wherein: The acquisition of multiple original logs includes: Get the original log whose length does not exceed the preset length.

4. The method according to claim 2, wherein: The preprocessing of the original log to obtain valid content of the original log includes: Invalid content in the original log is filtered out by using a preset regular expression library to obtain valid content of the original log, where the invalid content includes content irrelevant to the log analysis result.

5. The method according to claim 2, wherein: The method further comprises: The format of the valid content is converted according to the preset conversion rules to obtain valid content that meets the format requirements.

6. The method according to claim 5, wherein: The format conversion of the valid content according to the preset conversion rules to obtain valid content that meets the format requirements includes: Converting the writing format of letters in the valid content into a preset format; and / or, According to a preset word correspondence, the domain words in the valid content are replaced with preset words corresponding to the domain words. The word correspondence stores the domain words of the domain where the original log is located and the corresponding preset words.

7. The method according to claim 2, wherein: The extracting the log template from the standby log to obtain the log template corresponding to the standby log includes: Matching the backup log with the extracted log template; In response to the backup log failing to match the extracted log template, performing log template extraction on the backup log to generate a log template corresponding to the backup log; In response to the backup log matching the extracted log template, the log template of the backup log is determined to be the extracted log template matching the backup log.

8. The method according to claim 2, wherein: The method further comprises: Based on the device status label, the log template and the number of backup logs corresponding to the log template, an importance evaluation is performed on the log template to obtain the importance of the log template in determining the device status; The log template whose importance exceeds the first preset threshold is recorded as an important log template.

9. The method according to any one of claims 2 to 8, wherein obtaining the log template feature based on the device status tag and the backup log corresponding to the log template comprises: Counting multiple time series statistical features of the domain keywords of the log template, where the domain keywords of the log template include domain keywords corresponding to the backup log matching the log template, and the time series statistical features include statistical features of time series data of the domain keywords of the log template within a preset time period for the same device; Using an association rule mining algorithm, based on the device status tag and the backup log corresponding to the log template, performing association rule mining on the log template to obtain multiple log template association rules; A frequent sequence pattern mining algorithm is used to mine the log template based on the device status label and the backup log corresponding to the log template, so as to obtain a plurality of frequent sequence patterns of the log template.

10. A log processing device, comprising: A first acquisition module is configured to acquire multiple original logs; a preprocessing module configured to preprocess the original log to obtain valid content of the original log, wherein the valid content includes relevant content for determining a log analysis result; A filtering module configured to filter out domain keywords of the domain where the original log is located in the valid content to obtain a backup log; an extraction module configured to extract a log template from the backup log to obtain a log template corresponding to the backup log; a third acquisition module, configured to acquire a device status tag corresponding to the original log; a fourth acquisition module configured to acquire a log template feature based on the device status tag and the backup log corresponding to the log template, the log template feature comprising at least one of the following features: a time series statistical feature, an association rule of the log template, and a frequent sequence pattern of the log template; a second evaluation module configured to respectively evaluate the importance of the multiple time series statistical features, the multiple log template association rules, and the multiple log template frequent sequence patterns of the log template based on the device status label, and obtain the importance of the multiple time series statistical features, the multiple log template association rules, or the multiple log template frequent sequence patterns in determining the device status; a determination module configured to determine, among the multiple time series statistical features of the log template, a time series statistical feature whose importance exceeds a second preset threshold as an important time series feature of the log template; Determining a log template association rule whose importance exceeds a third preset threshold among the multiple log template association rules as an important log template association rule; Determining a log template frequent sequence pattern whose importance exceeds a fourth preset threshold among the multiple log template frequent sequence patterns as an important log template frequent sequence pattern; The important time series features, the important log template association rules, and the important log template frequent sequence patterns are used for server status diagnosis.

11. An electronic device comprising a memory and a processor, wherein: The memory is configured to store one or more computer instructions, and the one or more computer instructions are executed by the processor to implement the method according to any one of claims 2 to 9.

12. A readable storage medium having computer instructions stored thereon, wherein the computer instructions implement the method according to any one of claims 2 to 9 when executed by a processor.

13. A computer program product comprising computer instructions, wherein when the computer instructions are executed by a processor, the method according to any one of claims 2 to 9 is implemented.

Images