A log analysis method and related apparatus

Abstract

The application discloses a log analysis method, and relates to the technical field of computers, and comprises the following steps: converting unstructured system logs into structured system logs; classifying the structured system logs, and determining log weights of the structured system logs according to the categories of the structured system logs; obtaining the number of logs in a preset time slice according to the structured system logs generated in the preset time slice and corresponding log weights; determining a set of system logs with abnormal log numbers according to the number of logs; and assigning weights to different fields of the structured system logs in the set of system logs. The method can realize efficient and lightweight log analysis. The application also discloses a log analysis device, equipment and a computer readable storage medium, which all have the above technical effects.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a log analysis method; it also relates to a log analysis apparatus, device, and computer-readable storage medium. Background Technology

[0002] System logs contain a wealth of information, making them suitable for various analytical scenarios, such as device software / hardware status monitoring, network behavior analysis, user behavior analysis, intrusion detection, fault location and diagnosis, performance evaluation, and predictive maintenance. However, log analysis has the following characteristics: 1) System logs are unstructured and often vary depending on device type, vendor, model, and operating system. Therefore, most current log analysis is performed manually by operations engineers, which is time-consuming and labor-intensive. Furthermore, the complex log data often leads to the omission of anomalies due to the complexity of the logs. 2) The sheer volume of system logs means that during troubleshooting, to accurately identify faults, operations engineers often search the system logs of relevant devices within a specific time window, extracting useful information from a large amount of unstructured, heterogeneous logs. This process is often time-consuming and labor-intensive, and can easily overlook details buried in ordinary logs. Although log analysis can be performed using mathematical algorithms such as principal component analysis and singular value decomposition, as well as various machine learning algorithms, based on the textual nature and parsable structure of system logs, these machine learning algorithms are relatively complex and require significant system computational power. Data analysis methods such as principal component analysis and singular value decomposition are often inefficient when dealing with large amounts of unstructured text, and are not suitable for real-time detection, resulting in low portability.

[0003] Therefore, providing an efficient and lightweight log analysis solution has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] The purpose of this application is to provide a log analysis method that enables efficient and lightweight log analysis. Another purpose of this application is to provide a log analysis apparatus, device, and computer-readable storage medium, all of which achieve the aforementioned technical effects.

[0005] To address the aforementioned technical problems, this application provides a log analysis method, including:

[0006] Transform unstructured system logs into structured system logs;

[0007] The structured system logs are classified, and the log weights of the structured system logs are determined according to their categories.

[0008] The number of logs within the preset time slice is obtained based on the structured system logs generated within the preset time slice and the corresponding log weights.

[0009] Based on the log count, determine the system log set with an abnormal log count;

[0010] Assign weights to different fields of the structured system logs in the system log set.

[0011] Optionally, classifying the structured system logs includes:

[0012] The structured system logs are transformed into multidimensional vectors;

[0013] The multidimensional vectors are processed by a clustering algorithm to obtain the categories of the structured system logs.

[0014] Optionally, assigning weights to different fields of the structured system logs in the system log set includes:

[0015] Assign weights to the timestamps, event levels, log identifiers, and event details of the structured system logs.

[0016] Optionally, assigning weights to the event details of the structured system log includes:

[0017] Extract event details template;

[0018] Build an exception terminology database based on the event details template;

[0019] Assign weights to the words in the abnormal word library.

[0020] Optionally, the template for extracting event details includes:

[0021] Use FT-Tree to extract event details templates.

[0022] Optionally, the system log set used to determine abnormal log counts includes:

[0023] Anomaly detection of log counts is performed using statistical algorithms;

[0024] The system log set is determined by performing secondary filtering on the detection results of the statistical algorithm using preset filtering rules.

[0025] Optionally, the anomaly detection of log quantity using statistical algorithms includes:

[0026] The log count was anomaly detected using the NSIgma algorithm.

[0027] To address the aforementioned technical problems, this application also provides a log analysis apparatus, comprising:

[0028] The transformation module is used to transform unstructured system logs into structured system logs.

[0029] A classification module is used to classify the structured system logs and determine the log weight of the structured system logs according to their categories.

[0030] The log quantity determination module is used to obtain the log quantity within the preset time slice based on the structured system logs generated within the preset time slice and the corresponding log weights;

[0031] The log set determination module is used to determine the system log set with an abnormal number of logs based on the number of logs.

[0032] The allocation module is used to assign weights to different fields of the structured system logs in the system log set.

[0033] To address the aforementioned technical problems, this application also provides a log analysis device, comprising:

[0034] Memory, used to store computer programs;

[0035] A processor for executing the computer program to implement the steps of the log analysis method as described in any of the preceding claims.

[0036] To address the aforementioned technical problems, this application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the log analysis method described in any of the preceding claims.

[0037] The log analysis method provided in this application includes: converting unstructured system logs into structured system logs; classifying the structured system logs and determining the log weights of the structured system logs according to their categories; obtaining the number of logs within a preset time slice based on the structured system logs generated within that preset time slice and their corresponding log weights; determining a set of system logs with an abnormal number of logs based on the log count; and assigning weights to different fields of the structured system logs in the system log set.

[0038] As can be seen, the log analysis method provided in this application performs hierarchical log analysis. It automatically transforms unstructured system logs into structured system logs, assigns log weights to each log, determines the number of logs based on these weights, and then performs log quantity anomaly detection. Furthermore, it assigns weights to different fields of system logs with abnormal log quantities, facilitating anomaly detection based on system log fields and their weights. Compared to traditional log analysis methods, such as manual analysis and various machine learning algorithms, the log analysis method in this application achieves efficient and lightweight log analysis.

[0039] The log analysis apparatus, equipment, and computer-readable storage medium provided in this application all have the aforementioned technical effects. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the prior art and embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 A schematic flowchart illustrating a log analysis method provided in an embodiment of this application;

[0042] Figure 2 This is a schematic diagram of a log analysis process provided in an embodiment of this application;

[0043] Figure 3 This is a schematic diagram of a log analysis device provided in an embodiment of this application;

[0044] Figure 4 This is a schematic diagram of a log analysis device provided in an embodiment of this application. Detailed Implementation

[0045] The core of this application is to provide a log analysis method that enables efficient and lightweight log analysis. Another core aspect of this application is to provide a log analysis device, apparatus, and computer-readable storage medium, all of which possess the aforementioned technical effects.

[0046] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0047] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a log analysis method provided in an embodiment of this application. (Refer to...) Figure 1 As shown, the method includes:

[0048] S101: Convert unstructured system logs into structured system logs;

[0049] Step S101 aims to preprocess the system logs, transforming the raw, unstructured system logs into structured system logs. Although the definition of system logs varies across different manufacturers, models, and types of devices, system logs typically consist of multiple fields. These include, for example, a timestamp, device ID / device name, event level, log identifier, and event details. The timestamp refers to the system time when the system log was generated; the device ID / device name refers to the device ID / device name that generated the system log; the event level characterizes the severity of the event recorded in the system log; the log identifier is an information type descriptor; and the event details contain free-format text describing the event.

[0050] Raw, unstructured system logs can be processed using log management platforms such as ELK. ELK, short for Elasticsearch, Logstash, and Kibana, is a widely used log management platform. Logstash is responsible for extracting event data from system logs from different sources (devices) and transforming this unstructured raw event data into a series of structured event data, which is then stored in Elasticsearch. Elasticsearch acts like a database, storing the core data of the ELK platform. Kibana is a data visualization tool that reads structured log data from Elasticsearch and creates different charts according to user requirements for data analysis.

[0051] S102: Classify the structured system logs and determine the log weight of the structured system logs according to their categories;

[0052] S103: Based on the structured system logs generated within the preset time slice and the corresponding log weights, the number of logs within the preset time slice is obtained;

[0053] S104: Based on the stated number of logs, determine the system log set with an abnormal number of logs;

[0054] Steps S102 to S104 aim to detect abnormal log counts. A simple implementation of abnormal log count detection is to calculate the number of system logs within each fixed time slice and determine whether the number of system logs within each fixed time slice exceeds a threshold. However, this detection method has the following problems: 1) Detection will fail in time slices with a large amount of useless or duplicate information, such as "user login," "incorrect password," "timeout," and "illegal command." When similar situations occur, the number of logs within the time slice to be detected may surge, leading to detection errors. 2) The threshold needs to be manually optimized as business adjustments are made, resulting in inflexibility and weak scalability. When there are many indicators to be detected, it is difficult for operations and maintenance personnel to adjust the threshold in a timely and accurate manner, often causing false alarms or missed alarms in different scenarios. For example, a high threshold setting when business is at a low peak will lead to missed alarms; a low threshold setting when business is at a high peak will lead to a large number of duplicate alarms in a short period of time.

[0055] To avoid the above problems, this embodiment adopts the following method for detecting abnormal log counts:

[0056] According to Shannon's definition of "entropy" in information theory:

[0057] Uppercase P indicates different types of system logs, while lowercase p indicates the probability of different types of system logs appearing within a certain period of time.

[0058] The information a source sends is uncertain; its content can be measured by its probability of occurrence. A higher probability means a higher chance of occurrence and lower uncertainty, and vice versa. In system log analysis, some system logs frequently appear in many time slices or periodically appear in large quantities in specific time slices, which is considered normal. For example, user login information. Conversely, some system logs appear very infrequently and require close attention from operations and maintenance personnel. Therefore, frequently occurring system logs can be filtered, retaining those with significant changes for focused analysis, and minimizing the impact of periodicity. Thus, events with high frequency are assigned lower weights to make them easier to filter out, while events with low frequency are assigned higher weights to make significant changes more prominent. Based on this, this embodiment first classifies system logs and then assigns corresponding log weights to each category.

[0059] One approach is to categorize system logs based on a specific field, such as "event level." However, this method has low accuracy because the frequency of events is not solely determined by the event level field. A more reasonable approach would be to extract as much information as possible from the system logs to improve classification accuracy. Therefore, in one specific implementation, classifying the structured system logs includes: converting the structured system logs into multi-dimensional vectors; and processing the multi-dimensional vectors using a clustering algorithm to obtain the categories of the structured system logs.

[0060] Although different system logs may appear completely different on the surface, they are all composed of the same field categories, differing only in the content of each field. Therefore, a system log can be viewed as a multi-dimensional vector, where each dimension of the vector is a field, and the content of that field is the value of the multi-dimensional vector in that dimension.

[0061] To adapt to algorithm implementation, the text data of the fields can be digitized. The discrete field settings facilitate digitization, as a field often takes values ​​from a discrete set of states. For example, "event level" is usually represented by several fixed severity levels. In this case, simply assign different numbers to different levels, such as encoding 0-7 different severity levels. Other fields can be processed similarly. To improve the accuracy of subsequent model training, normalization can be added to the encoding. After the above steps, a large amount of text-based log data can be digitized without losing any information. Figure 2 As shown, a clustering algorithm (an unsupervised machine learning algorithm) is used to train a model on historical data within a certain time period for the encoded vector data. The trained model can efficiently and automatically classify future system logs.

[0062] After categorizing the system logs, assign different weights to each category. Weights can be set directly based on expert operational experience or through analysis of historical data over a fixed period. Specifically, select historical data within a given timeframe, ensuring the selected period encompasses the complete business / hardware performance cycle. Count the number of logs of different types. Based on the principle of entropy, the weight should be inversely proportional to the frequency of occurrence.

[0063] If the system's computing power allows, deep learning algorithms can be used to directly weight system logs. This involves training the model (using clustering algorithms and inverse weighting labels as training data) to classify unknown logs and then focusing on logs with lower frequency of occurrence. Furthermore, the deep learning model can be updated periodically.

[0064] After determining the log weight of the system logs, the number of logs and the system log sets with abnormal log counts are further determined.

[0065] In one specific implementation, determining the system log set with an abnormal log quantity includes: performing anomaly detection on the log quantity using a statistical algorithm; and performing secondary filtering on the detection results of the statistical algorithm using preset filtering rules to determine the system log set. The anomaly detection on the log quantity using a statistical algorithm may include: performing anomaly detection on the log quantity using the NSIgma algorithm.

[0066] Specifically, a time slice interval of τ is set, and the number of logs within the interval [t, t+τ] is counted. A sliding window mechanism is adopted, with a sliding time interval of μ (where μ < τ). The detection interval size and algorithm parameter n are set, and the statistical algorithm NSIgma is used to detect anomalies in the sequence of logs in the sliding window, detecting sudden increases and decreases in the number of logs.

[0067] The number of logs within a time slice is obtained by weighting and summing the system logs generated within the time slice interval and the corresponding weights of each system log.

[0068] Additionally, one can roughly determine whether an indicator exhibits periodicity and select different data segments for concatenation based on the length of the period. For example, if the application scenario lacks obvious periodicity, the t0-minute period before the current time can be selected as the sequence to be detected. When the periodicity is short, roughly on a daily basis, the t0-minute period before the current time can be concatenated with the t0-minute periods before and after the same time point of the previous day as the sequence to be detected. Similarly, when the periodicity is long, roughly on a weekly basis, the t0-minute period before the current time can be concatenated with the t0-minute periods before and after the same time point of the previous week as the sequence to be detected.

[0069] Preset filtering rules can be configured differently to better suit real-world application scenarios. For example, detection can be tailored to detect only sudden spikes in log counts, by adding a filter condition with a positive slope. In this case, when a spike or drop in log count is detected, the drop is filtered out, while the spike is retained. Alternatively, based on anomaly detection using the NSIgma statistical algorithm, a dynamic threshold can be used for secondary filtering. The dynamic threshold can be described as a polynomial, which can be represented as follows:

[0070] T = f(t1, t2, ..., t) n )=a1t1+a2t2+…+a n t n Where t1, t2, ..., t n This indicates factors affecting the dynamic threshold, such as: device level, the service level associated with the device, the order of magnitude of device traffic, etc.; a1, a2, ..., a n This indicates the weight of different factors in the dynamic threshold. This weight can be preset through operational experience or obtained through statistical fitting of historical data.

[0071] Furthermore, in practical applications, the number of logs on a single device or within the same functional area exhibits a certain periodicity. Therefore, when computing power allows, reinforcement learning-based algorithms can be used on some core devices or core functional areas to mine periodic patterns, thereby increasing detection accuracy. An agent is trained to accurately determine whether anomalies have occurred based on the observed number and content of logs. Successful detections receive positive feedback, while missed or false detections incur penalties. After long-term training, the agent can find a near-optimal criterion to maximize cumulative gain, thereby determining whether periodicity should be implemented, and if so, the specific duration of periodicity. In reinforcement learning algorithms, the state of the environment can be defined as the content or quantity of logs per unit time, the agent's action can be defined as whether an anomaly has been detected, and the reward mechanism is determined by the accuracy of the prediction.

[0072] S105: Assign weights to different fields of the structured system logs in the system log set.

[0073] By executing steps S101 to S104, a structured set of system logs representing the number of anomalies identified within a certain time slice can be obtained. Step S105 aims to select different weighting strategies and set corresponding weights based on the different amounts of information represented by different fields of the system logs in this system log set, in order to perform system log anomaly detection.

[0074] In one specific implementation, setting the weights of different fields of the structured logs in the log set with the abnormal number of logs includes:

[0075] Set the weights for the timestamp, event level, log identifier, and event details of the structured log.

[0076] Specifically, the timestamp represents the system time when the system log was generated, and its weight can be differentiated by determining whether it is a peak business period. For example, 09:00-17:00 each day can be considered a peak business period, and 17:00-09:00 the next day can be considered a low business period, with the weight of the peak business period being higher than that of the low business period.

[0077] Event levels represent the severity of an event, typically expressed as a numerical range or several discrete severity levels. For example, the event levels of Vendor 1's system logs might be represented by a numerical range of 0-7, with smaller numbers indicating greater criticality and higher priority. Vendor 2's system logs might use three discrete levels: "Major," "Warning," and "Error." Generally, the weights of different severity levels should be proportional to the severity they represent; therefore, a linear mechanism can be used. Alternatively, a logarithmic mechanism can be used. Taking the logarithm does not change the nature or correlation of the data, but it can compress the scale of the variables. Specifically, it means that the sensitivity to differences in smaller numerical values ​​is greater than that to differences in larger numerical values. Therefore, to increase sensitivity to higher severity levels, the weight selection strategy could be to take the reciprocal of the logarithmic mechanism.

[0078] Simple log anomaly detection can determine the importance of a log entry using the weighting strategy described above. However, in some complex situations, this field can only indicate anomalies in a single attribute of a single device and cannot indicate the actual impact of the event on services, the network, or other devices. Therefore, this embodiment does not directly use this field as an indicator for log anomaly detection, but rather as part of the system log analysis.

[0079] Log identifiers provide a brief description of events and their changes, as well as a general classification of different log types. Generally, a device has only a limited number of message types, such as "user login," "incorrect password," "timeout," and "illegal command." Therefore, the weight of a log identifier can be directly defined as several discrete values, which can be adjusted appropriately during the verification process based on the actual situation.

[0080] Event details are the string information actually output by each module to the information center. Each module fills in the details with each output, describing the specific content of the log. Different devices have different event description methods, but event details can mostly be parsed into an event template (constant part) with some specific parameters (variable part: e.g., IP address information, user information, interface information, etc.). The constant part consists of fixed plain text and represents the corresponding event type. The constant part of the event details is similar to a further subset of the message type field, so it is usually selected from a finite set of types (facilitating numerical conversion). The variable part records the log's differences / runtime information, status, and parameter values ​​(e.g., port number, etc.), which may vary depending on the occurrence of different events (this part is helpful for event localization and specific analysis).

[0081] Weights are assigned to different event details. First, the event details are classified. One way to do this is to automatically separate the constant and variable parts of the original system log, i.e., offline deparameterization or template extraction, and then match the system log with a fixed template online.

[0082] Template extraction can be achieved in different ways, such as: 1) using predefined regular expressions; 2) using association analysis algorithms; 3) using other rule-based or machine learning-based log parsers. For example: manually checking system logs or writing rules based on operational domain knowledge to detect errors, and adding keyword searches (such as "warning," "error," etc.); using LSTM (Long Short-Term Memory) based text recognition and classification, etc.

[0083] In one specific implementation, assigning weights to the event details of the structured system log includes: extracting an event details template; establishing an anomaly dictionary based on the event details template; and assigning weights to the words in the anomaly dictionary. The extraction of the event details template may include: using FT-Tree to extract the event details template.

[0084] FT-Tree is an intelligent algorithm based on a frequent pattern tree model, used to identify frequent combinations of words in system logs and generate message templates. Compared to template extraction methods based on rules or regular expressions, FT-Tree has higher accuracy and supports incremental learning. In this embodiment, the definition of the anomaly lexicon is not just a set of keywords such as "warning" or "error," but a lexicon of any words in historical data samples that may carry "anomaly" information. Specifically, it can be divided into the following three steps:

[0085] Using FT-Tree to extract event detail templates: A proper template is typically a combination of words that frequently appear in the system log. Furthermore, FT-Tree, while building the template tree, prunes the variable parts of the system log, thus automatically separating constant and variable components.

[0086] Specifically, the establishment of the abnormal vocabulary is as follows: A trained log template library can be obtained through FT-Tree (each member in this library is a combination of words representing event details). Based on this log template library, a usable abnormal vocabulary is extracted in the following ways: 1) Count the set of all words appearing in the log template library (regardless of part of speech); 2) Count the frequency of each word in the set in the training set; 3) Select appropriate intervals by manually setting or based on the statistical information after drawing a frequency distribution histogram, and define words whose frequencies appear in the intervals as being added to the abnormal vocabulary; 4) There can be multiple abnormal vocabulary libraries, and different abnormal vocabulary libraries have different abnormal levels and different selected word frequency intervals.

[0087] Assign weights to words in the abnormal vocabulary: The weight of each word in the abnormal vocabulary can be calculated through the following mechanisms: 1) Normalize the weight by taking the inverse ratio of the frequency of each word; 2) Based on the principle of "entropy", the weight is defined as the size of the "information content" of the word.

[0088] In summary, the log analysis method provided in this application performs hierarchical log analysis. It automatically transforms unstructured system logs into structured logs, assigns log weights to each log entry, determines the number of log entries based on these weights, and then performs anomaly detection. Furthermore, it assigns weights to different fields of system logs with abnormal log counts, facilitating anomaly detection based on these fields and their weights. Compared to manual analysis and traditional log analysis methods using various machine learning algorithms, this application's log analysis method achieves efficient and lightweight log analysis.

[0089] This application also provides a log analysis apparatus, which is described below and can be referred to in conjunction with the method described above. Please refer to... Figure 3 , Figure 3 This is a schematic diagram of a log analysis device provided in an embodiment of this application, combined with... Figure 3 As shown, the device includes:

[0090] Transformation module 10 is used to transform unstructured system logs into structured system logs;

[0091] The classification module 20 is used to classify the structured system logs and determine the log weight of the structured system logs according to the categories of the structured system logs.

[0092] The log quantity determination module 30 is used to obtain the log quantity within the preset time slice based on the structured system logs generated within the preset time slice and the corresponding log weights.

[0093] Log set determination module 40 is used to determine the system log set with an abnormal number of logs based on the number of logs.

[0094] The allocation module 50 is used to assign weights to different fields of the structured system logs in the system log set.

[0095] Based on the above embodiments, as a specific implementation method, the classification module 20 includes:

[0096] A transformation unit is used to transform the structured system logs into a multidimensional vector;

[0097] The processing unit is used to process the multidimensional vector through a clustering algorithm to obtain the categories of the structured system logs.

[0098] Based on the above embodiments, as a specific implementation method, the allocation module 50 is specifically used for:

[0099] Assign weights to the timestamps, event levels, log identifiers, and event details of the structured system logs.

[0100] Based on the above embodiments, as a specific implementation method, the allocation module 50 includes:

[0101] Extraction unit, used to extract event details template;

[0102] Establishment unit, used to establish an exception dictionary based on the event details template;

[0103] The assignment unit is used to assign weights to words in the abnormal vocabulary.

[0104] Based on the above embodiments, as a specific implementation method, the extraction unit is specifically used for:

[0105] Use FT-Tree to extract event details templates.

[0106] Based on the above embodiments, as a specific implementation method, the log set determination module 40 includes:

[0107] The detection unit is used to detect anomalies in the number of logs using statistical algorithms;

[0108] The filtering unit is used to perform secondary filtering on the detection results of the statistical algorithm according to preset filtering rules to determine the system log set.

[0109] Based on the above embodiments, as a specific implementation method, the detection unit is specifically used for:

[0110] The log count was anomaly detected using the NSIgma algorithm.

[0111] The log analysis device provided in this application performs hierarchical log analysis. It automatically transforms unstructured system logs into structured system logs, assigns log weights to each log, determines the number of logs based on these weights, and then performs anomaly detection for abnormal log counts. Furthermore, it assigns weights to different fields of system logs with abnormal log counts, facilitating anomaly detection based on system log fields and their weights. Compared to traditional log analysis methods, such as manual analysis or various machine learning algorithms, this log analysis device achieves efficient and lightweight log analysis.

[0112] This application also provides a log analysis device, for reference. Figure 4 As shown, the device includes a memory 1 and a processor 2.

[0113] Memory 1 is used to store computer programs;

[0114] Processor 2 is used to execute computer programs to perform the following steps:

[0115] Transform unstructured system logs into structured system logs;

[0116] The structured system logs are classified, and the log weights of the structured system logs are determined according to their categories.

[0117] The number of logs within the preset time slice is obtained based on the structured system logs generated within the preset time slice and the corresponding log weights.

[0118] Based on the log count, determine the system log set with an abnormal log count;

[0119] Assign weights to different fields of the structured system logs in the system log set.

[0120] For a description of the equipment provided in this application, please refer to the above method embodiments; further details will not be provided here.

[0121] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the following steps:

[0122] Transform unstructured system logs into structured system logs;

[0123] The structured system logs are classified, and the log weights of the structured system logs are determined according to their categories.

[0124] The number of logs within the preset time slice is obtained based on the structured system logs generated within the preset time slice and the corresponding log weights.

[0125] Based on the log count, determine the system log set with an abnormal log count;

[0126] Assign weights to different fields of the structured system logs in the system log set.

[0127] The computer-readable storage medium may include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0128] For a description of the computer-readable storage medium provided in this application, please refer to the above method embodiments; further details will not be repeated here.

[0129] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatuses, devices, and computer-readable storage media disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant details can be found in the method section.

[0130] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0131] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0132] The log analysis method, apparatus, device, and computer-readable storage medium provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A log analysis method, characterized in that, include: Transform unstructured system logs into structured system logs; The structured system logs are classified, and the log weights of the structured system logs are determined according to their categories. The number of logs generated within the time slice is obtained by weighted summation of the system logs generated within the time slice and their corresponding weights. Based on the log count, determine the system log set with an abnormal log count; Assign weights to different fields of the structured system logs in the system log set.

2. The log analysis method according to claim 1, characterized in that, The classification of the structured system logs includes: The structured system logs are transformed into multidimensional vectors; The multidimensional vectors are processed by a clustering algorithm to obtain the categories of the structured system logs.

3. The log analysis method according to claim 1, characterized in that, Assigning weights to different fields of the structured system logs in the system log set includes: Assign weights to the timestamps, event levels, log identifiers, and event details of the structured system logs.

4. The log analysis method according to claim 1, characterized in that, Assigning weights to the event details of the structured system log includes: Extract event details template; Build an exception terminology database based on the event details template; Assign weights to the words in the abnormal word library.

5. The log analysis method according to claim 4, characterized in that, The template for extracting event details includes: Use FT-Tree to extract event details templates.

6. The log analysis method according to claim 1, characterized in that, The system log set that determines the abnormal number of logs includes: Anomaly detection of log counts is performed using statistical algorithms; The system log set is determined by performing secondary filtering on the detection results of the statistical algorithm using preset filtering rules.

7. The log analysis method according to claim 6, characterized in that, The anomaly detection of log quantity using statistical algorithms includes: The log count was anomaly detected using the NSIgma algorithm.

8. A log analysis device, characterized in that, include: The transformation module is used to transform unstructured system logs into structured system logs. A classification module is used to classify the structured system logs and determine the log weight of the structured system logs according to their categories. The log count determination module is used to calculate the number of logs generated within a time slice by weighting and summing the system logs generated within the time slice and their corresponding weights. The log set determination module is used to determine the system log set with an abnormal number of logs based on the number of logs. The allocation module is used to assign weights to different fields of the structured system logs in the system log set.

9. A log analysis device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the log analysis method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the log analysis method as described in any one of claims 1 to 7.

Images