Abnormal log identification methods, models, equipment, media and log monitoring platform

Abstract

The present invention belongs to the technical field of big data analysis, and specifically discloses an abnormal log identification method, model, device, medium and log monitoring platform. The method includes searching for the first K logs similar to the log to be detected in the training set. If there is a log labeled as abnormal, the log to be detected is marked as a candidate abnormal log; the text of the candidate abnormal log is semantically verified based on a pre-trained language model to determine whether the candidate abnormal log is an abnormal log. The log monitoring platform includes a log collection module, a stream processing module, an abnormal log identification module, a database and a visualization module. The present invention is suitable for log abnormality identification and log monitoring, improves the accuracy of log identification, reduces the amount of calculation, and improves calculation efficiency; tracks the log status in real time, identifies abnormal logs, and sends log abnormality alarms to users.

Description

Technical Field

[0001] The present invention belongs to the technical field of big data analysis, and specifically relates to an abnormal log identification method, model, device, medium and log monitoring platform. Background Art

[0002] System logs record important data such as system runtime and status, playing a key role in maintaining system stability and reliability. Developers and operations personnel typically review logs using methods such as keyword searches, rule matching, or filtering algorithms. However, with the continuous expansion and increasing complexity of systems, modern systems are generating massive log volumes at a rate of approximately 50GB (approximately 120 million to 200 million rows) per hour. This explosive growth in log data poses significant challenges to log analysis, making it extremely difficult to identify issues from logs.

[0003] Traditional log anomaly detection methods rely on predefined rules and features. These methods require significant manpower and resources to maintain and update the rule base, are accompanied by high rates of false positives and false negatives, and often fail to effectively detect anomalies when faced with unknown and complex attack vectors. This is why intelligent log anomaly detection technology has emerged.

[0004] There are two mainstream approaches to intelligent log anomaly detection: the first, machine learning algorithms based on rule-based detection and simple statistical models, typically do not require complex computing resources and can process large amounts of log data in real time, ensuring efficient system operation. However, their accuracy is highly dependent on the accuracy of data annotation, and they perform poorly when there is little or no annotated data. This is particularly true when faced with complex patterns and dynamically changing environments. Their performance is limited by inherent simplifying assumptions and the model's generalization ability. Furthermore, when there are too many log types, they are prone to dimensionality explosion, significantly impacting performance. The second, deep learning models based on long short-term memory networks (LSTMs) or Transformers, rely heavily on the quality and diversity of training data and require significant computing resources and time to train. This poses limitations in actual production scenarios, and the model's performance often falls short of expectations. Summary of the Invention

[0005] The purpose of the present invention is to provide a method, model, device and medium for identifying abnormal logs to improve the accuracy and computational efficiency of log anomaly detection; another purpose of the present invention is to provide a log monitoring platform to track log status in real time, identify abnormal logs, and send log anomaly alarms to users.

[0006] In order to achieve the above-mentioned purpose, the technical methods adopted by the present invention are as follows:

[0007] A method for identifying abnormal logs, comprising:

[0008] Collect a labeled training set and search for the first K logs in the training set that are similar to the log to be detected, where K is a positive integer less than the number of logs in the training set. If there is a log labeled as abnormal, mark the log to be detected as a candidate abnormal log.

[0009] The text of the candidate anomaly log is semantically verified based on the pre-trained language model to determine whether the candidate anomaly log is an anomaly log.

[0010] As a limitation, the method for searching the top K logs similar to the log to be detected in the training set is:

[0011] Calculate the MinHash signature of each log in the training set;

[0012] Using the locality-sensitive hashing algorithm, the MinHash signatures of the logs in the training set are sharded and stored in hash buckets;

[0013] Calculate the MinHash signature of the log to be tested, query the hash bucket based on the MinHash signature of the log to be tested, calculate the overall similarity between the MinHash signature of the log to be tested and the MinHash signature of the log in the corresponding hash bucket, sort the logs in the corresponding hash bucket from high to low according to the overall similarity, and take the top K logs.

[0014] As a further qualification: the overall similarity is the product of count similarity and text similarity, that is:

[0015] ,

[0016] Where S 总体 is the overall similarity, S 计数 is the count similarity, S 文本 is the text similarity;

[0017] The formula for calculating count similarity is:

[0018] ,

[0019] Where n1 is the frequency of occurrence of the MinHash signature of the log to be detected, and n2 is the frequency of occurrence of the MinHash signature of a log in the corresponding hash bucket;

[0020] The text similarity adopts Jaccard similarity, and the calculation formula is:

[0021] ,

[0022] Where A is the MinHash signature of the log to be detected, and B is the MinHash signature of a log in the corresponding hash bucket.

[0023] As a further qualification, the MinHash signature is generated as follows:

[0024] Preprocess the logs and extract word sets;

[0025] Apply multiple hash functions to each word in the log, retain the minimum hash value, and form a multi-dimensional signature vector, which is the MinHash signature.

[0026] As a limitation, the method for semantic verification of the text of the candidate exception log is:

[0027] Construct a template containing mask marks, concatenate the candidate exception logs with the template, and generate the input text;

[0028] Feed the input text into a pre-trained BERT model or a derivative of the BERT model to predict the probability distribution of filler words at the masked token positions.

[0029] Establish a mapping relationship between label words and log categories. Based on the probability that the filler word at the mask mark position is the label word, calculate the probability that the candidate anomaly log is an anomaly log or a normal log. If the probability that the candidate anomaly log is an anomaly log is greater than the probability that the candidate anomaly log is a normal log, then the candidate anomaly log is judged to be an anomaly log.

[0030] As a further limitation, the mapping relationship between label words and log categories is established as follows:

[0031] Select multiple label words related to normal logs to form a normal label word set, select multiple label words related to abnormal logs to form an abnormal label word set, and introduce a weighted voting method to assign weights to different label words.

[0032] As a further limitation, the probability that a candidate abnormal log is a normal log is the sum of the products of the weight of each label word in the normal label word set and the probability that the filler word at the mask mark position is the corresponding label word. The probability that a candidate abnormal log is an abnormal log is the sum of the products of the weight of each label word in the abnormal label word set and the probability that the filler word at the mask mark position is the corresponding label word.

[0033] The present invention also provides an abnormal log identification model, including:

[0034] The fast screening module is used to collect a labeled training set and search the training set for the first K logs similar to the log to be detected, where K is a positive integer less than the number of logs in the training set. If there is a log labeled as abnormal, the log to be detected is marked as a candidate abnormal log.

[0035] The semantic verification module performs semantic verification on the text of the candidate anomaly log based on the pre-trained language model to determine whether the candidate anomaly log is an anomaly log.

[0036] The present invention also provides a log monitoring platform, comprising:

[0037] Log collection module collects logs in real time and sends them to the stream processing module;

[0038] Stream processing module, which receives log data streams and implements log distribution and caching;

[0039] The abnormal log identification module uses the above abnormal log identification model to obtain logs from the stream processing module, identify whether they are abnormal logs, and label the logs and store them in the first database;

[0040] The database includes a first database and a second database. The first database is used for storing and indexing logs. The second database is used to store user information tables and log alarm tables. The log alarm table records user-defined log sequence information that requires alarms.

[0041] The visualization module is used to provide a front-end interface and support log query and log exception alarm functions.

[0042] As a limitation, each log in the first database is stored as an independent document. Each document contains multiple fields. An inverted index is established for these fields, and an index segmentation strategy is based on time. The index is segmented regularly according to the frequency of log data generation.

[0043] As a limitation, the log query function can reproduce the log flow of the abnormal period according to the timeline and locate the log through the query statement; the log abnormality alarm function supports two abnormality matching modes. The first abnormality matching mode is fuzzy matching, which is suitable for log abnormality alarms of discontinuous sequences. As long as there are a preset number of abnormal logs in this sequence, an alarm will be triggered even if there are normal logs in the middle of this sequence. The user will be notified by email or SMS according to the time interval selected by the user; the second abnormality matching mode is strict matching, which uses the KMP algorithm. The alarm will only be triggered when a preset number of abnormal logs appear continuously. The user will be notified by email or SMS according to the time interval selected by the user.

[0044] The present invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor calls the computer program in the memory to execute the above-mentioned abnormal log identification method.

[0045] The present invention also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, it is used to implement the above-mentioned abnormal log identification method.

[0046] Due to the adoption of the above solution, the present invention has the following beneficial effects compared with the prior art:

[0047] (1) The abnormal log identification method, model, device, and medium provided by the present invention use the K-nearest neighbor classification algorithm to make a preliminary judgment on the log to be detected, and then use the BERT model or a derivative model of the BERT model to perform semantic verification on the candidate abnormal logs. This avoids the problem of the K-nearest neighbor classification algorithm causing normal logs to be incorrectly marked as abnormal logs in log identification, thereby improving the accuracy of log identification;

[0048] (2) The abnormal log identification method, model, device and medium provided by the present invention accelerate the search of logs by the K nearest neighbor classification algorithm through the local sensitive hashing algorithm. It is not necessary to calculate the similarity between all logs in the training set and the log to be detected. Instead, it is only necessary to calculate the similarity between the logs in the hash bucket corresponding to the log to be detected and the log to be detected, which greatly reduces the amount of calculation and improves the calculation efficiency and log search speed.

[0049] (3) The abnormal log identification method, model, device, and medium provided by the present invention achieve a joint judgment of text similarity and frequency anomaly by calculating the overall similarity between the log in the hash bucket corresponding to the log to be detected and the log to be detected, thereby improving the accuracy of log identification;

[0050] (4) The abnormal log identification method, model, device, and medium provided by the present invention, the BERT model or a derivative model of the BERT model, when using a pre-training plus fine-tuning strategy to cope with downstream tasks, has strong generalization capabilities, can better utilize its deep semantic understanding capabilities, and compensate for the accuracy loss caused by the K-nearest neighbor classification algorithm;

[0051] (5) The abnormal log identification method, model, device and medium provided by the present invention introduce a weighted voting mechanism, which selects multiple label words, each of which is assigned a certain weight. These weights reflect the importance of each word in judging the log status. The weighted voting of all label words can obtain a comprehensive score, thereby improving the accuracy and robustness of abnormal log identification.

[0052] (6) The log monitoring platform provided by the present invention can track the log status in real time, capture and analyze log data in a timely manner, identify abnormal logs, and send log abnormality alarms to users, so that appropriate repair measures can be taken quickly to ensure the smooth execution of operations.

[0053] In summary, the present invention is suitable for log anomaly identification and log monitoring, which improves the accuracy of log identification, reduces the amount of calculation, and improves the calculation efficiency; it tracks the log status in real time, identifies abnormal logs, and sends log anomaly alarms to users. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The present invention will be described in further detail below with reference to the accompanying drawings and specific embodiments.

[0055] Figure 1 This is a flow chart of the abnormal log identification method according to embodiment 1 of the present invention;

[0056] Figure 2 This is a structural block diagram of the abnormal log identification model in Example 2 of the present invention;

[0057] Figure 3 This is a structural block diagram of the log monitoring platform of Example 3 of the present invention;

[0058] Figure 4 This is a schematic structural diagram of an electronic device according to embodiment 4 of the present invention. DETAILED DESCRIPTION

[0059] The present invention will be further described below with reference to the following embodiments. However, those skilled in the art should understand that the present invention is not limited to the following embodiments, and any improvements and equivalent changes made based on the specific embodiments of the present invention are within the scope of protection of the claims of the present invention.

[0060] Example 1

[0061] A method for identifying abnormal logs, such as Figure 1 Shown, including:

[0062] S1. Preliminary judgment of the logs to be detected based on the local sensitive hashing algorithm and the K nearest neighbor classification algorithm, specifically including the following steps:

[0063] S11. Collect a labeled training set, where normal logs in the training set are labeled with a normal label, and abnormal logs in the training set are labeled with an abnormal label. Preprocess each log in the training set, extract a word set from each log, apply multiple hash functions to each word in the log, calculate multiple hash values, retain the minimum hash value, and form a multidimensional signature vector, namely, the MinHash signature.

[0064] S12. Use the locality-sensitive hashing algorithm to split the MinHash signatures of the logs in the training set into pieces and store them in hash buckets.

[0065] S13. Preprocess the log to be tested, extract a word set, apply multiple hash functions to each word, and use the same hash functions used to calculate the MinHash signature of the logs in the training set. Calculate multiple hash values, retain the minimum hash value, and form the MinHash signature of the log to be tested.

[0066] S14. Query the hash bucket based on the MinHash signature of the log to be detected, calculate the overall similarity between the MinHash signature of the log to be detected and the MinHash signature of the log in the corresponding hash bucket, sort the logs in the corresponding hash bucket from high to low according to the overall similarity, take the first K logs, K is a positive integer less than the number of logs in the training set, if there is a log labeled as abnormal among the first K logs, mark the log to be detected as a candidate abnormal log; the overall similarity is the product of the count similarity and the text similarity, that is:

[0067] ,

[0068] Where S 总体 is the overall similarity, S 计数 is the count similarity, S 文本 is the text similarity;

[0069] The formula for calculating count similarity is:

[0070] ,

[0071] Where n1 is the frequency of occurrence of the MinHash signature of the log to be detected, and n2 is the frequency of occurrence of the MinHash signature of a log in the corresponding hash bucket;

[0072] The text similarity adopts Jaccard similarity, and the calculation formula is:

[0073] ,

[0074] Where A is the MinHash signature of the log to be detected, and B is the MinHash signature of a log in the corresponding hash bucket.

[0075] S2. Perform semantic verification on candidate abnormal logs based on the pre-trained language model, specifically including the following steps:

[0076] S21. Construct a template containing a mask mark. The template is a prefix structure in the form of "The following log is a [MASK] log:", where [MASK] is a mask mark. Concatenate the candidate exception logs with the template to generate the input text.

[0077] S22. Input the input text into a pre-trained BERT model or a derivative model of the BERT model to predict the probability distribution of filler words at the mask mark position. This embodiment uses the MLM classifier, a derivative model of the BERT model;

[0078] S23. Establish a mapping relationship between label words and log categories. Select multiple label words related to normal logs. The label words "success", "completed", and "normal" indicate normal operation and form a normal label word set. Select multiple label words related to abnormal logs. The label words "fail", "uncompleted", and "error" indicate system failure or abnormality. Form an abnormal label word set. Introduce a weighted voting method to assign weights to different label words. The probability that a candidate abnormal log is a normal log is the sum of the weight of each label word in the normal label word set and the probability that the filler word at the mask mark position is the corresponding label word, that is:

[0079] ,

[0080] Where, P 正常 is the probability that the candidate abnormal log is a normal log, P 11 is the probability that the filler word at the mask mark position is the label word "success", W 11 is the weight of the label word “success”, P 12 is the probability that the filler word at the mask mark position is the label word "completed", W 12 is the weight of the label word “completed”, P 13 is the probability that the filler word at the mask mark position is the label word "normal", W 13 is the weight of the label word "normal";

[0081] The probability that a candidate abnormal log is an abnormal log is the sum of the weight of each label word in the abnormal label word set and the probability that the filler word at the mask mark position is the corresponding label word, that is:

[0082] ,

[0083] Where, P 异常 is the probability that the candidate abnormal log is an abnormal log, P 21 is the probability that the filler word at the mask mark position is the label word "fail", W 21 is the weight of the label word “fail”, P 22 is the probability that the filler word at the mask mark position is the label word “uncompleted”, W 22 is the weight of the label word “uncompleted”, P 23 is the probability that the filler word at the mask mark position is the label word "error", W 23 is the weight of the label word "error";

[0084] If the probability that the candidate abnormal log is an abnormal log is greater than the probability that the candidate abnormal log is a normal log, the candidate abnormal log is determined to be an abnormal log.

[0085] Traditional abnormal log identification methods use the DeepLog model, which uses an LSTM deep neural network to identify abnormal logs. This model relies heavily on the quality and diversity of training data and requires significant computing resources and time to train. However, the abnormal log identification method of this embodiment uses the locality-sensitive hashing (LSH) algorithm to accelerate the K-nearest neighbor (KNN) search for logs. This eliminates the need to calculate the similarity between all logs in the training set and the log to be detected. Instead, it only calculates the similarity between the logs in the hash bucket corresponding to the log to be detected and the log to be detected, significantly reducing the computational effort and improving log search speed. The BERT model or its derivatives performs semantic verification on candidate abnormal logs, avoiding the false positive problem of the K-nearest neighbor algorithm in log identification, where normal logs are mistakenly labeled as abnormal. The BERT model or its derivatives employ a pre-training and fine-tuning strategy for downstream tasks, demonstrating strong generalization capabilities. The fine-tuning strategy uses prompt-tuning, which better leverages its deep semantic understanding capabilities and compensates for the accuracy loss caused by the K-nearest neighbor algorithm. By constructing a template containing mask marks, a bridge is established between the pre-trained knowledge of the BERT model or its derivative models and downstream tasks, which can effectively improve the model's ability to understand text. In the log anomaly recognition task, the template is designed to judge the difference between normal and abnormal logs, guiding the model to focus on identifying these subtle differences. Even when there are very few available training samples, the model's pre-trained knowledge can be effectively utilized to achieve rapid adaptation to new tasks and give the model strong generalization capabilities. When there is insufficient labeled data, the BERT model or its derivative models can also generate pseudo-labels for the training set. The output judgment results can also be used to reversely correct the labels of the logs in the training set and generate pseudo-labels to optimize the KNN model.

[0086] The abnormal log identification method of this embodiment is applied to the Thunderbird dataset to identify abnormal logs, and the F1 score reaches 1.00, while the DeepLog model is used to identify abnormal logs on the Thunderbird dataset, and the F1 score is 0.045-0.06; the abnormal log identification method of this embodiment is applied to the BGL dataset to identify abnormal logs, and the F1 score is 0.996, while the DeepLog model is used to identify abnormal logs on the BGL dataset, and the F1 score is 0.678; the abnormal log identification method of this embodiment is applied in actual production testing, and the F1 score is 0.923, while the DeepLog model is used in actual production testing, and the F1 score is 0.504; In summary, the F1 score of the abnormal log identification method of this embodiment for identifying abnormal logs is significantly higher than the F1 score of the DeepLog model for identifying abnormal logs, and the abnormal log identification method of this embodiment has a high accuracy rate in identifying abnormal logs.

[0087] Example 2

[0088] This embodiment provides an abnormal log identification model, such as Figure 2 Shown, including:

[0089] The quick screening module is used to preliminarily judge the logs to be detected, specifically:

[0090] Collect a labeled training set. Normal logs in the training set are labeled with normal labels, and abnormal logs in the training set are labeled with abnormal labels. Preprocess each log in the training set to extract the word set of each log. Apply multiple hash functions to each word in the log to calculate multiple hash values. Keep the minimum hash value to form a multi-dimensional signature vector, which is the MinHash signature.

[0091] Using the locality-sensitive hashing algorithm, the MinHash signatures of the logs in the training set are sharded and stored in hash buckets;

[0092] Preprocess the logs to be tested, extract a word set, and apply multiple hash functions to each word. These hash functions are the same as those used to calculate the MinHash signature of the logs in the training set. Calculate multiple hash values and retain the minimum hash value to form the MinHash signature of the log to be tested.

[0093] The hash bucket is queried based on the MinHash signature of the log to be tested. The overall similarity between the MinHash signature of the log to be tested and the MinHash signature of the log in the corresponding hash bucket is calculated. The logs in the corresponding hash bucket are sorted from high to low according to the overall similarity. The first K logs are taken, where K is a positive integer less than the number of logs in the training set. If any of the first K logs is labeled as abnormal, the log to be tested is marked as a candidate abnormal log. The overall similarity is the product of the count similarity and the text similarity, that is:

[0094] ,

[0095] Where S 总体 is the overall similarity, S 计数 is the count similarity, S 文本 is the text similarity;

[0096] The formula for calculating count similarity is:

[0097] ,

[0098] Where n1 is the frequency of occurrence of the MinHash signature of the log to be detected, and n2 is the frequency of occurrence of the MinHash signature of a log in the corresponding hash bucket;

[0099] The text similarity adopts Jaccard similarity, and the calculation formula is:

[0100] ,

[0101] Where A is the MinHash signature of the log to be detected, and B is the MinHash signature of a log in the corresponding hash bucket.

[0102] The semantic verification module performs semantic verification on candidate exception logs based on the pre-trained language model. Specifically:

[0103] Construct a template containing a mask tag. The template is a prefix structure in the form of "The following log is a [MASK] log:", where [MASK] is a mask tag. Concatenate the candidate anomaly logs with the template to generate the input text.

[0104] Input the input text to a pre-trained BERT model or a derivative of the BERT model to predict the probability distribution of filler words at the masked marker positions. This embodiment uses the MLM classifier, a derivative of the BERT model.

[0105] Establish a mapping relationship between label words and log categories. Select multiple label words related to normal logs. The label words "success", "completed", and "normal" indicate normal operations and form a normal label word set. Select multiple label words related to abnormal logs. The label words "fail", "uncompleted", and "error" indicate system failures or abnormalities and form an abnormal label word set. Introduce a weighted voting method to assign weights to different label words. The probability that a candidate abnormal log is a normal log is the sum of the weight of each label word in the normal label word set and the probability that the filler word at the mask mark position is the corresponding label word, that is:

[0106] ,

[0107] Where, P 正常 is the probability that the candidate abnormal log is a normal log, P 11 is the probability that the filler word at the mask mark position is the label word "success", W 11 is the weight of the label word “success”, P 12 is the probability that the filler word at the mask mark position is the label word "completed", W 12 is the weight of the label word “completed”, P 13 is the probability that the filler word at the mask mark position is the label word "normal", W 13 is the weight of the label word "normal";

[0108] The probability that a candidate abnormal log is an abnormal log is the sum of the weight of each label word in the abnormal label word set and the probability that the filler word at the mask mark position is the corresponding label word, that is:

[0109] ,

[0110] Where, P 异常 is the probability that the candidate abnormal log is an abnormal log, P 21 is the probability that the filler word at the mask mark position is the label word "fail", W 21 is the weight of the label word “fail”, P 22 is the probability that the filler word at the mask mark position is the label word “uncompleted”, W 22 is the weight of the label word “uncompleted”, P 23 is the probability that the filler word at the mask mark position is the label word "error", W 23 is the weight of the label word "error";

[0111] If the probability that the candidate abnormal log is an abnormal log is greater than the probability that the candidate abnormal log is a normal log, the candidate abnormal log is determined to be an abnormal log.

[0112] Example 3

[0113] This embodiment provides a log monitoring platform that adopts a development model with front-end and back-end separation. The back-end adopts the Gin framework, the front-end adopts the React framework, and the visual display uses the ECharts chart library. The Gin framework is a high-performance Go language HTTP Web framework that provides functions such as routing, error handling, and static file serving. It is suitable for the rapid development of scalable applications. The Gin framework supports parameter parsing and validation, and its execution efficiency is better than most similar Go frameworks. The React framework is a JavaScript library open sourced by Facebook, which is used to build dynamic user interfaces. It improves development efficiency through a componentized approach, optimizes the rendering process using virtual DOM technology, and facilitates the development and maintenance of large-scale applications. The ECharts chart library is a JavaScript chart library open sourced by the Apache Foundation, which provides a rich variety of chart types and supports custom charts.

[0114] The log monitoring platform of this embodiment is as follows Figure 3 Shown, including:

[0115] The log collection module uses the FileBeat text collection tool to collect logs from different servers in real time and send them to the stream processing module;

[0116] The stream processing module uses Kafka distributed message queue to receive log data streams and implement log distribution and caching;

[0117] An abnormal log identification module, which uses the abnormal log identification model of Example 2 to obtain logs from the stream processing module, identify whether they are abnormal logs, and label the logs and store them in the first database;

[0118] The database includes a first database and a second database. The first database uses the ElasticSearch database for log storage and indexing, and integrates Lucene expression support to facilitate log query and analysis, helping users to quickly locate abnormal logs. Each log in the first database is stored as an independent document. Each document contains fields such as timestamp, log level, message content, and source service identifier. An inverted index is established for these fields, and a time-based index segmentation strategy is implemented. According to the frequency of log data generation, the index is regularly segmented, which can facilitate the management of historical data while maintaining high performance. The second database uses the MySQL database to store user information tables and log alarm tables. The user information table records the user's system account, including username and password, as well as the email address and mobile phone number for receiving platform alarm information. The log alarm table records user-defined log sequence information that requires alarms.

[0119] The visualization module is used to provide a front-end interface and supports log query function and log exception alarm function. The log query function can reproduce the log flow of the abnormal period according to the timeline and locate the log through the query statement; the log exception alarm function supports two exception matching modes. Sometimes the system is fault-tolerant. When an abnormal log appears, the system can automatically recover. If multiple error logs are output continuously or intermittently, it means that there is a problem with the system's automatic recovery. In this case, an alarm will be triggered; the first exception matching mode is fuzzy matching, which is suitable for log exception alarms in discontinuous sequences. As long as there are a preset number of abnormal logs in this sequence, an alarm will be triggered even if there are normal logs in the middle of this sequence. The user will be notified by email or text message according to the time interval selected by the user; the second exception matching mode is strict matching, using the KMP algorithm. The alarm will only be triggered when a preset number of abnormal logs appear continuously. The user will be notified by email or text message according to the time interval selected by the user.

[0120] The log monitoring platform of this embodiment is designed to meet the user's demand for real-time information. Users can choose different time intervals to receive alarm information according to monitoring requirements, such as updating the system status every minute, every hour, or every day. In addition, considering that jobs in a distributed system run on different physical nodes, the log monitoring platform of this embodiment uses the FileBeat text collection tool to collect logs from different servers in real time, uses the Kafka distributed message queue to receive log data streams, and uses the ElasticSearch database for distributed cache data. It provides the function of log monitoring from the global perspective of the cluster, allowing administrators to not only monitor a single node, but also track the log status of the entire cluster in real time, timely capture and analyze log data from different nodes, so as to quickly take appropriate repair measures to ensure the smooth execution of jobs.

[0121] Example 4

[0122] This embodiment provides an electronic device, including a memory and a processor. The memory stores a computer program, and the processor calls the computer program in the memory to execute the abnormal log identification method of Example 1. Figure 4 This is a schematic diagram of the structure of an electronic device provided in this embodiment. The electronic device may be a terminal device or a server. The terminal device may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 4 The electronic device shown is only an example and should not limit the functions and scope of use of this embodiment.

[0123] like Figure 4 As shown, electronic devices may include processing devices, such as a central processing unit (CPU) and a graphics processing unit (GPU), which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) or programs loaded from a storage device into random access memory (RAM). RAM also stores various programs and data required for the operation of the electronic device. The processing device, ROM, and RAM are interconnected via a bus. Input devices, output devices, communication devices, and storage devices are also connected to the bus via I / O interfaces.

[0124] Typically, the following devices can be connected to the I / O interface: input devices such as touch screens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices such as liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices such as magnetic tapes and hard disks; and communication devices. Communication devices allow electronic devices to communicate with other devices wirelessly or by wire to exchange data. Figure 4 The electronic device is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0125] Example 5

[0126] This embodiment provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program is used to implement the abnormal log identification method of Example 1. The computer-readable storage medium of this embodiment can be included in an electronic device or can exist independently without being incorporated into the electronic device.

[0127] The computer-readable storage medium of this embodiment may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. A method for identifying abnormal logs, characterized in that: include: Collect a labeled training set and search for the top K logs in the training set that are similar to the log to be tested, where K is a positive integer less than the number of logs in the training set. If any of the logs is labeled as abnormal, mark the log to be tested as a candidate abnormal log. Then, perform semantic verification on the text of the candidate abnormal log based on the pre-trained language model to determine whether the candidate abnormal log is an abnormal log. The method for searching the top K logs similar to the log to be detected in the training set is: Calculate the MinHash signature of each log in the training set; Using the locality-sensitive hashing algorithm, the MinHash signatures of the logs in the training set are sharded and stored in hash buckets; Calculate the MinHash signature of the log to be tested, query the hash bucket based on the MinHash signature of the log to be tested, calculate the overall similarity between the MinHash signature of the log to be tested and the MinHash signature of the log in the corresponding hash bucket, sort the logs in the corresponding hash bucket from high to low based on the overall similarity, and take the top K logs; Among them, the overall similarity is the product of count similarity and text similarity, that is: ， Where S 总体 is the overall similarity, S 计数 is the count similarity, S 文本 is the text similarity; The formula for calculating count similarity is: ， Where n1 is the frequency of occurrence of the MinHash signature of the log to be detected, and n2 is the frequency of occurrence of the MinHash signature of a log in the corresponding hash bucket; The text similarity adopts Jaccard similarity, and the calculation formula is: , Where A is the MinHash signature of the log to be detected, and B is the MinHash signature of a log in the corresponding hash bucket; The method for semantic verification of the text of the candidate exception log is: Construct a template containing mask marks, concatenate the candidate exception logs with the template, and generate the input text; Feed the input text into a pre-trained BERT model or a derivative of the BERT model to predict the probability distribution of filler words at the masked token positions. Establish a mapping relationship between label words and log categories. Based on the probability that the filler word at the mask mark position is the label word, calculate the probability that the candidate anomaly log is an anomaly log or a normal log. If the probability that the candidate anomaly log is an anomaly log is greater than the probability that the candidate anomaly log is a normal log, then the candidate anomaly log is judged to be an anomaly log.

2. The abnormal log identification method according to claim 1, characterized in that: The method for generating a MinHash signature is: Preprocess the logs and extract word sets; Apply multiple hash functions to each word in the log, retain the minimum hash value, and form a multi-dimensional signature vector, which is the MinHash signature.

3. The abnormal log identification method according to claim 1, characterized in that: The mapping relationship between label words and log categories is established as follows: Select multiple label words related to normal logs to form a normal label word set, select multiple label words related to abnormal logs to form an abnormal label word set, and introduce a weighted voting method to assign weights to different label words.

4. The abnormal log identification method according to claim 3, characterized in that: The probability that a candidate abnormal log is a normal log is the sum of the products of the weight of each label word in the normal label word set and the probability that the filler word at the mask mark position is the corresponding label word. The probability that a candidate abnormal log is an abnormal log is the sum of the products of the weight of each label word in the abnormal label word set and the probability that the filler word at the mask mark position is the corresponding label word.

5. An abnormal log identification device, characterized in that: include: The fast screening module is used to collect a labeled training set and search the training set for the first K logs similar to the log to be detected, where K is a positive integer less than the number of logs in the training set. If there is a log labeled as abnormal, the log to be detected is marked as a candidate abnormal log. The method for searching the top K logs similar to the log to be detected in the training set is: Calculate the MinHash signature of each log in the training set; Using the locality-sensitive hashing algorithm, the MinHash signatures of the logs in the training set are sharded and stored in hash buckets; Calculate the MinHash signature of the log to be tested, query the hash bucket based on the MinHash signature of the log to be tested, calculate the overall similarity between the MinHash signature of the log to be tested and the MinHash signature of the log in the corresponding hash bucket, sort the logs in the corresponding hash bucket from high to low based on the overall similarity, and take the top K logs; Among them, the overall similarity is the product of count similarity and text similarity, that is: ， Where S 总体 is the overall similarity, S 计数 is the count similarity, S 文本 is the text similarity; The formula for calculating count similarity is: ， Where n1 is the frequency of occurrence of the MinHash signature of the log to be detected, and n2 is the frequency of occurrence of the MinHash signature of a log in the corresponding hash bucket; The text similarity adopts Jaccard similarity, and the calculation formula is: , Where A is the MinHash signature of the log to be detected, and B is the MinHash signature of a log in the corresponding hash bucket; The semantic verification module performs semantic verification on the text of the candidate anomaly log based on the pre-trained language model to determine whether the candidate anomaly log is an anomaly log; The method for semantic verification of the text of the candidate exception log is: Construct a template containing mask marks, concatenate the candidate exception logs with the template, and generate the input text; Feed the input text into a pre-trained BERT model or a derivative of the BERT model to predict the probability distribution of filler words at the masked token positions. Establish a mapping relationship between label words and log categories. Based on the probability that the filler word at the mask mark position is the label word, calculate the probability that the candidate anomaly log is an anomaly log or a normal log. If the probability that the candidate anomaly log is an anomaly log is greater than the probability that the candidate anomaly log is a normal log, then the candidate anomaly log is judged to be an anomaly log.

6. A log monitoring system, characterized in that: include: Log collection module collects logs in real time and sends them to the stream processing module; Stream processing module, which receives log data streams and implements log distribution and caching; an abnormal log identification module, which uses the abnormal log identification device according to claim 5 to obtain logs from the stream processing module, identify whether they are abnormal logs, and label the logs and store them in the first database; The database includes a first database and a second database. The first database is used for storing and indexing logs. The second database is used to store user information tables and log alarm tables. The log alarm table records user-defined log sequence information that requires alarms. The visualization module is used to provide a front-end interface and support log query and log exception alarm functions.

7. The log monitoring system according to claim 6, characterized in that: Each log in the first database is stored as an independent document. Each document contains multiple fields. An inverted index is established for these fields, and an index splitting strategy based on time is implemented. The index is split regularly according to the frequency of log data generation.

8. The log monitoring system according to claim 6, characterized in that: The log query function can reproduce the log stream of the abnormal period according to the timeline and locate the log through the query statement; the log abnormality alarm function supports two abnormality matching modes. The first abnormality matching mode is fuzzy matching, which is suitable for log abnormality alarms in discontinuous sequences. As long as there are a preset number of abnormal logs in this sequence, an alarm will be triggered even if there are normal logs in the middle of this sequence. The user will be notified by email or SMS according to the time interval selected by the user; the second abnormality matching mode is strict matching, which uses the KMP algorithm. The alarm will only be triggered when a preset number of abnormal logs appear continuously. The user will be notified by email or SMS according to the time interval selected by the user.

9. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor calls the computer program in the memory to execute the abnormal log identification method according to any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, is used to implement the abnormal log identification method according to any one of claims 1 to 4.

Images